Quagmire is he who ventures into a new undertaking, seemingly sound, only to find himself inextricably entangled in a hopeless mess, has become quagmired, bogged down, on shaky ground.
The last term here is a nearly literal equivalent of the first, as quag seems certainly to be a variant of quake, “to shake,” and mire is “muddy or swampy ground.”
Thus a quagmire is a piece of ground that looks firm, but shakes when walked upon, only to engulf the traveler.
An obsolete form is quickmire, and the relation to quicksand, which has much the same treacherous properties, immediately comes to mind.