This expression means very dead, of course (or deaf, or dumb), completely and absolutely non-responsive.
It is very old, has been traced back to 1350 in literary use and was therefore probably used in common speech long before that, possibly for several centuries.
But just why our remote ancestors conceived a doornail to be very dead, or deaf or dumb, is something that has never been satisfactorily explained. Todd, who published a revision of Johnson’s English Dictionary in 1818, advanced the notion that the ancient doornail was a heavy stud against which the knocker was struck.
If such a nail was used for that purpose, perhaps some old-time wit proclaimed that it had been struck on the head so often as to be dead; or another, after pounding vainly upon it without response from within, proclaimed it to be deaf or dumb.
We don’t know, nor have we anything to support Todd’s theory. The earliest usage, as far as the records show, was with the adjective “dead.”
When “dumb” first appeared, in 1362, it related to the door only; as in Langland’s Piers Plowman, “As doumbe as a dore.” “Deaf did not show up until the sixteenth century, when it was applied indiscriminately either to the “doore” or the “doore nayle.”