The same thing that once made young American men go west decided if a house was a manor: No, not gold, land.
A manor, in most cases, was an estate or part of one.
It usually held a main, large house or small castle, a church, an apothecary, granaries, stables, etc.
It also had enough land for cultivation and for one’s serfs and indentured servants to have small houses and their own private plots of land as well.
In a sense, a large manor was like a very small working village.
Manors were popular in the late middle ages and many were fortified like castles. Some even had moats and ditches to keep the thieves and robbers out.
Some manors in medieval times were fortified country houses intended more for show than for defense.