There are some rules of thumb for estimates for a limited area, but estimates for whole countries or the world are just guesstimates.
Archaeologists tend to find and count all the formerly inhabited sites in a study area; make an educated guess about which ones were occupied at a certain period, usually based on pottery of a certain age found on the surface; and then from the total occupied area make a rough estimate.
Evidence from aerial photos and satellite images may also be used.
From studies of modern villages, archaeologists can say about 100 or 200 people occupy a hectare, or two and a half acres, of a modern settlement, and apply this estimate to the past. But there are tremendous uncertainties in such estimates.
For example, big sites covered with the pottery of one period may indicate big cities, and some, like old Damascus and old Jerusalem, show evidence of densities of 500 to 600 people per hectare.