There are very few tall buildings in between Lower Manhattan and Midtown because of the geology of Manhattan Island.
Very tall buildings need a very solid foundation, and Manhattan Island’s bedrock of schist, which is a granite-like rock, is particularly solid and stable.
In lower Manhattan the schist is about 80 feet (24 m) below street level, and in midtown it lies about 38 feet (12 m) below street level.
But in between the two areas, the schist is more than 200 feet (61 m) deep, way too far for skyscraper builders to dig to reach bedrock.