In the nineteenth century, railway tracks usually ran right through the center of town, and it was the prevailing winds that determined which was the right or wrong side to live on.
As the town developed, the wealthy built homes on the cleaner, windward side of the tracks, while industrial development and the working class were confined to the other, dirtier side.
To be from the “wrong side of the tracks” meant you were from a poor or working-class family.