The word buff, as we use it most frequently today, is a color, a light yellow.
But it is also the name given to a soft, undyed and unglazed leather, especially a leather made from a buffalo hide, for it was from this leather that the color got its name.
Someone, about three hundred years ago, facetiously referred to his own bare skin as his “buff,” perhaps because it was tanned by the sun and had the characteristic fuzzy surface of buffalo leather.
The name stuck, but is now rarely used except in the phrase above, which, of course, means divested of one’s clothing.