Bus is an abbreviation of omnibus, which is what they called the original horse-drawn vehicles used for public transportation.
The busman, of course, was the driver, and because the bus was drawn by the driver’s own horses, he was very concerned about their well-being.
It wasn’t uncommon for busmen to frequently come down to the barn during their vacation time to ensure that their horses were being well treated, which gave us the expression “busman’s holiday.”
We call a working vacation a “busman’s holiday”.