Dictionaries for the collegiate trade, slavishly following one another, continue to carry the term bandog, though, except in the works of Sir Walter Scott, a student might never run across it.
It simply means a dog, usually a large one such as a mastiff, that is kept tied, band here being used in the sense of that which binds, a strap, chain or the like.
The word used to be spelled, more properly, band-dog. Scott contracted it to bandog and so it remains.