The phrase “call off the dogs” means: Cease some objectionable line of conduct, procedure, conversation, inquiry, or the like; break off an unprofitable or disagreeable course.
The analogy is that of the chase, in which dogs following a wrong scent are called off.