When someone is “looking for dirt,” or scavenging for scandal in another person’s earlier life, if they dig long enough they’re bound to find something.
Of course if you’re scrounging through an attic trunk or old scrapbooks, you too are “digging up the past.”
The expression logically comes from the science of archeology.
It is in that field of study where people make a profession of “digging up dirt” to understand the present by looking into the past.