If your employer dismisses you, or if your boy friend ceases to call, or if the newcomer on the next street rejects your friendly advances, you are, in modern American slang, getting the brush off; you arc being put aside.
The thought seems to be that one is likened to an undesirable piece of lint or streak of dirt that is brushed from a garment.
But there is also the possibility that the allusion is to a Pullman porter who, sensing a poor tip from one of his charges, gives that person a few flicks with his brush and passes on to a more likely customer.