Yellow, meaning cowardly, is actually an abbreviation of yellow dog, an American insult that first appeared in the nineteenth century to describe a cowardly or worthless person.
In the early twentieth century, when employers were fighting trade unions, they insisted that new employees sign a pledge never to join a union.
This pledge was called a “yellow dog” contract by union members with the implication that anyone signing it was “yellow.”