Contrary to popular myth, it was not a Native American custom but a colonial one first.
Scalping on the American continent was first instituted by the Dutch colonial government around New York when they actually set a bounty for killing Indians.
They came up with the idea of requiring a scalp as concrete evidence of a kill before paying a claim. Other colonial governments thought this was a great idea, and the English and French began using the same system to tally kills.
Eventually, even the Native Americans picked it up for their own internal bookkeeping.