There was no definite source; the word fiddle was in the language and, four hundred years ago, it had taken on the meaning “to act aimlessly,” so, just like such duplications as flip-flop, jimjams, helter-skelter, and the like, someone turned it into fiddlefaddle.
Thus since the sixteenth century this nonsense word has implied aimless or idle action or conversation, or any thing or occupation that is of little consequence:
“Pete’s sudden interest in postage stamps is just so much fiddle-faddle.”
