From early in the sixteenth century traders from Milan flocked into England with the products of their city and of Lombardy.
Steel work was a large part of the manufactures they brought, but their chief articles of commerce were textile fabrics. Milan bonnets, Milan gloves, Milan lace, Milan ribbons were in great demand.
The English pronunciation of the name of that foreign city, however, was not that which we use today; they rimed it then with “villain.”
In fact, the English often spelled the name “Millain.” Merchants from Milan were naturally called Milaners, pronounced as if spelled milliners, and thus Milliner actually became the general spelling.
Hence, subsequently, any person dealing with products similar to those that had once come from Milan became known as a milliner.
Restriction of the term to women’s headgear and the like is recent.