Helium gas is not only very light, it is monatomic, its particles are all made of a single atom.
As a result, helium is made up of the smallest gaseous particles possible.
The atoms are only 0.1 nanometre in diameter, and are quite capable of diffusing through metal films.
Because it so readily diffuses through small pores, helium is used to help test for leaks in industrial and laboratory vacuum systems. Nitrogen and oxygen molecules have a much larger diameter than helium atoms which means that they are much less capable of diffusing through the balloon walls.
It’s like the difference between trying to get sand and small pebbles to pass through a sieve, the sand goes through much more easily because it’s made from smaller particles.
The second factor which helps to increase losses by diffusion is that balloons are made from viscoelastic materials whose structure is a tangled mass of polymer strands, a bit like a plate of spaghetti.
The polymer strands cannot pack closely together, and have channels through which the helium can diffuse, so even at low pressure the helium will diffuse out through the walls.
When the balloon is inflated, the polymer stretches, so the balloon walls become thinner, the helium has a shorter distance to diffuse out, the molecular structure becomes slightly more open, making diffusion much easier, and the increased pressure provides a driving force for the diffusion.
These are the reasons why deflation is very rapid to begin with, but then gradually slows down as the balloon gets smaller.
Commercial helium balloons are made from non-porous inelastic materials and are coated to reduce the losses even further, although even they still lose a significant percentage of helium per day, certainly enough to disappoint children, and grown-ups, the morning after buying a balloon.
The helium atom is very small and very light. It is able to diffuse through the thin, stretched rubber of the balloon quite easily, finding its way through atomic-sized pores.
Air molecules, oxygen and nitrogen mainly, are larger and heavier and diffuse through much more slowly. In addition to the increased pressure inside the balloon which pushes helium out through the sides, there is another factor that increases helium flow outwards.
Because there is almost no helium in the air, far more helium atoms are hitting the inside of the balloon than the outside, and there is a net flow outwards. However, you will notice that the balloon does not completely deflate. This is because some air moves in as, conversely, more air molecules hit the outside than the inside.
This leads to a truly bizarre effect if the balloon is filled with the gas sulphur hexafluoride, which has large, very heavy molecules which hardly diffuse through the rubber at all, and so cannot get out.
But once again, as in the helium example, there are more air molecules outside than in, so air diffuses inwards and the balloon slowly increases in size.