Ball lightning is a poorly understood and comparatively rare phenomenon.
So if you ever see one, try to take a picture.
Perhaps 1 percent of the population will see ball lightning in a lifetime.
Though there have been thousands of reports of ball lightning, it is still very hard to study, because scientists do not know how to reproduce it in the laboratory.
As for why it is not usually destructive, it seems to be some sort of derivative of a lightning flash without the intensity of the flash it self, a phenomenon that splits off from the main channel, not the 50,000 degrees and huge current of lightning.
A lot of trees hit by ball lightning survive.
Ball lightning is also commonly reported to travel through enclosed spaces, rather than in the open field.
It is possible that the building contains part of the charge or flash, but that is speculative.
It may be just that the observers are there.