A disproportionate number of U.S. presidents named James didn’t have kids: James Madison, James Polk, and James Buchanan.
George Washington had no children of his own, but raised Martha’s children from her previous marriage.
Her children certainly considered him a father, as is evidenced by the fact that they allowed him and Martha to raise their children while he served as President.
The U.S. presidents that did have kids mostly kept them out of the public spotlight, which is a smart move, as the presidency is a huge target for lots of people including the press and yes, even the public.