Hunt had fifteen children by three wives.

He married Lyda Bunker of Lake Village, Arkansas in November 1914, and remained married to her until her death in 1955.[4] His seven children by her were: Margaret (born 1915), Haroldson (“Hassie,” 1917), Caroline (1923), Lyda (born and died in 1925), Nelson Bunker (1926), William Herbert (1929), and Lamar (1932). Their home to the north of White Rock Lake in Dallas was an exact model of Mount Vernon.
 
His first son, Hassie, who was expected to succeed him in control of the family business, was lobotomized in response to increasingly erratic behavior. He outlived his father. Lamar founded the American Football League and created the Super Bowl, drawing on the assistance of his children in selecting the game's name. Two other children, William and Bunker, are famous for having purchased much of the world's silver, in an attempt to corner the market. They ultimately owned more silver than any government in the world, before their scheme was discovered and undone. Bunker Hunt was briefly one of the wealthiest men in the world, having discovered and taken title to the Libyan oil fields, before Muammar Gaddafi nationalized the properties.
 
While still married to Lyda, H. L. Hunt is said to have married Frania Tye of Tampa, Florida in November 1925, using the name Franklin Hunt. Frania claimed to have discovered the bigamous nature of her marriage in 1934, and in a legal settlement in 1941, Hunt created trust funds for each of their four children and she signed a document stipulating that no legal marriage between them had ever existed. About the same time she briefly married then divorced Hunt’s employee John Lee, taking the last name Lee for herself and her four children.[5] Her four children by Hunt were: Howard (born 1926), Haroldina (1928), Helen (1930), and Hugh (“Hue,” 1934). Frania Tye Lee died in 2002.[6]
 
Hunt supported and had children by Ruth Ray of Shreveport, Louisiana, whom he met when she was a secretary in his Shreveport office. They married in 1957 after the death of Hunt’s wife Lyda. His four children by her were: Ray (born 1943), June (1944), Helen (1949), and Swanee (1950).His youngest son, Ray Lee, ultimately inherited the business, and was a major supporter of President George W. Bush.
 
Hunt died, aged 85, in Dallas, Texas. He was buried there in the Sparkman-Hillcrest Memorial Park Cemetery.