Athletic competition has long been a part of the Phillips Academy tradition. As early as 1805, football was being played on school grounds, according to a letter that Henry Pearson wrote his father, Eliphalet Pearson in 1805, saying, "I cannot write a long letter as I am very tired after having played at football all this afternoon."The first ever interscholastic football game between high schools was in 1875, when Phillips Academy played againstAdams Academy.One of the oldest schoolboy rivalries in American football is the Andover/Exeter competition, started in 1878. That year, the first Andover/Exeter baseball game took place, and The Phillipian returned from hiatus, named its first Board and began publishing regularly.
Today, Phillips Academy is an athletic powerhouse among New England private schools. Since the Constitution of the Phillips Academy Athletic Association was drawn up in 1903 with the objective of "Athletics for All", Andover has established 29 different interscholastic programs, and 44 intramural or instructional programs, including fencing, tai chi, figure skating, and yoga.[dead link] Andover Athletes have been successful in winning over 110 New England Championships in these different sports over the last three decades alone,[dead link] and have even had the chance to compete abroad, in such competitions as the Henley Royal Regatta in Henley, England for crew.

The athletic directors of Andover and the other members of the Eight Schools Association (ESA) compose the Eight Schools Athletic Council, which organizes sports events and tournaments among ESA schools. Andover is also a member of the New England Preparatory School Athletic Council. As a way to encourage all students to try new things and stay healthy, all students are required to have an athletic commitment each term. A range of instructional sports are available for those who wish to try new things, and for those already established in a sport, most teams have at least a varsity and junior varsity squad.