Andover, often linked with Exeter,is often understood symbolically as an "elite New England prep school," connoting privilege. Writer William S. Dietrich II described Andover and other elite prep schools as being part of a "WASP ascendancy" during the first half of the twentieth century. Elite universities such as Yale and Princeton tended to accept disproportionate percentages of prep school students while using quotas to deny admission to minority applicants. An account in Time Magazine in 1931 described the two schools as having "flourished", and that both schools were "twin giants of prep schools in size and in prestige", essentially feeder schools for Ivy League universities such as Harvard and Yale, according to Joseph Lieberman. A cultural image from the 1960s was young men who had "perfect white teeth" and wore Lacoste shirts, with a look easy to identify by young women at the time:

They can tell just by looking at him whether a boy goes to an Eastern prep school or not. Not only that, they can tell which prep school, usually St. Paul's or Hotchkiss or Groton or Exeter or Andover, or whatever; just by checking his hair and his clothes.
—Tom Wolfe in his book Mauve Gloves & Madmen, Clutter & Vine

The WASP ascendancy began to break down around the 1960s and onwards when the admissions policies of elite prep schools and universities began to emphasize merit rather than affluence.Still, images of exclusivity based on unfairness tended to remain. Gore Vidal suggested that Andover and Exeter had a "style that was quite witty."If the WASP ascendancy has waned, the image of unaffordability continues to persist, with one writer deploring how the schools cost $30,000 and more annually. Recent reports from graduates, however, paint a picture of educational excellence and diversification. For example, Christina Hartmann, who attended Andover from 2001-2003, described the school as having a need blind admissions policy. She suggested the student body was mostly diversified, and that the school had dedicated buildings for specific subject areas, was challenging academically, and had flexible teachers and peers who were "smart and driven".She elaborated that Andover provided two sign language interpreters, free of charge, to help her academically to cope with her deafness.While the overall image may be changing to one which emphasizes greater diversity and respect for individual talent, the image of the school in the media continues to connote privilege, money, exclusivity, prestige, academic quality, and sometimes negatively connotes chumminess or clubbiness or arrogance.