She met her future husband, the Chinese-American cinematographer James Wong Howe, before World War II. They traveled to Paris in 1937 to marry,  but their marriage was not recognized in California. They would not cohabit due to his traditional Chinese views, so they had separate apartments in the same building. When the state supreme court overturned California's anti-miscegenation law (which prohibited marriage between people of different races),  it took them three days to find a judge who would agree to marry them. Even then, the judge reportedly remarked, "She looks old enough. If she wants to marry a chink, that's her business."  Due to the ban, Howe's studio contract "morals clause" prohibited him from publicly acknowledging their marriage.

In the early 1940s Babb was West Coast secretary of the League of American Writers. She edited the literary magazine The Clipper and its successor The California Quarterly, helping to introduce the work of Ray Bradbury and B. Traven, as well as running a Chinese restaurant owned by Howe.

During the early years of the HUAC hearings, Babb was blacklisted, and moved to Mexico City to protect the "graylisted" Howe from further harassment