Homeowners lost more than $100,000 in wealth in many U.S. counties following the recession — a loss housing experts say will be difficult to recover from even as the economy shows signs of improvement.

New data released on Thursday by the U.S. Census Bureau shows the dramatic effects that the recession had on real estate values. Nationally, homes lost 9% of their median values from what the Census describes as the three-year recession period (2007-09) through the three-year post-recession period (2010-12), according to the report, dropping from $191,900 to $174,600.

 By now the impact of the plummeting home values is well known to most Americans. Real estate is the most valuable purchase most Americans ever make and it accounts for a significant share of their overall wealth. Declining values result in poorer households who in turn have less leverage to tap into equity in their home to make other big-ticket purchases. Separately, declining home values pushed millions of homeowners into a position where they owed more on their mortgage than their home was worth.

Housing sales and prices have picked up in the past year. The median sales price of single-family homes increased 12.5% in the third quarter compared with a year prior, according to the National Association of Realtors. The last time the quarterly price was higher was the third quarter of 2007.

While the numbers look rosy, some analysts say they are not a reflection of what regular Americans are doing. In many markets, including Las Vegas, Miami and Phoenix — where median prices have risen 21% to 32% in one year — investors account for about 30% to 40% of home purchases over the past three years, says Jack McCabe, an independent housing analyst in Deerfield Beach, Fla. These are mostly hedge funds and high-net-worth individuals who have been buying homes at rock-bottom prices and mostly renting them out. Buyers from abroad who are purchasing vacation homes and pied-a-terre’s are also fueling purchases.

Some analysts have a brighter outlook. Assuming unemployment continues to drop and consumer confidence rises, more people will likely start buying homes for their use next year, says Brad Hunter, chief economist at Metrostudy, a housing research and consulting firm. The homeownership rate, which fell from 66.4% to 64.7% over the period tracked by the Census, should start to pick up as well, he says.

Here are five states where homeowners lost the most wealth:

California
Median property values fell $102,600
Three counties in California — Los Angeles, Riverside and San Bernardino — lost more than $100,000 in median property values from the three-year recession period (2007 to 2009) through the three-year post-recession period (2010 to 2012). Trailing right behind them, Orange County lost a median of $99,300. Aggressive flippers (people who bought homes and sold them quickly for a profit) and high demand among buyers helped create the bubble in this state. Once unemployment spiked, many buyers weren't able to hold on to their homes and went into foreclosure.

Recently, home prices in many hard-hit areas have been picking up. Homes in Los Angeles County, for instance, gained 29% from the beginning of last year through October 2013, says Steve Dutra, senior vice president of research at John Burns Real Estate Consulting, a real estate research firm that’s headquartered in Irvine, Calif. In the Riverside-San Bernardino metro area, prices rose 42% over the same period.

But prices still have a way to go before they can return to the pre-downturn bubble levels. Home values would need to rise another 26% in Los Angeles County and another 53% in Riverside-San Bernardino in order to reach their 2006 peak, says Dutra. “We don’t foresee that happening in the next five years,” he says.

Nevada
Median property values fell $99,400
A weak economy and high number of foreclosures have battered home values in this state — and led to homes shedding 38% of their values (the most of the five states in this article) from the recession to post-recession period, according to the Census. Job losses contributed to 224,207 foreclosure filings from 2007 through 2009, according to RealtyTrac, a real-estate data firm. Those dropped slightly to 210,662 from 2010 to 2012.

It’s no surprise that most of the real estate losses in Nevada came from Sin City. Median home values in Clark County, home to Las Vegas, fell 40.4% or $106,300 from the recession through the post-recession, according to the Census. The drop brought median prices in the county to $156,500.

With prices at record lows, investors entered the market and began boosting home values, says Hunter of Metrostudy. The median sales price of single-family homes increased 32% in this third quarter from a year prior, according to the National Association of Realtors.

Experts question how long the momentum will last. Nevada’s economy remains shaky with a 9.5% unemployment rate as of August 2013, according to the Bureau of Labor Statistics. While it’s the lowest since 2008, its jobless rate ranks as the highest among all the states, according to the BLS. Foreclosures are still rampant: Nevada had one of the five highest foreclosure rates in the nation last month, according to data released this week by RealtyTrac.

Arizona
Median property values fell $63,800

Another epicenter of the housing meltdown, Arizona’s home prices shed 28.5% of value from the recession to the post-recession period, according to the Census. Those losses were, of course, mostly in Phoenix; median home values in its county, Maricopa, fell by $76,300, surpassing the overall state losses.

Like Las Vegas, Phoenix’s prices began to rise in the last couple of years as investors swept in. Last year, investors accounted for 42% of all sales in Phoenix on average, says Dutra of John Burns Real Estate Consulting. In the third quarter of 2013, they had dropped to 33%, and the pace of overall sales in the city is slowing down, he says.

Housing analysts say price growth is likely to slow as well. Investors have pulled back largely because they believe prices in Phoenix are now too high, says Hunter. Should the number of for-sale listings increase — and if the time they spend on the market gets longer — prices will drop.

Florida
Median property values fell $55,900
Excessive building and foreclosures led to the demise of real estate in much of Florida. Median home prices fell from $210,800 to $154,900 from the recession through post-recession period, according to the Census. Homeownership in the state declined by 2.4 percentage points — one of the largest drops nationwide — with 66.9% of homes being owner occupied during the three-year period of 2010-12. Prices hit a bottom in 2011 and surged over the past year. “Some of the turnaround there has been remarkable,” says Dutra.

Consider Miami. Homes in the county lost a median of $91,400 from the recession through post-recession, but prices are rising again. The median price of single-family homes in the Miami-Fort Lauderdale metro area was $252,200 during the third quarter of this year, up 21.3% over a year prior, according to the NAR.

But much of the rebound in the state is due to real estate investors—rather than regular home buyers. During the last six quarters, investors (including those buying second homes and vacation homes) have accounted for roughly 40% of all real estate purchases in the state, says Dutra. That includes hedge funds and high-net-worth individuals who are buying homes to rent them out, as well as buyers from abroad, specifically South America, Canada and to a lesser extent England, who are buying vacation homes in the state, says McCabe. “The vast majority of buyers are corporate business structures — it’s not Mr. and Mrs. Smith as it always has been in the past,” he says.

Maryland
Median property values fell $45,800
Homeowners in Maryland lost 14% of their property value between the recession and post-recession, according to the Census, with median prices falling to $289,300. In Montgomery County, which borders Washington, D.C., homes shed 8.4% of their value and the homeownership rate fell almost four percentage points. In contrast, the national homeownership rate during this period fell 1.7 percentage points.

Since last year, there have been signs that real estate is turning around in the state. Housing starts, which is essentially when construction on new housing begins, increased 38% during the third quarter compared with a year prior, says Hunter of Metrostudy. That’s likely a sign that more buyers are in the market and builders are feeling optimistic about demand and prices.