NEW YORK ( MainStreet) — The housing slump caused by this year's unusually cold winter has yet to end , leaving investors to wonder if the Federal Reserve should take the housing market more seriously.

The Fed traditionally tracks employment and inflation when it comes to determining its future monetary policies, but at an appearance before the Joint Economic Committee of Congress late last week, Fed Chairwoman Janet Yellen was emphatic about the lackluster housing market.

"Readings on housing activity--a sector that has been recovering since 2011--have remained disappointing so far this year and will bear watching," Yellen said.

The Chairwoman went on say the housing slump could also last longer than expected.

The latest housing data mirrors this sentiment . Existing home sales fell 0.2% in March, the latest reading from the National Association of Realtors and are down 7.5% year-over-year. Building permits fell 2.4% in March and while housing starts, which reflect the volume of new construction homes, grew 2.8% in March, starts are still down 5.9% since March 2013 .

One culprit of the housing slowdown, aside from cold weather, is rising interest rates . The Fed has been scaling back its bond stimulus, which has kept interest rates low, by $10 billion each month since December 2013. Short-term interest rates have stayed near zero since December 2008, although investors expect that to change sometime in mid-2015.

"I think that the Fed has always been aware and focused on the housing market, but has been in a little bit of self denial over the last six to nine months over the impact of the higher rates," says ITG chief economist Steve Blitz.

Rates on a 30-year fixed mortgage average 4.21%, compared to 3.42% last year at this time, according to Freddie Mac. The yield on the 10-year Treasury note, which impacts consumer interest rates, including mortgage rates, currently stands at 2.50%, after crossing 3% in December, but is still up from 1.94% in May 2013.

"Interest rates are now rising to where the supply and demand for credit is in the marketplace, which is still low," Blitz adds. "But that increase is coming from the Fed, not the marketplace, so demand for credit will drop."

In Yellen's remarks last week, she said, "a high degree of monetary accommodation remains warranted." Should the Fed continue its tapering at the current level of $10 billion per month, the end of quantitative easing would come in October.

If the housing slump continues and starts to weigh down the rest of the economy, the Fed could taper by a smaller amount each month, which would indicate that the economy isn't ready just yet to stand on its own two feet.

The housing market will be watched closely in the coming months. A few factors including the potential for increased summer demand and banks' loosening of lending standards could be the key boost the housing market needs.