From the Wall Street Journal:

Fannie alone ranks as the world’s largest bank by assets, while Freddie is in the top ten. The firms’ subsidized mortgages encourage Americans to take on a lot of debt. Their balance sheets, consisting of long-term investments financed with short-term borrowing, make them highly susceptible to sudden credit freezes. Their government ownership means taxpayers stand to lose tens of billions of dollars if the firms get into trouble again.

Of course, the taxpayers are losing tens of billions of dollars even now, while the firms are enjoying relatively chipper times.  But the reason for Fannie and Freddie is not supporting the residential mortgage market.  The reason for Fannie and Freddie is laundering money.  They give the Chinese, et al, something to do with the massive dollars they acquire through running trade surpluses.   The trade surplus laundering operation is also why the two entities were not allowed to fail, and why they will not be reduced in size or importance, unless some other government entity arises to effectively take their place.