Why do the rich take more financial risk and hence earn a higher return on their portfolios on average? In this paper, I argue that understanding the interdependence of optimal housing decisions, debt taking and portfolio allocation over the wealth distribution is key to explaining this robust empirical pattern. As apart from being a means of investment, housing also serves as a consumption good, households with a lower financial wealth to human capital ratio optimally choose a higher share of housing out of wealth. On the one hand, this implies that for relatively wealth-poor homeowners risky liquid assets are mechanically crowded out from their portfolio. Second, since this mechanism also makes poorer households optimally more leveraged, the effects are magnified by the wedge between borrowing and lending rates: if the interest rate on debt is higher, indebted households effectively face a lower risk premium, and thus are provided with lower incentives to hold risky assets. Calibrating a rich life-cycle model to the saving and home ownership profiles over age in Swedish administrative data I find that these mechanisms enable matching the increasing risky share pattern over the wealth distribution. I decompose the effect of different channels and also show that the model predicts a higher marginal propensity of stock investments for the rich.