Housing environments can directly and indirectly affect sleep, and blacks are more likely than whites to live in suboptimal housing conditions, which may independently contribute to sleep disparities. However, few large-scale epidemiological studies consider the potential influence of housing type on sleep health. Using data from the 2004–2015 National Health Interview Survey, we investigated overall and Black-White differences in the association between housing type (house/apartment versus mobile home/trailer) and sleep duration as well as sleep difficulties among 226,208 adults in the U.S. Poisson regression with robust variance was used to estimate sex-specific prevalence ratios (PR) for sleep categories, first comparing houses/apartments to mobile homes/trailers and then blacks to whites within housing types. All models were adjusted for age, educational attainment, income, occupational class, self-reported general health status, and region of residence. Compared to participants living in houses/apartments, the prevalence of short sleep was higher for men (PR = 1.05 (95% confidence interval (CI): 1.02–1.08)) and women (PR = 1.07 (95% CI: 1.04–1.09)) in mobile homes/trailers. Black men (PR = 1.26 (95% CI: 1.21–1.30)) and women (PR = 1.24 (95% CI: 1.20–1.27)) in a house/apartment were more likely to be short sleepers than their white counterparts. There was generally no significant difference in sleep characteristics (except long sleep) between black and white men in mobile homes/trailers after adjustments, and black men in houses/apartments as well as black women in both housing types were less likely to report sleep difficulties although being more likely to report short sleep. Overall, individuals in mobile homes/trailers, which may represent suboptimal housing, had worse sleep than those in houses/apartments; and racial differences in the quality of houses and apartments are likely to greatly vary in ways that still contribute to sleep disparities. Race–sex group differences in sleep duration among residents in a house/apartment and even a lack of racial difference among individuals living in mobile homes/trailers support the need for more research on residential environments and eventually multi-level interventions designed to reduce sleep disparities.