The conventional wisdom is that racial diversity promotes positive race relations and reduces racial residential segregation between blacks and whites. We use data from the 1990–2010 decennial censuses and 2007–2011 ACS to test this so-called “buffering hypothesis.” We identify cities, suburbs, and small towns that are virtually all white, all black, all Asian, all Hispanic, and everything in between. The results show that the most racially diverse places—those with all four racial groups (white, black, Hispanic, and Asian) present—had the lowest black-white levels of segregation in 2010. Black-white segregation also declined most rapidly in the most racially diverse places and in places that experienced the largest recent increases in diversity. Support for the buffering hypothesis, however, is counterbalanced by continuing high segregation across cities and communities and by rapid white depopulation in the most rapidly diversifying communities. We argue for a new, spatially inclusive perspective on racial residential segregation.