Implicit racial bias remains widespread, even among individuals who explicitly reject prejudice. One reason for the persistence of implicit bias may be that it is maintained through structural and historical inequalities that change slowly. We investigated the historical persistence of implicit bias by comparing modern implicit bias with the proportion of the population enslaved in those counties in 1860. Counties and states more dependent on slavery before the Civil War displayed higher levels of pro-White implicit bias today among White residents and less pro-White bias among Black residents. These associations remained significant after controlling for explicit bias. The association between slave populations and implicit bias was partially explained by measures of structural inequalities. Our results support an interpretation of implicit bias as the cognitive residue of past and present structural inequalities.