In binocular rivalry, presentation of different images to the separate eyes leads to conscious perception alternating between the two possible interpretations every few seconds. During perceptual transitions, a stimulus emerging into dominance can spread in a wave-like manner across the visual field. These traveling waves of rivalry dominance have been successfully related to the cortical magnification properties and functional activity of early visual areas, including the primary visual cortex (V1). Curiously however, these traveling waves undergo a delay when passing from one hemifield to another. In the current study, we used diffusion tensor imaging (DTI) to investigate whether the strength of interhemispheric connections between the left and right visual cortex might be related to the delay of traveling waves across hemifields. We measured the delay in traveling-wave times (ΔTWT) in nineteen participants and repeated this test 6 weeks later to evaluate the reliability of our behavioral measures. We found large interindividual variability but also good test-retest reliability for individual measures of ΔTWT. Using DTI in connection with fiber tractography, we identified parts of the corpus callosum connecting functionally defined visual areas V1-V3. We found that individual differences in ΔTWT was reliably predicted by the diffusion properties of transcallosal fibers connecting left and right V1, but observed no such effect for neighboring transcallosal visual fibers connecting V2 and V3. Our results demonstrate that the anatomical characteristics of topographically specific transcallosal connections predict the individual delay of interhemispheric traveling waves, providing further evidence that V1 is an important site for neural processes underlying binocular rivalry.