The SELEX Collaboration has reported a very large isospin splitting of doubly charmed baryons. We show that this effect would imply that the doubly charmed baryons are very compact. One intriguing possibility is that such baryons have a linear geometry Q–q–Q where the light quark q oscillates between the two heavy quarks Q, analogous to a linear molecule such as carbon dioxide. However, using conventional arguments, the size of a heavy-light hadron is expected to be around 0.5 fm, much larger than the size needed to explain the observed large isospin splitting. Assuming the distance between two heavy quarks is much smaller than that between the light quark and a heavy one, the doubly heavy baryons are related to the heavy mesons via heavy quark–diquark symmetry. Based on this symmetry, we predict the isospin splittings for doubly heavy baryons including Ξcc, Ξbb and Ξbc. The prediction for the Ξcc is much smaller than the SELEX value. On the other hand, the Ξbb baryons are predicted to have an isospin splitting as large as (6.3±1.7) MeV. An experimental study of doubly bottomed baryons is therefore very important to better understand the structure of baryons with heavy quarks.