Background: Although birth weight is considered as a fetal determinant of the development of adult-onset diabetes mellitus (DM), its public health importance relative to adult body mass index (BMI) remains unclear. We aimed to examine the association between adult-onset DM and birth weight in relation to adult BMI. Methods: We conducted a self-administered questionnaire as a baseline survey of the Japanese Nurses' Health Study cohort between 2001 and 2007. Exclusion criteria were applied to the volunteer sample of 49,927 female nurses (age <30 years or unknown, current pregnancy, development of DM before the age of 30 years, unknown core variables), and data from 26,949 female nurses aged 30 years or older were used. The association between history of DM diagnosis and birth weight was analyzed using logistic regression. Results: A linear inverse association was observed between birth weight and DM, after adjustment for age, BMI, and parental history of DM. The odds ratio for developing DM per 100 g increase in birth weight was 0.93 (95% confidence interval [CI], 0.90–0.96). The association was unchanged when birth weight was converted to percentile for gestational age. In the BMI-stratified analysis, the odds ratio for DM in the <2500 g birth weight group reached 4.75 (95% CI, 1.22–18.44, compared to the reference 3000–3499 g group) among women with normal low BMI (18.5–20.9). Conclusions: Birth weight and its percentile for gestational age were associated with adult-onset DM. Attention should be paid to the risk of DM among women born with low weight, even when their current BMI is normal.