We investigated the cross-sectional and longitudinal effects of tobacco smoking on brain atrophy in a large cohort of healthy elderly participants (65 to 80 years). MRI was used for measuring whole brain (WB), gray matter (GM), white matter (WM), and hippocampus (HIP) volumes at study entry time (baseline, N=1,451), and the annualized rates of variation of these volumes using a 4-year follow-up MRI in a subpart of the cohort (N=1,111). Effects of smoking status (never, former, or current smoker) at study entry and of lifetime tobacco consumption on these brain phenotypes were studied using sex-stratified AN(C)OVAs, including other health parameters as covariates. At baseline, male current smokers had lower GM, while female current smokers had lower WM. In addition, female former smokers exhibited reduced baseline HIP, the reduction being correlated with lifetime tobacco consumption. Longitudinal analyses demonstrated that current smokers, whether men or women, had larger annualized rates of HIP atrophy, as compared to either current or former smokers, independent of their lifetime consumption of tobacco. There was no effect of smoking on the annualized rate of WM loss. In all cases, measured sizes of these tobacco-smoking effects were of the same order of magnitude than those of age, and larger than effect sizes of any other covariate. These results demonstrate gender- and tissue specific effects of tobacco smoking on brain atrophy. They indicate that tobacco smoking is a major factor of brain aging, with notable effects on the hippocampus annualized-rate of atrophy after the age of 65.