This study examined whether middle-aged adults exposed to poverty in childhood or current financial hardship have detectable brain differences from those who have not experienced such adversity. Structural magnetic resonance imaging (MRI) was conducted as one aspect of the Personality and Total Health (PATH) through life study: a large longitudinal community survey measuring the health and well-being of three cohorts from south-eastern Australia. This analysis considers data from 431 middle-aged adults in the aged 44-48 years at the time of the interview. Volumetric segmentation was performed with the Freesurfer image analysis suite. Data on socio-demographic circumstances, mental health and cognitive performance were collected through the survey interview. Results showed that, after controlling for well-established risk factors for atrophy, adults who reported financial hardship had smaller left and right hippocampal and amygdalar volumes than those who did not report hardship. In contrast, there was no reliable association between hardship and intra-cranial volume or between childhood poverty and any of the volumetric measures. Financial hardship may be considered a potent stressor and the observed results are consistent with the view that hardship influences hippocampal and amygdalar volumes through hypothalamic-pituitary-adrenal axis function and other stress-related pathways.