The transition between receiving a PhD and securing a tenure track faculty position is challenging for nearly every astronomer interested in working in academia. Here we use a publicly available database of recently hired faculty (the Astrophysics Job Rumor Mill) to examine the amount of time astronomers typically spend in this transitory state. Using these data as a starting point to examine the experiences of astronomy postdocs, we find that the average time spent between receiving a PhD and being hired into a faculty position is 4.9$\pm$0.3 years, with female astronomers hired on average 4.2$\pm$0.4 years after receiving a PhD while male astronomers are typically hired after 5.3$\pm$0.4 years. Using a simple model of the labor market, we attempt to recreate this gendered difference in time spent as a postdoc. We can rule out the role of the increasing representation of women among astronomy PhDs, as well as any bias in favor of hiring female astronomers in response to efforts to diversify the faculty ranks. Instead the most likely explanation is that female astronomers are leaving the academic labor market, at a rate that is 3-4 times higher than male astronomers. This scenario explains the distinct hiring time distributions between male and female astronomers, as well as the measured percentage of female assistant professors, and the fraction of female applicants within a typical faculty search. These results provide evidence that more work needs to be done to support and retain female astronomers during the postdoctoral phase of their careers.