Episodic memory impairment is the hallmark symptom of Alzheimer's Disease (AD). However, episodic memory has also been shown to decline across the lifespan. Here, we investigated whether episodic memory is differentially affected relative to other cognitive abilities before old age, and whether being an Apolipoprotein E (APOE) ε4 carrier -a genetic risk factor for developing AD-exacerbates any such impairments. We used general linear models to test for performance differences within 4 composite measures of cognition - episodic memory, semantic memory, speed of processing, and fluid reasoning-- as a function of age group (young, average age 30.21 vs. middle-aged, average age 50.84) and APOE-ε4 genotype status (ε4+ vs. ε4-). We replicated findings of age-related reductions in episodic memory, speed of processing, and fluid reasoning, and age-related increases in semantic memory. However, we also found that APOE genotype status moderated the age-related declines in episodic memory: APOE-ε4+ middle-aged adults exhibited impairments relative to both APOE-ε4- middle-aged participants, and APOE-ε4+ younger adults. These results suggest specific and dynamic alterations to episodic memory as a function of APOE allelic variation and age.