Socioeconomic status (SES) is associated with executive function (EF) and prefrontal cortex (PFC) development. However, understanding of the specific aspects of SES that influence development of EF and the PFC remains limited. We briefly review existing literature on proposed mechanisms linking SES with EF. Then, we present a novel conceptual model arguing that early cognitive stimulation shapes EF and PFC development. We propose that cognitive stimulation drives lower-level sensory and perceptual processes that may impact EF and PFC development through reciprocal connections between the ventral visual stream and PFC. We argue that caregivers guide attention and associative learning, which provides children the opportunity to regulate attention and gain semantic knowledge. This experience in turn allows for opportunities to train the PFC to resolve conflict between stimuli with overlapping features and engage in increasingly complex computations as visual processing systems develop; this may lay the groundwork for development of EF. We review existing evidence for this model and end by highlighting how this conceptual model could launch future research questions.