The lateral prefrontal cortex (LPF) plays a fundamental role in planning, organizing, and optimizing behavioral performance. Neuroanatomical and neurophysiological studies have suggested that in this cortical sector, information processing becomes more abstract when moving from caudal to rostral and that such processing involves parietal and premotor areas. We review studies that have shown that the LPF, in addition to its involvement in implementing rules and setting behavioral goals, activates during the execution of forelimb movements even in the absence of a learned relationship between an instruction and its associated motor output. Thus, we propose that the prefrontal cortex is involved in exploiting contextual information for planning and guiding behavioral responses, also in natural situations. Among contextual cues, those provided by others' actions are particularly relevant for social interactions. Functional studies of macaques have demonstrated that the LPF is activated by the observation of biological stimuli, in particular those related to goal-directed actions. We review these studies and discuss the idea that the prefrontal cortex codes high-order representations of observed actions rather than simple visual descriptions of them. Based on evidence that the same sector of the LPF contains both neurons coding own action goals and neurons coding others' goals, we propose that this sector is involved in the selection of own actions appropriate for reacting in a particular social context and for the creation of new action sequences in imitative learning.