The Early Smoking Experience (ESE) questionnaire is the most widely used questionnaire to assess initial subjective experiences of cigarette smoking. However, its factor structure is not clearly defined and can be perceived from two main standpoints: valence, or positive and negative experiences, and sensitivity to nicotine. This article explores the ESE’s factor structure and determines which standpoint was more relevant. It compares two groups of young Swiss men (German- and French-speaking). We examined baseline data on 3,368 tobacco users from a representative sample in the ongoing Cohort Study on Substance Use Risk Factors (C-SURF). ESE, continued tobacco use, weekly smoking and nicotine dependence were assessed. Exploratory structural equation modeling (ESEM) and structural equation modeling (SEM) were performed. ESEM clearly distinguished positive experiences from negative experiences, but negative experiences were divided in experiences related to dizziness and experiences related to irritations. SEM underlined the reinforcing effects of positive experiences, but also of experiences related to dizziness on nicotine dependence and weekly smoking. The best ESE structure for predictive accuracy of experiences on smoking behavior was a compromise between the valence and sensitivity standpoints, which showed clinical relevance.