This paper presents the study on the association between in-vehicle music listening, physiological and psychological response, and driving performance, using the driving simulator approach, with which personality (temperament) was considered. The performance indicators considered were the standard deviation of speed, lane crossing frequency, perceived mental workload, and mean and variability of heart rate. Additionally, effects of the presence of music and music genre (light music versus rock music) were considered. Twenty participants of different personalities (in particular five, four, seven, and four being choleric, sanguine, phlegmatic, and melancholic, respectively) completed a total of 60 driving simulator tests. Results of mixed analysis of variance (M-ANOVA) indicated that the effects of music genre and driver character on driving performance were significant. The arousal level perceived mental workload, standard deviation of speed, and frequency of lane crossing were higher when driving under the influence of rock music than that when driving under the influence of light music or an absence of music. Additionally, phlegmatic drivers generally had lower arousal levels and choleric drivers had a greater mental workload and were more likely distracted by music listening. Such findings should imply the development of cost-effective driver education, training, and management measures that could mitigate driver distraction. Therefore, the safety awareness and safety performance of drivers could be enhanced.