Background: Scientific workers play an important role in the development of science and technology. However, evidence is lacking with regard to the associations between their dietary factors and their health-related quality of life (HRQOL). Methods: A cross-sectional survey was conducted among 775 scientific workers from multiple universities and institutes in the Southwest region of China. A self-administered food-frequency questionnaire was used to collect the food consumption information, and the 36-item Short-Form Health Survey was used to assess physical HRQOL. Hierarchical multiple regression analysis was used to identify the factors associated with scientific workers’ HRQOL. Results: Physical HRQOL was negatively associated with age and intake of fresh pork (fat) and animal viscera, whereas consumption of vegetables, fruits, refined cereals and dairy products were positively correlated with physical HRQOL. Participants with daily intake of vegetable oils or mixed oils showed higher physical HRQOL scores than those with intake of animal oils. Conclusions: Dietary habits are closely associated with the physical HRQOL of scientific workers. The dietary patterns that had more vegetables and fruits, less fresh pork (fat) and animal viscera, and used vegetable oils during cooking corresponded to higher physical HRQOL scores. These findings are important for planning dietary strategies to improve physical health in scientific workers.