Purpose: Construction workers’ reactions to safety-related issues during operation vary from person to person due to their different occupational levels, which can be attributed to various influencing factors and their correspondingly complicated interactions. This research aims to propose an integrated framework to combine the concepts of these factors and provide a holistic interpretation of the interrelationship among them. Methods: Based on items that were mainly extracted from competency theory, exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were conducted to identify the critical factors from the data collected from 243 scaffolders on Wuhan Metro construction sites. The interactions among the identified factors were then analyzed, and the safety competency model was thus established with the use of structural equation modeling (SEM). Results: A total of 17 items were identified as critical to workers’ safety competency, and these were further tested and attributed to four factors: (1) individual character and inclination; (2) self-adjustment and adaptability; (3) working attitudes; (4) safety-related operation qualification. Subsequent analysis showed that all the factors significantly contributed to one’s safety competency, and individual character and inclination contributed most to the formation of one’s ability, while the intermediating effects of self-adjustment and adaptability should not be neglected both in theoretical and practical terms. The resultant safety competency model consisting of these four factors was revealed to share a hierarchical structure with the classical competency model. Significance: This study provided an integrated theoretical framework and a set of modeling approaches to combine the related concepts and facilitate a greater understanding of construction safety in terms of workers’ characteristics and behaviors. Practical implications: This study presented a tentative approach for assessing construction workers’ safety competency, as well as emphasized to the managers and professionals the necessity of developing training systems to ensure workers are integrated into a crew in an appropriate and smooth manner. Limitations and Future Work: The volume and the scope of samples impeded the study from achieving a more generalized result and a more cost-efficient data collection approach is in need of development for a comprehensive and in-depth investigation.