Large-scale rural labor transfer is considered an important means of promoting urbanization and poverty alleviation; however, it has been difficult to fully reveal the complexity of mountain labor transfer using traditional theory. In this study, macrostatistics and microsurvey data were employed to conduct an empirical analysis of rural labor transfer in Xichang, highlighting the regional features of rural labor transfer in the southwest mountainous areas of China. The results show that the employment structure of rural labor is dominated by agriculture; however, its proportion is decreasing annually. The development of secondary and tertiary industries significantly contributes to the non-agricultural employment of rural laborers, who transfer mainly to the building, industry, and consumption fields—especially consumption, which has the greatest ability to absorb surplus rural labor. Migration of the population and farmers’ per capita net income promotes rural labor transfer, while the amount of rural labor, the urbanization rate in townships, agricultural development, and the topography have lagged effects on the process. This study argues that rural labor transfer in the mountains has regionality, multi-pattern, and gradient features; therefore, the promotion of rural labor transfer should consider local urbanization and poverty alleviation in such a manner that aligns with the local natural and socioeconomic conditions of the mountainous areas.