Building resilient communities towards recurrent droughts is increasingly becoming an important element in development endeavours, particularly among communities vulnerable to shocks and stresses. Despite decades of remarkable efforts made by governmental and non-governmental organization, the resilience capacity of pastoralists in Ethiopia remains poor. The aim of this study is to test the statistical relationships among the resilience dimensions that emerged through community consultations, and to identify the intervention pathways for effective resilience building efforts. Data were collected from 1058 randomly sampled households in Arero and Dhas districts of Borana Zone, Southern Ethiopia. The data were collected through interviewer administered structured questionnaire and observational checklist. Principal component analyses were done to develop composite scores of the different resilience dimensions. Structural equation model (SEM) verified the theoretical model. The SEM also revealed that resilience towards impact of recurrent droughts was multi-dimensional and showed statistically significant (p < 0.05) relationships. Consequently, household food insecurity manifested as ultimate outcome of poor resilience. Infrastructure and social services (β = −0.24), livestock dimension (β = −0.21), human capital (β = −0.12), psychosocial distress (β = −0.1) dimensions significantly (p < 0.05) affected the status of household food insecurity. Furthermore, livestock and wealth (β = 0.16), wealth and infrastructure (β = 0.06), infrastructure and human capital (β = 0.18), livestock and psychosocial distress (β = −0.09) dimensions have structural relationships and significantly influence each other. Environment, and peace and security are found to be major underlying resilience factors and significantly associated with pastoralists’ resilience which affect other resilience dimensions. The intervention pathway indicated that intervention on the livestock and infrastructure dimensions could help to improve the resilience of these communities.