(1) Background: Teachers’ personal and strenuous working conditions reflect the realities of the teaching vocation that may result in increased stress levels and associated negative consequences, such as negative emotions. It is also well-known that teacher stress contributes to more violence against students. However, little is known about personal and school context factors that contribute to teachers’ stress. The current study examined whether, in addition to school-related factors, job perceptions, including the feeling of pressure at work and perceived school climate and teaching difficulties, contribute to teachers’ stress. (2) Methods: A representative sample of 291 teachers from 12 public secondary schools in southwestern Uganda responded to self-administered questionnaires. (3) Results: Teaching difficulties and feelings of pressure at work contributed to teachers’ stress. Furthermore, stress did not vary with teachers’ sociodemographic variables. (4) Conclusions: Teachers’ perceptions of their working conditions were associated with teacher stress levels. Therefore, more efforts need to be geared towards improving the working conditions of teachers as a way of reducing stress.