Populus is an effective model for genetic studies in trees. The genus Populus includes dioecious species, and the differences exhibited in males and females have been intensively studied. This review focused on the distinctions between male and female poplar and aspen plants under stress conditions, such as drought, salinity, heavy metals, and nutrient deficiency on morphological, physiological, proteome, and gene expression levels. In most studies, males of Populus species were more adaptive to the majority of the stress conditions and showed less damage, better growth, and higher photosynthetic capacity and antioxidant activity than that of the females. However, in two recent studies, no differences in non-reproductive traits were revealed for male and female trees. This discrepancy of the results could be associated with experimental design: different species and genotypes, stress conditions, types of plant materials, sampling sizes. Knowledge of sex-specific differences is crucial for basic and applied research in Populus species.