The effect of ambient pollutants on the male reproductive system is controversial. This retrospective study investigated the effect of environmental pollutants on male reproductive health.Male patients with primary infertility (n = 282) were identified from a single center between January 2016 and December 2017. Patients were physically examined for the presence of varicocele and for the volume of both testicles. Semen quality was measured in terms of the total sperm count, sperm concentration, and the percentage of sperm cells with motility and normal morphology. Data were acquired on the concentration of ambient pollutants, namely particulate matters of diameter < 2.5 μm, sulfur dioxide (SO), nitrogen oxides (NO), and ozone (O), measured on daily and hourly basis, from the Environmental Protection Administration Executive Yuan, Taiwan. Individual exposure to pollutants was estimated based on the reported residential address of each participant. Statistical analysis indicated the effect of each pollutant on the testicular volume, sex hormone profile, and semen parameters.The mean ± standard deviation of age was 36.7 ± 7.3 years. The average sperm count and concentration were 41.9 million/mL and 34.1 million/mL, respectively. The mean levels of serum testosterone, follicle-stimulating hormone, and luteinizing hormone were 3.57 ± 1.68 ng/mL, 7.59 ± 6.3 IU/L, and 4.68 ± 3.49 IU/L, respectively. According to the multivariate linear regression model, NO exposure was a risk factor for decreased sperm concentration and motility (p = 0.043 and 0.032). Furthermore, SO exposure was negatively associated and testicular volume (p < 0.01).NO and SO exposure were negatively associated with the seminal parameter and decreased testicular volume, respectively, in a population of men with infertility. However, additional prospective studies are needed to ascertain the cause-effect relation of current results.