Background: To evaluate the impact of poor glycemic control of type 2 diabetes mellitus (T2DM) on serum prostate-specific antigen (PSA) concentrations in men. Methods: We performed a prospective analysis of 215 consecutive patients affected by erectile dysfunction (ED). ED was evaluated using the IIEF-5 questionnaire and the poor glycemic control (PGC) of T2DM was assessed according to the HbA1c criteria (International Diabetes Federation). Patients were divided into PGC group (HbA1c ≥ 7%) and control group (CG) (HbA1c < 6%). Correlations between serum HbA1c levels and various variables were evaluated and multivariate logistic regression analyses were carried out to identify variables for PGC. Results: We compared 110 cases to 105 controls men ranging from 44 to 81 years of age, lower PSA concentrations were observed in men with PGC (PGC mean PSA: 0.9 ng/dl, CG mean PSA: 2.1 ng/dl, p < 0.001). Also mean prostate volume was 60% was smaller among men with PGC compared with men with CG (PGC mean prostate volume: 26 ml, CG prostate volume: 43 ml, p < 0.001). A strong negative correlation was found between serum HbA1c levels and serum PSA (p < 0.001 and r = −0.665) concentrations in men with PGC. We also found at the multivariate logistic regression model that PSA, prostate volume and peak systolic velocity were independent predictors of PGC. Conclusion: Our results suggest that there is significant impact of PGC on serum PSA levels in T2DM. Poor glycemic control of type 2 diabetes was associated with lower serum PSA levels and smaller prostate volumes.