In Italy, the Human Papillomavirus (HPV) vaccination was implemented for twelve years old girls in 2007, but its coverage was lower than the recommended level. Sicily is one of the Italian administrative regions with lower vaccination coverage, with a value of 59% for those born in 1996 increasing to 62% coverage for those born in 1999. The aim of the study was to investigate factors associated with the refusal of HPV vaccination among young adult women of Palermo, Italy. The study was approved by the Ethics Committee of the Policlinico “Paolo Giaccone” Hospital (Palermo 1) and the questionnaire was validated in a convenience sample representing 10% of the young women. A cross-sectional study was conducted through the administration of a telephone questionnaire, consisting of 23 items on HPV infection and vaccination knowledge based on the Health Belief Model framework. The eligible population were young women (18–21 years old) who had at least a vaccination among all included in the Sicilian vaccination schedule, without starting or completing HPV vaccination. Overall, 141 young women were enrolled (response rate 22%). Among them, 84.4% were unvaccinated and 15.6% had at least one dose of the HPV vaccine. In multivariate analysis, the factors associated with the refusal of the HPV vaccination were a bachelor’s as the education level (OR = 10.2, p = 0.041), lower participation at school seminar on HPV (OR = 0.2, p = 0.047) and lower perception of HPV vaccine benefits (OR = 0.4, p = 0.048). Public health educational program focusing and tailored on benefits perception of HPV vaccine and HPV disease severity, carried out at school or during medical visits, can be useful to improve HPV vaccination uptake.