Living with obesity is an experience that may affect multiple aspects of an individual’s life. Obesity is considered a relevant public health problem in modern societies. To determine the comparative efficacy of different treatments and to assess their impact on patients’ everyday life, it is important to identify factors that are relevant to the quality of life of obese patients. The present study aims to evaluate, in Portuguese obese patients, the simultaneous impact of several psychosocial factors on quality of life. This study also explores the mediating role of stigma in the relationship between positive/negative affect and quality of life. A sample of 215 obese patients selected from the main hospitals in Portugal completed self-report questionnaires to assess sociodemographic, clinical, psychosocial, and quality of life variables. Data were analysed using structural equation modeling. The model fitted the data reasonably well, CFI = 0.9, RMSEA = 0.06. More enthusiastic and more active patients had a better quality of life. Those who reflect lower perception of stigma had a better physical and mental health. Partial mediation effects of stigma between positive affect and mental health and between negative affect and physical health were found. The stigma is pervasive and causes consequences for psychological and physical health.