Cannabis use and disorders (CUD) are influenced by multiple genetic variants of small effect and by the psychosocial environment. However, this information has not been effectively incorporated into studies of gene-environment interaction (GxE). Polygenic risk scores (PRS) that aggregate the effects of genetic variants can aid in identifying the links between genetic risk and psychosocial factors. Using data from the Pasman et al. GWAS of cannabis use (meta-analysis of data from the International Cannabis Consortium and UK Biobank), we constructed PRS in the Collaborative Study on the Genetics of Alcoholism (COGA) participants of European (N: 7591) and African (N: 3359) ancestry. The primary analyses included only individuals of European ancestry, reflecting the ancestral composition of the discovery GWAS from which the PRS was derived. Secondary analyses included the African ancestry sample. Associations of PRS with cannabis use and DSM-5 CUD symptom count (CUDsx) and interactions with trauma exposure and frequency of religious service attendance were examined. Models were adjusted for sex, birth cohort, genotype array, and ancestry. Robustness models were adjusted for cross-term interactions. Higher PRS were associated with a greater likelihood of cannabis use and with CUDsx among participants of European ancestry (p < 0.05 and p < 0.1 thresholds, respectively). PRS only influenced cannabis use among those exposed to trauma (R: 0.011 among the trauma exposed vs. R: 0.002 in unexposed). PRS less consistently influenced cannabis use among those who attend religious services less frequently; PRS × religious service attendance effects were attenuated when cross-term interactions with ancestry and sex were included in the model. Polygenic liability to cannabis use was related to cannabis use and, less robustly, progression to symptoms of CUD. This study provides the first evidence of PRS × trauma for cannabis use and demonstrates that ignoring important aspects of the psychosocial environment may mask genetic influences on polygenic traits.