We aimed to develop an in-depth understanding of quality criteria for scholarly journals by analyzing journals and publishers indexed in blacklists of predatory journals and whitelists of legitimate journals and the lists' inclusion criteria. To quantify content overlaps between blacklists and whitelists, we employed the Jaro-Winkler string metric. To identify topics addressed by the lists' inclusion criteria and to derive their concepts, we conducted qualitative coding. We included two blacklists (Beall's and Cabells Scholarly Analytics') and two whitelists (the Directory of Open Access Journals' and Cabells Scholarly Analytics'). The number of journals per list ranged from 1,404 to 12,357, and the number of publishers ranged from 473 to 5,638. Seventy-two journals and 42 publishers were included in both a blacklist and a whitelist. Seven themes were identified in the inclusion criteria: (i) peer review; (ii) editorial services; (iii) policy; (iv) business practices; (v) publishing, archiving, and access; (vi) website; and (vii) indexing and metrics. Business practices accounted for almost half of the blacklists' criteria, whereas whitelists gave more emphasis to criteria related to policy. Criteria could be allocated to four concepts: (i) transparency, (ii) ethics, (iii) professional standards, and (iv) peer review and other services. Whitelists gave most weight to transparency. Blacklists focused on ethics and professional standards. Whitelist criteria were easier to verify than those used in blacklists. Both types gave little emphasis to quality of peer review. Overall, the results show that there is overlap of journals and publishers between blacklists and whitelists. Lists differ in their criteria for quality and the weight given to different dimensions of quality. Aspects that are central but difficult to verify receive little attention. Predatory journals are spurious scientific outlets that charge fees for editorial and publishing services that they do not provide. Their lack of quality assurance of published articles increases the risk that unreliable research is published and thus jeopardizes the integrity and credibility of research as a whole. There is increasing awareness of the risks associated with predatory publishing, but efforts to address this situation are hampered by the lack of a clear definition of predatory outlets. Blacklists of predatory journals and whitelists of legitimate journals have been developed but not comprehensively examined. By systematically analyzing these lists, this study provides insights into their utility and delineates the different notions of quality and legitimacy in scholarly publishing used. This study contributes to a better understanding of the relevant concepts and provides a starting point for the development of a robust definition of predatory journals.