Self-incompatibility (SI) is found in approximately 40% of flowering plant species and at least 100 families. Although orchids belong to the largest angiosperm family, only 10% of orchid species present SI and have gametophytic SI (GSI). Furthermore, a majority (72%) of Dendrobium species, which constitute one of the largest Orchidaceae genera, show SI and have GSI. However, nothing is known about the molecular mechanism of GSI. The S-determinants of GSI have been well characterized at the molecular level in Solanaceae, Rosaceae, and Plantaginaceae, which use an S-ribonuclease (S-RNase)-based system. Here, we investigate the hypothesis that Orchidaceae uses a similar S-RNase to those described in Rosaceae, Solanaceae, and Plantaginaceae SI species. In this study, two SI species (Dendrobium longicornu and D. chrysanthum) were identified using fluorescence microscopy. Then, the S-RNase- and SLF-interacting SKP1-like1 (SSK1)-like genes present in their transcriptomes and the genomes of Phalaenopsis equestris, D. catenatum, Vanilla shenzhenica, and Apostasia shenzhenica were investigated. Sequence, phylogenetic, and tissue-specific expression analyses revealed that none of the genes identified was an S-determinant, suggesting that Orchidaceae might have a novel SI mechanism. The results also suggested that RNase-based GSI might have evolved after the split of monocotyledons (monocots) and dicotyledons (dicots) but before the split of Asteridae and Rosidae. This is also the first study to investigate S-RNase-based GSI in monocots. However, studies on gene identification, differential expression, and segregation analyses in controlled crosses are needed to further evaluate the genes with high expression levels in GSI tissues.