Tissue-resident memory T cells (TRM) reside in barrier tissues and provide local immediate protective immunity. Here, we show that the salivary gland (SG) most-effectively induces CD8+ and CD4+ TRM cells against murine cytomegalovirus (MCMV), which persists in and spreads from this organ. TRM generation depended on local antigen for CD4+, but not CD8+, TRM cells, highlighting major differences in T cell subset-specific demands for TRM development. CMV-specific CD8+ T cells fail to control virus replication upon primary infection in the SG due to CMV-induced MHC I downregulation in glandular epithelial cells. Using intraglandular infection, we challenge this notion and demonstrate that memory CD8+ T cells confer immediate protection against locally introduced MCMV despite active viral immune evasion, owing to early viral tropism to cells that largely withstand MHC I downregulation. Thus, we unravel a yet-unappreciated role for memory CD8+ T cells in protecting mucosal tissues against CMV infection.