This paper describes the geochemistry of groundwater and its flow system in the closed Yatani mine in southern Yamagata Prefecture, Japan. The mine is located in a sulfide deposit containing pyrite and has been generating acid mine drainage (AMD). The study was intended to elucidate the formation of AMD and its flow patterns using geological, hydrological, geochemical, and isotopic techniques. The results indicate that AMD is formed by the interaction of groundwater with sulfide minerals, sand slime, and tailings back-filled into excavated mine areas. Groundwater recharge areas were identified on the mountain slope at an elevation of ~900 m. The formation of AMD in the drifts and shaft was more extensive than that in the deeper drainage levels. Principal component analysis was applied to the hydrogeochemical data to identify the causes of AMD formation. The first, second, and third principal components reveal that the increased ion concentrations in mine drainage are a result of water-mineral reactions in excavated mine areas, the contribution of groundwater in deep reductive environments, and isotopic fractionation during precipitation, respectively. A promising method of reducing AMD formation is to prevent contact between dissolved oxygen and sulfide minerals by increasing the drainage level or by filling the shallow underground excavated area with cementitious materials.