The ambient temperature–health relationship is of growing interest as the climate changes. Previous studies have examined the association between ambient temperature and mortality or morbidity, however, there is little literature available on the ambient temperature effects on year of life lost (YLL). Thus, we aimed to quantify the YLL attributable to non-optimum ambient temperature. We obtained data from 1 January 2013 to 31 December 2017 of 70 counties in Hunan, China. In order to combine the effects of each county, we used YLL rate as a health outcome indicator. The YLL rate was equal to the total YLL divided by the population of each county, and multiplied by 100,000. We estimated the associations between ambient temperature and YLL with a distributed lag non-linear model (DNLM) in a single county, and then pooled them in a multivariate meta-regression. The daily mean YLL rates were 22.62 y/(p·100,000), 10.14 y/(p·100,000) and 2.33 y/(p·100,000) within the study period for non-accidental, cardiovascular, and respiratory disease death. Ambient temperature was responsible for advancing a substantial fraction of YLL, with attributable fractions of 10.73% (4.36–17.09%) and 16.44% (9.09–23.79%) for non-accidental and cardiovascular disease death, respectively. However, the ambient temperature effect was not significantly for respiratory disease death, corresponding to 5.47% (−2.65–13.60%). Most of the YLL burden was caused by a cold temperature than the optimum temperature, with an overall estimate of 10.27% (4.52–16.03%) and 15.94% (8.82–23.05%) for non-accidental and cardiovascular disease death, respectively. Cold and heat temperature-related YLLs were higher in the elderly and females than the young and males. Extreme cold temperature had an effect on all age groups in different kinds of disease-caused death. This study highlights that general preventative measures could be important for moderate temperatures, whereas quick and effective measures should be provided for extreme temperatures.