Planetary gearbox torsional vibration signals are free from the extra amplitude modulation effect due to time-varying transmission paths and have simpler frequency structure than translational ones. Gear faults result in modulation on the torsional resonance vibration and are manifested by the modulation feature. These merits are exploited for planetary gearbox fault diagnosis in this paper. Gear fault induced torsional vibrations in resonance region are modelled as amplitude modulation and frequency modulation (AM-FM) processes, the explicit equation of Fourier spectrum is derived, and the sideband characteristics are summarized. To avoid complex sideband analysis, amplitude and frequency demodulation analysis methods are exploited. The equations of amplitude and frequency demodulated spectra are derived in closed form, and their frequency structures are revealed. For fault diagnosis based on above theoretical derivations, a resonance frequency identification approach is proposed through time-frequency analysis of torsional vibrations during variable speed processes, according to the independence nature of resonance frequency on running conditions. The theoretical derivations and proposed approach are illustrated by numerical simulated signal analysis and are further validated through dynamics modelling and lab experimental tests. Localized faults on the sun, planet, and ring gears are successfully diagnosed.