Forest height is of great significance in analyzing the carbon cycle on a global or a local scale and in reconstructing the accurate forest underlying terrain. Major algorithms for estimating forest height, such as the three-stage inversion process, are depending on the random-volume-over-ground (RVoG) model. However, the RVoG model is characterized by a lot of parameters, which influence its applicability in forest height retrieval. Forest density, as an important biophysical parameter, is one of those main influencing factors. However, its influence to the RVoG model has been ignored in relating researches. For this paper, we study the applicability of the RVoG model in forest height retrieval with different forest densities, using the simulated and real Polarimetric Interferometric SAR data. P-band ESAR datasets of the European Space Agency (ESA) BioSAR 2008 campaign were selected for experiments. The test site was located in Krycklan River catchment in Northern Sweden. The experimental results show that the forest density clearly affects the inversion accuracy of forest height and ground phase. For the four selected forest stands, with the density increasing from 633 to 1827 stems/Ha, the RMSEs of inversion decrease from 4.6 m to 3.1 m. The RVoG model is not quite applicable for forest height retrieval especially in sparsely vegetated areas. We conclude that the forest stand density is positively related to the estimation accuracy of the ground phase, but negatively correlates to the ground-to-volume scattering ratio.