The wide application of high-resolution SAR data, such as TanDEM-X and TerraSAR-X, has resulted in an increase of the data processing difficulty of interferograms, especially in urban areas with serious layovers caused by high-rise buildings. In this paper, a new method based on frequency estimation is proposed to extract and compensate the building layover phase without considering the building structure. We use a non-local algorithm to estimate the high-accuracy frequency in the range direction, which is utilized to extract the layover areas of a building. Then, a two-step method for estimating local frequencies is used for layover phase removal. Efficient frequency estimation and building extraction is demonstrated on real data in comparison with traditional methods. The results of the removal approach with both simulated and real TanDEM-X and TerraSAR-X images are presented to prove the potential of the method.