Unmanned aerial vehicles (UAVs) are now readily available worldwide and users can easily fly them remotely using smart controllers. This has created the problem of keeping unauthorized UAVs away from private or sensitive areas where they can be a personal or public threat. This paper proposes an improved radio frequency (RF)-based method to detect UAVs. The clutter (interference) is eliminated using a background filtering method. Then singular value decomposition (SVD) and average filtering are used to reduce the noise and improve the signal to noise ratio (SNR). Spectrum accumulation (SA) and statistical fingerprint analysis (SFA) are employed to provide two frequency estimates. These estimates are used to determine if a UAV is present in the detection environment. The data size is reduced using a region of interest (ROI), and this improves the system efficiency and improves azimuth estimation accuracy. Detection results are obtained using real UAV RF signals obtained experimentally which show that the proposed method is more effective than other well-known detection algorithms. The recognition rate with this method is close to 100% within a distance of 2.4 km and greater than 90% within a distance of 3 km. Further, multiple UAVs can be detected accurately using the proposed method.