Crowd analysis in general and counting in congested scenes, in particular, is an effective and vibrant research domain in computer vision due to its numerous applications. Understanding the risk analysis and safety aspects of crowd dynamics at various vital occasions related to sports cultural and religious activities, specifically, at Hajj and Umrah, is essential. Thousands of people gathered in a small area to carry out their rites. Localizing and counting the annotated head points is quite challenging due to occlusion and large-scale variation in the congested environment. To deal with these problems, a small and effective solution is to generate the density maps. However, the significant flaws of the density map have a blurry Gaussian blob which is less effective for counting and localizing head annotations in the congested scene. To overcome these issues, we propose Congested Scene Crowd Counting and Localization Network (CSCCL-Net) with a Focal inverse Distance Transform (FIDT) map that can count and localize the people simultaneously in the highly congested scene. To evaluate the proposed model’s efficiency, extensive tests were performed on the ShanghaiTech part A, ShanghaiTech part B, and JHU-CROWD++ datasets. The proposed model outperforms existing state-of-the-art techniques regarding high accuracy and low Mean Absolute Error (MAE) and Mean Square Error (MSE) values.