In Network Security, there is a major issue to secure the public or private network from abnormal users. It is because each network is made up of users, services and computers with a specific behavior that is also called as heterogeneous system. To detect abnormal users, anomaly detection system (ADS) is used. In this paper, we present a novel and hybrid Anomaly Detection System with the uses of IP gray space analysis and dominant scanning port identification heuristics used to detect various anomalous users with their potential behaviors. This methodology is the combination of both statistical and rule based anomaly detection which detects five types of anomalies with their three types of potential behaviors and generates respective alarm messages to GUI.