Automatic extraction of ground points, called filtering, is an essential step in producing Digital Terrain Models from airborne LiDAR data. Scene complexity and computational performance are two major problems that should be addressed in filtering, especially when processing large point cloud data with diverse scenes. This paper proposes a fast and intelligent algorithm called Semi-Global Filtering (SGF). The SGF models the filtering as a labeling problem in which the labels correspond to possible height levels. A novel energy function balanced by adaptive ground saliency is employed to adapt to steep slopes, discontinuous terrains, and complex objects. Semi-global optimization is used to determine labels that minimize the energy. These labels form an optimal classification surface based on which the points are classified as either ground or non-ground. The experimental results show that the SGF algorithm is very efficient and able to produce high classification accuracy. Given that the major procedure of semi-global optimization using dynamic programming is conducted independently along eight directions, SGF can also be paralleled and sped up via Graphic Processing Unit computing, which runs at a speed of approximately 3 million points per second.