Hydraulic power and other kinds of disturbance in a linear motor-direct drive actuator (LM-DDA) have a great impact on the performance of the system. A mathematical model of the LM-DDA system is established and a double-loop control system is presented. An extended state observer (ESO) with switched gain was utilized to estimate the influence of the hydraulic power and other load disturbances. Meanwhile, Radial Basis Function (RBF) neural network was utilized to optimize the parameters in this intelligent controller. The results of the dynamic tests demonstrate the performance with rapid response and improved accuracy could be attained by the proposed control scheme.