To overcome the influence of the nonlinear friction on the gimbaled servo-system of an inertial stabilized platforms (ISPs) with DC motor direct-drive, the methods of modeling and compensation of the nonlinear friction are proposed. Firstly, the inapplicability of LuGre model when trying to interpret the backward angular displacement in the prestiction regime is observed experimentally and the reason is deduced theoretically. Then, based on the dynamic model of direct-drive ISPs, a modified LuGre model is proposed to describe the characteristic of the friction in the prestiction regime. Furthermore, the state switch condition of the three friction regimes including presliding, gross sliding and prestiction is presented. Finally, a composite compensation controller including a nonlinear friction observer and a feedforward compensator based on the novel LuGre model is designed to restrain the nonlinear friction and to improve the control precision. Experimental results indicate that compared with those of the conventional proportion–integration–differentiation (PID) control method and the PID plus LuGre model-based friction compensation method, the dwell-time has decreased from 0.2 s to almost 0 s, the position error decreased to 86.7% and the peak-to-peak value of position error decreased to 80% after the novel compensation controller is added. It concludes that the composite compensation controller can greatly improve the control precision of the dynamic sealed ISPs.