In an atomic force microscope (AFM) system, the measurement accuracy in the scan images is determined by the displacement accuracy of piezo scanner. The hysteresis model of piezo scanner displacement is complex and difficult to correct, which is the main reason why the output displacement of the piezo scanner does not have high precision. In this study, an image pixel hysteresis model of the piezo scanner displacement in the AFM system was established. An AFM was used to scan a two-dimensional (2D) grating in the 0 ° and 90 ° directions and a polynomial fitting method was employed to obtain the image pixel hysteresis model parameters of the piezo scanner displacement in the X-direction and Y-direction. The image pixel hysteresis model was applied to correct the AFM scan image of regular octagons. The results showed that the relative measurement error in the X-direction was decreased from 12.47% to 0.52% after the correction and that in the Y-direction decreased from 28.57% to 0.35%. The image pixel hysteresis model can be applied in the post-processing software of a commercial AFM system. The model solves the hysteresis problem of the AFM system and improves the measuring accuracy of AFM in 2 degrees of freedom (2 DOF).