A visual-inertial odometer is used to fuse the image information obtained by a vision sensor with the data measured by an inertial sensor and recover the motion track online in a global frame. However, in an indoor environment, geometric transformation, sparse features, illumination changes, blurring, and noise will occur, which will either cause a reduction in or failure of the positioning accuracy. To solve this problem, a map matching algorithm based on an indoor plane structure map is proposed to improve the positioning accuracy of the system; this algorithm was implemented using a conditional random field model. The output of the attitude information from the visual-inertial odometer was used as the input of the conditional random field model. The feature function between the attitude information and the expected value was established, and the maximum probabilistic value of the attitude was estimated. Finally, the closed-loop feedback correction of the visual-inertial system was carried out with the probabilistic attitude value. A number of experiments were designed to verify the feasibility and reliability of the positioning method proposed in this paper.