Accurate indoor positioning technology provides location-based service for a variety of applications. However, most existing indoor localization approaches (e.g., Wi-Fi and Bluetooth-based methods) rely heavily on positioning infrastructure, which prevents their large-scale deployment and limits the range at which they are applicable. Here, we proposed an infrastructure-free indoor positioning and tracking approach, termed LiMag, which used ubiquitous magnetic field and ambient lights (e.g., fluorescent, incandescent, and light-emitting diodes (LEDs)) without containing modulated information. We conducted an in-depth study on both the advantages and the challenges in leveraging magnetic field and ambient light intensity for indoor localization. Based on the insights from this study, we established a hybrid observation model that took full advantage of both the magnetic field and ambient light signals. To address the low discernibility of the hybrid observation model, LiMag first generated a single-step fingerprint model by vectorizing consecutive hybrid observations within each step. In order to accurately track users, a lightweight single-step tracking algorithm based on the single-step fingerprints and the particle filter framework was designed. LiMag leveraged the walking information of users and several single-step fingerprints to generate long trajectory fingerprints that exhibited much higher location differentiation ability than the single-step fingerprint. To accelerate particle convergence and eliminate the accumulative error of single-step tracking algorithm, a long trajectory calibration scheme based on long trajectory fingerprints was also introduced. An undirected weighted graph model was constructed to decrease the computational overhead resulting from this long trajectory matching. In addition to typical indoor scenarios including offices, shopping malls and parking lots, we also conducted experiments in more challenging scenarios, including large open-plan areas as well as environments characterized by strong sunlight. Our proposed algorithm achieved a 75th percentile localization accuracy of 1.8 m and 2.2 m, respectively, in the office and shopping mall tested. In conclusion, our LiMag algorithm provided location-based service of infrastructure-free with significantly improved localization accuracy and coverage, as well as satisfactory robustness inside complex indoor environments.