The effect of deep-level defects is a key issue for the applications of CdZnTe high-flux photon counting devices of X-ray irradiations. However, the major trap energy levels and their quantitive relationship with the device’s performance are not yet clearly understood. In this study, a 16-pixel CdZnTe X-ray photon counting detector with a non-uniform counting performance is investigated. The deep-level defect characteristics of each pixel region are analyzed by the current–voltage curves (I–V), infrared (IR) optical microscope photography, photoluminescence (PL) and thermally stimulated current (TSC) measurements, which indicate that the difference in counting performance is caused by the non-uniformly distributed deep-level defects in the CdZnTe crystals. Based on these results, we conclude that the CdZnTe detectors with a good photon counting performance should have a larger Te cd 2 + and Cd vacancy-related defect concentration and a lower A-center and Tei concentration. We consider the deep hole trap Tei, with the activation energy of 0.638–0.642 eV, to be the key deep-level trap affecting the photon counting performance. In addition, a theoretical model of the native defect reaction is proposed to understand the underlying relationships of resistivity, deep-level defect characteristics and photon counting performance.