For state-of-the-art organic solar cells (OSCs) consisting of a large-bandgap polymer donor and a near-infrared (NIR) molecular acceptor, the control of the HOMO offset is the key to simultaneously achieve small energy loss (E) and high photocurrent. However, the relationship between HOMO offsets and the efficiency for hole separation is quite elusive so far, which requires a comprehensive understanding on how small the driving force can effectively perform the charge separation while obtaining a high photovoltage to ensure high OSC performance. By designing a new family of ZITI-X NIR acceptors (X = S, C, N) with a high structural similarity and matching them with polymer donor J71 forming reduced HOMO offsets, we systematically investigated and established the relationship among the photovoltaic performance, energy loss, and hole-transfer kinetics. We achieved the highest PCEs of 14.05 ± 0.21% in a ternary system (J71:ZITI-C:ZITI-N) that best optimize the balance between driving force and energy loss.