The emergence of quantization at the nanoscale, the quantum size effect (QSE), allows flexible control of matter and is a rich source of advanced functionalities. A QSE-induced transition into an insulating phase in semimetallic nanofilms was predicted for bismuth a half-century ago and has regained new interest with regard to its surface states exhibiting nontrivial electronic topology. Here, we reveal an unexpected mechanism of the transition by high-resolution angle-resolved photoelectron spectroscopy combined with theoretical calculations. Anomalous evolution and degeneracy of quantized energy levels indicate that increased Coulomb repulsion from the surface states deforms a quantum confinement potential with decreasing thickness. The potential deformation strongly modulates spatial distributions of quantized wave functions, which leads to acceleration of the transition beyond the original QSE picture. This discovery establishes a complete picture of the long-discussed transition and highlights a new class of size effects dominating nanoscale transport in systems with metallic surface states.