Confronted with the great challenges of globally growing populations and food shortages, society must achieve future food security by increasing grain output and narrowing the gap between potential yields and farmers’ actual yields. This study attempts to diagnose the climatic and agronomic dimensions of oat yield gaps and further to explore their restrictions. A conceptual framework was put forward to analyze the different dimensions of yield gaps and their limiting factors. We quantified the potential yield (Yp), attainable yield (Yt), experimental yield (Ye), and farmers’ actual yield (Ya) of oat, and evaluated three levels of yield gaps in a rain-fed cropping system in North and Northeast China (NC and NEC, respectively). The results showed that there were great differences in the spatial distributions of the four kinds of yields and three yield gaps. The average yield gap between Yt and Ye (YG-II) was greater than the yield gap between Yp and Yt (YG-I). The yield gap between Ye and Ya (YG-III) was the largest among the three yield gaps at most sites, which indicated that farmers have great potential to increase their crop yields. Due to non-controllable climatic conditions (e.g., light and temperature) for obtaining Yp, reducing YG-I is extremely difficult. Although YG-II could be narrowed through enriching soil nutrients, it is not easy to improve soil quality in the short term. In contrast, narrowing YG-III is the most feasible for farmers by means of introducing high-yield crop varieties and optimizing agronomic managements (e.g., properly adjusting sowing dates and planting density). This study figured out various dimensions of yield gaps and investigated their limiting factors, which should be helpful to increase farmers’ yields and regional crop production, as long as these restrictions are well addressed.