Cyclocarya paliurus has been used commonly to treat diabetes in China. However, the effective components and the effect of plant origin remain unclear. In this study, C. paliurus leaves with different chemical compositions were selected from five geographical locations, and their effects on streptozotocin (STZ)-induced diabetic mice were evaluated with both ethanol and aqueous extracts. Glucose levels, lipid levels, and biomarkers of liver and kidney function were measured. The principal components of both C. paliurus ethanol and aqueous extracts from different geographical locations differed quantitatively and qualitatively. Results showed that C. paliurus extracts with better antihyperglycemic effects were characterized by higher contents of total flavonoids, especially quercetin-3-O-glucuronide and kaempferol-3-O-glucuronide. Furthermore, significantly negative correlations were found between triterpenoids contents and lipid levels. These results revealed the potential antihyperglycemic capacity of C. paliurus flavonoids and the antihyperlipidemic effect of C. paliurus triterpenoids. Thus, we suggest that the composition of C. paliurus compounds might help to design therapeutic alternatives for the treatment of diabetes mellitus. However, geographic origins and the extraction solvents can also affect the effectiveness of the treatment as these factors influence the chemical compositions and thereby the biological activities.