Most of the RANS turbulence models solve the Reynolds stress by linear hypothesis with isotropic model. They can not capture all kinds of vortexes in the turbomachineries. In this paper, an improved nonlinear k-ε turbulence model is proposed, which is modified from the RNG k-ε turbulence model and Wilcox's k-ω turbulence model. The Reynolds stresses are solved by nonlinear methods. The nonlinear k-ε turbulence model can calculate the near wall region without the use of wall functions. The improved nonlinear k-ε turbulence model is used to simulate the flow field in a curved rectangular duct. The results based on the improved nonlinear k-ε turbulence model agree well with the experimental results. The calculation results prove that the nonlinear k-ε turbulence model is available for high pressure gradient flows and large curvature flows, and it can be used to capture complex vortexes in a turbomachinery.