This paper presents a new set-membership based hybrid Kalman filter (SM-HKF) by combining the Kalman filtering (KF) framework with the set-membership concept for nonlinear state estimation under systematic uncertainty consisted of both stochastic error and unknown but bounded (UBB) error. Upon the linearization of the nonlinear system model via a Taylor series expansion, this method introduces a new UBB error term by combining the linearization error with systematic UBB error through the Minkowski sum. Subsequently, an optimal Kalman gain is derived to minimize the mean squared error of the state estimate in the KF framework by taking both stochastic and UBB errors into account. The proposed SM-HKF handles the systematic UBB error, stochastic error as well as the linearization error simultaneously, thus overcoming the limitations of the extended Kalman filter (EKF). The effectiveness and superiority of the proposed SM-HKF have been verified through simulations and comparison analysis with EKF. It is shown that the SM-HKF outperforms EKF for nonlinear state estimation with systematic UBB error and stochastic error.