The top tension riser (TTR) is one of the most frequently used equipment in deep-sea petroleum engineering. At present, the research methods of its vortex-induced vibration (VIV) are mainly focused on finite element analysis and experiment. The understanding of its various nonlinear mechanical mechanisms would be inadequate via limited numerical or experimental studies rather than nonlinear analysis of its rich dynamics. Based on the Van der Pol wake oscillator model, the nonlinear dynamic model of the TTR subject to shear flow VIV is established. The proposed model includes the fluid-structure interaction of the TTR under shear flow. Dynamic behavior of the TTR in association with the variation of flow velocity is investigated. The dynamic behavior is simulated by computing the local maximum displacement response via the fifth-order Galerkin discretization. The Poincare map is then utilized to quantify the dynamic property of TTR under each individual flow velocity, which helps identifying the bifurcation path of the nonlinear system. The time history, phase diagram, FFT spectrum, and envelope diagram about the riser VIV at typical flow velocity in different regions of the bifurcation diagram are then given. It is found that the VIV response of the TTR depicts the Hopf bifurcation phenomena with bistable characteristics. Together with the structural eigen-analysis and the three-dimensional spectrum contour, the main dynamic features of the TTR in shear flow are more comprehensively understood. Such understandings may provide new ideas and references for the design and optimization of the riser structural parameters.