Two-dimensional (2D) effective continuous models are derived for the surface states and thin films of a three-dimensional topological insulator (3DTI). Starting from an effective model for 3DTI based on first-principles calculations (Zhang et al 2009 Nat. Phys. 5 438), we present solutions for both the surface states in a semi-infinite boundary condition and those in a thin film with finite thickness. The coupling between opposite topological surfaces and structure inversion asymmetry (SIA) gives rise to gapped Dirac hyperbolas with Rashba-like splittings in the energy spectrum. In addition, SIA leads to asymmetric distributions of wavefunctions for the surface states along the film growth direction, making some branches in the energy spectra much harder than others to probe by light. These features agree well with the recent angle-resolved photoemission spectra of Bi2Se3 films grown on SiC substrate (Zhang et al 2009 arXiv:0911.3706). More importantly, using the parameters fitted by experimental data, the result indicates that the thin film Bi2Se3 lies in the quantum spin Hall (QSH) region based on the calculation of the Chern number and Z2 invariant. In addition, strong SIA always tends to destroy the QSH state.