This paper presents an efficient, numerically stable algorithm for parallel transport of tangent vectors in the group of diffeomorphisms. Previous approaches to parallel transport in large deformation diffeomorphic metric mapping (LDDMM) of images represent a momenta field, the dual of a tangent vector to the diffeomorphism group, as a scalar field times the image gradient. This "scalar momenta" constraint couples tangent vectors with the images being deformed and leads to computationally costly horizontal lifts in parallel transport. This paper uses the vector momenta formulation of LDDMM, which decouples the diffeomorphisms from the structures being transformed, e.g., images, point sets, etc. This decoupling leads to parallel transport expressed as a linear ODE in the Lie algebra. Solving this ODE directly is numerically stable and significantly faster than other LDDMM parallel transport methods. Results on 2D synthetic data and 3D brain MRI demonstrate that our algorithm is fast and conserves the inner products of the transported tangent vectors.