Lung motion due to respiration causes image degradation in medical imaging, especially in nuclear medicine which requires long acquisition times. We have developed a method for image correction between the respiratory-gated (RG) PET images in different respiration phases or breath-hold (BH) PET images in an inconsistent respiration phase. In the method, the RG or BH-PET images in different respiration phases are deformed under two criteria: similarity of the image intensity distribution and smoothness of the estimated motion vector field (MVF). However, only these criteria may cause unnatural motion estimation of lung. In this paper, assuming the use of a PET-CT scanner, we add another criterion that is the similarity for the motion direction estimated from inhalation and exhalation CT images. The proposed method was first applied to a numerical phantom XCAT with tumors and then applied to BH-PET image data for seven patients. The resultant tumor contrasts and the estimated motion vector fields were compared with those obtained by our previous method. Through those experiments we confirmed that the proposed method can provide an improved and more stable image quality for both RG and BH-PET images.