Introduction Patient set-up optimization is required in radiotherapy to fill the accuracy gap between personalized treatment planning and uncertainties in the irradiation set-up. In this study, we aimed to develop a new method based on neural network to estimate patient geometrical setup using 4-dimensional (4D) XCAT anthropomorphic phantom. Materials and Methods To access 4D modeling of motion of dynamic organs, a phantom employs non-uniform rational B-splines (NURBS)-based Cardiac-Torso method with spline-based model to generate 4D computed tomography (CT) images. First, to generate all the possible roto-translation positions, the 4D CT images were imported to Medical Image Data Examiner (AMIDE). Then, for automatic, real time verification of geometrical setup, an artificial neural network (ANN) was proposed to estimate patient displacement, using training sets. Moreover, three external motion markers were synchronized with a patient couch position as reference points. In addition, the technique was validated through simulated activities by using reference 4D CT data acquired from five patients. Results The results indicated that patient geometrical set-up is highly depended on the comprehensiveness of training set. By using ANN model, the average patient setup error in XCAT phantom was reduced from 17.26 mm to 0.50 mm. In addition, in the five real patients, these average errors were decreased from 18.26 mm to 1.48 mm various breathing phases ranging from inhalation to exhalation were taken into account for patient setup. Uncertainty error assessment and different setup errors were obtained from each respiration phase. Conclusion This study proposed a new method for alignment of patient setup error using ANN model. Additionally, our correlation model (ANN) could estimate true patient position with less error.