To improve respiratory-gated radiotherapy accuracy, we developed a machine learning approach for markerless tumor tracking and evaluated it using lung cancer patient data. Digitally reconstructed radiography (DRR) datasets were generated using planning 4DCT data. Tumor positions were selected on respective DRR images to place the GTV center of gravity in the center of each DRR. DRR subimages around the tumor regions were cropped so that the subimage size was defined by tumor size. Training data were then classified into two groups: positive (including tumor) and negative (not including tumor) samples. Machine learning parameters were optimized by the extremely randomized tree method. For the tracking stage, a machine learning algorithm was generated to provide a tumor likelihood map using fluoroscopic images. Prior probability tumor positions were also calculated using the previous two frames. Tumor position was then estimated by calculating maximum probability on the tumor likelihood map and prior probability tumor positions. We acquired treatment planning 4DCT images in eight patients. Digital fluoroscopic imaging systems on either side of the vertical irradiation port allowed fluoroscopic image acquisition during treatment delivery. Each fluoroscopic dataset was acquired at 15 frames per second. We evaluated the tracking accuracy and computation times. Tracking positional accuracy averaged over all patients was 1.03 ± 0.34 mm (mean ± standard deviation, Euclidean distance) and 1.76 ± 0.71 mm (95th percentile). Computation time was 28.66 ± 1.89 ms/frame averaged over all frames. Our markerless algorithm successfully estimated tumor position in real time.