Aiming at the problem of low detection accuracy of small targets such as helmets and self-rescuers in complex scenarios in coal mines, a small target detection method based on improved Real-Time DEtection TRansformer (RTDETR) for underground coal mines is proposed. A new BasicBlock-PConv module was created by incorporating Partial Convolutions (PConv) into the conventional BasicBlock, which was based on the FasterNet network. This decreased the number of network parameters and computation. By introducing Deformable Attention in the coding part of the RTDETR algorithm, the deformable feature of this attention mechanism is used to improve the network's ability to extract effective image features. In order to increase the accuracy of tiny object detection and concentrate on the detail information in the shallow feature map, the small object detection layer P2 is simultaneously added to the Head of the coding section. Based on the improvement of the above three parts, the improved PDP-RTDETR working model in this paper achieves a Mean Average Precision (mAP) of 56.6% for detecting small targets on the self-constructed dataset, which is 11.2, 12.1, 9.9, and 5.2% better than that of the traditional models Yolov5s, Yolov7-Tiny, Yolov8n, and RTDETR, respectively. Meanwhile, the improved PDP-RTDETR algorithm parameter count is reduced by 2.6 M compared to the base model. The results suggest that the approach can successfully increase the detection accuracy of small targets in the mine scene, which gives a certain reference value for the application of small target detection.