Convolution sparse representation (CSR) is a novel compressive sensing technique proposed in 2016 and provides an excellent framework for extracting the impulses induced by bearing faults and the unevenness of wheel tread. However, its sparsity performance on extracting impulses is sensitive to the improper penalty parameter. So, a novel fault detection method, appropriately sparse impulse extraction, is proposed based on the combination of CSR, estimating the number of atom types (ENA), and crest factor. The type of atoms embedded in vibration signals is estimated by ENA. Aiming at the different types of atoms, the impulses with different sparse characteristic are spanned by CSR with different penalty parameters. The appropriately sparse impulses are selected for fault detection based on the maximal crest factor. The simulation validation, experiment verification, and practical application are conducted to validate the effectiveness of the proposed appropriately sparse impulses extraction. These results show that the proposed appropriately sparse impulse extraction not only can obtain fault-characteristic frequency and its harmonics for fault judgment but also describes the dynamic behaviour between elementary defects and their matching surfaces. In addition, the proposed appropriately sparse impulse extraction can isolate the impulses with different types of atoms and is very suitable for detecting the wheelset bearing faults.