The accurate and prompt recognition of a driver’s cognitive distraction state is of great significance to intelligent driving systems (IDSs) and human-autonomous collaboration systems (HACSs). Once the driver’s distraction status has been accurately identified, the IDS or HACS can actively intervene or take control of the vehicle, thereby avoiding the safety hazards caused by distracted driving. However, few studies have considered the time–frequency characteristics of the driving behavior and vehicle status during distracted driving for the establishment of a recognition model. This study seeks to exploit a recognition model of cognitive distraction driving according to the time–frequency analysis of the characteristic parameters. Therefore, an on-road experiment was implemented to measure the relative parameters under both normal and distracted driving via a test vehicle equipped with multiple sensors. Wavelet packet analysis was used to extract the time–frequency characteristics, and 21 pivotal features were determined as the input of the training model. Finally, a bidirectional long short-term memory network (Bi-LSTM) combined with an attention mechanism (Atten-BiLSTM) was proposed and trained. The results indicate that, compared with the support vector machine (SVM) model and the long short-term memory network (LSTM) model, the proposed model achieved the highest recognition accuracy (90.64%) for cognitive distraction under the time window setting of 5 s. The determination of time–frequency characteristic parameters and the more accurate recognition of cognitive distraction driving achieved in this work provide a foundation for human-centered intelligent vehicles.