To develop a dual-modal neural network model to characterize ultrasound (US) images of breast masses.A combined US B-mode and color Doppler neural network model was developed to classify US images of the breast. Three datasets with breast masses were originally detected and interpreted by 20 experienced radiologists according to Breast Imaging-Reporting and Data System (BI-RADS) lexicon ((1) training set, 103212 masses from 45,433 + 12,519 patients. (2) held-out validation set, 2748 masses from 1197 + 395 patients. (3) test set, 605 masses from 337 + 78 patients). The neural network was first trained on training set. Then, the trained model was tested on a held-out validation set to evaluate agreement on BI-RADS category between the model and the radiologists. In addition, the model and a reader study of 10 radiologists were applied to the test set with biopsy-proven results. To evaluate the performance of the model in benign or malignant classifications, the receiver operating characteristic curve, sensitivities, and specificities were compared.The trained dual-modal model showed favorable agreement with the assessment performed by the radiologists (κ = 0.73; 95% confidence interval, 0.71-0.75) in classifying breast masses into four BI-RADS categories in the validation set. For the binary categorization of benign or malignant breast masses in the test set, the dual-modal model achieved the area under the ROC curve (AUC) of 0.982, while the readers scored an AUC of 0.948 in terms of the ROC convex hull.The dual-modal model can be used to assess breast masses at a level comparable to that of an experienced radiologist.• A neural network model based on ultrasonic imaging can classify breast masses into different Breast Imaging-Reporting and Data System categories according to the probability of malignancy. • A combined ultrasonic B-mode and color Doppler neural network model achieved a high level of agreement with the readings of an experienced radiologist and has the potential to automate the routine characterization of breast masses.