This paper proposes an ultrasound video interpretation algorithm that enables novel classes or instances to be added over time, without significantly compromising prediction abilities on prior representations. The motivating application is diagnostic fetal echocardiography analysis. Currently in clinical practice, recording full diagnostic fetal echocardiography is not common. Diagnostic videos are typically available in varying length and summarize a number of diagnostic sub-tasks of varying difficulty. Although large clinical datasets may be available at onset to build ultrasound image-based models for automatic image analysis, data may also become available over extended time to assist in algorithm refinement. To address this scenario, we propose to use an incremental learning approach to build a hierarchical network model that allows for a parallel inclusion of previously unseen anatomical classes without requiring prior data distributions. Super classes are obtained by coarse classification followed by fine classification to allow the model to self-organize anatomical structures in a sequence of categories through a modular architecture. We show that this approach can be adapted with new variable data distributions without significantly affecting previously learned representations. Two extreme situations of new data addition are considered; (1) when new class data is available over time with volume and distribution similar to prior available classes, and (2) when imbalanced datasets arrive over future time to be learned in a few-shot setting. In either case, availability of data from prior classes is not assumed. Evolution of the learning process is validated using incremental accuracies of fine classification over novel classes and compared to results from an end-to-end transfer learning-derived model fine-tuned on a clinical dataset annotated by experienced sonographers. The modularization of subsequent learning reduces the depreciation in future accuracies over old tasks from 6.75% to 1.10% using balanced increments. The depreciation is reduced from 6.95% to 1.89% with imbalanced data distributions in future increments, while retaining competitive classification accuracies in new additions of fine classes with parameter operations in the same order of magnitude in all stages in both cases.