The adhesion between bitumen and aggregates strongly influences the lifetime of pavements. To improve adhesiveness, the road construction industry has been using additives to alter the interfacial energy and improve the affinity of materials in the presence of water. However, the water sensitivity varies according to the mixture design, since the interaction may occur differently depending on the materials chosen. As the use of alternative materials is increasing in road constructions, further analysis of its affinity with aggregates and bitumen is necessary. In that sense, this study evaluates the adhesion performance of mastics mixed with traditional fillers, such as limestone and dolomite, and residues, such as fly ash and red mud. To assess possible interactions with the red mud, the fillers are mixed in distinct percentages and tested for adhesiveness, wettability, penetration, and softening point. The results show the importance of hydrophilicity, asphalt viscosity, and physical-chemical properties to define adhesive interactions.