The asphalt-aggregate interface interaction (AAI) plays a significant role in the overall performances of asphalt mixture, which is caused due to the complicated physicochemical processes and is influenced by various factors, including the acid-base property of aggregates. In order to analyze the effects of the chemical constitution of aggregate on the AAI, the average structure C65H74N2S2 is selected to represent the asphaltene in asphalt and magnesium oxide (MgO), calcium oxide (CaO), aluminium sesquioxide (Al2O3), and silicon dioxide (SiO2) are selected to represent the major oxides in aggregate. The molecular models are established for asphaltene and the four oxides, respectively, and the molecular dynamics (MD) simulation was conducted for the four kinds of asphaltene-oxide system at different temperatures. The interfacial energy in MD simulation is calculated to evaluate the AAI, and higher value means better interaction. The results show that interfacial energy between asphaltene and oxide reaches the maximum value at 25°C and 80°C and the minimum value at 40°C. In addition, the interfacial energy between asphaltene and MgO was found to be the greatest, followed by CaO, Al2O3, and SiO2, which demonstrates that the AAI between asphalt and alkaline aggregates is better than acidic aggregates.