The particle size of sediment is one of the main factors that influence the phosphorus physical adsorption on sediment. In order to eliminate the effect of other components of sediment on the phosphorus physical adsorption the sediment mineral matrices were obtained by removing inorganic matter metal oxides, and organic matter from natural sediments, which were collected from the Nantong reach of the Yangtze River. The results show that an exponential relationship exists between the median particle size (D50) and specific surface area (Sg) of the sediment mineral matrices, and the fine sediment mineral matrix sample has a larger specific surface area and pore volume than the coarse sediment particles. The kinetic equations were used to describe the phosphorus adsorption process of the sediment mineral matrices, including the Elovich equation, quasi-first-order adsorption kinetic equation, and quasi-second-order adsorption kinetic equation. The results show that the quasi-second-order adsorption kinetic equation has the best fitting effect. Using the mass conservation and Langmuir adsorption kinetic equations, a formula was deduced to calculate the equilibrium adsorption capacity of the sediment mineral matrices. The results of this study show that the phosphorus adsorption capacity decreases with the increase of D50, indicating that the specific surface area and pore volume are the main factors in determining the phosphorus adsorption capacity of the sediment mineral matrices. This study will help understand the important role of sediment in the transformation of phosphorus in aquatic environments.