The correlation of the wettability of different lithology aggregates and the drying shrinkage of concrete materials is studied, and some influential factors such as wettability and wetting angle are analyzed. A mercury porosimeter is used to measure the porosities of different lithology aggregates accurately, and the pore size ranges that significantly affect the drying shrinkage of different lithology aggregate concretes are confirmed. The pore distribution curve of the different coarse aggregates is also measured through a statistical method, and the contact angle of different coarse aggregates and concrete is calculated according to the linear fitting relationship. Research shows that concrete strength is determined by aggregate strength. Aggregate wettability is not directly correlated with concrete strength, but wettability significantly affects concrete drying shrinkage. In all types’ pores, the greatest impacts on wettability are capillary pores and gel pores, especially for the pores of the size locating 2.5–50 nm and 50–100 nm two ranges.