Interfacial properties play a vital role in spin currents injection from ferromagnetic (FM) layer into non-magnetic (NM) layer. So far, impedance matching and spin-orbit coupling are two important well-known factors in spin current transport in FM/NM heterostructures. In this work, the spin current transport in Y3Fe5O12 (YIG)/NiO/Pt heterostructures was investigated by spin Hall magnetoresistance and inverse spin Hall effect measurements. By inserting the layer of the antiferromagnetic insulator NiO, the magnetic proximity effect affecting the Pt atoms owing to YIG and the anomalous spin Hall voltage can be efficiently blocked. Ferromagnetic resonance and spin pumping measurements verified that the ferromagnetic/antiferromagnetic exchange coupling inhibits transmission of the spin current at the YIG/NiO interface when the NiO layer is thick. Atomic force microscopy and spherical aberration corrected transmission electron microscopy proved that the strong interfacial roughness enhanced spin scattering between NiO and Pt can greatly increase both the inverse spin Hall voltage and spin Hall magnetoresistance when the NiO layer is thin or even discontinuous. This interface roughness dominated spin scattering mechanism based on YIG/NiO/Pt heterostructure is a new discovery and there is significant potential for exploiting this mechanism in the construction of low-dissipation spintronic devices with an efficient spin current injection.