In this study the mechanical and adhesion properties of an experimental methacrylate based dentin bonding system containing a combination of spherical and layered platelet nanoparticles were investigated. The nanoparticles were first modified through surface graft polymerization of methacrylic acid in order to make the particles surface compatible with the bonding matrix resin.Graft free radical polymerization in aqueous media was performed to attach Poly (methacrylic acid) (PMA) chains onto the surface of Na-MMT nanoclay (Cloisite® Na+) and silica nanoparticles (Aerosil® 200). The hybrid PMA grafted nanoparticles (PMA-g-NC-Sil) were characterized using GPC, FTIR, TGA, and X-ray diffraction (XRD). Dentin adhesives containing different amounts of the hybrid modified nanoparticles were photopolymerized and their characteristics were studied using FTIR, TEM, SEM, EDXA, and XRD techniques. The adhesives containing different amounts of PMA-g-NC-Sil were applied to the conditioned human premolar dentin to bond a dental composite to the teeth. The bond strength was then measured by microshear bond strength testing method. The results were analyzed and compared statistically. The stability of PMA-g-NC-Sil dispersion in the dentin adhesive was investigated using separation analysis (LUMi Reader) techniques.The grafting of PMA chains onto the surface of nanoclay was confirmed by FTIR and TGA analytical techniques. The intercalated-exfoliated structure for the nanoclay platelets in the photo-cured adhesive was observed using XRD and TEM. The surface modification of the nanoparticles significantly increased the dispersion stability of the fillers in the adhesive solution. The microshear test results indicated that the incorporation of the PMA-g-NC-Sil nanoparticles significantly enhanced the bond strength to dentin with the highest shear bond strength observed at 0.5 wt%.The incorporation of the PMA modified hybrid nanofillers into the dentin adhesive resulted in a dentin bonding agent with enhanced shear bond strength through reinforcing the adhesive matrix and potential interactions between their carboxylic acid groups and the tooth structure. The dispersion stability of the nanoparticles was also dramatically improved by the surface modification of the nanoparticles.