Nanocomposite dental resins with 0, 2, 5, and 10 wt % methacryl polyhedral oligomeric silsesquioxane (POSS) as filler in the resin matrix were prepared by a light curing method.The atomic force microscopy (AFM), fourier transform infrared spectroscopy (FTIR), nanoindentation, and nanoscratch tests were carried out to study the effect of POSS contents on the compatibility, double bond conversion, volumetric shrinkage, hardness, modulus, and resistance of the dental resins. POSS was very uniformly dispersed and showed a good compatibility with the matrix. The double bond conversion increased and the volume reduced with the addition of POSS. As the POSS addition increased, the mechanical properties increased initially. Small addition of POSS remarkably enhanced the hardness and scratch resistance of the resin matrix.