In this paper, the analysis of thermal instability of rotating nanofluid layer is revised with a physically more realistic boundary condition on the nanoparticle volumetric fraction i.e. the nanoparticle flux is assumed to be zero rather than prescribing the nanoparticle volumetric fraction on the rigid impermeable boundaries. This shows that the nanoparticle fraction value at the boundary adjusts accordingly. In this respect, the present model is more realistic physically than those of the previous investigations. The numerical computations are presented for water-based nanofluids with alumina and copper nanoparticles. For Alumina-water nanofluid, zero flux nanoparticle boundary condition has more destabilizing effect than the constant nanoparticle boundary conditions, while reverse for copper–water nanofluid. The effect of rotation is found to have a stabilizing effect. Further, volumetric fraction of nanoparticles ϕ0∗, the Lewis number Le, the density ratio Rρ and modified diffusivity ratio NA are found to destabilize the system.