Granular media is the second-most-manipulated substance on Earth, second only to water. However, simulation of granular media is still challenging due to the complexity of granular materials and the large number of discrete solid particles. As we know, dry granular materials could form a hybrid state between a fluid and a solid, therefore we propose a two-layer model and divide the simulation domain into a dilute layer, where granules can move freely as a fluid, and a dense layer, where granules act more like a solid. Motivated by the shallow water equations, we derive a set of shallow sand equations for modeling dry granular flows by depth-integrating three-dimensional governing equations along its vertical direction. Unlike previous methods for simulating a 2D granular media, our model does not restrict the depth of the granular media to be shallow anymore. To allow efficient fluid-solid interactions, we also present a ray casting algorithm for one-way solid-fluid coupling. Finally, we introduce a particle-tracking method to improve the visual representation. Our method can be efficiently implemented based on a height field and is fully compatible with modern GPUs, therefore allows us to simulate large-scale dry granular flows in real time.