The wireless monitoring system is always destroyed by the insufficient energy of the sensors in railway. Hence, how to optimize the communication protocol and extend the system lifetime is crucial to ensure the stability of system. However, the existing studies focused primarily on cluster-based or multihop protocols individually, which are ineffective in coping with the complex communication scenarios in the railway wireless monitoring system (RWMS). This study proposes a hybrid protocol which combines the cluster-based and multihop protocols (CMCP) to minimize and balance the energy consumption in different sections of the RWMS. In the first hierarchy, the total energy consumption is minimized by optimizing the cluster quantities in the cluster-based protocol and the number of hops and the corresponding hop distances in the multihop protocol. In the second hierarchy, the energy consumption is balanced through rotating the cluster head (CH) in the subnetworks and further optimizing the hops and the corresponding hop distances in the backbone network. On this basis, the system lifetime is maximized with the minimum and balance energy consumption among the sensors. Furthermore, the hybrid particle swarm optimization and genetic algorithm (PSO-GA) are adopted to optimize the energy consumption from the two-layer hierarchy. Finally, the effectiveness of the proposed CMCP is verified in the simulation. The performances of the proposed CMCP in system lifetime, residual energy, and the corresponding variance are all superior to the LEACH protocol widely applied in the previous research. The effective protocol proposed in this study can facilitate the application of the wireless monitoring network in the railway system and enhance safety operation of the railway.