Thermophotovoltaic (TPV) systems can convert radiant energy into electrical power. Here we explore the design of the TPV system cavity, which houses the emitter and the photovoltaic (PV) cells. Mirrors are utilized in the cavity to modify the spatial and spectral distribution within. After discussing the basic concentric tubular design, two novel cavity configurations are put forward and parametrically studied. The investigated variables include the shape, number, and placement of the mirrors. The optimization objectives are the optimized efficiency and the extended range of application of the TPV system. Through numerical simulations, the relationship between the design parameters and the objectives are revealed. The results show that careful design of the cavity configuration can markedly enhance the performance of the TPV system.