Ceramic alkali superionic conductor solid electrolytes (SICEs) play a prominent role in the development of rechargeable alkali-ion batteries, ranging from replacement of organic electrolytes to being used as separators in aqueous batteries. The aqueous stability of SICEs is an important property in determining their applicability in various roles. In this work, we analyze the aqueous stability of twelve well-known Li-ion and Na-ion SICEs using Pourbaix diagrams constructed from first principles calculations. We also introduce a quantitative free energy measure to compare the aqueous stability of SICEs under different environments. Our results show that though oxides are in general more stable in aqueous environments than sulfides and halide-containing chemistries, the cations present play a crucial role in determining whether solid phases are formed within the voltage and pH ranges of interest.