Many membrane proteins are thought to function as dimers or higher oligomers, but measuring membrane protein oligomerization in lipid membranes is particularly challenging. Förster resonance energy transfer (FRET) and fluorescence cross-correlation spectroscopy are noninvasive, optical methods of choice that have been applied to the analysis of dimerization of single-spanning membrane proteins. However, the effects inherent to such two-dimensional systems, such as the excluded volume of polytopic transmembrane proteins, proximity FRET, and rotational diffusion of fluorophore dipoles, complicate interpretation of FRET data and have not been typically accounted for. Here, using FRET and fluorescence cross-correlation spectroscopy, we introduce a method to measure surface protein density and to estimate the apparent Förster radius, and we use Monte Carlo simulations of the FRET data to account for the proximity FRET effect occurring in confined two-dimensional environments. We then use FRET to analyze the dimerization of human rhomboid protease RHBDL2 in giant plasma membrane vesicles. We find no evidence for stable oligomers of RHBDL2 in giant plasma membrane vesicles of human cells even at concentrations that highly exceed endogenous expression levels. This indicates that the rhomboid transmembrane core is intrinsically monomeric. Our findings will find use in the application of FRET and fluorescence correlation spectroscopy for the analysis of oligomerization of transmembrane proteins in cell-derived lipid membranes.