In this study, we present several demonstrations of color averaging between luminance boundaries. In each of the demonstrations, different black outlines are superimposed on one and the same colored surface. Whereas perception without these outlines comprises a blurry colored gradient, superimposing the outlines leads to a much clearer binary color percept, with different colors perceived on each side of the boundary. These demonstrations show that the color of the perceived surfaces is flexible, depending on the exact shape of the outlines that define the surface, and that different positioning of the outlines can lead to different, distinct color percepts. We argue that the principle of color averaging described here is crucial for the brain in building a useful model of the distal world, in which differences within object surfaces are perceptually minimized, while differences between surfaces are perceptually enhanced.