In four experiments, a total of 205 participants studied individual color patches and were given an old-new recognition test after a brief retention interval (0.5 or 5.0 s). The pattern of false alarms revealed hue memory shifting away from the prototype of the hue’s basic color category. The shifts demonstrate that hue memory is influenced by categorization early in processing. The shifts did not depend on intentional categorization; the shifts were found even when participants made preference ratings at encoding rather than labeling judgments. Overall, we found that categorization and memory are deeply intertwined from perception onward. We discuss the impact of the results on theories of memory and categorization, including the effects of category labels on memory (e.g., Lupyan, 2008). We also put forward the hypothesis that atypical shifts in hue are related to atypical shifts that have previously observed in face recognition (Rhodes et al., 1987).