Efficient retrieval of the synaptic vesicle (SV) membrane from the presynaptic plasma membrane, a process called endocytosis, is crucial for the fidelity of neurotransmission, particularly during sustained neural activity. Although multiple modes of endocytosis have been identified, it is clear that the efficient retrieval of the major SV cargos into newly formed SVs during any of these modes is fundamental for synaptic transmission. It is currently believed that SVs are eventually reformed via a clathrin-dependent pathway. Various adaptor proteins recognize SV cargos and link them to clathrin, ensuring the efficient retrieval of the cargos into newly formed SVs. Here, we summarize our current knowledge of the molecular signatures within individual SV cargos that underlie efficient retrieval into SV membranes, as well as discuss possible contributions of the mechanisms under physiological conditions.