The odorous steroid androstadienone, a putative male chemo-signal, was previously reported to evoke sex differences in hypothalamic activation in adult heterosexual men and women. In order to investigate whether puberty modulated this sex difference in response to androstadienone we measured the hypothalamic responsiveness to this chemo-signal in 39 prepubertal and 41 adolescent boys and girls by means of functional magnetic resonance imaging. We then investigated whether 36 prepubertal children and 38 adolescents diagnosed with Gender Dysphoria (GD; DSM-5) exhibited sex-atypical (in accordance with their experienced gender), rather than sex-typical (in accordance with their natal sex) hypothalamic activations during olfactory stimulation with androstadienone. We found that the sex difference in responsiveness to androstadienone was already present in prepubertal control children and thus likely developed during early perinatal development instead of during sexual maturation. Adolescent girls and boys with GD both responded remarkably like their experienced gender, thus sex-atypical. In contrast, prepubertal girls with GD showed neither a typically male nor female hypothalamic activation pattern and prepubertal boys with GD had hypothalamic activations in response to androstadienone that were similar to control boys, thus sex-typical. We present here a unique data set of boys and girls diagnosed with GD at two different developmental stages, showing that these children possess certain sex-atypical functional brain characteristics and may have undergone atypical sexual differentiation of the brain.