Oxytocin has been shown to stimulate social approach behaviors, although effects may depend on contextual and individual difference factors. Here, we examined intranasal oxytocin effects on interpersonal distance using an immersive Virtual Reality paradigm, taking into account early caregiving experiences and interpersonal context as potential moderators. Participants were 180 women who received 24 IU oxytocin or a placebo and had reported how often their mother used love withdrawal as a disciplinary strategy, involving withholding love and affection after a failure or misbehavior. We used a virtual stop-distance paradigm, instructing participants to approach a virtual person or to stop an approaching virtual person at a preferred distance (passive approach). In order to examine the role of interpersonal context in shaping oxytocin effects, facial expressions and bodily gestures of the virtual person were manipulated. The person showed a dynamical expression of sadness, happiness, anger, fear, disgust, or no emotional expression in six different emotion conditions. We found that oxytocin reduced interpersonal distance across the different emotion conditions, but only in individuals with lower levels of love withdrawal. In addition, oxytocin reduced anxiety levels during passive approach, in particular in the disgust condition, but only in individuals with lower levels of maternal disciplinary love withdrawal. Individuals with more love withdrawal experienced more anxiety while being approached by a virtual person displaying disgust or fear, but benefitted less from anxiety-reducing oxytocin effects. These results are consistent with previous research showing a dysregulated oxytocinergic system after childhood adversity and indicate that oxytocin may be less effective for individuals who are most in need of an intervention because of a problematic family background.