Similarity between targets and distracters is a key factor in generating distractibility, and exerts a large detrimental effect on aging. The present EEG study tested the role of a new stimulus dimension in generating distractibility in visual Working Memory (vWM), namely numerical similarity. In a change detection paradigm a varying number of relevant and irrelevant stimuli were presented simultaneously in opposite hemifields. Behavioral results indicated that young participants outperformed older individuals; however, in both groups numerical similarity per se did not modulate performance. At the electrophysiological level, in young participants the Contralateral Delay Activity (CDA, a proxy for item maintenance in vWM) was modulated by the numerosity of the relevant items regardless of numerical similarity. In older participants, the CDA was modulated by target numerosity only in the same numerical condition, where the total number of (relevant and irrelevant) items increased with increasing target numerosities. No effect was present in the dissimilar numerical condition, where the total number of items did not vary substantially across target numerosity. This pattern was suggestive of an age-related effect of the total number of (relevant and irrelevant) items on vWM. The additional analyses on alpha-band lateralization measures support this interpretation by revealing that older adults lacked selective deployment of attentional and vWM resources towards the relevant hemifield. Overall, the results indicate that, while numerical similarity does not modulate distractibility, there is an age-related redistribution of vWM resources across the two visual fields, ultimately leading to a general decrease in task performance of older adults.