It has been assumed that one can determine the particular, fixed memory capacity for each person in many working memory literatures. In the study reported here, we investigated whether visual working memory (VWM) capacity was stable irrespective of the number of to-be-remembered objects and ages using an individual-differences approach. Younger and older adult participants performed a change detection task using many objects defined by color. We found wide variability in the VWM capacity estimated by the change detection task, across memory set sizes, ages, and individuals. A marked drop in the number of objects held in VWM was observed in both younger and older adults with low memory capacity, but was not in high-capacity individuals, when set size went well beyond the limits of VWM capacity. In addition, a drop in the number of objects held in VWM was prevented in low-capacity younger adults by increasing time for VWM encoding, whereas a drop was still observed in low-capacity older adults. These findings suggest that low-capacity individuals are subject to a drop in VWM capacity induced by overload, and that aging exacerbates this deficit such that it cannot be recovered by simply increasing the encoding time for objects. Overall, our findings challenge the prevailing assumption that VWM capacity is fixed and stable, and call for a revision of the strict view that VWM capacity is constrained by a fixed number of distinct slots in which high-resolution object representations are stored.