The present study aimed at distinguishing processing of early learned L2 words from late ones for Chinese natives who learn English as a foreign language. Specifically, we examined whether the age of acquisition (AoA) effect arose during the arbitrary mapping from conceptual knowledge onto linguistic units. The behavior and ERP data were collected when 28 Chinese-English bilinguals were asked to perform semantic relatedness judgment on word pairs, which represented three stages of word learning (i.e., primary school, junior and senior high schools). A 3 (AoA: early vs. intermediate vs. late) × 2 (regularity: regular vs. irregular) × 2 (semantic relatedness: related vs. unrelated) × 2 (hemisphere: left vs. right) × 3 (brain area: anterior vs. central vs. posterior) within-subjects design was adopted. Results from the analysis of N100 and N400 amplitudes showed that early learned words had an advantage in processing accuracy and speed; there is a tendency that the AoA effect was more pronounced for irregular word pairs and in the semantic related condition. More important, ERP results showed early acquired words induced larger N100 amplitudes for early AoA words in the parietal area and more negative-going N400 than late acquire words in the frontal and central regions. The results indicate the locus of the AoA effect might derive from the arbitrary mapping between word forms and semantic concepts, and early acquired words have more semantic interconnections than late acquired words.