Goats play a major role in poor marginalized communities of South Africa for food security and socio-economic purposes. Majority of the goats are raised in villages with poor infrastructure and resources, therefore facing challenges that affect growth performance which leads to low mature weights. Investigating growth profiles will shed light on growth performances and will aid in goat improvement and selection. This study investigated the growth profiles and genomic structure of SA indigenous breeds raised in different production systems to unravel the genetic potential of indigenous goat populations. Live weights and morphological body measurements were collected from a total of 83 kids representing the commercial meat-producing SA Boer (n = 14); the indigenous veld goats (IVG) of NC Skilder (n = 14), Mbuzi (n = 13), and Xhosa lob (n = 14) raised under intensive systems; and nondescript village goat populations (n = 14) raised in intensive and others (n = 14) raised in extensive production systems. The remaining 72 of 83 phenotyped goats were genotyped using the Illumina Caprine SNP50K BeadChip. The SA Boer had a higher weight (28.96 ± 0.30 kg) gain as compared to other populations. The Mbuzi population was the smallest (14.83 ± 0.33 kg), while the village goats raised in Pella Village were relatively smaller (17.55 ± 0.37 kg) than those raised on the research farm (19.55 ± 0.36 kg). The study concluded that both genetics and management systems can lead to improved growth performance in goat production. The outputs of this study can be used to identify suitable breeds and potential genotypes for optimal growth and establish optimal goat management systems suitable for communal farmers for improved productivity.