In the present work, finite element (FE) analysis has been performed for evaluating effective axial Young’s modulus of CNT-based composites considering different types of representative volume elements (RVEs). Different types of matrix materials from low strength to high strength have been considered to observe the effect of axial Young’s modulus on matrix materials. Effect of volume fraction on axial Young’s modulus has also been investigated. The present computational results are also compared with the theoretical results obtained based on rule of mixture (ROM). It has been found that the cylindrical RVE underestimated the Young’s modulus but the square and hexagonal RVEs give more closer results to those obtained based on ROM. It has also been observed that the axial Young’s modulus of CNT-based composites for all the RVE models increase with increase in volume fraction.