A carbon fiber-reinforced polymer (CFRP) is a light and rigid composite applicable in various fields, such as in aviation and automobile industry. However, due to its low thermal conductivity, it does not dissipate heat sufficiently and thus accumulates heat stress. Here, we reported a facile and effective strategy to improve the through-thickness thermal conductivity of CFRP composites by using a layer-by-layer coating of inorganic crystals. They could provide efficient heat transfer pathways through layer-by-layer contact within the resulting composite material. The high thermally conductive CFRP composites were prepared by employing three types of inorganic crystal fillers composed of aluminum, magnesium, and copper on prepreg through the layer-by-layer coating process. The vertical thermal conductivity of pure CFRP was increased by up to 87% on using magnesium filler at a very low content of 0.01 wt %. It was also confirmed that the higher the thermal conductivity enhancement was, the better were the mechanical properties. Thus, we could demonstrate that the layer-by-layer inclusion of inorganic crystals can lead to improved through-thickness thermal conductivity and mechanical properties of composites, which might find applications in varied industrial fields.