One of the applications of bone marrow stromal cells (BMSCs) that are produced by ex vivo expansion is for use in in vivo bone tissue engineering. Cultured stromal cells are a mixture of cells at different stages of commitment and expansion capability, leading to a heterogeneous cell population that each time can differ in the potential to form in vivo bone. A parameter that predicts for in vivo bone forming capacity is thus far lacking. We employed single colony-derived BMSC cultures to identify such predictive parameters. Using limiting dilution, we have produced sixteen single CFU-F derived BMSC cultures from human bone marrow and found that only five of these formed bone in vivo. The single colony-derived BMSC strains were tested for proliferation, osteogenic-, adipogenic- and chondrogenic differentiation capacity and the expression of a variety of associated markers. The only robust predictors of in vivo bone forming capacity were the induction of alkaline phosphatase, (ALP) mRNA levels and ALP activity during in vitro osteogenic differentiation. The predictive value of in vitro ALP induction was confirmed by analyzing “bulk-cultured” BMSCs from various bone marrow biopsies. Our findings show that in BMSCs, the additional increase in ALP levels over basal levels during in vitro osteogenic differentiation is predictive of in vivo performance.