Measuring body sizes accurately and rapidly for optimal garment fit detection has been a challenge for fashion retailers. Especially for apparel e-commerce, there is an increasing need for digital and convenient ways to obtain body measurements to provide their customers with correct-fitting products. However, the currently available methods depend on cumbersome and complex 3D reconstruction-based approaches. In this paper, we propose a novel smartphone-based body size measurement method that does not require any additional objects of a known size as a reference when acquiring a subject’s body image using a smartphone. The novelty of our proposed method is that it acquires measurement positions using body proportions and machine learning techniques, and it performs 3D reconstruction of the body using measurements obtained from two silhouette images. We applied our proposed method to measure body sizes (i.e., waist, lower hip, and thigh circumferences) of males and females for selecting well-fitted pants. The experimental results show that our proposed method gives an accuracy of 95.59% on average when estimating the size of the waist, lower hip, and thigh circumferences. Our proposed method is expected to solve issues with digital body measurements and provide a convenient garment fit detection solution for online shopping.