Accurate occupancy is crucial for planning for sustainable buildings. Using massive, passively-collected mobile phone data, we introduce a novel framework to estimate building occupancy at unprecedented scale. We show that, at urban-scale, occupancy differs widely from current estimates based on building types. For commercial buildings, we find typical occupancy rates are 5 times lower than current assumptions imply, while for residential buildings occupancy rates vary widely by neighborhood. Our mobile phone based occupancy estimates are integrated with a state-of-the-art urban building energy model to understand their impact on energy use predictions. Depending on the assumed relationship between occupancy and internal building loads, we find energy consumption which differs by +1% to -15% for residential buildings and by -4% to -21% for commercial buildings, compared to standard methods. This highlights a need for new occupancy-to-load models which can be applied at urban-scale to the diverse set of city building types.