Optical imaging through diffusive, visually-opaque barriers and around corners is an important challenge in many fields, ranging from defense to medical applications. Recently, novel techniques that combine time-of-flight (TOF) measurements with computational reconstruction have allowed breakthrough imaging and tracking of objects hidden from view. These light detection and ranging (LiDAR)-based approaches require active short-pulsed illumination and ultrafast time-resolved detection. Here, bringing notions from passive radio detection and ranging (RADAR) and passive geophysical mapping approaches, we present an optical TOF technique that allows passive localization of light sources and reflective objects through diffusive barriers and around corners. Our approach retrieves TOF information from temporal cross-correlations of scattered light, via interferometry, providing temporal resolution that surpasses state-of-the-art ultrafast detectors by three orders of magnitude. While our passive approach is limited by signal-to-noise to relatively sparse scenes, we demonstrate passive localization of multiple white-light sources and reflective objects hidden from view using a simple setup.