The ever-growing concerns about making buildings more energy efficient and increasing the share of renewable energy used in them, has led to the development of ultra-low carbon buildings or passive houses. However, a huge potential still exists to lower the hot water energy demand, especially by harnessing heat from waste water exiting these buildings. Reusing this heat makes buildings more energy-efficient and this source is considered as a third-generation renewable energy technology, both factors conforming to energy policies throughout the world. Based on several theoretical and experimental studies, the potential to harness non-industrial waste water is quite high. As an estimate about 3.5 kWh of energy, per person per day could be harnessed and used directly, in many applications. A promising example of such an application, are low temperature fourth generation District Heating grids, with decentralized sources of heat. At the moment, heat exchangers and heat pumps are the only viable options to harness non-industrial waste heat. Both are used at different scales and levels of the waste-water treatment hierarchical pyramid. Apart from several unfavourable characteristics of these technologies, the associated exergetic efficiencies are low, in the range of 20–50%, even when cascaded combinations of both are used. To tackle these shortcomings, several promising trends and technologies are in the pipeline, to scavenge this small-scale source of heat to a large-scale benefit.