Mimicking the nanostructure of bone and understanding the interactions between the nanoscale inorganic and organic components of the extracellular bone matrix are crucial for the design of biomaterials with structural properties and a functionality similar to the natural bone tissue. Generally, these interactions involve anionic and/or cationic functional groups as present in the organic matrix, which exhibit a strong affinity for either calcium or phosphate ions from the mineral phase of bone. This study reviews the interactions between the mineral and organic extracellular matrix components in bone tissue as a source of inspiration for the design of novel nanocomposites. After providing a brief description of the various structural levels of bone and its main constituents, a concise overview is presented on the process of bone mineralization as well as the interactions between calcium phosphate (CaP) nanocrystals and the organic matrix of bone tissue. Bioinspired synthetic approaches for obtaining nanocomposites are subsequently addressed, with specific focus on chemical groups that have affinity for CaPs or are involved in stimulating and controlling mineral formation, that is, anionic functional groups, including carboxyl, phosphate, sulfate, hydroxyl, and catechol groups.