The biological functions of estrogens are regulated by estrogen receptors (ERα and ERβ), which contribute in the progression of several hormone-responsive cancer types via estrogen signaling mechanisms. The coordinated actions of ERs and extracellular matrix (ECM) macromolecules are principal mediators of ECM remodeling in the tumor and the adjacent stroma. In breast cancer, ERs are critical biomarkers as their expression in breast tumor determines the disease-free survival, yet guiding treatment decisions and predicting prognosis as well as response to endocrine therapy. In this article, we critically survey the current knowledge on dynamic interactions among ERs and major ECM macromolecules and effectors, such as growth factor receptors, proteoglycans and matrix metalloproteinases, in respect to their key effects in cancer progression, cancer cell functional properties, epithelial-to-mesenchymal transition and epigenetics. Understanding the ERs-mediated ECM reorganization during cancer progression may pave way in identifying novel targets for diagnosis and novel therapeutic approaches for cancer management.