Understanding large-scale crop growth and its responses to climate change are critical for yield estimation and prediction, especially under the increased frequency of extreme climate and weather events. County-level corn phenology varies spatially and interannually across the Corn Belt in the United States, where precipitation and heat stress presents a temporal pattern among growth phases (GPs) and vary interannually. In this study, we developed a long short-term memory (LSTM) model that integrates heterogeneous crop phenology, meteorology, and remote sensing data to estimate county-level corn yields. By conflating heterogeneous phenology-based remote sensing and meteorological indices, the LSTM model accounted for 76% of yield variations across the Corn Belt, improved from 39% of yield variations explained by phenology-based meteorological indices alone. The LSTM model outperformed least absolute shrinkage and selection operator (LASSO) regression and random forest (RF) approaches for end-of-the-season yield estimation, as a result of its recurrent neural network structure that can incorporate cumulative and nonlinear relationships between corn yield and environmental factors. The results showed that the period from silking to dough was most critical for crop yield estimation. The LSTM model presented a robust yield estimation under extreme weather events in 2012, which reduced the root-mean-square error to 1.47 Mg/ha from 1.93 Mg/ha for LASSO and 2.43 Mg/ha for RF. The LSTM model has the capability to learn general patterns from high-dimensional (spectral, spatial, and temporal) input features to achieve a robust county-level crop yield estimation. This deep learning approach holds great promise for better understanding the global condition of crop growth based on publicly available remote sensing and meteorological data.