Self-Supervised Vision Transformers for Crop Yield Prediction Using Multi-Spectral Satellite Imagery Streams

Accurate and timely crop yield prediction is critical for global food security, agricultural planning, and supply chain management. While deep learning models, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have shown promise in analyzing multi-spectral satellite imagery streams, they often suffer from the scarcity of labeled ground-truth yield data and struggle to capture complex, long-range spatial-temporal dependencies. In this paper, we propose a novel self-supervised learning framework based on Vision Transformers (ViTs) tailored for multi-spectral satellite imagery streams. Our approach leverages a Masked Autoencoder (MAE) pre-training strategy adapted for multi-spectral bands to learn robust spatial-spectral representations from vast quantities of unlabeled Sentinel-2 imagery. Subsequently, we fine-tune the pre-trained ViT backbone coupled with a temporal Transformer encoder to predict end-of-season crop yields. Evaluating our model on a multi-year dataset of corn and soybean yields in the US Midwest, the self-supervised ViT framework outperforms state-of-the-art supervised models (including 3D-CNNs and ConvLSTMs), achieving an R-squared (R²) of 0.84 for corn and 0.81 for soybeans. Notably, our self-supervised model demonstrates superior generalization and robustness under low-labeled data regimes, reducing the required labeled training data by up to 60% while maintaining competitive predictive performance. This study highlights the potential of self-supervised foundation models in remote sensing to advance sustainable agriculture and climate resilience.