Adaptive Feature Engineering and Uncertainty Quantification in High-Dimensional Energy Demand Forecasting

Accurate energy demand forecasting is fundamental to modern power grid operations, particularly given the rapid integration of intermittent renewable energy sources and the increasing frequency of extreme weather events. Traditional forecasting models often struggle with the high dimensionality, non-stationarity, and dynamic non-linear relationships present in complex smart grid ecosystems. In this paper, we propose a novel framework that integrates adaptive feature engineering with distribution-free uncertainty quantification for high-dimensional short-to-medium-term energy demand forecasting. Our methodology utilizes an online attention-guided dynamic feature selection mechanism that continuously recalibrates feature weights based on seasonal, temporal, and exogenous weather inputs. To provide reliable risk assessment for grid operators, we couple a deep quantile temporal neural network with conformal prediction techniques, generating statistically rigorous prediction intervals with guaranteed coverage probabilities. Evaluated on a massive, multi-year smart grid dataset comprising hourly load and meteorological observations, our adaptive model achieves a 14.2% reduction in Continuous Ranked Probability Score (CRPS) and an 11.8% decrease in Mean Absolute Percentage Error (MAPE) relative to state-of-the-art static baselines. Furthermore, the proposed uncertainty quantification pipeline maintains nominal target coverage under severe environmental volatility, offering grid operators an actionable tool for optimal reserve margin allocation and risk management.