Time series forecasting using transformer architecture variants

Context and relevance. Time series forecasting is critically important for decision-making, but classical methods (ARIMA) have limitations, and LSTM suffers from gradient vanishing. Specialized transformers require theoretical justification. Objective. To investigate the application of transformers for time series forecasting and determine the conditions for the effectiveness of their modifications. Hypothesis. Transformers outperform classical methods in forecasting accuracy, and architecture selection depends on data characteristics. Methods and materials. A theoretical analysis of transformer modifications was conducted, along with an experimental comparison of LSTM, baseline transformer, Autoformer, and FEDformer on an electricity consumption dataset (1 million records). Results. The baseline transformer showed the best results (RMSE = 0.5874), surpassing LSTM by 8.7%. The effectiveness of specialized architectures depends on data characteristics: for electricity consumption, the advantages of decomposition were minimal. Conclusions. Architecture selection should be based on the specifics of time series. Hybrid architectures and lightweight versions of transformers for working with limited computational resources are promising directions.