Time Series Forecasting

A time series is any sequence of observations indexed in time: daily sales, hourly electricity demand, monthly unemployment, second-by-second stock prices, annual rainfall, real-time server CPU usage. Time series forecasting predicts future values based on historical patterns - one of the most universally applicable ML tasks, with applications in virtually every industry.

Classical statistical methods dominated time series for decades. ARIMA (AutoRegressive Integrated Moving Average) models the current value as a linear function of recent past values and recent forecast errors. Exponential smoothing (Holt-Winters) weights recent observations more heavily with exponentially decreasing weights. Seasonal decomposition separates a time series into trend, seasonality, and residual components. Facebook's Prophet model packages a generalised additive model with automatic seasonality and changepoint detection into a user-friendly package suitable for business forecasting.

Neural network approaches to time series began with RNNs and LSTMs, which model sequential dependencies by maintaining a hidden state that captures information from prior time steps. LSTMs addressed the vanishing gradient problem that made vanilla RNNs ineffective for long-range dependencies. Temporal Convolutional Networks (TCN) apply dilated causal convolutions for efficient long-range temporal modelling with parallelisable computation. WaveNet (DeepMind, originally for audio) demonstrated that dilated convolutions could capture patterns at multiple timescales efficiently.

Transformer-based models have become dominant for time series following their success in NLP. Informer, Autoformer, FEDformer, and PatchTST adapt the Transformer architecture for time series, handling the computational challenge of applying full attention to long sequences. PatchTST divides the time series into patches (analogous to image patches in vision Transformers) and applies attention over patch tokens, achieving state-of-the-art results on long-horizon forecasting benchmarks.

N-BEATS and N-HiTS are deep learning architectures specifically designed for time series without the inductive biases of recurrent or attention architectures, using residual stacking and hierarchical interpolation. TimesFM and Moirai are recent foundation model approaches: trained on billions of time series points from diverse domains, they achieve strong zero-shot forecasting on new time series without task-specific fine-tuning.

Probabilistic forecasting extends point forecasts to full predictive distributions, quantifying uncertainty. DeepAR (Amazon) trains autoregressive models that output distribution parameters at each step. Normalising flow-based models and diffusion models for time series generate full probabilistic forecast trajectories.

Key challenges: non-stationarity (the statistical properties of the series change over time), multiple seasonalities (daily, weekly, annual patterns superimposed), irregular sampling, missing data, and distributional shift when exogenous events (economic shocks, policy changes) alter the underlying data generating process.