London Energy Consumption Forecasting

End-to-end data science project: joining weather and energy datasets, imputing missing values with a learned model, and predicting daily borough-level electricity consumption using a deep learning regression model with modern architectural patterns.

Final test set performance: 1.61% MAPE (R² = 0.998), beating a linear regression baseline (8.88% MAPE) and a borough-mean baseline (25.24% MAPE).

---

What this project does

1. Loads and joins two London datasets — daily weather (1979–2021) and daily residential energy consumption (2011–2014, 33 boroughs).
2. Explores both datasets statistically and visually, including a folium map of total consumption per borough.
3. Imputes ~4,000 missing cloud_cover values by training a deep learning classifier on the rows where it is present.
4. Engineers features including calendar variables (day-of-week, month, weekend indicator) and one-hot encoded boroughs.
5. Splits chronologically (not randomly) into train/test so the test period is strictly later than training — the right way to evaluate a time-series model.
6. Compares against baselines (borough-mean, linear regression) before training the deep model, so improvements are honestly measured.
7. Trains a deep learning regressor with inverted bottlenecks, residual connections, and GELU activations.
8. Evaluates with sklearn RMSE / MAE / MAPE / R², plus residual diagnostics and a per-borough error breakdown.
9. Documents limitations and sources of bias.

Results

Model	Test RMSE (MWH)	Test MAE (MWH)	Test MAPE
Borough-mean baseline	139.66	119.67	25.24%
Linear regression	44.98	34.38	8.88%
Deep learning (this project)	10.57	8.16	1.61%

R² on the test set: 0.998.

The deep learning model beats linear regression by ~5× in MAPE — non-linear interactions between weather, calendar, and borough are real and substantial.

Architecture

The deep learning models use the same modern architectural pattern for both the classifier (cloud cover imputation) and the regressor (energy prediction):

• Inverted bottleneck blocks: expand the hidden dimension to 4× width, then project back. The same pattern used in MobileNetV2 and modern transformer feed-forward layers.
• Residual connections around each block, easing gradient flow and acting as implicit ensembles.
• GELU activations rather than ReLU — smoother and used in modern transformers.
• Layer normalisation before each block.
• Dropout for regularisation.
• Adam optimiser with early stopping on validation loss.

Repository structure

london-energy-forecasting/
├── README.md                                   ← this file
├── requirements.txt
├── .gitignore
├── data/
│   └── raw/
│       ├── london_weather.csv                  ← daily weather, 1979–2021
│       ├── london_energy.csv                   ← daily borough energy, 2011–2014
│       └── README.md                           ← data sources and licensing
├── notebooks/
│   └── london_energy_forecasting.ipynb         ← the main notebook (pre-executed)
└── reports/
    └── figures/                                ← saved plots from the notebook run
        ├── 01_temperature_timeseries.png
        ├── 02_weather_correlation.png
        ├── 03_cloud_cover_distribution.png
        ├── 04_consumption_by_borough.png
        ├── 05_london_energy_map.html           ← interactive folium map
        ├── 06_classifier_training.png
        ├── 07_regressor_training.png
        ├── 08_residual_diagnostics.png
        └── 09_error_by_borough.png

How to run it

Option 1 — view only

Open notebooks/london_energy_forecasting.ipynb directly on GitHub. All cells are pre-executed with outputs and figures inline.

Option 2 — re-run locally

# Clone the repo
git clone https://github.com/IngaPosiunaite/london-energy-forecasting.git
cd london-energy-forecasting

# Create a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate          # macOS / Linux
# .venv\Scripts\activate            # Windows

# Install dependencies
pip install -r requirements.txt

# Launch Jupyter
jupyter lab notebooks/london_energy_forecasting.ipynb

End-to-end runtime on CPU: ~2 minutes (~30 seconds for the classifier, ~60 seconds for the regressor with early stopping).

Datasets

Both datasets are pre-aggregated versions of public Kaggle datasets, included in data/raw/:

File	Source	Records
london_weather.csv	London Weather Data (Kaggle)	15,341 daily records
london_energy.csv	London Homes Energy Data (Kaggle)	27,357 records (33 boroughs × 829 days)

See data/raw/README.md for details and licensing notes.

Methodology notes

A few decisions worth flagging for anyone reviewing the work:

• Chronological train/test split. A random 80/20 split on this panel would let the model see future weather during training. The split here uses a strict date cutoff (2013-09-16) so the test period is entirely later than the training period.
• Standardisation with training-only statistics. The mean and standard deviation used to standardise features are computed on the training set only, then applied to both. Computing them on the full dataset would leak test information into training.
• Baselines come first. A 1.6% MAPE sounds impressive in isolation, but it only means something compared to what a trivial model achieves. The borough-mean baseline (~25% MAPE) tells us that borough alone explains most of the variance; the linear regression baseline (~9% MAPE) tells us how much of the remaining signal is linear; the deep model captures the rest.
• Cloud cover imputation is honest. The classifier achieves 31% test accuracy on a 10-way problem (vs 16% for the most-frequent-class baseline). That's not high in absolute terms but is meaningful relative to the difficulty. The imputed values feed the downstream regressor — a noise source documented in the limitations.

Limitations

Documented in detail in the notebook (section 12). Summary:

• Borough is acting as a proxy for unobserved variables (housing type, dwelling count, income, fuel mix).
• The energy data is from 2011–2014 — heat pumps, EVs, and post-COVID demand patterns are not reflected.
• The test set covers only one season (autumn/winter 2013–14); a rolling-window evaluation would be more thorough.
• ~26% of weather rows had cloud cover imputed by a model with imperfect accuracy.

Tech stack

• Python 3.10+
• pandas, numpy
• matplotlib, seaborn, folium (visualisation)
• scikit-learn (baselines, metrics)
• TensorFlow / Keras (deep learning)
• Jupyter

Background

This project began as the capstone assessment for the PDA Data Science (Level 8) at Glasgow Clyde College (2026), covering four SQA units: Working with Data, Communicating with Data, Data Science Project, and Machine Learning. It has been substantially restructured and extended for a portfolio audience — adding baselines, fixing the time-aware split, adding calendar feature engineering, residual diagnostics, per-borough error analysis, and a documented limitations section.

Inga Posiunaite · GitHub