CO5420: Neural Networks & Deep Learning — Course Project
Department of Computer Engineering, Faculty of Engineering, University of Peradeniya, Sri Lanka

👥 Project Team & Contributors

Team Members

H.M.H.N. Aberathna

E/22/001

e22001@eng.pdn.ac.lk

T.H. Abeywickrama

E/22/008

e22008@eng.pdn.ac.lk

M.A.N.P. Anawarathna

E/22/027

e22027@eng.pdn.ac.lk

S.H.S. Hansara

E/22/130

e22130@eng.pdn.ac.lk

W.A.H. Sathsarani

E/22/362

e22362@eng.pdn.ac.lk

Project Supervisors

Dr. Damayanthi Herath

Senior Lecturer, Department of Computer Engineering

damayanthiherath@eng.pdn.ac.lk

Dr. Sampath Deegalla

Senior Lecturer, Department of Computer Engineering

sampath@eng.pdn.ac.lk

📑 Table of Contents

1. Executive Summary
2. Project Progression & Breakthroughs
3. Dataset & Problem Formulation
4. Advanced Feature Engineering Pipeline
5. Model Architectures
   • PyTorch Deep ResNet-1D BiLSTM
   • TensorFlow Bidirectional RNN Baselines
   • Gradient Boosted Decision Trees (GBDTs)
6. Convex SLSQP Ensembling & Blending
7. Experimental Results & Benchmark Tracking
8. Hardware Acceleration & Compute Infrastructure
9. Repository Structure
10. Getting Started & Reproduction
11. Academic Links & Resources

🌟 Executive Summary

Airborne fine particulate matter with aerodynamic diameter under 2.5 micrometers (PM2.5) constitutes a severe public health hazard, penetrating deep into human pulmonary and cardiovascular systems. Due to turbulent boundary-layer meteorology, complex photochemical transformations, wind transport kinematics, and sharp seasonal heating shifts, predicting hourly PM2.5 concentrations is an inherently non-linear spatio-temporal challenge.

This project engineer delivers a state-of-the-art forecasting system trained on multi-year hourly observational data across 12 national air quality monitoring stations in Beijing.

Starting from standard recurrent neural network baselines (LSTM, GRU, BiLSTM with RMSE 15.09 – 15.71), our solution progressed into an industry-grade Advanced SOTA Pipeline combining:

• Atmospheric Physics & Chemistry Domain Feature Engineering (166+ features including wind vector decomposition, Magnus relative humidity, dew point depression, ventilation index, and photochemical ratios).
• Custom Deep Temporal Architectures: Residual 1D Convolutional networks coupled with 2-layer Bidirectional LSTMs (Deep ResNet-1D BiLSTM).
• Hardware Acceleration: Mixed-precision CUDA acceleration on NVIDIA RTX 6000 Ada Generation (48 GB VRAM) with an in-memory streaming pipeline.
• Convex SLSQP Ensemble Blending: Achieving a top clean and verified benchmark score of 14.04457 (Test 3) and an Out-Of-Fold RMSE of 14.14202.

🚀 Project Progression & Breakthroughs

flowchart TD
    A[Initial RNN Baselines<br/>BiLSTM / GRU / LSTM<br/>RMSE: 15.09670] --> B[Exploratory Iterations<br/>Batch 32, Preprocessing, Augmentation<br/>RMSE: 15.16 - 15.71]
    B --> C[Advanced Pipeline Test 1<br/>166 Domain Features + 4-Model Ensemble<br/>RMSE: 14.36675]
    C --> D[Advanced Pipeline Test 2<br/>5-Model SLSQP Blending<br/>Score: 13.90609 - Weight Bug Identified]
    D --> E[Advanced Pipeline Test 3<br/>Pure CPU Multi-Core SOTA Pipeline<br/>RMSE: 14.04457 🏆 Best Verified]
    E --> F[Advanced Pipeline Test 4 & 5<br/>NVIDIA RTX 6000 Ada GPU In-Memory<br/>RMSE: 14.29812]

📊 Dataset & Problem Formulation

1. Dataset Overview

The dataset contains continuous, multi-year hourly meteorological and air pollutant records across 12 air quality monitoring stations in Beijing:

• Monitoring Stations: Aotizhongxin, Changping, Dingling, Dongsi, Guanyuan, Gucheng, Huairou, Nongzhanguan, Shunyi, Tiantan, Wanliu, Wanshouxigong.
• Pollutants: PM2.5, PM10, SO2, NO2, CO, O3.
• Meteorological Factors: Temperature (TEMP), Atmospheric Pressure (PRES), Dew Point (DEWP), Precipitation (RAIN), Wind Direction (wd), Wind Speed (WSPM).
• Scale: Over 315,648 hourly training observations.

2. Task Formulation

Given an hourly sliding observation window $X_{t-24:t}$ over the preceding 24 hours of atmospheric and chemical observations at station $s$, forecast the 1-hour ahead particulate concentration:

• Evaluation Metric: Root Mean Squared Error (RMSE)

🔬 Advanced Feature Engineering Pipeline

The final pipeline converts raw multivariate inputs into an information-dense 166-dimensional feature space grounded in atmospheric science and time-series kinematics:

🧠 Model Architectures

1. PyTorch Deep ResNet-1D BiLSTM

To leverage both localized feature abstractions and long-range sequential memory, we designed a custom Deep ResNet-1D BiLSTM in PyTorch:

• Input Projection: Linear layer projecting 166 input dimensions to a 256-dimensional latent space.
• Residual Blocks (ResNet1DBlock): Two stacked residual blocks with identity skip connections: \(\mathbf{h}_{\text{res}} = \text{ReLU}\left(\mathbf{W}_2 \cdot \text{ReLU}(\mathbf{W}_1 \mathbf{h} + \mathbf{b}_1) + \mathbf{b}_2 + \mathbf{h}\right)\)
• Bidirectional Temporal Core: 2-layer Bidirectional LSTM ($128$ hidden units per direction, dropout = 0.2) capturing bidirectional sequential dynamics.
• Regression Head: Multi-layer perceptron (Linear(256 -> 128) → ReLU → Dropout(0.1) → Linear(128 -> 64) → ReLU → Linear(64 -> 1)).
• Target Normalization: StandardScaler on features and targets, with cosine/plateau learning rate decay (ReduceLROnPlateau).

2. TensorFlow/Keras Bidirectional LSTM & GRU Baseline

Built in Air_Pollution_Forecasting_Using_Temporal_NN_new.ipynb for initial benchmark analysis:

• Station-aware Sequence Generator: Generates 24-hour temporal arrays without crossing station boundaries ((Samples, 24, 43)).
• BiLSTM Model: Input (24, 43) → Bidirectional(LSTM(128, return_sequences=True)) → Dropout(0.2) → Bidirectional(LSTM(64)) → Dense(64, activation='relu') → Dense(1)
• BiGRU Model: Input (24, 43) → Bidirectional(GRU(128, return_sequences=True)) → Dropout(0.2) → Bidirectional(GRU(64)) → Dense(64, activation='relu') → Dense(1)
• Target Compression: $\log(1 + y)$ target transformation with np.log1p to stabilize high-value outlier gradients.

3. Gradient Boosted Decision Trees (GBDTs)

• LightGBM Regressor: Fast histogram-based gradient boosting (learning_rate=0.025, num_leaves=63, max_depth=8, colsample_bytree=0.65, subsample=0.8).
• XGBoost Regressor: GPU-accelerated histogram algorithm (tree_method='hist', learning_rate=0.025, max_depth=8, reg_alpha=0.5, reg_lambda=1.5).
• CatBoost Regressor: Symmetric oblivious tree boosting robust against categorical overfitting (iterations=105000 with 100-round early stopping, learning_rate=0.025, depth=8).
• ExtraTrees Regressor: Extremely randomized trees (n_estimators=300, max_depth=16) providing uncorrelated ensemble diversity.

⚖️ Convex SLSQP Ensembling & Blending

Rather than naive uniform averaging, we applied Sequential Least Squares Programming (SLSQP) convex optimization to find the mathematically optimal weight vector $\mathbf{w} = [w_1, w_2, \dots, w_M]^T$:

⚠️ Diagnostic Note on Test 2 (13.90609): In Test 2, a weight convergence anomaly was identified during diagnostic evaluation. Hence, Test 3 (14.04457) stands as our verified, reproducible, and clean production benchmark.

📈 Experimental Results & Benchmark Tracking

⚡ Hardware Acceleration & Compute Infrastructure

Training 105,000 boosting estimators across 5 folds and deep recurrent neural networks over 315,000+ time-series records demands specialized compute:

• GPU Compute: NVIDIA RTX 6000 Ada Generation (Compute Capability 8.9)
• VRAM: 47.37 GB GDDR6 with Error-Correcting Code (ECC)
• Software Stack: Python 3.12, PyTorch 2.1.3+cu130, CUDA 13.0
• CPU Optimizations: OpenMP thread management (n_jobs=8) on LightGBM to prevent thread lock contention.
• In-Memory Streaming: Fold predictions cached directly in high-speed unified GPU memory to eliminate disk I/O bottlenecks.

📁 Repository Structure

Air-Pollution-Forecasting-Using-Temporal-NNs/
│
├── README.md                                          # Root repository documentation
├── powershell.cmd                                     # Environment execution shim
│
├── docs/                                              # 🌐 Live GitHub Pages Documentation
│   ├── README.md                                      # Site home page
│   ├── _config.yml                                    # Jekyll site configuration
│   ├── images/                                        # Assets (Banner, Logo, SVGs)
│   │   ├── VectorX.png                                # Official Team Logo
│   │   ├── project_banner.svg                         # High-res project banner
│   │   ├── architecture_pipeline.svg                  # Model architecture
│   │   ├── ensemble_weights.svg                       # SLSQP weights breakdown
│   │   └── benchmark_chart.svg                        # Experimental performance chart
│   └── data/                                          # Portal metadata & cover images
│       ├── index.json                                 # Project card metadata
│       ├── cover_page.jpg                             # Panoramic cover banner (940x352)
│       └── thumbnail.jpg                              # Project thumbnail (640x360)
│
└── code/                                              # 🔬 Core Codebase & Experimental Pipelines
    ├── train_raw.csv                                  # Primary Beijing air quality dataset
    ├── Air_Pollution_Forecasting_Using_Temporal_NN_new.ipynb  # RNN Baseline (LSTM, GRU, BiLSTM)
    │
    ├── advanced pipeline model/                       # 🌟 SOTA High-Performance Pipeline
    │   ├── test 1/                                    # 166 Features + 4-Model Ensemble (14.36675)
    │   ├── test 2/                                    # 5-Model SLSQP Ensemble (13.90609)
    │   ├── test 3/                                    # Pure CPU SOTA Pipeline (14.04457 🏆)
    │   ├── test 4/                                    # GPU Ada CUDA Diagnostics
    │   ├── test 5/                                    # In-Memory GPU Ada Pipeline (14.29812)
    │   └── test 7/                                    # Alternative Validation Setup (14.55754)
    │
    ├── 3 model/                                       # Baseline 3-model exploration
    ├── 3 model with 32 batch/                         # Batch size 32 test
    ├── new mdel with pre processing steps/            # Preprocessing iteration 1
    ├── add more pre procesing steps/                  # Preprocessing iteration 2
    └── increased train data with test data/           # Augmented training set test

🛠️ Getting Started & Reproduction

1. Environment Setup

# Clone the repository
git clone https://github.com/cepdnaclk/e22-co542-Air-Pollution-Forecasting-Using-Temporal-NNs-Team-VectorX.git
cd e22-co542-Air-Pollution-Forecasting-Using-Temporal-NNs-Team-VectorX/code

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install required dependencies
pip install lightgbm xgboost catboost scikit-learn pandas numpy torch tensorflow matplotlib seaborn scipy joblib

2. Running the Verified SOTA Pipeline (Recommended)

Open code/advanced pipeline model/test 3/new-cpu-pipeline.ipynb to execute the verified CPU benchmark:

jupyter notebook "advanced pipeline model/test 3/new-cpu-pipeline.ipynb"

3. Running the GPU High-Power Pipeline (CUDA Mode)

Open code/advanced pipeline model/test 5/gpu_ada (2).ipynb in a CUDA-enabled GPU environment:

jupyter notebook "advanced pipeline model/test 5/gpu_ada (2).ipynb"

🔗 Academic Links & Resources

• Project Repository
• Live Project Page
• Department of Computer Engineering
• Faculty of Engineering, University of Peradeniya
• University of Peradeniya