🤖🧠 Deep mind AI blog series

6. Hyperparameter Tuning Guide

Oct 06, 20254 min read

Why Hyperparameters Have Limited Impact?

They Don’t Add Information, it control HOW the model learns, not WHAT it can learn. In machine learning, the quality and quantity of information in your features matters 10-20× more than hyperparameter tuning.

🎯 Current Performance

  • Validation Accuracy: 76.70%
  • Training Accuracy: 80.94%
  • Overfitting Gap: 4.24%
  • Status: ✅ Target achieved (70-80% range)

🚀 Improvement Strategies

1. Reduce Overfitting (Priority: HIGH)

Current gap of 4.24% suggests mild overfitting.

A. Increase Dropout

# Current: dropout_rate: 0.1
# Try:
dropout_rate: 0.2 # Moderate increase
dropout_rate: 0.3 # Aggressive regularization

Expected Impact: ↓ Train accuracy, ↑ Val accuracy, ↓ Overfitting

B. Add L2 Regularization (Weight Decay)

# Current: weight_decay: 0.0
# Try:
weight_decay: 0.0001 # Light regularization
weight_decay: 0.001 # Medium regularization

Expected Impact: Smoother decision boundaries

2. Optimize Learning Process (Priority: MEDIUM)

A. Learning Rate Tuning

# Current: learning_rate: 0.0005
# Try:
learning_rate: 0.001 # Faster convergence (may overfit)
learning_rate: 0.0003 # Slower, more stable
learning_rate: 0.0007 # Middle ground

B. Enable Learning Rate Scheduling

 
lr_scheduler:
 
enabled: true
 
type: "step"
 
step_size: 20 # Reduce LR every 20 epochs
 
gamma: 0.5 # Multiply LR by 0.5

Expected Impact: Better convergence, avoid plateaus

C. Alternative: Cosine Annealing

 
lr_scheduler:
 
enabled: true
 
type: "cosine"
 
T_max: 50 # Restart every 50 epochs

3. Architecture Experiments (Priority: LOW)

A. Wider Network (More Capacity)

 
hidden1_size: 128 # Current: 64
 
hidden2_size: 64 # Current: 32

Trade-off: More parameters, may increase overfitting

B. Deeper Network (More Layers)

Add a third hidden layer:

 
hidden1_size: 64
 
hidden2_size: 48
 
hidden3_size: 32

C. Narrower Network (Less Overfitting)

 
hidden1_size: 48 # Current: 64
 
hidden2_size: 24 # Current: 32

4. Training Process Optimization

A. Batch Size Experiments

 
# Current: batch_size: 32
 
# Try:
 
batch_size: 64 # Faster training, more stable gradients
 
batch_size: 16 # More updates per epoch, may generalize better

B. Early Stopping Adjustment

 
early_stopping:
 
patience: 30 # Current: 55 (reduce for faster experiments)
 
min_delta: 0.0005 # Current: 0.0001 (more strict)

📊 Recommended Experiment Plan

Experiment 1: Reduce Overfitting

Goal: Improve validation accuracy to 78-80%

 
model:
 
dropout_rate: 0.2 # Increase from 0.1
 
  
 
training:
 
learning_rate: 0.0005 # Keep same
 
weight_decay: 0.0001 # Add L2 regularization

Expected Result: Train ~78%, Val ~78%, Gap ~0-1%

Experiment 2: Learning Rate Schedule

Goal: Better convergence

 
training:
 
learning_rate: 0.001 # Start higher
 
lr_scheduler:
 
enabled: true
 
type: "step"
 
step_size: 15
 
gamma: 0.5

Expected Result: Faster initial learning, better final accuracy

Experiment 3: Optimal Balance

Goal: Best overall performance

 
model:
 
hidden1_size: 64
 
hidden2_size: 32
 
dropout_rate: 0.25 # Balanced regularization
 
  
 
training:
 
learning_rate: 0.0007 # Slightly higher
 
batch_size: 64 # More stable gradients
 
weight_decay: 0.0001
 
lr_scheduler:
 
enabled: true
 
type: "cosine"
 
T_max: 40

Expected Result: Train ~79%, Val ~78-80%, Gap ~1-2%

🧪 How to Run Experiments

Option 1: Manual Config Editing

  1. Edit configs/nn_config.yaml

  2. Run training: python scripts/train_nn.py

  3. Record results

Create a hyperparameter sweep script:

 
import subprocess
 
import yaml
 
  
 
experiments = [
 
{"name": "baseline", "dropout": 0.1, "lr": 0.0005, "wd": 0.0},
 
{"name": "high_dropout", "dropout": 0.2, "lr": 0.0005, "wd": 0.0001},
 
{"name": "lr_schedule", "dropout": 0.2, "lr": 0.001, "wd": 0.0001},
 
]
 
  
 
for exp in experiments:
 
# Update config
 
# Run training
 
# Save results

📈 Performance Tracking

Metrics to Monitor

  1. Validation Accuracy (primary metric)

  2. Training Accuracy (check overfitting)

  3. Loss Curves (convergence behavior)

  4. Training Time (efficiency)

Success Criteria

  • ✅ Val Accuracy > 78%

  • ✅ Overfitting Gap < 3%

  • ✅ Stable training (no oscillations)

🎓 Key Insights

What We Learned

  1. Memory/History Works: 76.7% vs 50% baseline (+26%)

  2. Mild Overfitting: 4.2% gap is manageable

  3. More Info = Better Decisions: 21 features >> 9 features

Next Level Performance (Beyond 80%)

To push beyond 80%, consider:

  1. Solution 2: Multi-modal architecture (separate perception/history paths)

  2. Attention Mechanisms: Let network focus on relevant features

  3. Ensemble Methods: Combine multiple models

  4. More Data: Generate 5000+ environments

⚡ Quick Wins (Try These First)

1. Increase Dropout (30 sec to change)

 
# Edit nn_config.yaml: dropout_rate: 0.1 → 0.2
 
python scripts/train_nn.py

2. Add Weight Decay (30 sec to change)

 
# Edit nn_config.yaml: weight_decay: 0.0 → 0.0001
 
python scripts/train_nn.py

3. Enable LR Scheduling (1 min to change)

 
# Edit nn_config.yaml: lr_scheduler.enabled: false → true
 
python scripts/train_nn.py

📝 Results Template

Track your experiments:


Experiment: [Name]

Date: [Date]

Config Changes:

- dropout_rate: 0.2

- weight_decay: 0.0001

Results:

- Train Acc: ___%

- Val Acc: ___%

- Test Acc: ___%

- Overfitting Gap: ___%

- Training Time: ___ min

Notes:

- [Observations]

- [Next steps]

🎯 Goal Summary

Current: 76.7% validation accuracy ✅

Next Target: 78-80% validation accuracy

Ultimate Goal: 80%+ with <2% overfitting

Expected Improvement Path:

  • Baseline: 50% → Current: 76.7% → Target: 78-80% → Stretch: 80-85%

Graph View

Backlinks