Cross Validation and hyperparameter selection correct procedure

Cross‑Validation and Hyperparameter Selection: The Correct Procedure

In the world of artificial intelligence and machine learning, building a model that truly generalises to unseen data is a core challenge. Two concepts lie at the heart of this challenge:

  • Cross‑validation (CV) – a systematic way to estimate a model’s performance on unseen data.
  • Hyperparameter selection – the process of finding the best configuration of algorithmic settings (e.g., learning rate, regularisation strength, tree depth).

When these steps are combined incorrectly, you risk optimism bias – the illusion that your model is better than it really is. Below is the recommended, error‑free workflow for AI practitioners.

1. Split the Data Once: Train, Validation, and Test Sets

Start with a single outer split:

  1. Training set – used for fitting the model.
  2. Validation set (optional) – used for early‑stopping or quick sanity checks.
  3. Test set – held completely untouched until the final evaluation.

Typical ratios are 60‑70% training, 15‑20% validation, and 15‑20% test. The test set must never influence any part of model training or hyperparameter optimisation.

2. Perform Nested Cross‑Validation on the Training Portion

To obtain an unbiased estimate of hyperparameter performance, use nested cross‑validation:

  • Outer loop (e.g., 5‑fold CV) creates multiple training/validation splits. Each outer fold mimics the final train/test split.
  • Inner loop (e.g., 3‑fold CV) runs on the training portion of each outer fold and searches the hyperparameter space.

This structure ensures that the hyperparameter search never sees the data used for the outer‑fold evaluation, preserving a genuine out‑of‑sample estimate.

3. Choose a Hyperparameter Search Strategy

Depending on resources and problem complexity, pick one of the following:

StrategyWhen to Use
Grid SearchSmall parameter spaces, exhaustive exploration needed.
Random SearchLarge spaces; you want a good trade‑off between coverage and speed.
Bayesian Optimisation (e.g., Tree‑Parzen, Gaussian Processes)Expensive models; you need smarter sampling.
Population‑based methods (e.g., Hyperband, BOHB)Very high‑dimensional spaces with limited compute budget.

Regardless of the method, the search must be confined to the inner CV loop. The outer CV’s validation scores are used to pick the best hyperparameter configuration.

4. Evaluate the Chosen Model on the Held‑Out Test Set

After the nested CV process:

  1. Retrain the model on the entire training set using the best hyperparameters identified.
  2. Compute final performance metrics (accuracy, F1, ROC‑AUC, etc.) on the test set that has remained untouched.

This final step provides an unbiased estimate of how the model will behave in production.

5. Common Pitfalls to Avoid

  • Leakage through preprocessing: Any scaling, encoding, or feature engineering must be fitted on the training fold only and applied to validation/test folds separately.
  • Using the test set for early stopping: Early stopping should rely on a separate validation split inside the training data.
  • Repeating hyperparameter search on the full data: Once you have a test‑set estimate, avoid further tuning; otherwise you invalidate the test results.

6. A Minimal Code Blueprint (Python/Scikit‑Learn)

from sklearn.model_selection import GridSearchCV, KFold
from sklearn.metrics import accuracy_score
import numpy as np

# 1. Outer split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)

# 2. Outer CV
outer_cv = KFold(n_splits=5, shuffle=True, random_state=42)

outer_scores = []

for train_idx, val_idx in outer_cv.split(X_train):
    X_tr, X_val = X_train[train_idx], X_train[val_idx]
    y_tr, y_val = y_train[train_idx], y_train[val_idx]

    # 3. Inner CV + hyperparameter search
    param_grid = {'C': [0.01, 0.1, 1, 10], 'kernel': ['linear', 'rbf']}
    inner_cv = KFold(n_splits=3, shuffle=True, random_state=42)
    grid = GridSearchCV(SVC(), param_grid, cv=inner_cv, scoring='accuracy')
    grid.fit(X_tr, y_tr)

    # 4. Evaluate best model on outer validation fold
    best_model = grid.best_estimator_
    pred = best_model.predict(X_val)
    outer_scores.append(accuracy_score(y_val, pred))

print('Nested CV accuracy:', np.mean(outer_scores))

# 5. Final training on full training data
final_model = grid.best_estimator_
final_model.fit(X_train, y_train)
test_acc = accuracy_score(y_test, final_model.predict(X_test))
print('Test accuracy:', test_acc)

Conclusion

The correct procedure for cross‑validation combined with hyperparameter selection is nested cross‑validation, strict data segregation, and a final unbiased test‑set evaluation. Following this workflow protects your AI models from over‑optimistic performance claims and prepares them for reliable deployment in real‑world scenarios.