Train / Validation / Test Lab

Predict first: If you inspect test error for five candidate models and choose the model with the smallest test error, is that test error still an unbiased final assessment?

The experiment

TrainingValidationTestFresh future dataSelected model

Training
60%

Validation
20%

Test
20%

Candidate models

Degree	Train RMSE	Validation RMSE	Test RMSE

Run the methodology

Example dataset

Selected degree3

Decision basisManual

Step 1. Fit each candidate on training data. The fitted coefficients come only from green points.

Step 2. Lock the model. Only then may we use the red test data for the final estimate.

Test values are hidden. Choose a model using training/validation information.

Final test RMSE🔒

Fresh future RMSE🔒

What should students notice?

Training error answers “how well did we fit the examples used to estimate the weights?” It is not a fair basis for choosing capacity.

Validation error can be used to choose degree, architecture, k, regularization, or another model choice.

Test error is meaningful as a final assessment only while the test examples have not influenced model selection.

The purple “future” sample exists only to demonstrate what happens after test-set peeking. In practice we do not keep inventing new test sets.

Why do we need three data splits?

The experiment

Candidate models