Hyperparameter Tuning with Optuna
Bayesian optimisation over GridSearch. Define a search space, let Optuna find the best hyperparameters with far fewer trials.
GridSearchCV evaluates 200 combinations blindly. Optuna evaluates 30 combinations intelligently — and usually finds a better answer.
Your gradient boosting model has 6 hyperparameters to tune: n_estimators, learning_rate, max_depth, subsample, colsample_bytree, reg_alpha. If you try 4 values per parameter with GridSearchCV, that is 4⁶ = 4,096 combinations. With 5-fold CV each combination trains 5 models — 20,480 model fits. At 30 seconds each, that is 7 days of compute.
RandomizedSearchCV cuts this to 100 random combinations — still blind, just fewer. It does not learn from previous trials. If learning_rate=0.01 consistently underperforms learning_rate=0.1, random search keeps wasting trials on learning_rate=0.01 anyway.
Optuna uses Bayesian optimisation. After each trial it builds a probabilistic model of which hyperparameter regions are likely to produce good scores. It uses this model to choose the next trial — focusing on promising regions and skipping areas already known to be bad. With 30–50 trials it typically matches or beats GridSearchCV on 200+ combinations.
Finding the best hyperparameters is like prospecting for gold in a mountain range. GridSearch digs at every point on a fixed grid — systematic but wasteful. Random search digs at random spots — faster but still uninformed. Optuna is a geologist who studies the rock formations after each dig. If gold appeared near a granite outcrop, they dig near other granite outcrops first. They learn from each result to make the next dig smarter.
Optuna's probabilistic model of the search space is called a surrogate model. The strategy for choosing the next trial from the surrogate is called an acquisition function. Together they make Optuna far more sample-efficient than any exhaustive or random search.
Grid vs Random vs Bayesian — what each one does and when it wins
Optuna in three steps — study, objective, optimize
Optuna has a simple API built around three concepts. A study is the optimisation session — it stores all trials and their results. An objective function is the function Optuna calls for each trial — it receives a trial object, samples hyperparameters from it, trains the model, and returns a score.optimize() runs the objective n_trials times, using previous results to guide each new trial.
Inside the objective function, you use the trial object to suggest hyperparameter values. Optuna chooses values based on its surrogate model — not randomly and not from a fixed grid.
Pruning, callbacks, and persistence — Optuna at scale
For expensive models, Optuna's pruning feature terminates unpromising trials early — after seeing partial results. If a trial looks bad after fold 2 of 5-fold CV, Optuna stops it and moves on. This can cut total compute by 30–50% with no loss in final quality.
Tuning XGBoost and LightGBM with Optuna — the full workflow
In production you will tune XGBoost or LightGBM far more often than sklearn's GradientBoostingClassifier. Both have native Optuna integration. The search spaces for these models are well-established and the code below gives you a production-ready starting template.
A systematic tuning workflow — what to tune first, how many trials
Tuning all hyperparameters simultaneously with a flat search space is inefficient. Some parameters matter far more than others. A systematic order dramatically reduces the trials needed.
Find the right learning rate range. Low lr needs many trees. High lr needs few. Fix the relationship before tuning anything else.
Control model complexity. Deeper trees = more capacity but more overfitting. min_child_samples prevents leaf overfitting.
Fine-tune generalisation. These parameters have diminishing impact — tune after structure is fixed.
Final polish. The search space is now small and well-targeted. Optuna finds the global optimum quickly.
Every common tuning mistake — explained and fixed
How tuning actually happens on a real ML team — not 200 trials on a laptop
Every code example in this module so far runs in a notebook and finishes in a few minutes. On a real team, tuning is a scheduled job, not something an engineer babysits interactively until they get bored of watching numbers scroll by. It runs inside the training pipeline, triggered nightly or weekly, against a fixed compute budget approved in advance — usually expressed as "n_trials, or six hours of GPU time, whichever comes first" rather than "however many the engineer is willing to wait for."
For anything larger than a single-GPU sklearn or LightGBM model — fine-tuning a transformer, tuning a two-tower recommendation model — Optuna alone is not enough, because a single study running on one machine cannot use twenty GPUs at once. Teams reach for Ray Tune, which distributes trials across a cluster and adds early-stopping schedulers like ASHA that kill the worst-performing half of trials at each checkpoint instead of letting every trial run to completion. Every trial — whichever tool ran it — gets logged automatically to an experiment tracker (MLflow or Weights and Biases), because "best_params printed to stdout" does not survive a container restart.
A notebook cell, run interactively, restarted whenever the kernel dies.
A scheduled CI job on a shared cluster, triggered nightly or on a retraining cadence.
GridSearchCV or a single Optuna study, in-memory, no persistence.
Optuna with a Postgres-backed study, or Ray Tune with ASHA across many GPUs at once.
"However many trials I feel like waiting for."
A fixed, approved budget — n_trials or wall-clock time, because GPU-hours cost real money.
If the process dies, the study is gone. Start over.
Persistent storage means a preempted spot-instance job resumes the same study the next night.
best_params printed to the console, copy-pasted into the next cell.
Logged to an experiment tracker, versioned, and promoted to a model registry if it beats the incumbent.
In practice, search ranges are rarely as wide as a tutorial's first attempt. A team that has run a hundred studies on similar gradient boosting models already knows learning_rate below 0.005 never wins and max_depth above seven consistently overfits on their data — so the next study's search space starts narrower, warm-started from what past studies already learned. Pruning stops being optional once a single trial costs real money: fine-tuning a transformer for one trial can take hours, not seconds, so a scheduler like ASHA or Optuna's MedianPruner that kills a clearly-losing trial after the first checkpoint is the difference between a nightly job finishing on time and one that blows through its budget without ever reaching its planned number of trials.
Five things people get wrong about hyperparameter tuning
Grid search guarantees the best combination within the grid you specified — nothing more. If you searched learning_rate in {0.05, 0.1, 0.2} and the true optimum is 0.07, grid search will never find it; it only ever evaluates the discrete points you listed. This is a genuine blind spot, not a minor approximation — the discretisation decision (which values to include, how coarse the spacing is) silently caps how good a result grid search can ever return, no matter how many folds or how much compute you throw at it.
This is backwards for most real search spaces. Bergstra and Bengio's 2012 result shows that when only a few hyperparameters actually matter (nearly always true — max_depth might dominate while min_samples_leaf barely moves the score), grid search wastes a huge fraction of its budget varying unimportant parameters while barely varying the important ones across their range. Random search, by contrast, samples every parameter independently on every trial, so it explores far more distinct values of the parameters that matter, for the same total budget. This is not "sometimes true" — it is the expected outcome whenever hyperparameter importance is uneven.
TPE's advantage comes from learning across trials, which requires trials to run mostly sequentially — each new suggestion depends on the results so far. If you have massive parallel compute (say, 500 machines) and a cheap objective function, running 500 random trials at once often finds a comparably good result faster in wall-clock time than running Optuna's mostly-sequential search, because Optuna's benefit from smarter sampling doesn't outweigh the parallelism it gives up. Bayesian optimisation wins on sample efficiency (fewer trials for the same quality), not on being universally faster or better in every deployment scenario.
Every trial that reuses the same CV folds is effectively another look at the same finite validation data, and Optuna is explicitly searching for whatever configuration scores best on it — including configurations that happen to fit noise in those specific folds rather than genuine signal. This is hyperparameter overfitting, and it happens without ever touching the test set: more trials on a small dataset with few folds increases the chance that the "best" hyperparameters found were the luckiest on this particular split, not the best in general. This module's own error section shows CV AUC of 0.94 collapsing to test AUC of 0.81 for exactly this reason.
Tuning searches for the best configuration of a fixed model family and a fixed set of features — it cannot manufacture signal that isn't in the data, and it cannot make a fundamentally unsuitable algorithm suddenly capture a relationship it structurally cannot represent. A linear model tuned across every regularisation strength available still cannot fit a strongly non-linear relationship; 500 Optuna trials on a feature set missing the one variable that actually explains the target will not out-tune that missing feature. Tuning is the last 5–10% of model quality, applied after the model family and features are already validated as reasonable — not a substitute for getting those right first.
Hyperparameter tuning — 5 questions interviewers actually ask
With 6 hyperparameters, even 4 values each is 4⁶ = 4,096 combinations — combinatorial explosion makes grid search computationally infeasible well before you'd want to stop. Optuna uses Bayesian optimisation (TPE by default): after each trial it builds a probabilistic surrogate model of which regions of the search space score well, and chooses the next trial to focus on promising regions instead of blindly covering the whole grid. In practice 30–50 Optuna trials routinely match or beat 200+ grid search combinations, because Optuna spends its budget where it is likely to pay off instead of spreading it uniformly, including over regions already known to be bad.
Grid search's systematic coverage is actually its weakness when hyperparameters have unequal importance, which is the normal case. Say learning_rate matters a lot and min_samples_leaf barely matters at all — a 3×3 grid only tries 3 distinct learning rates no matter how many min_samples_leaf values you add, wasting evaluations varying a parameter that doesn't move the score. Random search samples every parameter independently on every trial, so for the same 9 trials it tries 9 distinct learning rates. For the same budget, random search covers the important dimensions more densely — this is Bergstra and Bengio's core argument, and it's a structural property of how the two methods spend budget, not an occasional fluke.
This is hyperparameter overfitting: 500 trials searched an enormous space of configurations against the same small set of CV folds, and with only 800 rows some configuration was bound to fit the noise in those specific folds unusually well by chance — that configuration doesn't generalise, hence the 16-point drop on the untouched test set. Fixes: cap n_trials relative to dataset size (30–50 trials for a dataset this small is plenty), use more folds or RepeatedStratifiedKFold so each trial's score is harder to get lucky on, and use nested cross-validation so the hyperparameter search itself is evaluated out-of-sample rather than just the final model. The test set must never be seen during the Optuna study, only for the final, one-time check.
They solve independent problems. The sampler (e.g. TPESampler) decides which hyperparameter values to try next, based on a surrogate model of previous results — it controls where in the search space you look. The pruner (e.g. MedianPruner) decides whether to abandon a trial that is already running, based on its intermediate results — for example, stopping a cross-validation fold early if the score after fold 2 is already worse than the median of completed trials. You want both because they save different kinds of waste: a good sampler avoids wasting whole trials on unpromising regions, while a good pruner avoids wasting compute finishing a trial that's already clearly bad partway through. Combined, they can cut total compute 30–50% with no loss in final quality.
With unlimited compute and an arbitrarily fine grid, grid search's coverage does approach exhaustive search over the space, so in the limit it would find configurations at least as good as anything Optuna finds. But "unlimited compute" is doing a lot of work in that question — grid size grows exponentially with each additional hyperparameter (a finer grid on 6 parameters multiplies trials by that factor across all 6 dimensions simultaneously), so the compute required to make grid search competitive grows far faster than Optuna's requirement for the same quality. The honest answer is: grid search's ceiling is real, but the compute needed to reach it makes it impractical exactly in the high-dimensional cases where Optuna's sample efficiency matters most.
You can tune any model. Next: explain any prediction.
You have built, evaluated, calibrated, and tuned models. The final module in the Evaluation section answers the question stakeholders always ask after seeing the model performance: why did the model make this specific prediction? Module 39 covers SHAP and LIME — the two most widely used techniques for explaining individual predictions from any model. SHAP was introduced briefly in Module 30 for XGBoost. Module 39 covers it comprehensively across all model types including black-box models with no direct feature importance.
Explain any individual prediction. Global feature importance, local explanations, and how to present model decisions to regulators.
🎯 Key Takeaways
- ✓GridSearchCV evaluates every combination exhaustively — combinatorial explosion makes it unusable beyond 3 parameters. RandomizedSearchCV samples n_iter random combinations — better but still learns nothing between trials. Optuna uses Bayesian optimisation (TPE) to focus each new trial on promising regions based on all previous results.
- ✓The Optuna API has three pieces: create_study (the session), an objective function (trains and evaluates one hyperparameter combination, returns a score), and study.optimize (runs the objective n_trials times). Everything else — sampling, pruning, persistence — builds on this core.
- ✓Use trial.suggest_float with log=True for parameters that span orders of magnitude: learning_rate (0.001 to 0.3), reg_alpha (1e-8 to 10). Log-uniform sampling ensures equal exploration at each magnitude. Use trial.suggest_int for discrete parameters like n_estimators, max_depth, num_leaves.
- ✓Pruning stops unpromising trials early — report intermediate scores with trial.report() and check trial.should_prune() inside the CV loop. MedianPruner prunes any trial whose intermediate score falls below the median of completed trials at the same step. Saves 30–50% compute on expensive models.
- ✓Tune in phases for large search spaces: learning_rate + n_estimators first (biggest impact), then tree structure, then regularisation, then joint refinement in a narrow range around the best values. This finds the optimum with far fewer total trials than a flat all-parameters-at-once search.
- ✓Optuna studies are persistent — save to SQLite or PostgreSQL with the storage argument, set load_if_exists=True to resume. This lets you run 20 trials today, stop, and add 20 more tomorrow. The surrogate model continues improving from where it left off.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.