Classical HPO vs frontier LLMs: an update

What we already wrote

In the original paper we benchmarked 9 HPO methods (classical, LLM-based, hybrid) on Karpathy's autoresearch task, optimizing a 50M-parameter GPT-2-class transformer on Climbmix-400B. Each method got an identical 24h training-time budget per seed. We tested two LLM optimizer "shapes":

• Karpathy Agent (14 HPs): the LLM suggests the next config inside a fixed 14-HP search space (DEPTH, head dim, batch sizes, learning rates, etc.).
• Karpathy Agent (Code): the LLM edits train.py directly. No fixed search space, so OOM and "off-spec" trials are possible.

We also introduced Centaur (CMA-ES + LLM): a CMA-ES inner loop where, on a fraction of trials, the LLM overrides the proposal with a config informed by CMA-ES's internal state (its current mean and covariance estimate). The paper concluded that classical methods consistently outperform pure LLM-based agents within a fixed search space, that code-editing freedom narrows but does not close the gap with frontier models such as Opus 4.6 and Gemini 3.1 Pro Preview, and that Centaur was the only LLM-flavored setup that landed near the classical baselines.

That was the picture at n=3 seeds per method.

What changed

Three things shifted since publication; one further thing shifted between the original 2026-05-19 update and this 2026-06-20 revision:

1. Full n=8 campaigns on every method now. TPE: n=3 → n=8. Centaur (CMA-ES + LLM) for every Anthropic generation: n=8. Karpathy Agent (14 HPs) for every Opus generation: n=8. Karpathy Agent (Code) is at n=8 for Opus 4.6/4.7, still n=5 for Opus 4.8 and Sonnet 4.6.
2. Three new Anthropic generations. Claude Opus 4.7 (April 2026), Claude Sonnet 4.6 (May 2026), and Claude Opus 4.8 (June 2026). All tested via the Claude Code SDK with thinking=False, identical SUGGEST_PROMPT to the paper's Opus 4.6 runs, identical VRAM cap.
3. A live tracker. Auto-updates the comparison and the paired Wilcoxon p-values whenever new seeds clear the 95% training-budget threshold.
4. A mechanistic finding for the Centaur Opus 4.7 underperformance (added 2026-06-20). It's an interaction effect between Centaur's _override_trial_params + FAILURE_PENALTY bookkeeping and a small uptick in Opus 4.7's willingness to suggest OOM-prone configs. See Why Centaur with Opus 4.7 underperforms below.

Updated results

The current state of the 12 LLM × generation cells plus 4 classical baselines (all n=8 unless noted):

Reading the table:

• Two of twelve LLM × generation variants beat TPE with marginal statistical significance (p < 0.10): Centaur (CMA-ES + LLM) [Opus 4.6] at p=0.055, and Karpathy Agent (14 HPs) [Opus 4.7] at p=0.098. Both are 8-seed paired comparisons.
• Five more LLM cells beat TPE numerically without significance (Centaur Opus 4.8, Karpathy Agent (14 HPs) Opus 4.6 / Opus 4.8, Karpathy Agent (Code) Opus 4.6 / Opus 4.8).
• One Centaur generation loses to TPE: Opus 4.7 at p=0.844 (only Centaur row above TPE). We diagnose it below as a composition pathology, not an Opus 4.7 LLM-quality problem.
• Sonnet 4.6 variants cluster above TPE in every method. Same prompt, same VRAM, weaker model.
• Of the four classical methods, TPE is by far the strongest. CMA-ES alone (0.9776) sits 12th overall, ahead of only the worst three rows. SMAC and Random are not competitive at this budget.

Surprises worth a paragraph

1. Centaur with Opus 4.6 holds up; Opus 4.8 reproduces the direction

The same recipe (same prompt, same CMA-ES state hand-off, same VRAM cap, same 30% LLM-override ratio) gives p=0.055 with Opus 4.6 at n=8. It reproduces the direction with the newer Opus 4.8 (mean 0.9748, beats TPE numerically, p=0.32 at n=8) but not the significance. Sonnet 4.6 lands above TPE (p=0.97). And Opus 4.7 falls all the way to p=0.84, by a wide margin the worst Centaur generation. We chased that down: it's an interaction effect, see below.

2. Why Centaur with Opus 4.7 underperforms: a composition pathology

Centaur 4.7 is the only Centaur cell that loses to TPE (mean 0.9760, paired Wilcoxon vs Centaur 4.6 is p=0.027, 7/8 seeds worse than 4.6). The puzzle: Karpathy Agent (14 HPs) with the very same Opus 4.7 is the runner-up across the whole table at p=0.098. Same LLM, same prompt, same search space, but one wins and the other loses. The difference is the CMA-ES scaffolding.

We reconstructed which trials were LLM-suggested vs CMA-suggested per seed and split the failure rates:

• Opus 4.6 LLM-suggested trials fail (OOM) at 1.8%; pure-CMA trials at 25%.
• Opus 4.7 LLM-suggested trials fail at 41% overall and 66-68% in seeds 5-7; pure-CMA trials at 52% overall, climbing to 74-77% in the same seeds.
• Opus 4.8 LLM-suggested trials fail at 0.2%; pure-CMA trials at 16%.

Opus 4.7 is roughly 5× more willing than 4.6 to suggest OOM-prone configs in the pure-LLM setting (10.8% fail vs 2.3% in Karpathy Agent (14 HPs)). In a Centaur shell that's amplified catastrophically. Here is the loop:

1. CMA-ES asks the Optuna sampler for trial T; gets config X.
2. Centaur calls the LLM, which returns config Y (an OOM-prone Opus 4.7 suggestion). Centaur calls _override_trial_params(trial, Y) so Optuna's bookkeeping treats Y as if its Gaussian sampler had emitted it.
3. Y is evaluated, fails. Centaur reports FAILURE_PENALTY=100.0 to the sampler.
4. CMA-ES updates its mean and covariance using Y (an off-distribution point) with a 100.0-loss penalty. Its Gaussian's parameters drift toward the LLM's distribution; the FAILURE_PENALTY tells it that region is bad, so it tries to back away.
5. But the LLM keeps re-injecting overrides in the same OOM-prone region. CMA can't escape faster than the LLM injects.
6. CMA-ES's pure-CMA suggestions on subsequent trials end up adjacent to the OOM region (because the Gaussian's parameters got dragged there) and themselves OOM at a 74-77% rate.

Karpathy Agent (14 HPs) doesn't have step 4: there's no sampler covariance for the LLM to poison. A 10.8% LLM fail rate there just costs 10% of trial efficiency. Karpathy Agent (Code) doesn't hit it either because the LLM tends to write conservative Python code rather than push numerical bounds (its fail rate is 2.5% for Opus 4.7, the lowest of any 4.7 cell).

None of these 4.7 failures are infrastructure problems. The SLURM logs show every failed trial is a torch.OutOfMemoryError from training a model that doesn't fit in 76 GB of VRAM. The 24h training-time budget is fully consumed. The 0.9760 mean is a real measurement of "Centaur composed with Opus 4.7" on this benchmark, not a corrupted run.

Possible mitigations (out of scope for the current paper): replace the hard-coded FAILURE_PENALTY=100.0 with a running 90th-percentile of successful val_bpb; or skip study.tell entirely on LLM-overridden failed trials, so CMA-ES's covariance never sees off-distribution samples. Both should make Centaur more robust to an LLM that explores OOM edges.

3. Karpathy Agent (Code) [Opus 4.6] is now tied with TPE, not below it

The paper wrote: "Allowing the LLM to directly edit source code narrows the gap but does not close it, even with frontier models such as Claude Opus 4.6." Updated data at n=8: Karpathy Agent (Code) [Opus 4.6] mean = 0.9755, TPE mean = 0.9755. The two means are identical to 4 decimal places. Paired Wilcoxon p=0.53, fully inside noise. So "doesn't significantly close" is still defensible; "doesn't close" was overstated at the paper's n=3.

4. Part of the closing is TPE getting better data

TPE n=3 → n=8 moved its mean from approximately 0.9760 to 0.9755. That is roughly the same magnitude as the Karpathy Agent (Code) [Opus 4.6] gap closure. At n=3, paired tests on a 0.001 effect at this noise level have very wide confidence intervals; the paper's headline numbers were on thin statistical ice and we should have noted that more clearly. This is the most important methodological lesson from the update.

5. Karpathy Agent (14 HPs) [Opus 4.7] is the second-best row overall

Pure LLM-driven, no CMA scaffolding, no code-editing freedom. Mean 0.9741 with std 0.0027 at n=8, paired Wilcoxon vs TPE one-sided p=0.098. This is the strongest pure-LLM result we have so far on this benchmark, and it comes from the very model that gets crushed inside Centaur. The contrast strongly suggests Opus 4.7 is a capable HP-proposal model, and the Centaur shell is what holds it back.

6. CMA-ES alone is not the worst classical method (SMAC and Random are worse)

Pure CMA-ES: 0.9776. TPE: 0.9755. SMAC: 0.9816. Random: 0.9869. Among the classical baselines, TPE is by far the strongest at this budget; CMA-ES is mid-pack; SMAC and Random are not competitive. But hand CMA-ES's internal state to a frontier LLM (Centaur with Opus 4.6) and the combined system is the best of everything tested. The hybrid's lift is not "the LLM rescues a bad classical method": it's that exposing the CMA-ES internal state to the LLM gives the LLM a foothold that pure-LLM agents (Karpathy Agent 14 HPs) and pure CMA-ES alone do not have. When that foothold goes wrong (Centaur 4.7), the same scaffolding becomes a liability.

What we cannot claim yet

• No GPT-5 in the comparison. OpenAI's flagship is still absent. Adding GPT-5 is on the queue; results will land in the live tracker as they complete.
• No Claude Fable 5. Released 2026-06-09. We have not run it yet on cost grounds (~2× Opus 4.8 list price). Expect it in the tracker eventually.
• One task. Climbmix-400B language modeling is one downstream. We have not tested whether the Centaur-Opus-4.6 effect carries to other tasks.
• n=8 is still small. The Wilcoxon p-value at n=8 sits at 0.055; with additional seeds it will move and we cannot predict the direction.
• Karpathy Agent (Code) at Opus 4.8 and Sonnet 4.6 are at n=5, not n=8. Those rows have wider confidence intervals than the others.
• We do not know what triggers the Opus 4.7 mode collapse only on seeds 5-7. The CMA-ES sampler is seeded; seeds 5-7 happen to land Opus 4.7's first LLM prompts in a region of the search space where the model latches onto OOM-edge configs. The same seeds with Opus 4.6 / 4.8 produce normal trajectories. We do not have an explanation for the seed sensitivity, only the downstream loop.

The live tracker

https://ferreirafabio.github.io/autoresearch-automl/#tab=tracker

Four sections:

• 1. Convergence. All method × generation cells, val_bpb vs cumulative wall-time, mean ± std band. Filter by Claude generation.
• 2. Slopegraph. Per-method progression across the Claude generations we've tested.
• 3. Wilcoxon forest. Δ vs TPE with paired p-values.
• 4. Per-generation summary cards. One card per Anthropic release; last-updated timestamp; current leader.

The overview table at the top of the tracker now includes the four classical baselines (TPE, CMA-ES, SMAC, Random) alongside the 12 LLM-driven rows. Each new Anthropic release lands automatically once its 8-seed campaign clears the 95% training-budget threshold.

What's next

• GPT-5. Backend scaffolding in progress; will fold into the live tracker once seed runs complete.
• Finish the n=5 cells: Karpathy Agent (Code) [Opus 4.8] and Karpathy Agent (Code) [Sonnet 4.6] both currently at n=5; seeds 5-7 are running.
• Mitigation patch for Centaur: change FAILURE_PENALTY to be less destructive to the CMA-ES covariance (running p90, or skip-tell on LLM-overridden failures). Out of scope for this paper but a likely follow-up artifact.
• arXiv v2 with the updated numbers, the Opus 4.7 composition finding, and a more cautious framing of the n=3 results in the original.