The first hardware engineering benchmark built with the world’s most advanced hardware companies
HW-Bench measures frontier model performance on hardware engineering work across accuracy, cost, and latency
Leaderboard
Fig. 02 / Leaderboard
Early results based on six tasks within one job (Designing a 2.45 GHz antenna). Rankings may shift as we add jobs. Read more.
Updated
| # | Model | Pass RatePassᵏ: the chance that k runs of the same task all pass every critical criterion, averaged across tasks. | Weighted share of accuracy criteria passed (0 to 1). High accuracy score can still mean low passᵏ, since failing one critical criterion fails the run. | Mean USD per run of a task, plotted on a log scale. | Mean wall-clock time per run of a task, plotted on a log scale. | ||
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAI | pass1pass¹: 67% of runs passed every critical criterion | pass3pass³: 67% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 67% of tasks passed every critical criterion in all 5 runs | Accuracy0.810.81 mean accuracy (95% CI 0.79–0.82) | Cost$50.49 mean cost per job (95% CI $45.45–$55.60) | Time1h 28m mean time per job (95% CI 1h 21m–1h 35m) |
| 2 | GPT-6.1 SolOpenAI | pass1pass¹: 57% of runs passed every critical criterion | pass3pass³: 42% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 33% of tasks passed every critical criterion in all 5 runs | Accuracy0.780.78 mean accuracy (95% CI 0.74–0.82) | Cost$8.42 mean cost per job (95% CI $7.64–$9.19) | Time1h 45m mean time per job (95% CI 1h 36m–1h 53m) |
| 3 | GPT-6 SolOpenAI | pass1pass¹: 37% of runs passed every critical criterion | pass3pass³: 13% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.710.71 mean accuracy (95% CI 0.66–0.76) | Cost$8.07 mean cost per job (95% CI $7.11–$9.12) | Time53m mean time per job (95% CI 48m–58m) |
| 4 | Claude Opus 5Anthropic | pass1pass¹: 33% of runs passed every critical criterion | pass3pass³: 3% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.710.71 mean accuracy (95% CI 0.64–0.77) | Cost$28.87 mean cost per job (95% CI $24.72–$32.47) | Time1h 43m mean time per job (95% CI 1h 28m–1h 59m) |
| 5 | GPT-5.6 SolOpenAI | pass1pass¹: 30% of runs passed every critical criterion | pass3pass³: 7% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.730.73 mean accuracy (95% CI 0.68–0.79) | Cost$15.05 mean cost per job (95% CI $13.57–$16.58) | Time1h 07m mean time per job (95% CI 59m–1h 15m) |
| 6 | Claude Opus 4.8Anthropic | pass1pass¹: 30% of runs passed every critical criterion | pass3pass³: 17% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 17% of tasks passed every critical criterion in all 5 runs | Accuracy0.670.67 mean accuracy (95% CI 0.62–0.71) | Cost$22.36 mean cost per job (95% CI $20.59–$24.17) | Time1h 28m mean time per job (95% CI 1h 22m–1h 34m) |
| 7 | GPT-6 LunaOpenAI | pass1pass¹: 30% of runs passed every critical criterion | pass3pass³: 18% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 17% of tasks passed every critical criterion in all 5 runs | Accuracy0.590.59 mean accuracy (95% CI 0.54–0.64) | Cost$0.55 mean cost per job (95% CI $0.47–$0.65) | Time1h 41m mean time per job (95% CI 1h 26m–2h 00m) |
| 8 | Claude Fable 5.1Anthropic | pass1pass¹: 23% of runs passed every critical criterion | pass3pass³: 2% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.680.68 mean accuracy (95% CI 0.64–0.73) | Cost$33.84 mean cost per job (95% CI $30.67–$37.82) | Time1h 25m mean time per job (95% CI 1h 17m–1h 34m) |
| 9 | Claude Opus 5.5Anthropic | pass1pass¹: 20% of runs passed every critical criterion | pass3pass³: 0% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.710.71 mean accuracy (95% CI 0.65–0.78) | Cost$16.72 mean cost per job (95% CI $14.52–$19.20) | Time1h 23m mean time per job (95% CI 1h 11m–1h 37m) |
| 10 | Claude Sonnet 5.5Anthropic | pass1pass¹: 20% of runs passed every critical criterion | pass3pass³: 2% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.660.66 mean accuracy (95% CI 0.61–0.72) | Cost$5.75 mean cost per job (95% CI $5.27–$6.20) | Time1h 19m mean time per job (95% CI 1h 04m–1h 33m) |
| 11 | Grok 4.7xAI | pass1pass¹: 20% of runs passed every critical criterion | pass3pass³: 0% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.600.60 mean accuracy (95% CI 0.55–0.66) | Cost$17.56 mean cost per job (95% CI $15.36–$19.94) | Time1h 09m mean time per job (95% CI 56m–1h 27m) |
| 12 | GLM-5.3Z.ai | pass1pass¹: 17% of runs passed every critical criterion | pass3pass³: 7% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.590.59 mean accuracy (95% CI 0.54–0.64) | Cost$18.16 mean cost per job (95% CI $14.57–$22.10) | Time1h 56m mean time per job (95% CI 1h 32m–2h 23m) |
| 13 | Grok 4.6xAI | pass1pass¹: 13% of runs passed every critical criterion | pass3pass³: 7% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.610.61 mean accuracy (95% CI 0.58–0.64) | Cost$15.43 mean cost per job (95% CI $13.99–$16.97) | Time2h 17m mean time per job (95% CI 2h 02m–2h 33m) |
| 14 | Kimi K3Moonshot | pass1pass¹: 7% of runs passed every critical criterion | pass3pass³: 0% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.400.40 mean accuracy (95% CI 0.33–0.46) | Cost$26.18 mean cost per job (95% CI $16.17–$37.07) | Time3h 10m mean time per job (95% CI 2h 23m–4h 00m) |
| 15 | Grok 4.5xAI | pass1pass¹: 3% of runs passed every critical criterion | pass3pass³: 0% chance that 3 random runs of a task (of 5 runs) all pass every critical criterion | pass5pass⁵: 0% of tasks passed every critical criterion in all 5 runs | Accuracy0.500.50 mean accuracy (95% CI 0.47–0.54) | Cost$6.53 mean cost per job (95% CI $5.43–$7.94) | Time1h 00m mean time per job (95% CI 51m–1h 09m) |
95% CI: bootstrap resampling each model’s 5 runs within each task (Efron 1979; Agarwal et al. 2021)
Accuracy vs cost and latency
Early results based on six tasks within one job (Designing a 2.45 GHz antenna). Rankings may shift as we add jobs. Read more.
Updated
Frontier: no model is both more accurate and cheaper (or faster) Other models
95% CI: bootstrap resampling each model’s 5 runs within each task (Efron 1979; Agarwal et al. 2021)
Fig. 04 / Jobs
Jobs
Design & simulateBuild & testOperate & sustain
Electrical
Mechanical
Software & firmware
Systems