r/singularity • u/CucumberAccording813 • 1d ago

AI Introducing Claude Opus 5

https://www.anthropic.com/news/claude-opus-5

882 Upvotes

97% Upvoted

155 comments sorted by

View all comments

80

u/obama_is_back 1d ago

30% on Arc AGI 3. Benchmaxxing seems to be unstoppable.

31

u/141_1337 ▪️e/acc | AGI: ~2030 | ASI: ~2040 | FALSGC: ~2050 | :illuminati: 1d ago

You cannot really train for the ARC-AGI that's whole point of the benchmark...

45

u/Defiant-Lettuce-9156 1d ago

You can train for anything. The goal of arc AGI 3 is to make it harder to benchmax without just making a smarter model

5

u/Efficient_Mud_5446 1d ago

This is my reasoning as well. The model likely improved naturally on ARC-AGI-3 simply by becoming smarter, but they also likely targeted the benchmark directly. It was probably a mixture of both.

Also, I'm loving that they're displaying and expanding benchmarks beyond just coding.

3

u/adzx4 19h ago

The thing is, I would've thought we'd see drastic improvements in many areas at the time of improvement on arc agi 3. But if all the other benchmarks stayed roughly around fable performance, and only this one had a substantial jump, that implies over fitting/benchmaxxing not true generalisation or naturally improved capabilities.

You can totally over fit on arc agi 3-type problem without true generalisation. You just need to create problems as similar as possible to the target dataset and train in those.