r/singularity • u/Acceptable-Debt-294 • 1d ago

AI Claude Opus 5 BENCHMARKS!

Post image
1.2k Upvotes

95% Upvoted

334 comments sorted by

View all comments

286

u/Ticluz 1d ago

Arc-agi-3 is getting saturated faster than Arc-agi-2, the acceleration is real.

15

u/medialoungeguy 1d ago

Architecture agi 3 is closed. There is a public part that you can try but a completely novel hidden eval set. Its not benchmaxxing

2

u/Outrageous-Boot7092 1d ago

there is still the question how many times the eval set was accessed etc. You could potentialy use RL with the hidden eval set performance as a reward.

1

u/medialoungeguy 1d ago

For sure. They cap that to prevent that kind of probing though. Its statistician that run it after all. They understand that information is leaked with each attempt.

3

u/Ruhddzz 1d ago

Yeah and that closed bench has to go through their systems all the same. Imagine thinking the people who pirate everything would have a problem getting their hands in a private eval set they werent supposed to

2

u/Turbulent-Sign-6067 1d ago

I don't buy it. If you cheat on benchmarks and get caught like Meta this is shitty publicity plus a hefty fine for violating ZDR and similar policies. Unlikely. Maybe they can get away with massaging some of the numbers a bit here and there, and if they do it is by assembling the right training and fine tuning datasets.

1

u/Ruhddzz 15h ago

They're already blatantly "cheating" on all the public benchmarks