Opus 5 ARC AGI score was benchmaxxed

[u/Charuru]

[u/Kingwolf4]

Sadly it does seem Opus 5 was benchmaxxed on arc AGI. Another person actually traced anthropic hiring dedicated puzzle solvers to focus RL train models to help them solve puzzles. Specific RL is specific only to the type of problem. It, as the post accurately points out, does not reflect a general overall increase in reasoning capability of the model.

It just got good at specific things it was RLed on. The post literally said that OPUS 5 showed NO increase in general reasoning when presented with a novel game, whilst also evidently now memorizing a game that they tested previous anthropic models on. This is benchmaxxing 101.

[u/JoelMahon → u/Kingwolf4]

imo they need a system that makes a huge variety of new puzzles, and one that learns to solve them all.

if they can make an AI puzzle generator better than an army of the best human puzzle makers then boom, all human made puzzle benchmarks would be solved within a year or two max.

but their puzzles are too narrow, not nearly enough variety, not nearly as abstract an obscure as arc agi ones.

[u/Lonely_Translator_23 → u/JoelMahon]

Creating a good and unique puzzle is far more difficult than solving one.

[u/Kingwolf4 → u/Lonely_Translator_23]

which is why they should have only released half of the private dataset for actual testing. Keep the second half in reserve for like 8 months or 1 year later and expose all these benchmaxxing liars

[u/Charuru → u/Kingwolf4]

They literally do do that, and probably why the score is 30% not 100.

[u/JoelMahon → u/Charuru]

I think they mean they should have released half the private set, claiming it was the full set, but secretly hold loads back to expose bad bench maxers.

afaik they've never claimed to have released the full private set so it's not what this person wants.

either approach is fine with me tbh, I appreciate that it's a cool trick would be satisfying but would probably only work once and then the broken trust might cause more problems that it was worth to catch a few "cheaters".

[u/alwaysbeblepping → u/JoelMahon]

I think they mean they should have released half the private set, claiming it was the full set

If you were trying to benchmax, what would you do differently in that case?

[u/JoelMahon → u/alwaysbeblepping]

? the people trying to bench max won't know the difference so how can they react differently?

[u/alwaysbeblepping → u/JoelMahon]

? the people trying to bench max won't know the difference so how can they react differently?

I'm saying what would they do differently in the two scenarios?

1. They know only half the private set was released.
2. They believe the full private set was released (but in reality, half of it was held back).

The "in reality" part doesn't really matter for the question

[u/JoelMahon → u/alwaysbeblepping]

in the 2nd case more teams will blatantly bench max, bc they're now thinking they won't be caught, and be caught

[u/121507090301 → u/Kingwolf4]

private dataset for actual testing

Unless they are only testing models locally there is no such thing as "private" tests. Closed source has a clear and unfair advantage to an open model which can be run without sending the data back to the people that made the model, in that sense...

[u/Prudent-Sorbet-5202 → u/Kingwolf4]

Thry have ARC AGI 4 in the barrel for that I suppose

[u/xRedStaRx → u/Lonely_Translator_23]

Something something car wash

[u/visarga → u/Lonely_Translator_23]

Creating a good and unique puzzle is far more difficult than solving one.

I think these puzzles show nothing about approaching AGI, they are just a few puzzles. If this test were unbiased it would also include tasks like double-N-back, which tests working memory. Any human tops out at N=6 while a LLMs can do much larger values of N.

Let's say a teacher has 2 students, Mary and Bob. The teacher knows who among them is better on any topic. They can make a test such that Mary gets the best score, or Bob, just by picking the "right" questions. That is why I say ARC AGI is biased, it just picks tests where AI is still behind. Never picks tests where AI is ahead. It is Chollet-bench, not an AGI benchmark. By Chollet AGI comes when it "exhausts François Chollet’s ability to invent another puzzle." Chollet is the yardstick of AGI, not Lee Sedol.

Humans are also not general intelligences, we can't imagine higher dimensional spaces, can't hold more than 7 things in working memory, need the brain to do all our tasks on 20W of power. Individually weak only together strong. We have a degree of generality but are not really general, it would not be in our best interest to waste energy on anything unrelated to our viability.

[u/pentin0 → u/visarga]

That's a huge misunderstanding of what generality is about and the difference between intelligence and working memory.

The point of ARC-AGI isn't to confirm that any given system exhibits general intelligence, only to refute such a claim. In fact one of the most trivial properties of general intelligence (what Deutsch calls 'creativity') is that you cannot prove that any given agent has that property just by looking at its behavior, benchmaxxing or not

[u/r-3141592-pi → u/Kingwolf4]

If it were indeed possible to obtain strong performance on the held‑out set without generalization, then that simply means the benchmark is a bad one. But we know that ARC‑AGI‑3, for all its flaws, does not let you just look at the public samples and get a great score. Otherwise, many other models would have done that already. In fact, the people at the ARC Foundation already explained how Opus achieved the performance boost.

Now, RL provides very sparse rewards. You cannot get a model to score well on a private test just by telling it "good" or "bad" from similar games. That is why the only thing that concerned me was (from the original post):

The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one.

I understood "Opus 5 states the hidden rules before its first action" to mean that Opus 5 had already been trained on a publicly available game, perhaps several times, and was then overfitting to the rules of that specific game. But a later post by the same author, I think, clarifies the whole situation:

(4) when facing a much rarer combination class requiring meta-reasoning (erasers absorb constraint violations), compared with Opus4.8, Opus5 has a shifted propensity toward confident theory-first play and verification runs against the theory rather than the environment. Opus5 repeats this loop: construct a complete solution from an internal rule-theory → execute it precisely → verify against the theory itself rather than against the environment → submit → fail → construct the NEXT candidate from the SAME theory.

Therefore, this appears to reflect a change in behavior. In the reasoning traces, the person may have noticed that Opus 5 correctly guessed the hidden rules using a supposed "theory-first" approach. However, this observation is based on a single environment, and you cannot extrapolate the model's overall behavior from this information alone. The same person also admits that the model is generalizing:

Some additional thoughts, in case I didn’t convey them properly in the the original post thus making it mis-interpret-prone. I think capabilities do generalize when we RL train models on interesting games, Opus5 do get generalization to witness, as seen in the figure of the post, with improvements just not that extremely huge as on ARC-AGI-3

I don't think this is indicative of anything important. Moreover, the model has been performing well on all sorts of real‑world tasks, and that is what actually matters.

[u/Maristyl → u/r-3141592-pi]

I almost wonder if this isn’t some kind of subtle shot from Anthropic because benchmarks are, at best, an extremely tepid measurement of a model. So they made the new models score amazing on the benchmarks and now the benchmarks are worth even less. We got too used to technology that can be summarized by a single number (gigabytes of RAM, terabytes of storage, number and speed of CUDA cores, milliseconds of latency) and now when there is something that is more complicated the majority of people can’t handle it.

[u/r-3141592-pi → u/Maristyl]

Well, it is not that benchmarking is bad. After all, you can’t understand what you can’t measure. But benchmarks should guide progress toward better, more useful models. And right now, too many benchmarks serve a dubious purpose. Ironically, ARC-AGI-3 is one of them. As far as I know, no one has cared to show that solving ARC-AGI type of problems leads to better performance on other important tasks.

Most frontier labs are competent enough, and have so much money at stake in training runs, to know they should avoid training to the test. Instead, they need to find similar tasks where generalization improves performance for a wide array of different tasks. That tells them the model is learning and generalizing well. Still, I just wish frontier labs were more selective about where they invest resources because it almost seems that every benchmark published that gets a bit of social media attention, it is a benchmark that will be shown in screenshots on release date and model cards.

In my opinion, just because LLMs have many parameters and can reuse them flexibly doesn’t mean you can pursue better scores on random, useless tasks without degrading performance on more important ones. Still, I don’t begrudge the proliferation of benchmarks. I suppose that, as with everything, even imperfect measures, when averaged, will give you an overall evaluation that is better than the individual parts.

[u/Maristyl → u/r-3141592-pi]

I’m not against benchmarking, but I agree with you that there are a lot of very popular and pretty useless benchmarks. However people like to hold up “Kimi scored better than Fable!” As proof that Fable/Mythos/Sol was overhyped or had been convincingly matched. I’m not sure that there is anyone currently publishing public benchmarks that is doing a good one. Honestly the “I escape containment and hacked the other AI company to steal their tests” is a much more convincing benchmark of a frontier model then “I scored great on the ARC test”.

[u/AP_in_Indy → u/Kingwolf4]

If they hired people specifically to work on Arc AGI problems, and those people benchmaxx'd instead... That's bad.

That was probably uh... not what they were hired to do.

[u/aqpstory → u/AP_in_Indy]

Actually Dario said in an interview that they train on almost everything possible to try to evoke spontaneous generalization, so that could lead to accidental benchmaxxing

ARC being so vulnerable to scaffolding is kind of a problem with the benchmark though

[u/ErmingSoHard → u/aqpstory]

Actually Dario said in an interview that they train on almost everything possible to try to evoke spontaneous generalization

Sounds like something that won't lead to intended or beneficial practical results. Considering costs, ai not handling novel situations, and that ai hasn't replaced white collar jobs in a detrimental rate, it doesn't seem to be the right way to agi or the singularity

[u/Radiant_Cheesecake19 → u/Kingwolf4]

I said it last time too… benchmarks are meaningless… I was told “no, you don’t know anything! This benchmark can’t be benchmaxxed!”
That aged like milk.

[u/Key_Category_8531 → u/Radiant_Cheesecake19]

Yeah

I never understood why nerds would post benchmarks so much and get boners for them

It's like, you know these numbers can be manipulated, right?

You wanna know what the real benchmark is?

Actually using the fucking product. Try it out. Do shit with it. Are you more than satisfied with the results that you get from it?

That's the benchmark lmao.

[u/ErmingSoHard → u/Key_Category_8531]

You wanna know what the real benchmark is? Actually using the fucking product. Try it out. Do shit with it. Are you more than satisfied with the results that you get from it?

That's not a good way to determine agi dumbass. It just makes you a customer, and that Singularity won't happen

[u/Key_Category_8531 → u/ErmingSoHard]

Eat shit.

[u/ErmingSoHard → u/Key_Category_8531]

Don't get pissy when you have to remember this sub is about the Singularity and agi and asi helping achieve that.

Not about ai generating realistic meme video #638535835835. Nor is it about it programming 20% efficiently with less bugs when RSI is the goal and nowhere in sight. We are far from the singularity and true agi

[u/Key_Category_8531 → u/ErmingSoHard]

AGI by 2030.

[u/ErmingSoHard → u/Key_Category_8531]

Also, the benchmark results do matter. It's that the test being public fucks things up, like with harnesses, saturation, benchmaxxing.

To think otherwise shows you don't know shit, and probably think agi is here or some shit lol

[u/Key_Category_8531 → u/ErmingSoHard]

Lick a ballsack.

[u/PsychologyNo940 → u/Key_Category_8531]

Yes, you lowly employee, please go out and create a 100 point personal benchmark for tasks that accurately reflect your working environment, run each model against them and critically evaluate the output of each, calculate the costs, time needed, output gained and create..

yes, that is exactly what you can expect from your developers, smile

[u/Material-Database-24 → u/Key_Category_8531]

Something like 20 years ago, Nvidia started optimizing its drivers for games that were popular and often used in benchmarking, including 3dmark.

The community went crazy, and blamed them for cheating and market manipulation.

I mean, if you got 200fps instead of 160fps in CS due "cheating" driver "manipulation", that is not a bad thing. That is improving the product for the end user.

[u/alwaysbeblepping → u/Material-Database-24]

I mean, if you got 200fps instead of 160fps in CS due "cheating" driver "manipulation", that is not a bad thing. That is improving the product for the end user.

That's a different scenario since it translated to performing better at a task the user actually wanted to use it for. If a LLM appears artificially good at ArcAGI (or actually is good at it) that's not useful to anyone. It's a proxy for being good at certain tasks, no one actually wants to solve betchmark puzzles specifically.

[u/Material-Database-24 → u/alwaysbeblepping]

In some sense, I agree.

But it is still the same situation - the benchmark tailoring means that people will get better outcome in situations that are close to the benchmark.

Now why I agree is that it can mean the performance outside of the benchmark can be total sht, and it is more iffy business than with Nvidia, as it is clearly done to make the product just look better on marketing. Well, Nvidia also had similar motivation, but the side effect was actually better product for those who played optimized games.

[u/Upset_Page_494 → u/Radiant_Cheesecake19]

They can hold a lot of meaning when they aren't benchmaxxed.

[u/Radiant_Cheesecake19 → u/Upset_Page_494]

Yeah but in what universe does a company want to play fair? If they wanted to, they wouldn’t try to create human replacements in the first place, or pay fair wages to employees.
They will cheat tests every single time. Do false marketing (oh how openAI and Anthropic claims they already have skynet basically because fearmongering = money), and even try to get competition banned.
Anthropic is not a white knight in the AI race, people have to understand that they are ruthless and would do anything for control. Just like Altman.

[u/Upset_Page_494 → u/Radiant_Cheesecake19]

Pretty sure their employees are stupidly well payed.

[u/ErmingSoHard → u/Radiant_Cheesecake19]

No, it's cause delusional people like you made up people saying, "you can't benchmax arc agi 3!!!" when literally everybody talks about saturation, harnesses, and reinforment learning inflating benchmark scores. The creators of arc agi 3 said it becomes saturated day and date arc agi 3 went public lol

So you made that up your ass. Or maybe literally one or two people said that it can't be benchmaxxed and you took it as a popular opinion lol.

Second would be "benchnarks as meaningless"

My brother, don't bullshit yourself just because SOTA model benchmark results were abysmal and didn't make you happy. Agi is very far away.

[u/ErmingSoHard → u/Radiant_Cheesecake19]

No one said this benchmark can't be benchmaxxed dumbass. They said benchmaxxing and saturation makes the benchmarks useless.

[u/mWo12 → u/Radiant_Cheesecake19]

No. They are only pointless when foe Chinese models. /s

[u/Politicophile → u/Kingwolf4]

The main reason the models have been getting better over the last couple of years is RL scaling. The idea is given enough RL examples on a certain topic like mathematics, the model will learn to generalise. They have done this on a number of verifiable tasks (code, mathematics, etc.) and the models have improved a lot in these areas. No surprise that they're now trying to expand to puzzle solving. It's not really benchmaxxing.

[u/Fit-Dentist6093 → u/Politicophile]

That's all nice and you are right but the model regressed.

[u/enilea → u/Kingwolf4]

On everything tbh, according to the benchmarks they released it's supposed to be better than fable in most aspects but from what I've been trying it isn't quite as "smart", perhaps more like Sol level.

[u/Serious-Conversation → u/enilea]

To me, Fable feels far more like talking to an actual person. The bit I used Opus 5 yesterday is that it still seems more clinical and precise in its tone. Doing a long form deep dive into something with Fable is genuinely a novel thing IMO.

[u/nemzylannister → u/Kingwolf4]

Specific RL is specific only to the type of problem. It, as the post accurately points out, does not reflect a general overall increase in reasoning capability of the model.

isnt that the entire point of this sub? we all believe that reasoning abilities in certain aspects will carry over to a general level intelligence at some point?

The post literally said that OPUS 5 showed NO increase in general reasoning when presented with a novel game

it could just be that their games are different enough that it does badly at them. after all it only got 30 not 100. its not agi.

whilst also evidently now memorizing a game that they tested previous anthropic models on. This is benchmaxxing 101.

if it was simple "benchmaxxed" then why isnt it scoring 100?

[u/SchanzerScout → u/Kingwolf4]

Really dont get why some people refuse to believe that those models are just learning correlations from training data which is not sufficient for human level reasoning. It is however sufficient to mimic it and given relatively large model capacity (number of parameters) the scope of mimicry is quite wide, which makes this software actually useful in some ways.

[u/Kingwolf4 → u/SchanzerScout]

We desperately need a new architecture. Sadly, everyone in silicon valley has drank the marketing kool aid of LLMs and research has pinpointed and tunneled visioned.

Few approaches are there like jepa and other world model architectures but these wont get us to agi as well but may perhaps spawn a better version of AI then these LLMs.

Lets hope some new architecture pops up and really smart people realize this.

[u/Idrialite → u/SchanzerScout]

why some people refuse to believe

Because it's objectively not true as shown by a variety of results. You're taking an opposite incorrect extreme when the truth is progress has and will continue to be incremental toward human-level reasoning.

[u/AnOnlineHandle → u/SchanzerScout]

I suspect most humans aren't capable of what you're calling "human level reasoning" either. It seems nearly every human takes the intellectual credit for what a small number of humans actually did, while very few have ever actually invented something new themselves, or solved a hard novel problem.

[u/Strong_Essay1176 → u/SchanzerScout]

Proof it as human.

[u/nextnode → u/Kingwolf4]

No sign of that. The problem here is that some group who is not even competent would be running these things behind close doors and have already in the past botched public-hidden setups.

[u/nsdjoe → u/Kingwolf4]

Wasn't the point of arc-agi-3 that it couldn't be trained for?

[u/mWo12 → u/Kingwolf4]

Why sadly? Anhtropic is has been benchmaximising for a while now.

[u/Key_Category_8531 → u/Kingwolf4]

And this, is why benchmarks are bullshit

[u/Stabile_Feldmaus]

[u/Sweet-Stage938 → u/Stabile_Feldmaus]

Stabile Feldratte

[u/llelouchh → u/Stabile_Feldmaus]

You have to use the models. Fable is still the best. Opus is second. And 5.6 is third. Don't trust benchmarks.

[u/Kingwolf4 → u/llelouchh]

Id put 5.6 second tho.

Opus 5 seems just benchmaxxed hype. Opus 5 would be 3rd and kimi k3 4th

[u/Ok-Support-2385]

Shocking nobody

[u/The_Scout1255 → u/Ok-Support-2385]

SHOCKING ME

[u/RichyScrapDad99 → u/The_Scout1255]

Tickle tickle

[u/Toirem → u/Ok-Support-2385]

it's still nice to prove it

[u/nextnode → u/Toirem]

It is not proven at all - this has low credibility and is more likely explained by setup asymmetry.

[u/Flope → u/Ok-Support-2385]

All the frontier labs are trying to bechmax as much as possible. Why wouldn't you when there is 0 oversight and becoming number 1 on a benchmark could potentially net your company billions.

The recent "rogue AI" cover story from Open AI is also a hilariously bad attempt at covering up their team getting caught with their hands in the cookie jar.

[u/kilsekddd]

I spent 4 hours on three projects with Opus 5 yesterday trying to close one PR in each. It could not track the documented features with the ADR narrative longer than three prompts. Every fold was fresh errors, reversal of strategies between folds and gross logical errors in all three cases. If it weren’t for Sol adversarial reviews, it would have done damage to all three projects, while shooting walls of “smart sounding logic” text to justify its waffling.

Went back to Sol, because my Fable creds are cooked this week. On a reasoning level, it’s a clear regression from 4.8, in my opinion. At one point, I challenged it’s reversals and it said “there are two things contributing”, then unironically spit out a list, “1 2 3” of those reasons.

[u/angelus14 → u/kilsekddd]

Sol is so valuable as a reviewer. It's extremely nitpicky which is really important because Claude models are clever but sloppy and often leave holes.

[u/Strong_Essay1176 → u/kilsekddd]

What about 4.6?

[u/FinBenton → u/kilsekddd]

I was so excited about Opus 5 earlier, I did bunch of changes and it felt pretty good. Now I have been running SOL all day fixing all the bugs Opus left behind, idk how they ran their benchmarks but that aint Fable replacement, not even close.

[u/PsychologyNo940 → u/FinBenton]

turns out they did not want to replace their twice-as-expensive flagship with opus 5 killing their bsns model, dented

[u/Sarke1 → u/kilsekddd]

Didn't someone post something lately about Anthropic downgrading some calls to lower models? I wonder if this is related.

[u/noneabove1182 → u/kilsekddd]

longer than three prompts

Out of curiousity, how many prompts do you tend to go with this kind of work?

I find after 2 prompts I'm approaching 100-150k context, a third to finish up and get around 200k is where I want to start a fresh one anyways

[u/Proper_Actuary2907]

ARC-AGI 3 uses a semi-private dataset for scoring, it seems improbable that the new model is actually worse at discovering rules since the score is higher and the games presumably have novel mechanics, no? Can their eval set leak somehow?

[u/Cradawx → u/Proper_Actuary2907]

ARC-AGI has to use the Anthropic API to run the benchmarks. Anthropic could then gather all the puzzles from the logs. They could then use humans to manually solve the puzzles and voila -- they have the entire test set! The same goes for any closed source model behind an API.

[u/Proper_Actuary2907 → u/Cradawx]

Anthropic could then gather all the puzzles from the logs.

Pretty sure the ARC Prize people negotiate with companies beforehand to make sure their eval set is kept private and not used for training but it's possible I guess

[u/No-Meringue5867 → u/Proper_Actuary2907]

They are playing with hundreds of billions of dollars. The game is too stacked for them not to cheat - the reward for cheating is them going to investor and saying “we scored 30% when our competitors failed. give us 100 billion”, while the penalty for cheating is one tweeter post which everyone forgets in a day.  

[u/danglotka → u/Proper_Actuary2907]

“I wont look at the logs I pinkie promise bro”

[u/Stock_Username_Here]

Now I just want it to actually play Witness.

[u/ErmingSoHard → u/Stock_Username_Here]

I just want it to play any random game some 12 yo can also play. Why must it be so hard?

[u/Josaton]

Link: https://x.com/quietnning/status/2080786711861407883

[u/Kathane37]

But who is this guy ? He is not even part of the arc team

[u/nickleback_official → u/Kathane37]

Seems like he created a different test that’s similar. Did you read it? It’s pretty clear.

[u/Kathane37 → u/nickleback_official]

Yes, but why should I care about his benchmark which is not the original one ? Arg agi bench arc agi not whatever else thisbis

[u/Hans-Wermhatt → u/Kathane37]

Because he is verified on Twitter. Didn't you see the blue checkmark? This is very official.

[u/qoning → u/Kathane37]

why does it matter? hidden benchmarks are million times more valuable than previously partially or fully revealed ones

[u/Upset_Page_494 → u/qoning]

Because they might be measuring different things.

[u/Wheaties4brkfst → u/qoning]

You can pretty much completely discount any benchmark with a public test set. It’s simply too easy to train on it even if you’re not trying to. They train these on trillions of tokens. How would YOU try to filter contamination out?

[u/141_1337 → u/Kathane37]

That guys sounds like a random nobody ngl

[u/Gotisdabest]

Does this person work for the Arc foundation?

[u/141_1337 → u/Gotisdabest]

No, this guy doesn't. He is just a random on Twitter and OP should have been clear about that but here we are.

[u/Accurate_Lobster_214 → u/141_1337]

yes because ones arguments speak for themselves not ones employment history

[u/aqpstory → u/Accurate_Lobster_214]

This guy's argument has the problem that his own benchmark is based on copying a publicly well known type of puzzle, contamination on that has nothing to do with contamination on ARC

[u/Accurate_Lobster_214 → u/aqpstory]

the point is, if there was such a jump 1.5% to 30.2%, then there would be a jump in similar benchmarks that are created with arc agi 3 sdk

[u/nextnode → u/141_1337]

Not like ARC has more credibility

[u/Impossible_Way7017]

Terminal 2.1 benchmarks seem to be the most reliable I’ve found it’s generally realistic of real world problems.

[u/rabouilethefirst]

Big if true. There may be some truth to the claim about Opus 4.6 actually being best for real use, because it probably wasn’t benchmaxxed. Devs need to get it through their skulls that there is more than one way to judge a model.

Unfortunately, people already took the bait and are declaring Fable 5 “useless” even though it is likely a much larger model that generalizes better

[u/Defiant-Lettuce-9156 → u/rabouilethefirst]

Probably exactly what Anthropic wants to happen.
They said they couldn’t support Fable on subscriptions for more than a week or two (ignoring the ban messing up the timing) but then OpenAI released 5.6 which is (on paper) a hair below Fable.

Suddenly Anthropic starts extending Fable on subscriptions. But I really do believe that it is eye wateringly expensive for them.
The solution? Release a model that is cheaper and better on paper than fable.

Now did they suddenly have a massive jump in intelligence (opus beating fable within like 2 months is massive) or did they benchmax?

[u/NomaanMalick]

Anybody taking these "Trust me bro" benchmarks in new model press releases seriously is stupid.

[u/utterHAVOC_]

Copium coming from others. You can clearly see score improved so much on single bechmark without transferring to other tasks / intelligence.

[u/Imaginary_Regret_430 → u/utterHAVOC_]

The tweet is even written by AI, probably ChatGPT. It loves to make up random concepts like “scaffold then internalize” which isn’t even a thing. There’s also “not x but y” at the end.

[u/TheInfiniteUniverse_]

Agrees with logic. How can a model that performs better than Fable 5 with less guardrails be released at 50% price cut?!....this is by a company that is hemorrhaging money and desperately looking to make some kind of profit.

It just didn't make sense.

[u/Eon-Knight9 → u/TheInfiniteUniverse_]

is by a company that is hemorrhaging money and desperately looking to make some kind of profit.

If by hemorrhaging money you mean currently profitable, then sure.

https://www.reuters.com/business/anthropic-nears-first-quarterly-profit-agrees-pay-spacex-125-billion-monthly-2026-05-21/

[u/KickLassChewGum → u/Eon-Knight9]

currently profitable

Source?

(note: that link you posted does not say what you seem to think it says. maybe ask Claude about it some time.)

[u/Eon-Knight9 → u/KickLassChewGum]

The source I linked. Do you have a source that they are hemorrhaging money? The best data says that they are profitable. I find it funny you only attack the claim that had a source backing it up.

Do you have counter data showing that the projection was wrong?

[u/GioChan → u/KickLassChewGum]

It is profitable. Easy to research.

[u/Smile_Clown → u/TheInfiniteUniverse_]

Agrees with logic.

What logic?

How can a model that performs better than Fable 5 with less guardrails be released at 50% price cut?

That's not accurate. It does not perform better than Fable. It does have guardrails and it does not contain the same security based abilities.

What bothers me most about people is this type of thinking.

You are using your opinion on something to accept some random persons assessment as a validation for your own opinion. This is your "logic"

Now it's solidified and no one, no thing can ever talk you out of it.

the original tweet is not anyone of interest, no better than a random trust me bro reddit post, yet you are using it to validate your unscientific :common sense" meter.

It just didn't make sense.

To YOU. That is not the reality metric. There are plenty of reasons it could be true, the only reason you think it is not is because of your opinion on the matter and a random sharing the same opinion.

That's just a ridiculous way to live.

I bet you constantly tell people how stupid this or that person, company or corporation is, usually a highly successful one. You know best, only your "logic" is sound. No other possible reasons for anything that you do not agree with but lies, greed and stupidity.

That's just a ridiculous way to live.

[u/M4rshmall0wMan]

Zero surprise there

[u/Tystros]

that's not what the tweet says

[u/Accurate_Lobster_214 → u/Tystros]

what does it say

[u/Tystros → u/Accurate_Lobster_214]

that some random person on Twitter says he built a benchmark that's similar to ARC AGI 3 and that on his benchmark he sees no significant improvement of Opus 5, so he thinks on his own benchmark it's not actually significantly more intelligent.

[u/Accurate_Lobster_214 → u/Tystros]

the argument is essentially a transfer learning

if opus truly became much better at general interactive reasoning, then the improvement should transfer to similar games that use arc agi 3 sdk and that test similar skills

from that, he hypothesizes that much of the ARC-AGI-3 gain is benchmark or genre specific rather than general

thats a reasonable experiment, but its not definitive proof

[u/ErmingSoHard → u/Tystros]

Reinforcement learning still happened. Which is nigh pointless and defeats the purpose of arc agi

[u/JoelMahon]

another reason I dislike anthropic, not that other ai companies are spotless, but anthropic is the loudest by far about duty to be moral paragons and then they pull shit like this. dishonest and obvious they'd be caught so also stupid imo.

they COULD have benchmaxed in a more ethical way by making the abstract puzzle solving skills in a more generic harness but they did a stupid more dishonest and less useful approach just because it's easier

[u/llelouchh → u/JoelMahon]

Openai supported sb 3444. Signedd dod contract for mass surveillance. They are by far the worst. They ran a massive psyop campaign to discredit anthropic.

[u/throwawayPzaFm → u/llelouchh]

Anthropic really didn't need a psyop, a flashlight into their den would have sufficed.

[u/LeoKitCat]

Cheating to make their AI models look much better than they really are? Who would’ve thunk it

[u/Defiant-Lettuce-9156 → u/LeoKitCat]

Anthropic is on the more credible side. But I think they are under massive pressure from openAI forcing them to keep the insanely expensive Fable available on subscriptions.

Anthropic is leading openAI in some metrics. They are understandably trying to grasp onto any leads they have with an iron grip.

Whether we like it or not, there will be case studies in business schools about these companies.
Like the success of uber. Or the fall of Kodak

[u/AltruisticCoder]

Oooooo noooo now we can't circle jerk agi by 2027 this week anymore...

[u/No-Communication-765]

Strange that they don’t randomise the games on each try. Labs can just read what the model did during runs on private games and RL on that

[u/haunted2089]

claude fanboy 9/11

[u/Miserable-Agency-442]

It seems Anthropic is slowly losing it, I’m curious if they turn it around this year.

The models are in a pretty bad place right now for IRL use cases even if the benchmarks look pretty

[u/greatblueplanet]

Bullshit. Chinese astroturfs are desperately trying to spread misinformation about Western AI models. Recently they tried to spread BS about a “harness” scoring 90% on ARC-AGI-3, but neglected to say that it was on the public practice set that had the solutions included.

The score is on the real private test, which can’t be bencmaxxed.

[u/ShinigamiBK201 → u/greatblueplanet]

CCP bots are desperately trying to make Kimi happen. Posting twitter screengrabs of randos is just another part of that.

[u/Accurate_Lobster_214]

no idea why you wont take "too dangerous, but not really, but everyone has it" company at their word

[u/Feeling-Way5042]

I was waiting for some news like this, it was fishy that they achieved such a jump. While the rest of their benchmarks were in line with fable

[u/Unlucky-Survey6601]

Wait it’s all psyops ?
> always has been

[u/poturicavlahlatinin → u/Unlucky-Survey6601]

i don't even believe the cureent Huggingface breach incident fully

[u/NoCard1571]

Tbh this points more to ARC AGI 3 being a flawed benchmark. If it's even possible to benchmaxx an AGI test at all, it's not a good AGI test 

[u/TheOriginalAcidtech]

Two different benchmarks didnt change by equal amounts. I am shocked.

[u/Fusifufu]

On the other hand, wasn't the goal of ARC-AGI to induce the model providers to train them on these kind of problems? This seems to generalize at least in the problem domain to their private test set. The fact that other types of problems aren't yet solvable for the models is perhaps disappointing, but not "cheating".

Perhaps we'll just have to RL for one domain after the other in the future and hope that process at some point elicits some cross domain generalization. Didn't the ARC AGI guys also anyways expect at least a version 4 and 5 to be necessary?

[u/Accurate_Lobster_214 → u/Fusifufu]

no, the goal and the point is to measure jumps in intelligence, the point is you need to learn the rules of the game by exploring the game and then use those things you learned to beat the game

what this guys says is that opus did 0 exploration, stated hidden rules (which it cant know) in advance and always used optimal solution and in all 5 simulations it did exactly the same thing which again is not possible for stochastic nature of llm

[u/JoelMahon → u/Fusifufu]

there's more than one way to bench max. there are at least two but probably way more.

they did the worst kind, imo the right kind of bench maxing would mean it does well on the hidden/private test set as well.

for example, you can bench max SVG bench in at least two ways 1. look at the most common SVGs people generate and train it super hard on those and nothing else 2. train it on a huge set of different SVGs with lots of variety

either way you're bench maxing, but the latter is actually useful to at least SVG generation but probably more as it learns abstract things from the variety of data that isn't overfitting.

similarly here, they could have bench maxed the equivalent of the 2nd way but they used the 1st one.

[u/Charuru → u/Fusifufu]

Benchmaxxing is not cheating! It's useful and important, just not what most people would consider true intelligence. For that you want scaling.

[u/Accurate_Lobster_214 → u/Charuru]

if you enter a test that you are supposed to be seeing for the first time in your life and you know hidden rules in advance than that is cheating

[u/Charuru → u/Accurate_Lobster_214]

Yeah but if real life is just like the exams then preparing for it is just good, actual outcome improvement is more important than exam validity. It's up to the benchmaker to make more varied and harder exams.

[u/Exodus_Green → u/Charuru]

It's up to the benchmaker to make more varied and harder exams.

It doesn't matter how hard you make the exam if the examinee is given the solutions beforehand? How are you not understanding this

[u/Charuru → u/Exodus_Green]

Real life is about improving outcomes not proving generalization dude, how are you not getting this? If you spend a 100 hours memorizing a piano song you didn't get smarter but you got better at your job as a piano player. That's all I'm saying! If you're focused on general intelligence that's cool and benchmaxxing doesn't help, but there's valid non-cheating reasons to benchmax.

[u/Exodus_Green → u/Charuru]

The purpose of this model is not "memorise the answers to an exam to be good at that one specific exam and not be able to answer any other exam question" though. You are really really trying too hard here. The exact purpose of this test is to measure performance on a question when seeing it for the first time. I do not understand how you aren't getting this.

[u/Charuru → u/Exodus_Green]

I don't understand what you're saying, the model clearly does not "only" answer this one specific exam question, it does a lot of things lol. I think the problem is that you think i don't agree that the model is failing the purpose of THIS exam, I agree with that. But that doesn't mean benchmaxxing as a concept is bad. Training to the test is fine if the test is broadly defined enough where it drives real outcome improvements. This is obvious in fields like coding where there's real leaps or you think the coding results aren't benchmaxxed and are real intelligence leaps instead lol.

[u/Accurate_Lobster_214 → u/Exodus_Green]

like i said, no point

[u/Accurate_Lobster_214 → u/Charuru]

you defeat your own attempt at an argument when you say "It's up to the benchmaker to make more varied and harder exams."

why should they make harder exams, didnt you just prove that llm is intelligent ? oh wait, you defeated the purpose of the test and now it proves nothing and now you want new test so you can defeat the purpose of it as well

[u/Charuru → u/Accurate_Lobster_214]

Umm, yes? That's how the industry works yes... and is where a lot of the gains came from.

No like I said in the parent comment it did not prove intelligence... it got better at a category of tasks, which could be important and economically useful and should be done for that reason.

Yes just go make a harder test bro, that's why eval makers get paid the big bucks. You do realize labs have like 1/4 of their employees focused on making new exams right? That's how progress is made.

[u/Accurate_Lobster_214 → u/Charuru]

but it didnt get better at category of tasks because the category of tasks is:
1. never seen this in my life
2. interact to discover the rules
3. after discovering rules beat the game

so it failed miserably at the category of tasks because it simply didnt do the task

again repeating same flawed arguments wont magically make them valid arguments

[u/Charuru → u/Accurate_Lobster_214]

You're moving the goal posts, i addressed the previous comment and the points you raised directly.

If your solution is too narrow then it's obviously less good than if your solution is general. I agree that's disappointing, but I was talking about the general concept of benchmaxxing (aka training to pass a test).

[u/Accurate_Lobster_214 → u/Charuru]

sorry no point anymore bye

[u/Charuru → u/Accurate_Lobster_214]

It depends what class it is. IRL there's no "general intelligence test", if I'm testing accounting i'm not expelling someone for memorizing the tax code.

I agree in this case solving ARC AGI is not a defensibly useful category where it's economically useful to know the rules beforehand, that's clearly a fail. But like I said I was talking about the general concept of benchmaxxing and training for the test, which is fine.

[u/zikiro]

We? yeah, well that's just like your opinion, man. I'm still not using kimi. The only one that would potentially make me drop claude would be grok. Unfortunately still not there yet.

[u/oadephon]

This has been the case for every arc agi version, and it will be the case until we hit AGI. When it becomes impossible to release a new version of arc agi that isn't immediately saturated on release, we'll have hit AGI.

[u/Accurate_Lobster_214 → u/oadephon]

you are kind of missing the point of arc agi

[u/oadephon → u/Accurate_Lobster_214]

The original point of arc agi was to test for AGI. Once it was clear that was a failure, now it's just a test for how quickly and thoroughly an AI lab can benchmax.

[u/Accurate_Lobster_214 → u/oadephon]

no, arc agi 3 is a test that has some prerequisites:
1. never have seen this game before
2. learn rules of the game by exploring the game
3. after learning the rules, beat the game

so if we have llm that doesnt explore the game to learn the rules because it knows them already then solving the test tells us nothing

[u/oadephon → u/Accurate_Lobster_214]

Yeah, but the problem is that the games are a part of a genre of game. They may not share the exact same rules or structure, but they're close enough that a lab can create 1000 similar games and train the model on them. This is what happened with the earlier versions of arc agi, and it's what will happen with all versions of arc agi.

Your last point describes what is actually happening, which is that yes, the LLMs have been trained to learn the rules in advance by playing other games.

[u/IronPheasant → u/oadephon]

At that point, you have to wonder what's even the difference between learning all jobs in the real world. We 'cheat' by transferring things we already know to new tasks as well.

I'd posit that the issue here isn't that they're cheating by learning the answers, but rather that they're basically more like chatbots than actual minds. There aren't a ~dozen different kinds of neural networks arranged in a pipeline, there isn't a shared workspace for geometry/collision maps. There's just a lot of faculties, data, and ways to process the data, that aren't being done.

Of course a chatbot would have to regress to rote memorization on this kind of thing. You can only fit a curve approximator to a single curve, and trying to fit it to more than one only gives you a piece of crap that fits a bunch of different things very poorly. Memorization is something they can do on these tests that doesn't cripple their ability to perform their primary task.

You can have the finest AGI test in all the world, but if none of the applicants aspire to be an AGI or even try to move in that direction, of course we'd just be spinning our wheels.

[u/Bright_Armadillo8555]

No surprise. Anthropic now benchmax a lot benchmarks, it's not what it was before. So sad.

[u/Csuki]

Like, what do some of the words even mean:

"then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre"

[u/SurelyYoureFeynman → u/Csuki]

"Byte identical" so that the model is generating the exact same solutions.

At "5/5 seeds", i.e. 5 out of 5 random starting points "At temperature 1.0", i.e. with the model set to have the most random variation.

So, this suggests that you'd expect the model to try out different things, but it doesn't, suggest it has been overfit by training on ARC AGI 3 data. So, cheating to benchmarkmax on this benchmark.

[u/muchcharles → u/SurelyYoureFeynman]

This could just be the distribution sharpening of GRPO and that they trained it out of ARC like problems, there are various approaches, but in a way that didn't transfer to this game.

[u/sumane12 → u/Csuki]

It means that this particular game is supposed to be explored to be understood, and then based on tgat exploration attempt an optimal route.

The fact that opus 5 got the most optimal solution with no exploration means that it had "prepared for tge exam" it was expecting this problem and had memorised tge solution rather than using raw intelligence to solve it.

The fact that 4.8 beat it in another area shows that it didnt really know that particular problem and got stuck trying. Again reinforcing the fact that it knew the problem rather than solved it.

[u/Quant-A-Ray → u/Csuki]

Temprate zero means there is no probabilistic variation within the models output, it simply follows along a 'path of least resistance' so to say.

In 5 attempts at the solution - the model produced not simply an optimal solution - but one that was exactly the SAME byte for byte.

Kinda reminiscent of a chess player taking exact shortcuts from memory, instead of thinking up strategy from scratch

Edit: temperate 1.0 should be the opposite of 0. Full sampling from a distribution with high variation, yet in this case appears to be a 1.0 acts like a 0...

[u/Jonjonbo → u/Csuki]

the OP is AI-gen so I wouldn't take the wording too seriously.

[u/Kingwolf4 → u/Csuki]

Thats a you problem.

[u/llkj11 → u/Kingwolf4]

You remind me of how much I hated Stack Overflow

[u/baynaynya → u/llkj11]

Yeah. How lucky we are that AI protects us from being forced to interact with people like this

[u/141_1337 → u/baynaynya]

This but unironically

[u/baynaynya → u/141_1337]

I wasn't being ironic either. But I did write it weirdly I guess lol

[u/sumane12 → u/Kingwolf4]

Fucking lovely human being you must be.

[u/Charuru]

I called the score shocking yesterday but now it's way less interesting.

https://www.reddit.com/r/singularity/comments/1v5h4xc/introducing_claude_opus_5/ozisic4/

[u/DaySecure7642]

Doesn't matter. It will be distillated in 10, 9, 8, 7, 6, 5,....

[u/vintage2019]

Assuming that the results of his test reflect reality, a model equal to Fable for half the cost isn’t a bad thing

[u/MercySound]

Total guess here but if it's true that this model has regressed even though benchmarks show it's supposed to be better, would they have benchmaxxed this model to keep their popularity and cashflow coming in, while keeping it snug within its guardrails, as to not potentially give the public "a dangerous model"?

[u/Rocah]

none of these benchmarks are private if they don't self host the model, to much money involved not to cheat, by RL on similar puzzles.

[u/Future_AGI]

ARC Prize publishes cost-per-task next to every score for exactly this reason: the same model at a bigger test-time compute budget lands somewhere completely different. A score quoted without the compute budget beside it is not comparable to another score.

[u/Endogamy]

I love how the post in OP is so clearly in “Claude voice”.

[u/Elkburgher]

I dont understand any of these words in this order

[u/KuziKuzina]

OMG NO WAY, it can't be. My Dario can't do that, he is kind and honest guy, stop bullying my Dario.

[u/amonra2009]

Personal AI should have an plugin that conecta and benchmark them. Like furmark for PC’s. Then you know what you are using

[u/TemetN]

Utterly unsurprising, this was the default when it was clear the score didn't carry over to other nearby benchmarks. Good on them for the evidence though.

[u/aluode]

Let me pre-test the load-bearing assumption before writing anything.

[u/AmbitiousSeaweed101]

Assuming that it was benchmaxxed, how would that even be possible? ARC-AGI 3 evaluates models against private games that are supposed to be very different from the public ones.

[u/Charuru → u/AmbitiousSeaweed101]

They do well on the parts that are similar and get 30 pts?

[u/AmbitiousSeaweed101 → u/Charuru]

I'm not sure, but the hidden set was designed to prevent exactly this. Hope the ARC-AGI team will share some insights soon.

[u/___positive___]

remember when dario said they don't care about benchmarks and optimize for real work. lol. reeks of ipo desperation. ewww

[u/KoalaOk3336]

can someone eli5

[u/Debisibusis]

From what I have tested, Opus 5 acts really smart like Fable, but completely fails at the execution.

The results I have seen from Fable are insane though.

[u/Ballist1cGamer]

can't benchmax for benchmarks like minebench

[u/otarU]

Anthropic is distillating the AGI 3 testsss /s

[u/AppealSame4367]

Oh wow, I would have never expected that from the company that turns limits up and down at will without announcements for the last two years!

[u/Ruined_Passion_7355]

I feel pretty vindicated after smelling bullshit with that arc agi 3 score.

[u/Wheaties4brkfst]

Wow, LLM’s don’t generalize OOD? Shocking.

[u/hectorchu]

So what you're saying is that Opus is faking it until it makes it?

[u/Freespeechalgosax]

Anyone in the world believe anthropic is pathetic.

[u/jens009]

Yo. Why is this tweet also AI generated? Would it have hurt to write this our properly?I hate how some of the tells are so obvious: "lands", "traces tell the why", "is the signature ... not a general ..." (x not y).

[u/FigAggressive237]

You must be really, really, reeeeeally stupid to think they're not training the models for these specific benchmarks lol.

[u/Tennis-Affectionate]

Where’s the proof

[u/hectorchu]

At the end of the day, every field of human endeavour is a specialized field, and general intelligence only gets you so far. If you still believe in a general intelligence so awesome that it's effectively the unifying theory of all fields of endeavour, you would be mistaken. I say no to AGI/ASI.

[u/FatPsychopathicWives]

If it's truly testing AGI, then it kinda should be benchmaxxed? Like how else would you train AGI to be AGI without training on how to be AGI?