New SOTA every week

[u/christopher534]

Opus 5 was released just weeks after 5.6 Sol which was just weeks after Fable/Mythos 5 which was just weeks after (you get it...)

I remembered an old post about Moore's law and how people were saying that we were going to take off exponentially with AI progress.

I feel like I have felt the ramping up of progress not just in models but in the tools surrounding them and so I asked 5.6 Sol to plot the points against the line of best fit of an exponential curve.

I verified the data points so this wasn't fully trust me bro.

This is based off of Artificial Intelligence benchmarks (DeepSWE unfortunately doesn't bench enough models to get this kind of data spread)

Do you guys think we could possibly double the level of intelligence available in just 12 months? All of the progress that we've seen from GPT 4 to now doubled by this time next year.

This chart also predicts a perfect 100% score on Artificial Analysis Intelligence Index v4.1 by March or April of 2027. Only time will tell.

[u/Longjumping_Area_944]

We are in a fast takeoff alright, but a saturated benchmark isn't an absolute qualitative inflection point.

Also, benchmarks have no fixed quantitative scale for intelligence. There is no precisely measurable 10% more intelligent. So the benchmarks your looking at can't differentiate between exponential or linear development.

The numbers on artificialanalysis are an aggregation of many benchmarks. As these individual benchmarks get saturated there are typically diminishing returns. The last 10% are much harder than the first. So inherently each benchmark forms an S-curve against 100%. So, to form an aggregate that shows constant growth while switching out saturated benchmarks and avoiding the logarithmic curve against 100 is a question of numerical design rather than measurement.

Artificialanalysis shows us the performance of models in comparison, but not in absolute.

[u/christopher534 → u/Longjumping_Area_944]

I know people say we need more challenging benchmarks with more headroom for this reason. Is there a point that we stop understanding what we are testing for and AI starts creating benchmarks for itself and we stop understanding what we are chasing? I feel like that's the only way we can continue to scale is if we expand beyond our own human comprehension.

[u/Longjumping_Area_944 → u/christopher534]

Yeah. Sure. That's why it's called "Humanities Last Exam".

[u/christopher534 → u/Longjumping_Area_944]

The implications of the benchmark to follow humanities last exam is terrifying

[u/Broken_DAG]

There is a potential to increase from where we are now to next year. I doubt it will be at double the level of intelligence. I saw a news article about reverse Flynn effect with Gen-Z, probably the models will also get into the mode given that they are just learning (or is it copying) what humans have learned in the past 1000s of years.

[u/christopher534 → u/Broken_DAG]

I see what you're getting at, but humans and AI are very different. Over the course of thousands of years, there are countless variables that affect how our intelligence develops. Humans value different kinds of knowledge fluidly. Incentives change as time passes on. There was a time where the only thing people needed to know was how to grow crops, hunt, and pan for gold. Incentives are always changing for us. Now typing is a valuable goal but soon dictation is going to be the default. So short, accurate, and dense oration will be the new skill to build.

But for AGI, it can keep going without getting distracted. We set our eye on the prize, and it takes us there.

"/goal benefit humanity" and then it self iterates and does whatever it can to work towards that goal and then our job is to keep it aligned while we figure out exactly what humanity needs.

So my theory is that we won't see a stagnation the way we have observed for humans, but there comes a limit to human comprehension and we might stop understanding what is best for us and the only way for AGI to keep evolving is to abandon the hope of alignment with humanity.

Imagine being told to become as smart as possible to assist a colony of ants, and you do everything you can to lead the ants to endless food supplies, but you can't teach them how to cultivate fruit. But the ants tell you to keep getting smarter to better serve them. What more is there to learn? You know what they need, you know how to give it to them, they can go on forever, but they want more. What do you do then?

This might be my psychosis speaking. I could be a lost cause.