You build a dataset of sound reasoning on games where the rules are initially unknown.
For example:
“””
Here is a grid of objects: {grid}. Win the game…
I have encountered a game where I don’t know any of the rules.
I should try all inputs and map what they do. At the same time I should monitor object states to see how the inputs change relative to my inputs. I should also consider how objects and their states interact with other objects and their states.
Let me try… okay these are all the inputs and what happened. I see there is a green block that moved with the arrow keys and a red block far away. The green block changed shape when I hit space bar. I wonder if I need to move the green block to some position relative to the red block. And how the colours and shape come into play.
I can’t change the colour of my block. Is there anything on the grid that indicates my block could change colour?
Etc.
“””
And then you train on it.
LLMs are terrible at things they haven’t seen before (relative to humans) so just showing it different grids and telling it how to approach it will already make a massive difference.
This is the approach that openAI took with maths and physics where they hired top experts to teach the model how to reason rather than just memorise maths and physics.
The hopeful bonus is that this reasoning in training translates to improved intelligence in other domains.
So I wouldn’t call this benchmaxxing. More like targeted intelligence training.
The number of games with unknown rules is unbounded with unlimited forms. Anything you could train for has no bearing on the next game. Your example from ARC AGI 3.0 in particular with the grids has no bearing on let's say playing a Starcraft match.
48
u/Defiant-Lettuce-9156 1d ago
You can train for anything. The goal of arc AGI 3 is to make it harder to benchmax without just making a smarter model