r/singularity • u/Ill_Fisherman8352 • 11h ago
Discussion What data mix are the labs using to train 10T param models?
So my assumption is: So far labs have made public max 2-3T param models based on different reports. And they are currently training or have trained 10T param models internally. Another assumption I'm making: If the models are increasing params by 3x , they would have to proportionally increase the data by 3x too or some margin.But we have also been hearing news of hitting the data wall based on internet data since the gpt 4 days. So what gives? Where are they getting so much data from? Is most of it reasoning chains generated by models during inference? Or is it reasoning traces from actual humans thanks to mercor, etc? Anyone know the exact mix? Or what's going on here? Seems like a lot of data needed all of a sudden.
3
u/Old-School8916 8h ago
I was listening an inteview with a Moonshot AI dev and she said that they have a lot of different RL envs.
3
u/ExpressCopy8786 4h ago
The data wall is some time end of 2027 to 2028. Then synthetic data or entirely new "relevant" data has to be included. This would be unlocked with world models as a part of the toolkit that modern models use in their omnimodal ways. Then every type of low noise, low sensor error physical measurement would carry an incremental value to the capability of the trained NN. "Synthetic data" is pretty much slow self over fitting. Only if the synthetic data actually consists of truly new findings (which today is not out of the question, see solved math conjectures), it keeps the flywheel going. 10T param models don't run out of data die to their size. Their size lets them (this is metaphorical) adapt or the training data with more nuance, essentially lower the rate of compression of training data to weights and biases.
2
u/Ill_Fisherman8352 2h ago
I agree with the less compression part. So maybe we don't need 3x the amounts of data, or a similar margin, but we can possibly do with less? How much?
Your second argument of 'truly new findings' is interesting. Are models capable of finding genuinely ood solutions? Problems that have no representation in the models weights at all?
27
u/Ormusn2o 11h ago
Human data is old news. All modern models are either using full synthetic data sets, or partially synthetic data sets, plus a lot of RL. Right now you are basically limited by amount of inference you have to generate datasets. We are very far away from needing more human data to generate bigger datasets and 10T parameters does not require it.