Sorry, I assumed you would have actually read the DELEGATE-52 study linked instead of just the abstract. For “a model optimized for replacing StackOverflow” that is “better at writing papers than most students” LLMs sure did pretty bad at those tasks over multiple rounds.
As the chart on page 7 of the paper shows, LLMs are good at exactly the kind of tasks you’d expect (producing and manipulating language), and bad at exactly the kind of tasks you’d expect (doing almost anything else). All this paper shows is that (1) they aren’t AGI, and (2) as a consequence of not being AGI they aren’t good unsupervised.
What the fuck? The only task that didn’t degrade across most models was Python. Very basic things like JSON, Makefiles, and schemas got screwed. Fiction, emails, and food menus got screwed. Did you even bother to read the legend? If you consider a single pass to be “producing and manipulating language” you didn’t bother to read the idiotic article you started this thread in support of. Good luck.
Edit: why do you lie?
Catastrophic corruption (80 and below) occurs in more than 80% of model, domain combinations.
The only task that didn’t degrade across most models was Python.
Yeah, after 20 cycles of unsupervised iteration on the task. Gemini 3.1 Pro doing as well as it did under that experiment setup is quite remarkable actually.
Sorry, I assumed you would have actually read the DELEGATE-52 study linked instead of just the abstract. For “a model optimized for replacing StackOverflow” that is “better at writing papers than most students” LLMs sure did pretty bad at those tasks over multiple rounds.
As the chart on page 7 of the paper shows, LLMs are good at exactly the kind of tasks you’d expect (producing and manipulating language), and bad at exactly the kind of tasks you’d expect (doing almost anything else). All this paper shows is that (1) they aren’t AGI, and (2) as a consequence of not being AGI they aren’t good unsupervised.
Why do you lie like this?
What the fuck? The only task that didn’t degrade across most models was Python. Very basic things like JSON, Makefiles, and schemas got screwed. Fiction, emails, and food menus got screwed. Did you even bother to read the legend? If you consider a single pass to be “producing and manipulating language” you didn’t bother to read the idiotic article you started this thread in support of. Good luck.
Edit: why do you lie?
Yeah, after 20 cycles of unsupervised iteration on the task. Gemini 3.1 Pro doing as well as it did under that experiment setup is quite remarkable actually.
The paper does not show what you are arguing.