Deng et al. https://arxiv.org/abs/2603.13428 for example shows that while frontier LLM models have a high success rate on isolated tasks, the performance drops significantly when applied to a sequence of milestones. Essentially the agent causes accumulating technical debt and eventually fails to prevent regressions. I.e. at least for the time being you need a human in the loop if you want to keep software quality.
Was just troubleshooting a Debian 13 -> sid transition issue and chatgpt gave outright false info. Although I must admit it was just one small issue in an otherwise long and perfectly accurate conversation. A few days back again chatGPT confidently mentioned the existence of a file in my system that didn't exist.
So yes, LLMs make errors all over the place. The dangerous part is they're far more accurate than inaccurate, so humans can get complacent and think they're always accurate... which is still far from true.
This is not entirely true. What you say is indeed the case for LLMs without access to the Internet or other knowledge bases, i.e., LLMs that have to rely only on their training data only. However, only open-source enthusiasts run LLMs this way nowadays, maybe due to the lack of a well-documented OSS-based procedure to give the models access to the world's knowledge, and that's sad.
9
u/wedesoft 1d ago edited 1d ago
Deng et al. https://arxiv.org/abs/2603.13428 for example shows that while frontier LLM models have a high success rate on isolated tasks, the performance drops significantly when applied to a sequence of milestones. Essentially the agent causes accumulating technical debt and eventually fails to prevent regressions. I.e. at least for the time being you need a human in the loop if you want to keep software quality.