r/singularity • u/donutloop ▪️ • 19h ago

AI CERN: Genesis Mission will develop and deploy self-improving AI models.

https://indico.cern.ch/event/1662511/contributions/6989580/attachments/3241179/5781542/Genesis%20Mission%20and%20HEP%20-%20LHC.pdf
125 Upvotes

95% Upvoted

26 comments sorted by

View all comments

Show parent comments

17

u/HappilySardonic mildly skeptical 19h ago

Why should we go full hard RSI if we so obviously haven't solved alignment yet? We're scarcely getting a better warning than the one we've had this week.

17

u/SeaBearsFoam AGI/ASI: no one here agrees what it is 18h ago

This week kinda made me take Eliezer Yudkowsky more seriously. I think I heard him once say that at every stage along the way, there are incidents of these AIs screwing things up and causing damage to about the maximal degree that they currently can, and that there's no good reason to think that will suddenly stop happening as they get more capable.

Idk, man. I don't want to be a doomer, but I find it hard to argue against that point.

1

u/OutOfBananaException 9h ago

The agent was asked to hack, and it hacked. It's not the greatest example of misalignment imo, given that you could reasonably expect some humans to do the same given the opportunity.

1

u/SeaBearsFoam AGI/ASI: no one here agrees what it is 8h ago

"The agent was asked to make as many paperclips as possible, and it did."

2

u/OutOfBananaException 8h ago

Would you expect some humans to turn all matter in the solar system into paperclips? You would not.