"Company creates machine learning model to discover new drug compounds. To make sure the molecules they discover are non-toxic, their model explicitly predicts the toxicity of any given molecule and discards ones that are considered too toxic.
Then they hear worries about misuse of AI technology, and it occurs to them to try what would happen if, instead of minimizing toxicity, they told their model to create *maximally* toxic molecules instead. Could their system be misused to do something dangerous?
A couple of tweaks and six hours of runtime later, the model has not only reinvented "nerve agent VX, one of the most toxic chemical warfare agents developed during the twentieth century", but also several other known chemical warfare agents, *and* several novel ones that the model predicts to be even more deadly than any of the previously known ones." -- Kaj Sotala
Then they hear worries about misuse of AI technology, and it occurs to them to try what would happen if, instead of minimizing toxicity, they told their model to create *maximally* toxic molecules instead. Could their system be misused to do something dangerous?
A couple of tweaks and six hours of runtime later, the model has not only reinvented "nerve agent VX, one of the most toxic chemical warfare agents developed during the twentieth century", but also several other known chemical warfare agents, *and* several novel ones that the model predicts to be even more deadly than any of the previously known ones." -- Kaj Sotala