Update to Friendly AI theory
Eliezer Yudkowsky writes "I've posted an update to the Friendliness part of Friendly AI theory. The essay is called Collective Volition and it's currently located on the SL4 Wiki. The essay isn't technical; the update describes my most recent thoughts about the what-it-does of a Friendly AI, the thing that this ultrapowerful superintelligent recursively self-improving optimization process is supposed to do, if the Singularity Institute can get it running. If I don't hear any objections between now and the Singularity, I'll assume the entire human species is okay with this and I can go ahead and do it."
June 15th, 2004 at 5:18 PM
Evil Robot Army
You talk of tool-AI vs mind-AI and say that mind-AI can actually be safer than tool-AI. Tools are neither good nor evil, they can be used for either, but a mind can be constructed with a conscience. People are constructed with a conscience, and most of us are, yet we're capable of unspeakable evil. Perhaps that's human nature, so let's ignore that. You talk of an AI recognising that information it has received came from humans and therefore it will listen to arguments from humans. Presumably this is intended to make an AI "listen to reason" and prevent runaway harmful acts of stupidity, much like the 15 or so episodes of classic Star Trek where Kirk deals with a robot set on destroying the Enterprise. But what if those whispering into the ear of an AI are not acting in the best interests of humanity? An AI that finds religion could be the most unfriendly of them all.