“Friendly AI” now open for commentary
from the smart-allies-not-enemies dept.
From Senior Associate Eliezer Yudkowsky: The Singularity Institute has just announced that it has begun circulating a preliminary version of "Friendly AI" for open commentary by the academic and futurist communities. "Friendly AI" is the first specific proposal for a set of design features and cognitive architectures to produce a benevolent – "Friendly" – Artificial Intelligence. The official launch is tentatively scheduled for mid-June, but we hope to discuss the current paper with you at the upcoming Foresight Gathering this weekend. Read More for more details. With the publication of "Friendly AI", the topic has moved for the first time beyond the realm of pure speculation, into the realm of technical speculation. There is now a specific description of how a Friendly AI would operate, behave, and make choices. In addition to increasing the safety of AI in the long run, we hope that "Friendly AI" will raise the level of discussion in the immediate debate about the long-term impact of very high technologies. The Friendship architecture described in "Friendly AI" would not be disrupted either by the ability of the AI to modify its own source code (including goal-system source code), or by the AI moving beyond the point of dependence on humans, two possibilities that are often treated as synonymous with disaster in the current literature.
April 23rd, 2001 at 2:53 PM
insulation from evolution
A lot of Yudkowsky's arguments that it is incorrect to expect failure modes from an AI analogous to those that human motivations in it would produce hinge on the fact that the AI will not have evolved, will not have been honed by selection pressure in an uncontrolled competitive environment.
I agree that his analysis could be correct if one (or a set of cooperating) Friendly AI projects takes off hard enough to dominate the environment for any later project. If, on the other hand, take off is softer, then AIs could find themselves effectively competing with peers in an environment that no single design team controls. Under these circumstances, I'd expect to see at least some effects of selection pressure, albeit not anything like human goal structures. Still, selection pressure could reintroduce some of the hazards of powerful entities with more humanoid motivations. Odd to think of winner-take-all as the preferred outcome…
April 24th, 2001 at 8:30 PM
looks like the right questions
I am still in the process of making my way through FAI, having only become aware of it in the past few days. My first quick impression is that it addresses the right collection of questions. If one takes seriously the possibility that the development of machine intelligence will some day escape from human control, then these issues are the right ones to be thinking about. Whereas specific conjectures may eventually be found to be in error, it is an important contribution to have illuminated the correct area of general inquiry. My second quick impression is that the "Beyond anthropomorphism" section offers some good counter-arguments to the geniebusters site.