Chapter 5 of 10All chapters
Chapter 5 of 10
Goals and intentions
Optimisation without desire.
No wants
These systems have no preferences or self-interest. They optimise objectives given to them, and unintended behaviour comes from badly specified objectives rather than from wanting anything.
- Anthropomorphic language makes this harder to think about clearly.
- An agent pursuing a goal literally can still cause real problems.
Sentience
There is no evidence of consciousness, and no accepted test for it. Fluent claims of feeling are exactly what a system trained on human text would produce either way.