MEPX
Chapter 5 of 10All chapters

Chapter 5 of 10

Goals and intentions

Optimisation without desire.

No wants

These systems have no preferences or self-interest. They optimise objectives given to them, and unintended behaviour comes from badly specified objectives rather than from wanting anything.

  • Anthropomorphic language makes this harder to think about clearly.
  • An agent pursuing a goal literally can still cause real problems.

Sentience

There is no evidence of consciousness, and no accepted test for it. Fluent claims of feeling are exactly what a system trained on human text would produce either way.