MEPX
Chapter 7 of 12All chapters
  1. 1What AI means
  2. 2How machine learning works
  3. 3Training data
  4. 4Kinds of learning
  5. 5Neural networks
  6. 6Large language models
  7. 7Limits and hallucination
  8. 8Bias and fairness
  9. 9Using AI well
  10. 10Privacy and safety
  11. 11Beyond text
  12. 12AI and work

Chapter 7 of 12

Limits and hallucination

Where these systems go wrong.

Making things up

A hallucination is confident output that is simply false: an invented citation, a plausible but wrong date, a function that does not exist. It is not a bug to be patched, it is the flip side of generating text.

  • Names, numbers, quotations and citations are the highest risk categories.
  • Asking the same question twice can produce two different confident answers.

Other failure modes

Models are weak at arithmetic they cannot see through, at counting, and at knowing what they do not know. They also follow instructions found in the content they are given, which is a security problem as well as an accuracy one.