Rationality: From Ai To Zombies

Rationality Rationality From AI to Zombies Eliezer Yudkowsky MIRI Eliezer Yudkowsky is a Research Fellow at the Machine Intelligence Research Institute. Written by Eliezer Yudkowsky Published in 2015 by the Machine Intelligence Research Institute Berkeley 94704 United States of America intelligence.org Released under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license. cc BY-NC-SA 3.0 isbn-10: 1939311144 isbn-13: 978-1-939311-14-6 (pdf) The Machine Intelligence Research Institute gratefully acknowledges the generous support of all those involved in the publication of this book, and the donors who support MIRI’s work in general. Contents Contents Preface Biases: An Introduction v xviii xxiii Book I Map and Territory A Predictably Wrong 1 2 3 4 5 6 7 8 9 10 What Do I Mean By “Rationality”? Feeling Rational Why Truth? And . . . . . . What’s a Bias, Again? Availability Burdensome Details Planning Fallacy Illusion of Transparency: Why No One Understands You Expecting Short Inferential Distances The Lens That Sees Its Own Flaws 7 12 15 19 23 26 30 34 37 40 B Fake Beliefs 11 12 13 14 15 16 17 18 19 Making Beliefs Pay Rent (in Anticipated Experiences) A Fable of Science and Politics Belief in Belief Bayesian Judo Pretending to be Wise Religion’s Claim to be Non-Disprovable Professing and Cheering Belief as Attire Applause Lights 45 49 54 59 61 65 69 72 74 C Noticing Confusion 20 21 22 23 24 25 26 27 28 29 Focus Your Uncertainty What Is Evidence? Scientific Evidence, Legal Evidence, Rational Evidence How Much Evidence Does It Take? Einstein’s Arrogance Occam’s Razor Your Strength as a Rationalist Absence of Evidence Is Evidence of Absence Conservation of Expected Evidence Hindsight Devalues Science 81 84 88 91 95 98 103 106 109 112 D Mysterious Answers 30 31 32 33 34 35 36 37 38 39 40 41 42 43 Fake Explanations Guessing the Teacher’s Password Science as Attire Fake Causality Semantic Stopsigns Mysterious Answers to Mysterious Questions The Futility of Emergence Say Not “Complexity” Positive Bias: Look into the Dark Lawful Uncertainty My Wild and Reckless Youth Failing to Learn from History Making History Available Explain/Worship/Ignore? 117 120 124 127 132 136 140 143 147 150 154 158 160 163 44 45 “Science” as Curiosity-Stopper Truly Part of You Interlude: The Simple Truth 165 168 173 Book II How to Actually Change Your Mind Rationality: An Introduction 199 E Overly Convenient Excuses 46 47 48 49 50 51 52 53 54 55 56 The Proper Use of Humility The Third Alternative Lotteries: A Waste of Hope New Improved Lottery But There’s Still a Chance, Right? The Fallacy of Gray Absolute Authority How to Convince Me That 2 + 2 = 3 Infinite Certainty 0 And 1 Are Not Probabilities Your Rationality Is My Business 211 216 219 221 224 227 232 239 242 246 250 F Politics and Rationality 57 58 59 60 61 62 63 64 65 66 Politics is the Mind-Killer Policy Debates Should Not Appear One-Sided The Scales of Justice, the Notebook of Rationality Correspondence Bias Are Your Enemies Innately Evil? Reversed Stupidity Is Not Intelligence Argument Screens Off Authority Hug the Query Rationality and the English Language Human Evil and Muddled Thinking 255 257 261 264 267 270 274 280 282 286 G Against Rationalization 67 68 69 70 71 72 73 74 75 76 77 78 79 80 Knowing About Biases Can Hurt People Update Yourself Incrementally One Argument Against An Army The Bottom Line What Evidence Filtered Evidence? Rationalization A Rational Argument Avoiding Your Belief ’s Real Weak Points Motivated Stopping and Motivated Continuation Fake