Matt McAlister's List: Reading Yudkowsky

• Our minds respond less readily to [training] than our hands. Muscles are evolutionarily ancient subjects of neural control, while cognitive reflectivity is a comparatively more recent innovation
• These fields give us new focusing lenses through which to view the landscape of our own minds.
• A bias is “a certain kind of obstacle to our goal of obtaining truth.”
• error is the norm, not the exception
• As the proverb goes, “There are forty kinds of lunacy but only one kind of common sense.”
• Error is not an exceptional condition; it is success which is a priori so improbable that it requires an explanation.
• we seem to label as “biases” those obstacles to truth which are produced, not by the cost of information, nor by limited computing power, but by the shape of our own mental machinery.
• Biases are obstacles to truth not caused by the cost of information, or by limited computing power, or by adopted beliefs or moral values, or by brain damage. Instead, “biases arise from machinery that is humanly universal.”
• The vast majority of appeals that I witness to “rationalist’s humility” are excuses to shrug. The one who buys a lottery ticket, saying, “But you can’t know that I’ll lose.” The one who disbelieves in evolution, saying, “But you can’t prove to me that it’s true.” The one who refuses to confront a difficult-looking problem, saying, “It’s probably too hard to solve.” The problem is motivated skepticism aka disconfirmation bias – more heavily scrutinizing assertions that we don’t want to believe. Humility, in its most commonly misunderstood form, is a fully general excuse not to believe something; since, after all, you can’t be sure. Beware of fully general excuses!
• To be humble is to take specific actions in anticipation of your own errors.  To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty.
• lthough accuracy necessarily implies consistency, consistency does not necessarily imply accuracy.  If I sit in my living room with the curtains drawn, and make up five maps that are consistent with each other, but I don’t actually walk around the city and make lines on paper that correspond to what I see, then my maps will be consistent but not accurate.  When genuine Bayesians agree in their probability estimates, it’s not because they’re trying to be consistent – Aumann’s Agreement Theorem doesn’t invoke any explicit drive on the Bayesians’ part to be consistent.  That’s what makes AAT surprising!  Bayesians only try to be accurate; in the course of seeking to be accurate, they end up consistent.  The Modesty Argument, that we can end up accurate in the course of seeking to be consistent, does not necessarily follow.
• Eliezer: “I don’t know” is usually just a screen that people think is defensible and unarguable before they go on to do whatever they feel like, and it’s usually the wrong thing because they refused to admit to themselves what their guess was, or examine their justifications, or even realize that they’re guessing
• Eliezer: “I don’t know” is often a good way of not getting in trouble
• “Sometimes you just can’t save people from themselves.”

12 more annotations...

• We underestimate the power of science, and overestimate the power of personal observation. A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes. Our own eyes can deceive us. People can fool themselves, hallucinate, and even go insane. The controls on publication in major journals are more trustworthy than the very fabric of your brain. If you see with your own eyes that the sky is blue, and Science says it is green, then sir, I advise that you trust in Science.
• this lack of correlation should cause us to “Sit bolt upright in alarm.”
• I cannot but conclude that this person literally doesn’t know why you have to look at things. They may have been taught a certain ritual of experimentation, but they don’t understand the reason for it – that to map a territory, you have to look at it – that to gain information about the environment, you have to undergo a causal process whereby you interact with the environment and end up correlated to it…
• while the evidence for matters of fact (like biological evolution) can often be one-sided, this need not be the case for policies, which often have surprising, double-sided effects

• Burch’s Law:

   

  If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold.
• We tend to judge things – for example, a technology – with either a positive or negative attitude. We add up the positive points and negative points and we have our result.
• the facts we use in the debate are logically distinct. The facts don’t know which “side” they’re on.
• So is the Photoshopped model on the magazine cover. No live, physical woman on the planet has ever pushed that many buttons in the male sex drive so perfectly. Only hours of makeup and lighting and Photoshop can do that.

• I leave you with a final argument from fictional evidence: Simon Funk’s online novel After Life depicts (among other plot points) the planned extermination of biological Homo sapiens – not by marching robot armies, but by artificial children that are much cuter and sweeter and more fun to raise than real children. Perhaps the demographic collapse of advanced societies happens because the market supplies ever-more-tempting alternatives to having children, while the attractiveness of changing diapers remains constant over time. Where are the advertising billboards that say “BREED”? Who will pay professional image consultants to make arguing with sullen teenagers seem more alluring than a vacation in Tahiti?

   

  “In the end,” Simon Funk wrote, “the human species was simply marketed out of existence.”

7 more annotations...