Matt McAlister's List: Less Wrong 1 - Mysterious Answers to Mysterious Questions

• The moral is to ask "What experiences do I anticipate?" not "What statements do I believe?"
• The remarkable thing is that they know in advance exactly which experimental results they shall have to excuse, indicating that some part of their mind knows what's really going on
• by drawing an empirical inference from his religion, he had opened up his religion to empirical disproof
• This seemed weirder than "belief in belief" (she didn't act like she needed validation) or "religious profession" (she didn't try to act like she took her religion seriously). So, what was she doing? She was cheering for paganism — cheering loudly by making ridiculous claims.
• it is very easy for people to be passionate about group identification - sports teams, political sports teams - and this may account for the passion of beliefs worn as team-identification attire
• Sneering at narrowness is rather reminiscent of ancient Greeks who thought that going out and actually looking at things was manual labor, and manual labor was for slaves.
• Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.
• Absence of proof is not proof of absence. But absence of evidence is always evidence of absence.
• If you are a true Bayesian, you cannot seek evidence to confirm your theory, because you do not expect any evidence to do that. You can only seek evidence to test your theory.
• Hindsight bias distorts the testing of our models by observation, making us think that our models are better than they really are.
• This unfairly devalues the contributions of researchers. Worse, it prevents us from noticing when we are seeing evidence that doesn't fit what we really would have expected. We need to make a conscious effort to be shocked enough
• Real explanations constrain anticipation. Ideally, you could explain only the observations that actually happened. Fake explanations could just as well "explain" the opposite of what you observed.
• students incorrectly learn to guess at passwords in the face of strange observations rather than admit their confusion. Don't do that: any explanation you give should have a predictive model behind it.

11 more annotations...

• Though the two argue, one saying "No," and the other saying "Yes," they do not anticipate any different experiences.  The two think they have different models of the world, but they have no difference with respect to what they expect will happen to them.
• these two beliefs have an inferential consequence that is a direct sensory anticipation
• To anticipate sensory experiences as precisely as possible, we must process beliefs that are not anticipations of sensory experience.

  • Something profound.
• It is a great strength of Homo sapiens that we can, better than any other species in the world, learn to model the unseen. It is also one of our great weak points. Humans often believe in things that are not only unseen but unreal.
• but this belief yielded no advance predictions; the link from phlogiston to experience was always configured after the experience, rather than constraining the experience in advance
• The same brain that builds a network of inferred causes behind sensory experience, can also build a network of causes that is not connected to sensory experience, or poorly connected.
• We can build up whole networks of beliefs that are connected only to each other - call these "floating" beliefs. It is a uniquely human flaw among animal species, a perversion of Homo sapiens's ability to build more general and flexible belief networks.
• what experience will happen to you?
• The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict - or better yet, prohibit.
• It is even better to ask: what experience must not happen to you?
• A null answer means that your belief does not constrain experience; it permits anything to happen to you.  It floats.
• When you argue a seemingly factual question, always keep in mind which difference of anticipation you are arguing about. If you can't find the difference of anticipation, you're probably arguing about labels in your belief network - or even worse, floating beliefs, barnacles on your network.
• Above all, don't ask what to believe - ask what to anticipate. Every question of belief should flow from a question of anticipation, and that question of anticipation should be the center of the inquiry. Every guess of belief should begin by flowing to a specific guess of anticipation, and should continue to pay rent in future anticipations. If a belief turns deadbeat, evict it.
• In principle there are many true beliefs for which I have no evidence, but in practice I can never know what these true beliefs are, or even focus on them enough to think them explicitly, because they are so vastly outnumbered by false beliefs for which I can find no evidence.
• failure to see the map-territory distinction, and the resulting reification of categories
• Occam's razor is not so much a rule of science but an operating guideline for doing science. It could be reduced to "test simple theories first".
• Hickum's dictum in particular was a great find (generalized beyond medicine, it could be that explanations for unexplained events can be as complex as they damn well please).
• there is no more searching test of a theology than to submit it to dramatic handling; nothing so glaringly exposes inconsistencies in a character, a story, or a philosophy as to put it upon the stage and allow it to speak for itself.
• But a professional writer of fiction like Sayers really ought to have known better than to suggest that truth can be distinguished from untruth by seeing how easily each can be formed into art.

17 more annotations...

• Carl Sagan used this parable to illustrate the classic moral that poor hypotheses need to do fast footwork to avoid falsification.
• The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he'll need to excuse.
• As if the human brain only had enough disk space to represent one belief at a time!
• Yet it is a physical fact that you can write "The sky is green!" next to a picture of a blue sky without the paper bursting into flames.
• We're supposed to constantly ask our beliefs which experiences they predict, make them pay rent in anticipation. But the dragon-claimant's problem runs deeper, and cannot be cured with such simple advice.
• No, this invisibility business is a symptom of something much worse.
• As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it.
• What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it's not even clear what it would mean to believe it - what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green.  Dennett calls this "belief in belief".
• if you believe in belief, you cannot admit to yourself that you only believe in belief, because it is virtuous to believe, not to believe in belief, and so if you only believe in belief, instead of believing, you are not virtuous.
• You may believe in belief explicitly; you may recite in your deliberate stream of consciousness the verbal sentence "It is virtuous to believe that the Ultimate Cosmic Sky is perfectly blue and perfectly green."
• Maybe the dragon-claimant fears the public ridicule that he imagines will result if he publicly confesses he was wrong
• Maybe the dragon-claimant flinches away from the prospect of admitting to himself that there is no dragon, because it conflicts with his self-image as the glorious discoverer of the dragon, who saw in his garage what all others had failed to see.
• If all our thoughts were deliberate verbal sentences like philosophers manipulate, the human mind would be a great deal easier for humans to understand. Fleeting mental images, unspoken flinches, desires acted upon without acknowledgement - these account for as much of ourselves as words.
• we need a wider concept of belief, not limited to verbal sentences. "Belief" should include unspoken anticipation-controllers.  "Belief in belief" should include unspoken cognitive-behavior-guiders. It is not psychologically realistic to say "The dragon-claimant does not believe there is a dragon in his garage; he believes it is beneficial to believe there is a dragon in his garage."  But it is realistic to say the dragon-claimant anticipates as if there is no dragon in his garage, and makes excuses as if he believed in the belief.
• To correctly anticipate, in advance, which experimental results shall need to be excused, the dragon-claimant must (a) possess an accurate anticipation-controlling model somewhere in his mind, and (b) act cognitively to protect either (b1) his free-floating propositional belief in the dragon or (b2) his self-image of believing in the dragon.
• I think it is very difficult to attribute a belief in dragons to this "dragon-believer". Only a small subset of his actions - those involving verbal avowals - make sense if you attribute a belief in dragons to him. There is a conflict with the remainder of his beliefs, as can be seen when he nonchalantly enters his garage, or confabulates all sorts of reasons why his dragon can't be demonstrated.
• But as you have shown, everything makes sense if you attribute a related, but slightly different belief, namely "I should avow a genuine, heartfelt belief in dragons". Perhaps we can say that this man (and the religious man, since this is the real point) doesn't just believe in belief, but they believe that they believe. He tries to make a second-order belief do the work of a first-order belief.

15 more annotations...

• There was a pause, as the one realized he had just made his hypothesis vulnerable to falsification

• I still find myself struggling for words to describe what I saw as this woman spoke. She spoke with... pride? Self-satisfaction? A deliberate flaunting of herself?
• That strange pride/satisfaction/flaunting clearly had something to do with her knowing that her beliefs were scientifically outrageous.
• She didn't recite her creation myth with the fanatical faith of someone who needs to reassure herself. She didn't act like she expected us, the audience, to be convinced - or like she needed our belief to validate her.
• Dennett, in addition to suggesting belief in belief, has also suggested that much of what is called "religious belief" should really be studied as "religious profession".
• Even if a sentence is essentially meaningless, you can still know when you are supposed to chant the response aloud.
• If you had to profess a religious belief to satisfy a priest, or satisfy a co-religionist - heck, to satisfy your own self-image as a religious person - you would have to pretend to believe much more convincingly than this woman was doing.
• by launching into a five-minute diatribe about the primordial cow, she was cheering for paganism, like holding up a banner at a football game. A banner saying "GO BLUES" isn't a statement of fact, or an attempt to persuade; it doesn't have to be convincing - it's a cheer.
• That strange flaunting pride... it was like she was marching naked in a gay pride parade.
• It wasn't just a cheer, like marching, but an outrageous cheer, like marching naked - believing that she couldn't be arrested or criticized, because she was doing it for her pride parade.
• That's why it mattered to her that what she was saying was beyond ridiculous. If she'd tried to make it sound more plausible, it would have been like putting on clothes.

8 more annotations...

• I have so far distinguished between belief as anticipation-controller, belief in belief, professing and cheering.  Of these, we might call anticipation-controlling beliefs "proper beliefs" and the other forms "improper belief".  A proper belief can be wrong or irrational, e.g., someone who genuinely anticipates that prayer will cure her sick baby, but the other forms are arguably "not belief at all".
• Yet another form of improper belief is belief as group-identification - as a way of belonging.  Robin Hanson uses the excellent metaphor of wearing unusual clothing, a group uniform like a priest's vestments or a Jewish skullcap, and so I will call this "belief as attire".
• You cannot say the phrases "heroic self-sacrifice" and "suicide bomber" in the same sentence, even for the sake of accurately describing how the Enemy sees the world.   The very concept of the courage and altruism of a suicide bomber is Enemy attire - you can tell, because the Enemy talks about it.  The cowardice and sociopathy of a suicide bomber is American attire.  There are no quote marks you can use to talk about how the Enemy sees the world; it would be like dressing up as a Nazi for Halloween.
• Belief-as-attire may help explain how people can be passionate about improper beliefs.  Mere belief in belief, or religious professing, would have some trouble creating genuine, deep, powerful emotional effects.
• People who've stopped anticipating-as-if their religion is true, will go to great lengths to convince themselves they are passionate, and this desperation can be mistaken for passion.
• Identifying with a tribe is a very strong emotional force.  People will die for it.  And once you get people to identify with a tribe, the beliefs which are attire of that tribe will be spoken with the full passion of belonging to that tribe.

4 more annotations...

• Will bond yields go up, or down, or remain the same? If you're a TV pundit and your job is to explain the outcome after the fact, then there's no reason to worry. No matter which of the three possibilities comes true, you'll be able to explain why the outcome perfectly fits your pet market theory
• But it rapidly becomes clear that plausibility can't help you here - all three events are plausible. Fittability to your pet market theory doesn't tell you how to divide your time.
• Maybe anticipation should be treated as a conserved resource, like money.

1 more annotation...

• Outside their own professions, people often commit the misstep of trying to broaden a word as widely as possible, to cover as much territory as possible.  Is it not more glorious, more wise, more impressive, to talk about all the apples in the world?  How much loftier it must be to explain human thought in general, without being distracted by smaller questions, such as how humans invent techniques for solving a Rubik's Cube.  Indeed, it scarcely seems necessary to consider specific questions at all; isn't a general theory a worthy enough accomplishment on its own?
• You might think it poetic, to give one word many meanings, and thereby spread shades of connotation all around. But even poets, if they are good poets, must learn to see the world precisely. It is not enough to compare love to a flower. Hot jealous unconsummated love is not the same as the love of a couple married for decades. If you need a flower to symbolize jealous love, you must go into the garden, and look, and make subtle distinctions - find a flower with a heady scent, and a bright color, and thorns. Even if your intent is to shade meanings and cast connotations, you must keep precise track of exactly which meanings you shade and connote.
• A fully connected graph, with an edge between every two vertices, conveys the same amount of information as a graph with no edges at all.  The important graphs are the ones where some things are not connected to some other things.
• When the unenlightened ones try to be profound, they draw endless verbal comparisons between this topic, and that topic, which is like this, which is like that; until their graph is fully connected and also totally useless. The remedy is specific knowledge and in-depth study. When you understand things in detail, you can see how they are not alike, and start enthusiastically subtracting edges off your graph.
• Good hypotheses can only explain some possible outcomes, and not others.
• But rationalists - and also poets - need narrow words to express precise thoughts; they need categories which include only some things, and exclude others. There's nothing wrong with focusing your mind, narrowing your categories, excluding possibilities, and sharpening your propositions.

4 more annotations...

• Alas, belief is easier than disbelief; we believe instinctively, but disbelief requires a conscious effort.
• So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened.  And I knew how embarrassing this was. 
• Your strength as a rationalist is your ability to be more confused by fiction than by reality.  If you are equally good at explaining any outcome, you have zero knowledge.
• I should have paid more attention to that sensation of still feels a little forced. It's one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. 
• "EITHER YOUR MODEL IS FALSE OR THIS STORY IS WRONG."

3 more annotations...

• When we see evidence, hypotheses that assigned a higher likelihood to that evidence, gain probability at the expense of hypotheses that assigned a lower likelihood to the evidence.  This is a phenomenon of relative likelihoods and relative probabilities.  You can assign a high likelihood to the evidence and still lose probability mass to some other hypothesis, if that other hypothesis assigns a likelihood that is even higher.
• But the likelihood is still higher that the absence of a Fifth Column would perform an absence of sabotage.
• Whatever the likelihood that a Fifth Column would do no sabotage, the probability P(E|H1), it cannot be as large as the likelihood that no Fifth Column does no sabotage, the probability P(E|H2).  So observing a lack of sabotage increases the probability that no Fifth Column exists.
• Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge.  The strength of a model is not what it can explain, but what it can't, for only prohibitions constrain anticipation.  If you don't notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.

  • #Important

2 more annotations...

• Spee acted as confessor to many witches; he was thus in a position to observe every branch of the accusation tree, that no matter what the accused witch said or did, it was held a proof against her. 
• scientists write down their experimental predictions in advance
• The expectation of the posterior probability, after viewing the evidence, must equal the prior probability
• Therefore, for every expectation of evidence, there is an equal and opposite expectation of counterevidence.
• the mere expectation of encountering evidence - before you've actually seen it - should not shift your prior beliefs
• If you claim that "a good and proper life" is evidence that a woman is a witch, then an evil and improper life must be evidence that she is not a witch.  If you argue that God, to test humanity's faith, refuses to reveal His existence, then the miracles described in the Bible must argue against the existence of God.
• For a true Bayesian, it is impossible to seek evidence that confirms a theory.
• You can only ever seek evidence to test a theory, not to confirm it.
• This realization can take quite a load off your mind.  You need not worry about how to interpret every possible experimental result to confirm your theory.  You needn't bother planning how to make any given iota of evidence confirm your theory, because you know that for every expectation of evidence, there is an equal and oppositive expectation of counterevidence.  If you try to weaken the counterevidence of a possible "abnormal" observation, you can only do it by weakening the support of a "normal" observation, to a precisely equal and opposite degree.  It is a zero-sum game.  No matter how you connive, no matter how you argue, no matter how you strategize, you can't possibly expect the resulting game plan to shift your beliefs (on average) in a particular direction.
• Getting the right shape of the probability distribution of future belief is the whole skill in testing a hypothesis.
• you don't realize that Bayes's Theorem is precisely what describes the normative update in beliefs over time? That this is the whole point of Bayes's Theorem?
• Before black swans were observed, no one expected to encounter a black swan, and everyone expected to encounter another white swan on occasion. A black swan is huge evidence against, a white swan is tiny additional evidence for. Had they been normative, the two quantities would have balanced exactly.
• The word "normative" has stood in the way of my understanding what you mean
• The heuristics and biases people use it to mean "correct," because social scientists aren't allowed to use that word. I think there's a valuable lesson about academics, institutions, or taboos in there, but I'm not sure what it is. As far as I can tell, they are the only people that use it this way.

12 more annotations...

• Four experimental groups were respectively told that these four outcomes were the historical outcome.  The fifth, control group was not told any historical outcome.  In every case, a group told an outcome assigned substantially higher probability to that outcome, than did any other group or the control group.
• Hindsight bias matters in legal cases, where a judge or jury must determine whether a defendant was legally negligent in failing to foresee a hazard (Sanchiro 2003).
• In an experiment based on an actual legal case, Kamin and Rachlinski (1995) asked two groups to estimate the probability of flood damage caused by blockage of a city-owned drawbridge. The control group was told only the background information known to the city when it decided not to hire a bridge watcher. The experimental group was given this information, plus the fact that a flood had actually occurred. Instructions stated the city was negligent if the foreseeable probability of flooding was greater than 10%. 76% of the control group concluded the flood was so unlikely that no precautions were necessary; 57% of the experimental group concluded the flood was so likely that failure to take precautions was legally negligent. A third experimental group was told the outcome andalso explicitly instructed to avoid hindsight bias, which made no difference: 56% concluded the city was legally negligent.
• But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning sign which seemed as severe as the O-ring problem, without benefit of hindsight.  It could have been done, but it would have required a general policy much more expensive than just fixing the O-Rings.
• Because we don't see the cost of a general policy, we learn overly specific lessons.
• We don't learn the general lesson: the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.

4 more annotations...

• Do your thought processes at this point, where you really don't know the answer, feel different from the thought processes you used to rationalize either side of the "known" answer?
• Hindsight will lead us to systematically undervalue the surprisingness of scientific findings, especially the discoveries we understand - the ones that seem real to us, the ones we can retrofit into our models of the world. 
• This unfairly devalues the contribution of the researchers; and worse, will prevent you from noticing when you are seeing evidence that doesn't fit what you really would have expected.
• We need to make a conscious effort to be shocked
• These examples emphasize the benefit of frequently taking calibration tests, where we assign probabilities to answers and then checks those answer for calibration errors. Perhaps someone could create a website where we could do this regularly? Just collect a large list of questions like the ones above, questions with true answers but where we have intuitions about what the answer might be, and then have us answer those questions with probabilities, and then show us a calibration chart for the last X questions.
• Economic facts are facts about large numbers of people interacting, sometimes very indirectly, and so are a kind of fact which people have a hard time seeing, since they only encounter small numbers of other people at any given time, so they do not see the whole.

4 more annotations...

• Once upon a time, there was an instructor who taught physics students.  One day she called them into her class, and showed them a wide, square plate of metal, next to a hot radiator.  The students each put their hand on the plate, and found the side next to the radiator cool, and the distant side warm.  And the instructor said, Why do you think this happens?  Some students guessed convection of air currents, and others guessed strange metals in the plate.  They devised many creative explanations, none stooping so low as to say "I don't know" or "This seems impossible."
• If you say "heat conduction", what experience does that lead you to anticipate?  Under normal circumstances, it leads you to anticipate that, if you put your hand on the side of the plate near the radiator, that side will feel warmer than the opposite side.  If "because of heat conduction" can also explain the radiator-adjacent side feeling cooler, then it can explain pretty much anything.
• It is anticipation-isomorphic to saying "magic".  It feels like an explanation, but it's not.
• The deeper error of the students is not simply that they failed to constrain anticipation.  Their deeper error is that they thought they were doing physics.

2 more annotations...

• When I was older, and I began to read the Feynman Lectures on Physics, I ran across a gem called "the wave equation".  I could follow the equation's derivation, but, looking back, I couldn't see its truth at a glance.  So I thought about the wave equation for three days, on and off, until I saw that it was embarrassingly obvious.  And when I finally understood, I realized that the whole time I had accepted the honest assurance of physicists that light was waves, sound was waves, matter was waves, I had not had the vaguest idea of what the word "wave" meant to a physicist.
• There is an instinctive tendency to think that if a physicist says "light is made of waves", and the teacher says "What is light made of?", and the student says "Waves!", the student has made a true statement.  That's only fair, right?
• If I hear the syllables "bea-ver" and think of a large rodent, that is a fact about my own state of mind, not a fact about the syllables "bea-ver".
• And this leads into an even worse habit.  Suppose the teacher presents you with a confusing problem involving a metal plate next to a radiator; the far side feels warmer than the side next to the radiator.  The teacher asks "Why?"  If you say "I don't know", you have no chance of getting a gold star - it won't even count as class participation.  But, during the current semester, this teacher has used the phrases "because of heat convection", "because of heat conduction", and "because of radiant heat".  One of these is probably what the teacher wants.  You say, "Eh, maybe because of heat conduction?"
• This is not a hypothesis about the metal plate.  This is not even a proper belief.  It is an attempt to guess the teacher's password.
• If you aren't using the diffusion equation - putting in numbers and getting out results that control your anticipation of particular experiences - then the connection between map and territory is severed as though by a knife.  What remains is not a belief, but a verbal behavior.
• What if there is no teacher to tell you that you failed? 
• This is how human beings think, unless they are trained not to fall into the trap.  Humanity stayed stuck in holes like this for thousands of years.

6 more annotations...

• I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts.  If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection.  It would be worse than finding rabbit fossils in the pre-Cambrian.  If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.
• If the scientific in-crowd instead used the phrase "because of intelligent design", they would just as cheerfully use that instead - it would make no difference to their anticipation-controllers.  Saying "because of evolution" instead of "because of intelligent design" does not, for them, prohibit Storm.  Its only purpose, for them, is to identify with a tribe.
• They aren't speaking from within a model of cognition.  They don't realize they need a model.  They don't realize that science is about models. 
• Is there anything in science that you are proud of believing, and yet you do not use the belief professionally?  You had best ask yourself which future experiences your belief prohibits from happening to you.  That is the sum of what you have assimilated and made a true part of yourself.  Anything else is probably passwords or attire.

2 more annotations...

• At this point, some people say, "God!"
• What could possibly make anyone, even a highly religious person, think this even helped answer the paradox of the First Cause?  Why wouldn't you automatically ask, "Where did God come from?"  Saying "God is uncaused" or "God created Himself" leaves us in exactly the same position as "Time began with the Big Bang."  We just ask why the whole metasystem exists in the first place, or why some events but not others are allowed to be uncaused.
• Saying "God!" is a way of belonging to a tribe, which gives people a motive to say it as often as possible
• saying "God!" doesn't help.  It doesn't make the paradox seem any less paradoxical even if true.  How could anyone not notice this?
• Jonathan Wallace suggested that "God!" functions as a semantic stopsign - that it isn't a propositional assertion, so much as a cognitive traffic signal: do not think past this point.
• I know someone whose answer to every one of these questions is "Liberal democracy!"  That's it.  That's his answer.  If you ask the obvious question of "How well have liberal democracies performed, historically, on problems this tricky?" or "What if liberal democracy does something stupid?" then you're an autocrat, or libertopian, or otherwise a very very bad person.  No one is allowed to question democracy.
• I once called this kind of thinking "the divine right of democracy".  But it is more precise to say that "Democracy!" functioned for him as a semantic stopsign.  If anyone had said to him "Turn it over to the Coca-Cola corporation!", he would have asked the obvious next questions:  "Why?  What will the Coca-Cola corporation do about it?  Why should we trust them?  Have they done well in the past on equally tricky problems?"
• What distinguishes a semantic stopsign is failure to consider the obvious next question.

6 more annotations...

• Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem.  Imagine pointing to a market crash and saying "It's not a quark!"  Does that feel like an explanation?  No?  Then neither should saying "It's an emergent phenomenon!"
• It's the noun "emergence" that I protest, rather than the verb "emerges from".  There's nothing wrong with saying "X emerges from Y", where Y is some specific, detailed model with internal moving parts. 
• Now suppose I should say that gravity is explained by "arisence" or that chemistry is an "arising phenomenon", and claim that as my explanation.
• "Emergence" is commonly used as an explanation in its own right.
• What do you know, after you have said that intelligence is "emergent"?  You can make no new predictions.  You do not know anything about the behavior of real-world minds that you did not know before.  It feels like you believe a new fact, but you don't anticipate any different outcomes. Your curiosity feels sated, but it has not been fed.  The hypothesis has no moving parts - there's no detailed internal model to manipulate.  Those who proffer the hypothesis of "emergence" confess their ignorance of the internals, and take pride in it; they contrast the science of "emergence" to other sciences merely mundane.
• "Emergence" is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Emergence is popular because it is the junk food of curiosity. You can explain anything using emergence, and so people do just that; for it feels so wonderful to explain things. Humans are still humans, even if they've taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed up in the literary genre of "science" but still the same species psychology.
• It might be more useful to describe things in reverse " X are the components of phenomenon Y". Such as "Neurons firing are the known components of intelligence". Because when we observe something, it can be useful to ask "what are its components"?

  • Comments
• It is, rather, usually an attempt to delineate a subject matter which can be further investigated.
• I believe that the hope is that a general theory of emergence is possible
• That we can vastly simplify and idealize something without losing all that much predictive power seems to be a characteristic of many so-called emergent phenomena.
• In line with previous comments, I'd always understood the idea of emergence to have real content: "systems whose high-level behaviors arise or 'emerge' from the interaction of many low-level elements" as opposed to being centrally determined or consciously designed (basically "bottom-up" rather than "top-down"). It's not a specific explanation in and of itself, but it does characterise a class of explanations, and, more importantly, excludes certain other types of explanation.
• After all these posts on how the strength of an idea is what it excludes, forbids, prohibits, people are still citing positive examples as proof of the power of emergence? Tell me what it isn't!
• If X is not "emergent", then: a) X does not have a cellular automata-like model; there are no readily identifiable components of X that follow relatively simple, computable rules which generate the observed behavior of the system. (The game of tennis doesn't look like an emergent property of any of its obvious components, unless you decide that tennis emerges from human brains.) OR b) X does not have a high-level model that describes the overall behavior of X without modeling the behavior of each individual component of X separately. (A list of telephone numbers is not emergent.)
• In contrast, a lot of traffic jams 'just happen.' The previous sentence is false, strictly speaking; the jams come from somewhere. But you can't point to an individual cause of them; they arise from the local effects of millions of local actions taken by individual drivers. Removing any one of these actions wouldn't eliminate the jam; it's a cumulative product of all of them. So people searching for an explanation of why it takes two hours to dive ten miles in rush hour get really frustrated, because there's no good explanation to give them. And people trying to fix rush hour get even more frustrated, because there's no good angle to attack the problem from.
• "emergence" is useful as a shorthand for "the causes are complicated and interconnected, and I can't pick one out and tell you, 'here it is, this is why that happened.'" It's important not to get confused, and not to think an explanation of why we don't understand something is the same as an explanation of that thing. But as long as you remember that, it's a useful thing to remember.
• Emergence theory can be helpful in breaking down complex systems to gain a better understanding of how they work. If the behavior of a system is dominated by a decentralized and internal control mechanism, then it is emergent. If the behavior is dominated by a centralized or an external control mechanism then it is non-emergent.
• I do wish that Eliezer would stop pretending that the word "emergence" has to have an explanatory function in order to be useful. It has a classificatory function. It collects together a class of models which have in common the property that naive reductionists fail to understand them. It is classification, not explanation.

15 more annotations...

• correct complexity is only possible when every step is pinned down overwhelmingly.
• "Do not attempt long chains of reasoning or complicated plans."
• Even after I invented my "answer", the phenomenon was still a mystery unto me, and possessed the same quality of wondrous impenetrability that it had at the start.
• the young Eliezer was careful to ensure that his Mysterious Answer made a bold prediction of future experience
• But my hypothesis made no retrospective predictions.  According to Traditional Science, retrospective predictions don't count - so why bother making them?  To a Bayesian, on the other hand, if a hypothesis does not today have a favorable likelihood ratio over "I don't know", it raises the question of why you today believe anything more complicated than "I don't know".

  • Avenues for future research?
• Traditional Rationality still wasn't enough to get it right.  It just led me into different mistakes than the ones it had explicitly forbidden.
• it sheds light on the question of why people who call themselves "rationalists" do not rule the world.  You need one whole hell of a lot of rationality before it does anything but lead you into new and interesting mistakes. 
• Traditional Rationality is a walk, not a dance.  It's designed to get you to the truth eventually, and gives you all too much time to smell the flowers along the way.
• Traditional Rationalists can agree to disagree.  Traditional Rationality doesn't have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence.  In Traditional Rationality, you're allowed to guess, and then test your guess.  But experience has taught me that if you don't know, and you guess, you'll end up being wrong.

  • Crucial.
• I think it should be possible to start with Bayes, but I've never seen it done. Lessons on Traditional Rationality appeal to built-in human intuitions, like "Reality is either a certain way or it's not", so you'd appeal to the same intuitions but use them to introduce probability principles like "Your probabilities shouldn't sum to more than 1.0."

8 more annotations...

• The substance of a democracy is the specific mechanism that resolves policy conflicts.  If all groups had the same preferred policies, there would be no need for democracy - we would automatically cooperate.  The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an AI, but it has to be something.  What does it mean to call for a "democratic" solution if you don't have a conflict-resolution mechanism in mind?
• I think it means that you have said the word "democracy", so the audience is supposed to cheer.  It's not so much a propositional statement, as the equivalent of the "Applause" light that tells a studio audience when to clap.
• There are plenty of legitimate reasons for uttering a sentence that would be uninformative in isolation.  "We need to balance the risks and opportunities of AI" can introduce a discussion topic; it can emphasize the importance of a specific proposal for balancing; it can criticize an unbalanced proposal.
• if no specifics follow, the sentence is probably an applause light.
• Maybe audiences respond to speeches by summing the emotion vectors of each word in the speech, rather than parsing sentences.
• The reward of a roaring applause is of course enough to bias a speaker to start pouring on more applause lights.
• I think that for many people the "substance" of democracy is not the specific mechanism, but rather the general mechanism, and the nature of the output. The mechanism must include at least some formal representation of every member. The details of this don't matter so much: it might be direct voting (strictly equal power), or it might be a representative system (so long as the reps for each voter are more or less equal in power). And the general nature of the output is that it should be fair. Exactly what fair is, is a good question, and probably varies a lot. But at least this: conflicts should not always be resolved in favor of the same person or group or class.
• Dictators can whip people's muscles into obedience, maybe even their lower-end skills ("do this arithmetic or DIE!"), but when you want to engage the creativity of a gang of PhDs, it is not going to work until there is a mechanism for their dissent to be heard and addressed.
• It is almost universally felt that when we call a country democratic we are praising it: consequently the defenders of every kind of regime claim that it is a democracy, and fear that they might have to stop using that word if it were tied down to any one meaning.
• If you simply issue a call for "democracy", why, no one can disagree with that - it would be like disagreeing with a call for apple pie. As soon as you propose a specific mechanism of democracy, whether it is Congress passing a law, or an AI polling people by phone, or government funding of a large research project whose final authority belongs to an appointed committee of eminent scientists, et cetera, people can disagree with that, because they can actually visualize the probable consequences.
• So there is a tremendous motive to avoid criticism, to keep to the safely vague areas where people will applaud you, and not to make the concrete proposals where people might - gasp! - disagree.
• I think that the real value in these discussions comes from people willing to make concrete proposals and expose themselves to criticism.
• I get the impression the speaker is implying that someone they strongly disagree with does believe this obvious fact is not true, or that this ridiculous notion is.
• I think this is also the reason why the speaker's supporters applaud to statements like that - it implies the issue is very simple and clear-cut, only one side (ours) is remotely sensible, and you'd have to be insane to disagree. One-sidedness feels good. Very blatant one-sidedness feels even better.

12 more annotations...