Matt McAlister's List: Less Wrong 2 - A Human's Guide to Words

• A word fails to connect to reality in the first place.
• Your argument, if it worked, could coerce reality to go a different way by choosing a different word definition.
• You try to establish any sort of empirical proposition as being true "by definition".
• You unconsciously slap the conventional label on something, without actually using the verbal definition you just gave.
• The act of labeling something with a word, disguises a challengable inductive inference you are making.
• You try to define a word using words, in turn defined with ever-more-abstract words, without being able to point to an example.
• The extension doesn't match the intension.
• Your verbal definition doesn't capture more than a tiny fraction of the category's shared characteristics, but you try to reason as if it does.
• You try to treat category membership as all-or-nothing, ignoring the existence of more and less typical subclusters.
• A verbal definition works well enough in practice to point out the intended cluster of similar things, but you nitpick exceptions.
• You ask whether something "is" or "is not" a category member but can't name the question you really want answered.
• You treat intuitively perceived hierarchical categories like the only correct way to parse the world, without realizing that other forms of statistical inference are possible even though your brain doesn't use them.
• You argue about a category membership even after screening off all questions that could possibly depend on a category-based inference.
• You talk about categories as if they are manna fallen from the Platonic Realm, rather than inferences implemented in a real brain.
• if your brain's categorizing neural network contains a (metaphorical) central unit corresponding to the inference of blegg-ness, it may still feel like there's a leftover question.
• You allow an argument to slide into being about definitions, even though it isn't what you originally wanted to argue about.
• You think a word has a meaning, as a property of the word itself; rather than there being a label that your brain associates to a particular concept.
• You argue over the meanings of a word, even after all sides understand perfectly well what the other sides are trying to say.
• When you each understand what is in the other's mind, you are done.
• You pull out a dictionary in the middle of an empirical or moral argument.
• You pull out a dictionary in the middle of any argument ever.
• You defy common usage without a reason, making it gratuitously hard for others to understand you. 
• You use complex renamings to create the illusion of inference.
• You get into arguments that you could avoid if you just didn't use the word.
• When a word poses a problem, the simplest solution is to eliminate the word and its synonyms.
• The existence of a neat little word prevents you from seeing the details of the thing you're trying to think about.
• You have only one word, but there are two or more different things-in-reality, so that all the facts about them get dumped into a single undifferentiated mental bucket.
• You see patterns where none exist, harvesting other characteristics from your definitions even when there is no similarity along that dimension.
• You try to sneak in the connotations of a word, by arguing from a definition that doesn't include the connotations.
• You claim "X, by definition, is a Y!"  On such occasions you're almost certainly trying to sneak in a connotation of Y that wasn't in your given definition.
• You claim "Ps, by definition, are Qs!"
• You try to establish membership in an empirical cluster "by definition".
• Your definition draws a boundary around things that don't really belong together.

31 more annotations...

• "How?!" cried the jester in horror, as he was dragged away.  "It's logically impossible!"

   

  "It is entirely possible," replied the king.  "I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one."

• The fundamental problem with arguing that things are true "by definition" is that you can't make reality go a different way by choosing a different definition.
• mere guessing.  It wouldn't be, y'know, absolutely and eternally certain.  The Greek philosophers - like most prescientific philosophers - were rather fond of certainty.
• Mortality is one of several properties that are individually necessary, and together sufficient, to determine membership in the class "human".  The statement "All humans are mortal" is a logically valid truth, absolutely unquestionable.
• if Socrates is human, he must be mortal: it is a logical deduction, as certain as certain can be.
• But then we can never know for certain that Socrates is a "human" until after Socrates has been observed to be mortal.
• If mortality is defined to mean finite lifespan, then you can never really know if someone was human, until you've observed to the end of eternity - just to make sure they don't come back.
• It's also logically valid within neighboring Everett branches in which, due to a slightly different evolved biochemistry, hemlock is a delicious treat rather than a poison.  And it's logically valid even in universes where Socrates never existed, or for that matter, where humans never existed.

  • Thus it doesn't tell us what we need to know about the Hemlock.
• The Bayesian definition of evidence favoring a hypothesis is evidence which we are more likely to see if the hypothesis is true than if it is false. 
• the syllogism will be logically valid whether that proposition is true or false
• logic can only tell us that which, in some sense, we already know
• Syllogisms are valid in all possible worlds, and therefore, observing their validity never tells us anything about which possible world we actually live in.
• we can view logical reasoning as resolving our uncertainty about impossible possible worlds - eliminating probability mass in logically impossible worlds which we did not know to be logically impossible.  In this sense, logical argument can be treated as observation.
• But when you talk about an empirical prediction like "Socrates is going to keel over and stop breathing" or "Socrates is going to do fifty jumping jacks and then compete in the Olympics next year", that is a matter of possible worlds, not impossible possible worlds.
• Logic can tell us which hypotheses match up to which observations, and it can tell us what these hypotheses predict for the future - it can bring old observations and previous guesses to bear on a new problem.  But logic never flatly says, "Socrates will stop breathing now."  Logic never dictates any empirical question; it never settles any real-world query which could, by any stretch of the imagination, go either way.
• This is an area where AIs will have an advantage over us, because they should be able to reason about objects directly without having to compress them in to a definition first.

13 more annotations...

• So I say "blue", with a dutiful patina of humility.  For I am a sophisticated rationalist-type person, and I keep track of my assumptions and dependencies - I guess, but I'm aware that I'm guessing... right?
• But when a large yellow striped feline-shaped object leaps out at me from the shadows, I think, "Yikes!  A tiger!"  Not, "Hm... objects with the properties of largeness, yellowness, stripedness, and feline shape, have previously often possessed the properties 'hungry' and 'dangerous', and thus, although it is not logically necessary, it may be an empirically good guess that aaauuughhhh CRUNCH CRUNCH GULP."

• It is a common misconception that you can define a word any way you like.

   

  This would be true if the brain treated words as purely logical constructs, Aristotelian classes, and you never took out any more information than you put in.
• if mortality is part of your logical definition of "human", you can't logically classify Socrates as human until you observe him to be mortal.
• Aristotle's brain placed Socrates in the "human" category as efficiently as your own brain categorizes tigers, apples, and everything else in its environment:  Swiftly, silently, and without conscious approval.
• Aristotle laid down rules under which no one could conclude Socrates was "human" until after he died.  Nonetheless, Aristotle and his students went on concluding that living people were humans and therefore mortal; they saw distinguishing properties such as human faces and human bodies, and their brains made the leap to inferred properties such as mortality.
• Misunderstanding the working of your own mind does not, thankfully, prevent the mind from doing its work.  Otherwise Aristotelians would have starved, unable to conclude that an object was edible merely because it looked and felt like a banana.
• So the Aristotelians went on classifying environmental objects on the basis of partial information, the way people had always done.  Students of Aristotelian logic went on thinking exactly the same way, but they had acquired an erroneous picture of what they were doing.

  • #Important
• The Aristotelians were still the same people, they retained their original natures, but they had acquired incorrect beliefs about their own functioning.  They looked into the mirror of self-awareness, and saw something unlike their true selves: they reflected incorrectly.
• Your brain doesn't treat words as logical definitions with no empirical consequences, and so neither should you.  The mere act of creating a word can cause your mind to allocate a category, and thereby trigger unconscious inferences of similarity.  Or block inferences of similarity; if I create two labels I can get your mind to allocate two categories.  Notice how I said "you" and "your brain" as if they were different things?
• Making errors about the inside of your head doesn't change what's there; otherwise Aristotle would have died when he concluded that the brain was an organ for cooling the blood.

  • #Nice
• If you believe that you can "define a word any way you like", without realizing that your brain goes on categorizing without your conscious oversight, then you won't take the effort to choose your definitions wisely.
• What alternative model would you propose? I'm not quite ready yet to stop using words that imperfectly place objects into categories. I'll keep the fact that categories are imperfect in mind.

  • [Comments]
• The tiger, on the other hand, is a committed Platonist.

• Our tendency to unconsciously draw inferences through inductive thought is a real problem.

  The issue of word definitions is just a red herring.
• Innumerable beings who made inferences in a way different from ours perished; for all that, their ways might have been truer, Those, for example, who did not know how to find often enough what is "equal" as regards both nourishment and hostile animals– those, in other words, who subsumed things too slowly and cautiously– were favored with a lesser probability of survival than those who guessed immediately upon encountering similar instances that they must be equal. [ . . . ] The course of logical ideas and inferences in our brain today corresponds to a process and a struggle among impulses that are, taken singly, very illogical and unjust. We generally experience only the result of this struggle because this primeval mechanism now runs its course so quickly and is so well concealed. (The Gay Science, Section 111)

14 more annotations...

• To give an "intensional definition" is to define a word or phrase in terms of other words, as a dictionary does.  To give an "extensional definition" is to point to examples, as adults do when teaching children.
• In Hollywood Rationality and popular culture generally, "rationalists" are depicted as word-obsessed, floating in endless verbal space disconnected from reality.
• Peirce isn't actually showing you a piece of lithium.  He didn't have pieces of lithium stapled to his book.  Rather he's giving you a treasure map - an intensionally defined procedure which, when executed, will lead you to an extensional example of lithium.  This is not the same as just tossing you a hunk of lithium, but it's not the same as saying "atomic weight 7" either.
• The actual intension of my "tiger" concept would be the neural pattern (in my temporal cortex) that inspects an incoming signal from the visual cortex to determine whether or not it is a tiger.
• The actual extension of my "tiger" concept is everything I call a tiger.
• Similarly, if I say "dangerous yellow-black striped animal", without pointing to anything, the listener may visualize giant hornets.
• Intensional definitions don't capture entire intensions; extensional definitions don't capture entire extensions. 
• You can't capture in words all the details of the cognitive concept - as it exists in your mind - that lets you recognize things as tigers or nontigers.  It's too large.  And you can't point to all the tigers you've ever seen, let alone everything you would call a tiger.
• The strongest definitions use a crossfire of intensional and extensional communication to nail down a concept.  Even so, you only communicate maps to concepts, or instructions for building concepts - you don't communicate the actual categories as they exist in your mind or in the world.
• except in mathematics, definitions are usually treasure maps, not treasure
• But most of the brain's work of applying intensions happens sub-deliberately.
• Just as you can't usually convey a concept's whole intension in words because it's a big complicated neural membership test, you can't control the concept's entire intension because it's applied sub-deliberately.  This is why arguing that XYZ is true "by definition" is so popular.  If definition changes behaved like the empirical nullops they're supposed to be, no one would bother arguing them.  But abuse definitions just a little, and they turn into magic wands - in arguments, of course; not in reality.
• I have noticed the use of the term 'technoscience' or 'technoscientific' in rhetoric concerning science and technology. The use of this term is striking given the pretty obvious differences between science and technology as domains and activities in the real world. One must be making a very imprecise point for it to apply equally well to both science and technology in one breath. Use of this term might be nothing more than a symptom of this imprecision, but can also be thought of as stipulating an unsupported conclusion in itself. That is, anyone meeting the argument on its terms implicitly agrees that technology and science are identical for purposes of reasoning about them.

11 more annotations...

• We make complex tools to make complex tools, we use syntactical combinatorial language, we harness critical fission reactions as a source of energy: these things may serve out to single out only humans, but not all humans - many of us have never built a fission reactor.
• No dictionary, no encyclopedia, has ever listed all the things that humans have in common.
• With the right set of necessary-and-sufficient gene sequences you could single out all humans, and only humans - at least for now - but it would still be far from all that humans have in common.
  
• The initial clue only has to lead the user to the similarity cluster - the group of things that have many characteristics in common.  After that, the initial clue has served its purpose, and I can go on to convey the new information "humans are currently mortal", or whatever else I want to say about us
• A dictionary is best thought of, not as a book of Aristotelian class definitions, but a book of hints for matching verbal labels to similarity clusters, or matching labels to properties that are useful in distinguishing similarity clusters.

3 more annotations...

• "typicality effects" or "prototype effects"
• if you ask subjects to press a button to indicate "true" or "false" in response to statements like "A robin is a bird" or "A penguin is a bird", reaction times are faster for more central examples.
• which of these statements strikes you as more natural:  "98 is approximately 100", or "100 is approximately 98"?  If you're like most people, the first statement seems to make more sense.  (Sadock 1977.)  For similar reasons, people asked to rate how similar Mexico is to the United States, gave consistently higher ratings than people asked to rate how similar the United States is to Mexico.  (Tversky and Gati 1978.)
• A simpler interpretation is that people are using the (demonstrated) similarity heuristic as a proxy for the probability that a disease spreads, and this heuristic is (demonstrably) asymmetrical.
• So once again we see that Aristotle's notion of categories - logical classes with membership determined by a collection of properties that are individually strictly necessary, and together strictly sufficient - is not a good model of human cognitive psychology.
• When surveying people who aren't disease biologists, it doesn't matter if there are specific one-way paths in the cutting edge research, what matters are the processes that inform the decisions. In the absence of any biological information, there's no indication to tilt the scales one way or another.
• we're dealing with people who call everything even incidentally related to the subject 'evidence' and suggest that it should change the outcome of a Bayesian evaluation.

5 more annotations...

• This is the benefit of viewing robins as points in space:  You couldn't see the linear lineup as easily if you were just imagining the robins as cute little wing-flapping creatures.
• If you think that's extravagant, quantum physicists use an infinite-dimensional configuration space, and a single point in that space describes the location of every particle in the universe.  So we're actually being comparatively conservative in our visualization of thingspace - a point in thingspace describes just one object, not the entire universe.
• "Radial categories" are how cognitive psychologists describe the non-Aristotelian boundaries of words.  The central "mother" conceives her child, gives birth to it, and supports it. Is an egg donor who never sees her child a mother?  She is the "genetic mother".  What about a woman who is implanted with a foreign embryo and bears it to term?  She is a "surrogate mother".  And the woman who raises a child that isn't hers genetically?  Why, she's an "adoptive mother".  The Aristotelian syllogism would run, "Humans have ten fingers, Fred has nine fingers, therefore Fred is not a human" but the way we actually think is "Humans have ten fingers, Fred is a human, therefore Fred is a 'nine-fingered human'."
• We can think about the radial-ness of categories in intensional terms, as described above - properties that are usually present, but optionally absent.  If we thought about the intension of the word "mother", it might be like a distributed glow in thingspace, a glow whose intensity matches the degree to which that volume of thingspace matches the category "mother".  The glow is concentrated in the center of genetics and birth and child-raising; the volume of egg donors would also glow, but less brightly.
• we can think about the radial-ness of categories extensionally
• Then the center of all birdness would be densely populated by many neighboring tight clusters, robins and sparrows and canaries and pigeons and many other species.  Eagles and falcons and other large predatory birds would occupy a nearby cluster.  Penguins would be in a more distant cluster, and likewise chickens and ostriches.
• The result might look, indeed, something like an astronomical cluster: many galaxies orbiting the center, and a few outliers.
• Or we could think simultaneously about both the intension of the cognitive category "bird", and its extension in real-world birds:  The central clusters of robins and sparrows glowing brightly with highly typical birdness; satellite clusters of ostriches and penguins glowing more dimly with atypical birdness, and Abraham Lincoln a few megaparsecs away and glowing not at all.
• I prefer that last visualization - the glowing points - because as I see it, the structure of the cognitive intension followed from the extensional cluster structure.  First came the structure-in-the-world, the empirical distribution of birds over thingspace; then, by observing it, we formed a category whose intensional glow roughly overlays this structure.
• A natural cluster, a group of things highly similar to each other, may have no set of necessary and sufficient properties - no set of characteristics that all group members have, and no non-members have.
• Definitions can't be expected to exactly match the empirical structure of thingspace in any event, because the map is smaller and much less complicated than the territory.  The point of the definition "feathered flying things" is to lead the listener to the bird cluster, not to give a total description of every existing bird down to the molecular level.
• When you draw a boundary around a group of extensional points empirically clustered in thingspace, you may find at least one exception to every simple intensional rule you can invent.
• But if a definition works well enough in practice to point out the intended empirical cluster, objecting to it may justly be called "nitpicking".
• The universe is not actually quantized in pieces that have isolated effects on each other. However, causes and effects are "things", they are points in Thingspace and as "things" they actually represent aggregates, bunches of variable values that when recognized as a whole have, by definition, unique cause-effect relationships with other "things".
• words are centers of (indefinitely extending) clusters rather than boundaries of sets.

13 more annotations...

• Why are you calling this object a "blegg"?  A "blegg" was originally defined as blue and egg-shaped - the qualification of blueness appears in the very name "blegg", in fact.  This object is not blue.  One of the necessary qualifications is missing; you should call this a "purple egg-shaped object", not a "blegg".
• You've got to expect a few color defects.
• "Incidentally," Susan says casually, "it may interest you to know that bleggs contain small nuggets of vanadium ore, and rubes contain shreds of palladium, both of which are useful industrially."
• Surface characteristics, like blue color and furredness, do not determine whether an object is a blegg; surface characteristics only matter because they help you infer whether an object is a blegg, that is, whether the object contains vanadium.
• Thus, if you want to guess whether the object glows like a blegg, or remains dark like a rube, you should guess that it glows like a blegg.
• if there are any unknown characteristics of the object you need to infer, you'll infer them as if the object were a blegg, not a rube - group it into the similarity cluster of blue egg-shaped furred flexible opaque things, and not the similarity cluster of red cube-shaped smooth hard translucent things.

• The question "Is this object a blegg?" may stand in for different queries on different occasions.

   

  If it weren't standing in for some query, you'd have no reason to care.
• How could a dictionary possibly decide whether an empirical cluster of atheists is really substantially different from an empirical cluster of theologians?  How can reality vary with the meaning of a word?  The points in thingspace don't move around when we redraw a boundary.
• people often don't realize that their argument about where to draw a definitional boundary, is really a dispute over whether to infer a characteristic shared by most things inside an empirical cluster...

• Quote: My favorite example is, Is a fetus a person?

    

  I can answer this one: A foetus is not a person prior to 20 weeks gestation (18 weeks of pregnancy), but may be a person from that point onwards.

    

  A body with one mind is one person. A body with two minds is two people (conjoint twins). A body with three minds would be three people. A heart transplant does not switch a person into a different body. A lung transplant does not switch a person into a different body. A brain transplant (and therefore a mind transplant) would switch a person into a different body. It is minds, not bodies, that defines people.

    

  The mind exists, if at all, in the brain, or more specifically the cerebral cortex. The cerebral cortex begins to develop connections no earlier than 20 weeks gestation, therefore there is not a person before this time (though the body does have reflexes).
• Humans also have obsessions with (a) how far your cluster is from mine
• (b) given one empirical cluster, how can I pick a characteristic, however minor, which will allow me to split it into 'us vs them'
• sharp distinctions regarding fuzzy concepts are meaningless
• I think another great example of a disguised query is the recurring debate, "Is this art?" People have really widely varying definitions of "art" (e.g., some people's definition includes "aesthetically interesting," other people's definition merely requires "conceptually interesting") -- and in one sense, once both parties explain how they use the word "art," the debate should resolve pretty quickly.
• But of course, since it's a disguised query, the question "Is this art?" should really be followed up with the question "Why does it matter?" As far as I can tell, the disguised query in this case is usually "does this deserve to be taken seriously?" which can be translated in practice into, "Is this the sort of thing that deserves to be exhibited in a gallery?" And that's certainly a real, non-semantic debate. But we can have that debate without ever needing to decide whether to apply the label "art" to something -- in fact, I think the debate would be much clearer if we left the word "art" out of it altogether.

13 more annotations...

• And suppose I'm fairly naive about ANNs:  I've read excited popular science books about how neural networks are distributed, emergent, and parallel just like the human brain!!
• 
• A neural network needs a learning rule.  The obvious idea is that when two nodes are often active at the same time, we should strengthen the connection between them - this is one of the first rules ever proposed for training a neural network, known as Hebb's Rule.
• And lo, Network 1 exhibits this behavior even though there's no explicit node that says whether the object is a blegg or not.  The judgment is implicit in the whole network!!  Bleggness is an attractor!! which arises as the result of emergent behavior!! from the distributed!! learning rule.
• And there are other problems, like double-counting the evidence when messages bounce back and forth:  If you suspect that an object glows in the dark, your suspicion will activate belief that the object contains vanadium, which in turn will activate belief that the object glows in the dark.
• 
• In this network, a wave of activation converges on the central node from any clamped (observed) nodes, and then surges back out again to any unclamped (unobserved) nodes.  Which means we can compute the answer in one step, rather than waiting for the network to settle - an important requirement in biology when the neurons only run at 20Hz.  And the network architecture scales as O(N), rather than O(N²).
• in Network 1, there is no unit that represents blegg-ness; blegg-ness emerges as an attractor in the distributed network.
• Network 1 is not more useful on most real-world problems, where you rarely find an animal stuck halfway between being a cat and a dog.
• Neither Network 1, nor Network 2, are biologically realistic.  But it still seems like a fair guess that however the brain really works, it is in some sense closer to Network 2 than Network 1.  Fast, cheap, scalable, works well to distinguish dogs and cats: natural selection goes for that sort of thing like water running down a fitness landscape.

• It seems like an ordinary enough task to classify objects as either bleggs or rubes, tossing them into the appropriate bin.  But would you notice if sea-blue objects never glowed in the dark?

   

  Maybe, if someone presented you with twenty objects that were alike only in being sea-blue, and then switched off the light, and none of the objects glowed.  If you got hit over the head with it, in other words. 
• It wouldn't be easy or intuitive to notice, the way that distinguishing cats and dogs is easy and intuitive.
• So the brain decides, once and for all, that Socrates is human; and from there, infers that Socrates is mortal like all other humans thus yet observed.  It doesn't seem easy or intuitive to ask how much wearing clothes, as opposed to using language, is associated with mortality.  Just, "things that wear clothes and use language are human" and "humans are mortal".
• Are there biases associated with trying to classify things into categories once and for all?  Of course there are. 
• Emergence isn't, on its own, bad. It's just generally unhelpful. That doesn't mean it's wrong; my consciousness is in fact emergent from neurons firing. It just doesn't tell me anything extra. If instead, you said something like, "I've got a detailed model of the brain, that simulates the neurons, and through (insert technical explanation here) my model shows consciousness emerging as a property made up of smaller elements", nobody would complain. Emergence is only a bad explanation if it's the only explanation given.
• So in that sense, the naive bayes network, with the implicit "but is it ACTUALLY a blegg?" style question would seem to reflect something legitimate, not just a name for the cluster. Specifically, such a question would translate to something like "which of, say, two distinct physical processes produced this?"

14 more annotations...

• A key idea of the heuristics and biases program is that mistakes are often more revealing of cognition than correct answers.
• In one case, the question "Is it a blegg?" stands in for the disguised query, "Which bin does it go in?".  In the other case, the question "Is it a blegg?" stands in for the disguised query, "Will it glow in the dark?"
• Network 1's structure does have one major advantage over Network 2:  Every unit in the network corresponds to a testable query.  If you observe every observable, clamping every value, there are no units in the network left over
• Network 2, however, is a far better candidate for being something vaguely like how the human brain works:  It's fast, cheap, scalable - and has an extra dangling unit in the center, whose activation can still vary, even after we've observed every single one of the surrounding nodes.
• Which is to say that even after you know whether an object is blue or red, egg or cube, furred or smooth, bright or dark, and whether it contains vanadium or palladium, it feels like there's a leftover, unanswered question:  But is it really a blegg?
• Usually, in our daily experience, acoustic vibrations and auditory experience go together.  But a tree falling in a deserted forest unbundles this common association.  And even after you know that the falling tree creates acoustic vibrations but not auditory experience, it feels like there's a leftover question:  Did it make a sound?
• It takes a deliberate effort to visualize your brain from the outside - and then you still don't see your actual brain; you imagine what you think is there, hopefully based on science, but regardless, you don't have any direct access to neural network structures from introspection.
• When you look at Network 2, you are seeing from the outside; but the way that neural network structure feels from the inside, if you yourself are a brain running that algorithm, is that even after you know every characteristic of the object, you still find yourself wondering:  "But is it a blegg, or not?"
• we don't instinctively see our intuitions as "intuitions", we just see them as the world
• when people argue over whether the falling tree makes a sound, or whether Pluto is a planet, they don't see themselves as arguing over whether a categorization should be active in their neural networks.  It seems like either the tree makes a sound, or not.
• If you were a mind constructed along the lines of Network 1, you wouldn't say "It depends on how you define 'planet'," you would just say, "Given that we know Pluto's orbit and shape and mass, there is no question left to ask."  Or, rather, that's how it would feel - it would feel like there was no question left - if you were a mind constructed along the lines of Network 1.
• Before you can question your intuitions, you have to realize that what your mind's eye is looking at is an intuition - some cognitive algorithm, as seen from the inside - rather than a direct perception of the Way Things Really Are.
• People cling to their intuitions, I think, not so much because they believe their cognitive algorithms are perfectly reliable, but because they can't see their intuitions as the way their cognitive algorithms happen to look from the inside.
• While "reifying the internal nodes" must indeed be counted as one of the great design flaws of the human brain, I think the recognition of this flaw and the attempt to fight it are as old as history.
• words are not the concepts they represent
• Naive Bayes assumes that within the blegg cluster, or within the rube cluster, all remaining variance in the characteristics is independent; or to put it another way, once we know whether an object is a blegg or a rube, this screens off any other information that its shape could tell us about its color. This isn't the same as assuming that the only causal influence on a blegg's shape is its blegg-ness - in fact, there may not be anything that corresponds to blegg-ness.
• Network 1, being totally connected, is slower than network 2. But network 2 has a node which corresponds to no observable thing. It can leave one with the feeling that some question has not been completely answered even though all the observables have known states.

15 more annotations...

• Or Barry could be a skeptic who, upon hearing Albert's argument, reflexively scrutinized it for possible logical flaws; and then, on finding a counterargument, automatically accepted it without applying a second layer of search for a counter-counterargument; thereby arguing himself into the opposite position.
• if Barry didn't have a differing intuition before, he sure has one now.
• Whenever you feel tempted to say the words "by definition" in an argument that is not literally about pure mathematics, remember that anything which is true "by definition" is true in all possible worlds, and so observing its truth can never constrain which world you live in.
• the shift usually passes unnoticed
• Dictionary editors are historians of usage, not legislators of language. Dictionary editors find words in current usage, then write down the words next to (a small part of) what people seem to mean by them.  If there's more than one usage, the editors write down more than one definition.
• I would just like to point out - in a mournful sort of way - that Albert and Barry seem to agree on virtually every question of what is actually going on inside the forest, and yet it doesn't seem to generate any feeling of agreement.
• "Personally I'd say that if the issue arises, both sides should switch to describing the event in unambiguous lower-level constituents, like acoustic vibrations or auditory experiences.  Or each side could designate a new word, like 'alberzle' and 'bargulum', to use for what they respectively used to call 'sound'; and then both sides could use the new words consistently.  That way neither side has to back down or lose face, but they can still communicate.  And of course you should try to keep track, at all times, of some testable proposition that the argument is actually about.  Does that sound right to you?"

• Albert:  "Oh, yeah:  If a tree falls in the forest, and no one hears it, does it make a sound?"

   

  Barry:  "It makes an alberzle but not a bargulum.  What's the next question?"
• This remedy doesn't destroy every dispute over categorizations.  But it destroys a substantial fraction.
• I am so sick of arguing with people who's definition of the issue constitutes 99% of their argument, and who aren't willing to acknowledge that their definition needs consensus before their point is even meaningful let alone valid.
• Equivocation can be a powerful teaching tool and rhetorical device, but more often it serves as a way to lie plausibly, both to others and to oneself.
• Another time you can't just generate new words is when a category boundary like "person" or "human" or "baby" makes a direct appearance in your utility function.
• You are the smartest person in the group and convince everyone that language is important. They agree to reform the language, but aren't capable of constructing or learning a new one, and aren't interested in teaching their children one. What simple reforms might be a good idea?
• It will no longer be correct to say that something is (a color or similar property). One must say it "seems" a color, as well as to whom. Not "Snow is white", rather, "Snow seems white to me".

12 more annotations...

• Telepathy, or "language", or whatever you want to call our partial thought transfer ability, is more complicated than it looks.

• But it would be quite inconvenient to go around thinking, "Now I shall partially transduce some features of my thoughts into a linear sequence of phonemes which will invoke similar thoughts in my conversational partner..."

   

  So the brain hides the complexity - or rather, never represents it in the first place - which leads people to think some peculiar thoughts about words.
• Considering this as a design constraint on the human cognitive architecture, you wouldn't want any extra steps between when your auditory cortex recognizes the syllables "tiger", and when the tiger concept gets activated.
• 
• visualize a direct connection running from the unit that recognizes the syllable "blegg", to the unit at the center of the blegg network.
• what that algorithm feels like from inside is that the label, and the concept, are very nearly identified; the meaning feels like an intrinsic property of the word itself.
• yet another case of E. T. Jaynes's "Mind Projection Fallacy".  It feels like a word has a meaning, as a property of the word itself; just like how redness is a property of a red apple
• you'll think, I wonder what "blegg" means, and not, I wonder what mental category Susan associates to the auditory label "blegg".
• on most occasions, the brain will not distinguish at all between the word and the meaning - only bothering to separate the two while learning a new language, perhaps.
• sound.meaning == auditory experiences
  forest.sound == false
• myBrain.FindConcept("sound") == concept_AuditoryExperience
  concept_AuditoryExperience.match(forest) == false
• humans have not evolved to know this, anymore than humans instinctively know the brain is made of neurons.

10 more annotations...

• Not all definitional disputes progress as far as recognizing the notion of common usage.  More often, I think, someone picks up a dictionary because they believe that words have meanings, and the dictionary faithfully records what this meaning is.
• Dictionary editors read what other people write, and record what the words seem to mean; they are historians.  The Oxford English Dictionary may be comprehensive, but never authoritative.
• the quiet cooperation on which all speech depends.
• In everyday life, shared language usually does not arise from a deliberate agreement, as of the IEEE.  It's more a matter of infection, as words are invented and diffuse through the culture.
• it is in our mutual interest that we use the same word for similar concepts - preferably, concepts similar to the limit of resolution in our brain's representation thereof - even though we have no obvious mutual interest in using any particular word for a concept
• But, while we have a mutual interest, it is not strictly necessary that you and I use the similar labels internally; it is only convenient.  If I know that, to you, "oto" means sound - that is, you associate "oto" to a concept very similar to the one I associate to "sound" - then I can say "Paper crumpling makes a crackling oto."  It requires extra thought, but I can do it if I want.
• Preferably, words we use frequently should be short
• Albert assumes this for the sole purpose of rhetorically accusing Barry of breaking the agreement, and endangering the public good. 
• A rationalist, with the discipline of hugging the query active, would notice that the conversation had gone rather far astray.
• If it's just a namespace conflict that has gotten blown out of proportion, and nothing more is at stake, then the two sides need merely generate two new words and use them consistently.
• When both sides know what the other side wants to say, and both sides accuse the other side of defecting from "common usage", then whatever it is they are about, it is clearly not working out a way to communicate with each other. 
• Yet often categorizations function as hidden inferences and disguised queries.
• If the question is how to cluster together similar things for purposes of inference, empirical predictions will depend on the answer; which means that definitions can be wrong.  A conflict of predictions cannot be settled by an opinion poll.
• If everyone believes that the red light in the sky is Mars the God of War, the dictionary will define "Mars" as the God of War.  If everyone believes that fire is the release of phlogiston, the dictionary will define "fire" as the release of phlogiston.
• As for the omnipotence-with-rocks argument, the solution is to recognize that the question is nonsense because the concepts it uses are self-contradictory. What happens when an immovable object meets an irresistible force?
• Can an omnipotent being create a black calico cat?
• When an irresistible force meets an immovable object, it goes through without leaving a hole.

15 more annotations...

• It's not just that I can vary the label, but that I can get along just fine without any label at all - the rules for Aristotelian classes work purely on structures like [A, C, D].  To call one of these structures "zawa", or attach any other label to it, is a human convenience (or inconvenience) which makes not the slightest difference to the Aristotelian rules.
• All [mortal, ~feathers, bipedal] are mortal.
  Socrates is a [mortal, ~feathers, bipedal].
  Therefore, Socrates is mortal.
• Here the illusion of inference comes from the labels, which conceal the premises, and pretend to novelty in the conclusion.  Replacing labels with definitions reveals the illusion, making visible the tautology's empirical unhelpfulness.  You can never say that Socrates is a [mortal, ~feathers, biped] until you have observed him to be mortal.
• the Aristotelian system works fine without any labels at all - it cranks out exactly the same stream of tautologies, they just look a lot less impressive.  The labels are only there to create the illusion of inference.
• "Definitions don't need words."

3 more annotations...

• Clearly, since one says "sound" and one says "~sound", we must have a contradiction, right?  But suppose that they both dereference their pointers before speaking
• Now there is no longer an apparent collision - all they had to do was prohibit themselves from using the word sound.
• You get a very different picture of what people agree or disagree about, depending on whether you take a label's-eye-view (Albert says "sound" and Barry says "not sound", so they must disagree) or taking the test's-eye-view (Albert's membership test is acoustic vibrations, Barry's is auditory experience).
• Get together a pack of soi-disant futurists and ask them if they believe we'll have Artificial Intelligence in thirty years, and I would guess that at least half of them will say yes.  If you leave it at that, they'll shake hands and congratulate themselves on their consensus.  But make the term "Artificial Intelligence" taboo, and ask them to describe what they expect to see, without ever using words like "computers" or "think", and you might find quite a conflict of expectations hiding under that featureless standard word.
• The illusion of unity across religions can be dispelled by making the term "God" taboo, and asking them to say what it is they believe in; or making the word "faith" taboo, and asking them why they believe it.  Though mostly they won't be able to answer at all, because it is mostly profession in the first place, and you cannot cognitively zoom in on an audio recording.

  • Brilliant.
• When you find yourself in philosophical difficulties, the first line of defense is not to define your problematic terms, but to see whether you can think without using those terms at all.  Or any of their short synonyms.  And be careful not to let yourself invent a new word to use instead.  Describe outward observables and interior mechanisms; don't use a single handle, whatever that handle may be.
• I would advise Albert and Barry to describe what it is that they think people do, or do not have, without using the phrase "free will" at all.  (If you want to try this at home, you should also avoid the words "choose", "act", "decide", "determined", "responsible", or any of their synonyms.)
• This is one of the nonstandard tools in my toolbox, and in my humble opinion, it works way way better than the standard one.
• The rationalist's game involves seeing at a lower level of detail. Not thinking up synonyms and keywords that weren't on the card.
• It has an obvious failure mode if you try to communicate something too difficult without requisite preliminaries, like calculus without algebra. Taboo isn't magic, it won't let you cross a gap of months in an hour.

8 more annotations...

• find a way of describing without the standard concept-handle
• What does it take to play the rationalist version of Taboo, in which the goal is not to find a synonym that isn't on the card, but to find a way of describing without the standard concept-handle?

• Is that a "bat"?  No, it's a long, round, tapering, wooden rod, narrowing at one end so that a human can grasp and swing it.

   

  Is that a "ball"?  No, it's a leather-covered spheroid with a symmetrical stitching pattern, hard but not metal-hard, which someone can grasp and throw, or strike with the wooden rod, or catch.
• You have to visualize.  You have to make your mind's eye see the details, as though looking for the first time.  You have to perform an Original Seeing.
• The chief obstacle to performing an original seeing is that your mind already has a nice neat summary, a nice little easy-to-use concept handle.  Like the word "baseball", or "bat", or "base".  It takes an effort to stop your mind from sliding down the familiar path, the easy path, the path of least resistance, where the small featureless word rushes in and obliterates the details you're trying to see.  A word itself can have the destructive force of cliche; a word itself can carry the poison of a cached thought.
• The chief obstacle to performing an original seeing is that your mind already has a nice neat summary, a nice little easy-to-use concept handle.  Like the word "baseball", or "bat", or "base".  It takes an effort to stop your mind from sliding down the familiar path, the easy path, the path of least resistance, where the small featureless word rushes in and obliterates the details you're trying to see.  A word itself can have the destructive force of cliche; a word itself can carry the poison of a cached thought.
• Playing the game of Taboo - being able to describe without using the standard pointer/label/handle - is one of the fundamental rationalist capacities.  It occupies the same primordial level as the habit of constantly asking "Why?" or "What does this belief make me anticipate?"
• Playing the game of Taboo - being able to describe without using the standard pointer/label/handle - is one of the fundamental rationalist capacities.  It occupies the same primordial level as the habit of constantly asking "Why?" or "What does this belief make me anticipate?"
• • Pragmatism, because seeing in this way often gives you a much closer connection to anticipated experience, rather than propositional belief;
   
  • Reductionism, because seeing in this way often forces you to drop down to a lower level of organization, look at the parts instead of your eye skipping over the whole;
   
  • Hugging the query, because words often distract you from the question you really want to ask;
   
  • Avoiding cached thoughts, which will rush in using standard words, so you can block them by tabooing standard words;
   
  • The writer's rule of "Show, don't tell!", which has power among rationalists;
   
  • And not losing sight of your original purpose.
• Hugging the query, because words often distract you from the question you really want to ask
• Why are you going to "school"?  To get an "education" ending in a "degree".  Blank out the forbidden words and all their obvious synonyms, visualize the actual details, and you're much more likely to notice that "school" currently seems to consist of sitting next to bored teenagers listening to material you already know, that a "degree" is a piece of paper with some writing on it, and that "education" is forgetting the material as soon as you're tested on it.
• Leaky generalizations often manifest through categorizations:  People who actually learn in classrooms are categorized as "getting an education", so "getting an education" must be good; but then anyone who actually shows up at a college will also match against the concept "getting an education", whether or not they learn.
• You want to learn, so you need an "education"; and then as long as you're getting anything that matches against the category "education", you may not notice whether you're learning or not.  Or you'll notice, but you won't realize you've lost sight of your original purpose, because you're "getting an education" and that's how you mentally described your goal.
• A mental category, that imperfectly matches your goal, can produce the same kind of incentive failure internally.  You want to learn, so you need an "education"; and then as long as you're getting anything that matches against the category "education", you may not notice whether you're learning or not.  Or you'll notice, but you won't realize you've lost sight of your original purpose, because you're "getting an education" and that's how you mentally described your goal.
• To categorize is to throw away information.
• You want to use categories to throw away irrelevant information, to sift gold from dust, but often the standard categorization ends up throwing out relevant information too.  And when you end up in that sort of mental trouble, the first and most obvious solution is to play Taboo.
• The rationalist version isn't a game, which means that you can't win by trying to be clever and stretching the rules.  You have to play Taboo with a voluntary handicap:  Stop yourself from using synonyms that aren't on the card.  You also have to stop yourself from inventing a new simple word or phrase that functions as an equivalent mental handle to the old one.  You are trying to zoom in on your map, not rename the cities; dereference the pointer, not allocate a new pointer; see the events as they happen, not rewrite the cliche in a different wording.
• The rationalist version isn't a game, which means that you can't win by trying to be clever and stretching the rules.  You have to play Taboo with a voluntary handicap:  Stop yourself from using synonyms that aren't on the card.  You also have to stop yourself from inventing a new simple word or phrase that functions as an equivalent mental handle to the old one.  You are trying to zoom in on your map, not rename the cities; dereference the pointer, not allocate a new pointer; see the events as they happen, not rewrite the cliche in a different wording.
• If you can look with fresh eyes, as though for the first time, you will see yourself doing things that you would never dream of doing if they were not habits.
• If you see your activities and situation originally, you will be able to originally see your goals as well.  If you can look with fresh eyes, as though for the first time, you will see yourself doing things that you would never dream of doing if they were not habits.
• Purpose is lost whenever the substance (learning, knowledge, health) is displaced by the symbol (a degree, a test score, medical care).  To heal a lost purpose, or a lossy categorization, you must do the reverse
• Replace the symbol with the substance; replace the signifier with the signified; replace the property with the membership test; replace the word with the meaning; replace the label with the concept; replace the summary with the details; replace the proxy question with the real question; dereference the pointer; drop into a lower level of organization; mentally simulate the process instead of naming it; zoom in on your map.

• Purpose is lost whenever the substance (learning, knowledge, health) is displaced by the symbol (a degree, a test score, medical care).  To heal a lost purpose, or a lossy categorization, you must do the reverse:

   

  Replace the symbol with the substance; replace the signifier with the signified; replace the property with the membership test; replace the word with the meaning; replace the label with the concept; replace the summary with the details; replace the proxy question with the real question; dereference the pointer; drop into a lower level of organization; mentally simulate the process instead of naming it; zoom in on your map.
• Bayes's Rule itself describes "evidence" in pure math, without using words like "implies", "means", "supports", "proves", or "justifies".  Set out to define such philosophical terms, and you'll just go in circles.
• Ask yourself whether the document fulfilled its purpose, not whether it followed the rules.
• Bayes's Rule itself describes "evidence" in pure math, without using words like "implies", "means", "supports", "proves", or "justifies".  Set out to define such philosophical terms, and you'll just go in circles.
• More relevantly in my own life, a degree is a tag attached to my name that changes the way a variety of real-world sorting algorithms (e.g., employers) evaluate it.

25 more annotations...

• I was told that so long as I prayed in Hebrew, I didn't need to know what the words meant, it would work anyway.
• As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. 
• It is now being suggested in several sources that an actual majority of published findings in medicine, though "statistically significant with p<0.05", are untrue.  But so long as p<0.05 remains the threshold for publication, why should anyone hold themselves to higher standards, when that requires bigger research grants for larger experimental groups, and decreases the likelihood of getting a publication?  Everyone knows that the whole point of science is to publish lots of papers, just as the whole point of a university is to print certain pieces of parchment, and the whole point of a school is to pass the mandatory tests that guarantee the annual budget.  You don't get to set the rules of the game, and if you try to play by different rules, you'll just lose.

  • Seriously?
• (Though for some reason, physics journals require a threshold of p<0.0001.  It's as if they conceive of some other purpose to their existence than publishing physics papers.)
• It's another matter when incentives must flow through large organizations - or worse, many different organizations and interest groups, some of them governmental.  Then you see behaviors that would mark literal insanity, if they were born from a single mind.
• Someone gets paid every time they open a car door, because that's what's measurable; and this person doesn't care whether the driver ever gets paid for arriving at the supermarket, let alone whether the buyer purchases the chocolate, or whether the eater is happy or starving.
• consider the No Child Left Behind Act.  The politicians want to look like they're doing something about educational difficulties; the politicians have to look busy to voters this year, not fifteen years later when the kids are looking for jobs.  The politicians are not the consumers of education.  The bureaucrats have to show progress, which means that they're only interested in progress that can be measured this year.  They aren't the ones who'll end up ignorant of science.  The publishers who commission textbooks, and the committees that purchase textbooks, don't sit in the classrooms bored out of their skulls.
• Politicians are too busy being re-elected to study all the data themselves
• The textbook publishers have no motive to produce bad textbooks, but they know that the textbook purchasing committee will be comparing textbooks based on how many different subjects they cover, and that the fourth-grade purchasing committee isn't coordinated with the third-grade purchasing committee, so they cram as many subjects into one textbook as possible.  Teachers won't get through a fourth of the textbook before the end of the year, and then the next year's teacher will start over.  Teachers might complain, but they aren't the decision-makers, and ultimately, it's not their future on the line, which puts sharp bounds on how much effort they'll spend on unpaid altruism...
• Want to see the problem really solved?  Make the politicians go to school.
• A single human mind can track a probabilistic expectation of utility as it flows through the conditional chances of a dozen intermediate events - including nonlocal dependencies, places where the expected utility of opening the car door depends on whether there's chocolate in the supermarket.  But organizations can only reward today what is measurable today, what can be written into legal contract today, and this means measuring intermediate events rather than their distant consequences.  These intermediate measures, in turn, are leaky generalizations - often very leaky.  Bureaucrats are untrustworthy genies, for they do not share the values of the wisher.
• I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult.
• The essential thing in the art of epistemic rationality is to understand how every rule is cutting through to the truth in the same movement.  The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see how every expected utility cuts through to utility.  You must thoroughly research this.
• "Your sword has no blade. It has only your intention. When that goes astray you have no weapon."
• People can handle goal-tracking for driving to the supermarket just fine, when it's all inside their own heads, and no genies or bureaucracies or philosophies are involved.  The trouble is that real civilization is immensely more complicated than this.  Dozens of organizations, and dozens of years, intervene between the child suffering in the classroom, and the new-minted college graduate not being very good at their job.  (But will the interviewer or manager notice, if the college graduate is good at looking busy?)  With every new link that intervenes between the action and its consequence, intention has one more chance to go astray.  With every intervening link, information is lost, incentive is lost.  And this bothers most people a lot less than it bothers me, or why were all my classmates willing to say prayers without knowing what they meant?  They didn't feel the same instinct to look-to-the-generator.
• Do you care about threats to your civilization?  The worst metathreat to complex civilization is its own complexity, for that complication leads to the loss of many purposes.
• I look back, and I see that more than anything, my life has been driven by an exceptionally strong abhorrence to lost purposes.  I hope it can be transformed to a learnable skill.

15 more annotations...

• The only life-size, atomically detailed, 100% accurate map of California is California.  But California has important regularities, such as the shape of its highways, that can be described using vastly less information - not to mention vastly less physical material - than it would take to describe every atom within the state borders.

  • #Good
• A paper map of California, at a scale of 10 kilometers to 1 centimeter (a million to one), doesn't have room to show the distinct position of two fallen leaves lying a centimeter apart on the sidewalk.  Even if the map tried to show the leaves, the leaves would appear as the same point on the map; or rather the map would need a feature size of 10 nanometers, which is a finer resolution than most book printers handle, not to mention human eyes.
• "The map is not the territory, but you can't fold up the territory and put it in your glove compartment."
• Reality is very large - just the part we can see is billions of lightyears across.  But your map of reality is written on a few pounds of neurons, folded up to fit inside your skull. 
• Inevitably, then, certain things that are distinct in reality, will be compressed into the same point on your map.
• But what this feels like from inside is not that you say, "Oh, look, I'm compressing two things into one point on my map."  What it feels like from inside is that there is just one thing, and you are seeing it.
• To realize that there are two distinct events, underlying one point on your map, is an essentially scientific challenge
• There is just one mental folder in the filing system, labeled "sound", and everything thought about "sound" drops into that one folder.
• the more dangerous fallacy of compression arises from having no idea whatsoever that two distinct entities even exist
• 'Tis easier to question one's facts than one's ontology.
• The map of reality contained in a human brain, unlike a paper map of California, can expand dynamically when we write down more detailed descriptions.  But what this feels like from inside is not so much zooming in on a map, as fissioning an indivisible atom - taking one thing (it felt like one thing) and splitting it into two or more things.
• We also label "conscious" the state of being awake, rather than asleep, in our daily cycle.  But they are all different concepts going under the same name, and the underlying phenomena are different scientific puzzles.
• The part of "consciousness" that people find most interesting is reflectivity, self-awareness, realizing that the person I see in the mirror is "me"; that and the hard problem of subjective experience as distinguished by Chalmers. 
• You can explain being awake without explaining reflectivity or subjectivity.
• Fallacies of compression also underlie the bait-and-switch technique in philosophy - you argue about "consciousness" under one definition (like the ability to think about thinking) and then apply the conclusions to "consciousness" under a different definition (like subjectivity).  Of course it may be that the two are the same thing, but if so, genuinely understanding this fact would require first a conceptual split and then a genius stroke of reunification.
• Where you see a single confusing thing, with protean and self-contradictory attributes, it is a good guess that your map is cramming too much into one point - you need to pry it apart and allocate some new buckets.  This is not like defining the single thing you see, but it does often follow from figuring out how to talk about the thing without using a single mental handle.

14 more annotations...

• It was Karl Landsteiner's discovery of this fact, and how to test for compatible blood types, that made it possible to transfuse blood without killing the patient.  (1930 Nobel Prize in Medicine.)
• The stereotypes must have arisen strictly from the mere existence of the labels.
• the chief difficulty of doing such an experiment would be finding a protocol that didn't mislead the subjects into thinking that, since the label was given you, it must be significant somehow
• mere labeling had power over all things, at least in the human imagination
• You can see this in terms of similarity clusters: once you draw a boundary around a group, the mind starts trying to harvest similarities from the group.  And unfortunately the human pattern-detectors seem to operate in such overdrive that we see patterns whether they're there or not; a weakly negative correlation can be mistaken for a strong positive one with a bit of selective memory.
• Or you can see this in terms of the boundless human ability to make stuff up out of thin air and believe it because no one can prove it's wrong.  As soon as you name the category, you can start making up stuff about it.  The named thing doesn't have to be perceptible; it doesn't have to exist; it doesn't even have to be coherent.
• Any way you look at it, drawing a boundary in thingspace is not a neutral act.  Maybe a more cleanly designed, more purely Bayesian AI could ponder an arbitrary class and not be influenced by it.  But you, a human, do not have that option.  Categories are not static things in the context of a human brain; as soon as you actually think of them, they exert force on your mind.  One more reason not to believe you can define a word any way you like.

5 more annotations...