Matt McAlister's List: Less Wrong 7 - Fun Theory

• They're just there to collect credit for the Deep Wisdom of asking the question.  It's enough to imply that the question is unanswerable, and therefore, we should all drop dead.
• The primary experimental result in hedonic psychology - the study of happiness - is that people don't know what makes them happy.
• When people are asked to imagine how happy or sad an event will make them, they anchor on the moment of first receiving the news, rather than realistically imagining the process of daily life years later.
• Consider what the Christians made of their Heaven, meant to be literally eternal.  Endless rest, the glorious presence of God, and occasionally - in the more clueless sort of sermon - golden streets and diamond buildings.  Is this eudaimonia?  It doesn't even seem very hedonic.
• Ideology is one major wellspring of flawed Utopias, containing things that the imaginer believes should be enjoyed, rather than things that would actually be enjoyable.
• Heaven is not configured as a nice place to live.  It is rather memetically optimized to be a nice place for an exhausted peasant to imagine.

• We need to be ready to correct for such biases - the world that is fun to live in, may not be the world that sounds good when spoken into our ears.

   

  And the second lesson is that there's nothing in the universe out of which to construct Fun Theory, except that which we want for ourselves or prefer to become.
• There's no outside force to scold us for making the wrong decision.
• The thought occurred to me that I didn't actually want to bloat up immediately into a superintelligence, or have my world transformed instantaneously and completely into something incomprehensible.  I'd prefer to have it happen gradually, with time to stop and smell the flowers along the way.
• We don't need to transform the universe into something we feel dutifully obligated to create, but isn't really much fun - in the same way that a Christian would feel dutifully obliged to enjoy heaven - or that some strange folk think that creating orgasmium is, logically, the rightest thing to do.
• It doesn't get any better than fun.

9 more annotations...

• So this is the ultimate end of the prophecy of technological progress - just staring at a screen that says "YOU WIN", forever.
• eternal laziness "sounds like good news" to your present self who still has to work.
• futuristic ideals phrased in terms of "getting rid of work" would be better reformulated as "removing low-quality work to make way for high-quality work".
• if we're looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
• for a long-run meaning of life, we need games that are fun to play and not just to win.
• If you work at creating a cancer cure for twenty years through your own efforts, learning new knowledge and new skill, making friends and allies - and then some alien superintelligence offers you a cancer cure on a silver platter for thirty bucks - then you shut up and take it.
• But "curing cancer" is a problem of the 3D-predicate sort: you want the no-cancer predicate to go from False in the present to True in the future.  The importance of this destination far outweighs the journey; you don't want to go there, you just want to be there.  There are many legitimate goals of this sort, but they are not suitable as long-run fun.  "Cure cancer!" is a worthwhile activity for us to pursue here and now, but it is not a plausible future goal of galactic civilizations.
• "Because it's human nature" is not an explicit justification for anything.  There is human nature, which is what we are; and there is humane nature, which is what, being human, we wish we were.
• since you never designed your own leg muscles, you are racing using strength that isn't yours.

7 more annotations...

• It's not a worthy use of a human-level intelligence.
• How large is Fun Space?
• Learning is fun, but it uses up fun: you can't have the same stroke of genius twice.  Insight is insight because it makes future problems less difficult, and "deep" because it applies to many such problems.
• And the smarter you are, the faster you learn - so the smarter you are, the less total fun you can have.
• An even stronger constraint is that if you do something many times, you ought to learn from the experience and get better - many problems of the same difficulty will have the same "learnable lessons" embedded in them, so that doing one consumes some of the fun of others.
• There are insights you can't have without prerequisite insights.
• there are insights you can't understand at all without having a brain of sufficient size and sufficient design.
• One suspects that in a better-designed existence, the eudaimonic rate of intelligence increase would be bounded below by the need to integrate the loot of your adventures - to incorporate new knowledge and new skills efficiently, without swamping your mind in a sea of disconnected memories and associations - to manipulate larger, more powerful concepts that generalize more of your accumulated life-knowledge at once.
• And one also suspects that part of the poignancy of transhuman existence will be having to move on from your current level - get smarter, leaving old challenges behind - before you've explored more than an infinitesimal fraction of the Fun Space for a mind of your level.  If, like me, you play through computer games trying to slay every single monster so you can collect every single experience point, this is as much tragedy as an improved existence could possibly need.
• how long would you have to stay human, to collect every gold coin in the dungeons of history?
• the dimension of how much novelty we can expect to find available to introduce into our fun.
• our descendants and future selves won't exhaust Fun Space to the point that there is literally nothing left to do but carve the 162,329th table leg.

10 more annotations...

• Is there enough fun in the universe, sufficiently accessible, for a transhuman to jog off the hedonic treadmill - improve their life continuously, at a sufficient rate to leap to an even higher hedonic level before they had a chance to get bored with the previous one?
• Robin, in "For Discount Rates", pointed out that an investment earning 2% annual interest for 12,000 years adds up to a googol (10^100) times as much wealth; therefore, "very distant future times are ridiculously easy to help via investment".
• "I know of no law limiting economic value per atom."
• Ultradense information does not give you ultradense economic value or ultradense fun unless you can also use that information in a way that consumes few resources.  Otherwise it's just More Fun Than You Can Handle.
• So I haven't lost hope.  But I haven't lost despair, either; that would be faith.
• If you can get by with increasing the size in bits of your mind at a linear rate, then you can last for quite a while.  Until the end of the universe, in many versions of cosmology.
• But if you have to grow exponentially, in order for your ever-larger mind and its ever-larger memories not to end up uncomfortably squashed into too small a brain - squashed down to a point, to the point of it being pointless - then a transhuman's life is measured in subjective eons at best, and more likely subjective millennia.  Though it would be a merry life indeed.
• I just want to live one more day, and tomorrow I will still want to live one more day.  The part about "wanting to live forever" is just an induction

• If no environmental improvement ever has much effect on subjective well-being, the way you are now, because you've got a more or less genetically set level of happiness that you drift back to, then...

   

  Well, my usual heuristic is to imagine messing with environments before I imagine messing with minds.

   

  But in this case?  Screw that.  That's just stupid.  Delete it without a qualm.

7 more annotations...

• Failing to escape the computer tiger would also have fewer long-term consequences than failing to escape a biological tiger - it would be less a part of the total story of your life - meaning you're also likely to be less emotionally involved.
• And there's the sense of touch that indicates the wind on your skin; and the proprioceptive sensors that respond to the position of your limbs; and the nerves that record the strain on your muscles.  There's a whole strip of sensorimotor cortex running along the top of your brain, that would be much more intensely involved in "real" running.
• Physics is physics, so it's not that one environment is less real than another.  But our brains are more adapted to interfacing with jungles than computer code.
• "There is no limit to the talent you can use in computer programming."  It's as close as human beings have ever come to playing with the raw stuff of creation - but our grasp on it is too distant from the jungle.  All our involvement is through letters on a computer screen.  I win, and I'm happy, but there's no wind on my face.
• If there were something around that was smart enough to rewrite human brains without breaking them - not a trivial amount of smartness - then it would be possible to expand the range of things that are sensually fun.
• our ancestors also got helpful reinforcement from their taste buds about what to eat - reinforcement which is now worse than useless, because of the marketing incentive to reverse-engineer tastiness using artificial substances. 
• But I do want to nudge people into adopting something of a questioning attitude toward the senses we have now, rather than assuming that the existing senses are The Way Things Have Been And Will Always Be.  A sex organ bears thousands of densely packed nerves for signal strength, but that signal - however strong - isn't as complicated as the sensations sent out by taste buds.  Is that really appropriate for one of the most interesting parts of human existence?  That even a novice chef can create a wider variety of taste sensations for your tongue, than - well, I'd better stop there.  But from a fun-theoretic standpoint, the existing setup is wildly unbalanced in a lot of ways.  It wasn't designed for the sake of eudaimonia.

5 more annotations...

• I don't know if I'd call agriculture a mistake.  But one should at least be aware of the downsides.
• In Guessing the Teacher's Password I talked about the (well-known) problem in which schools end up teaching verbal behavior rather than real knowledge.  In Truly Part of You I suggested one test for false knowledge:  Imagine deleting a fact from your mind, and ask if it would grow back.
• Borrowing someone else's knowledge really doesn't give you anything remotely like the same power level required to discover that knowledge for yourself.
• If he'd personally seen the planets reduced from gods to spheres in a telescope?
• We stand on the shoulders of giants, and in doing so, the power that we wield is far out of proportion to the power that we could generate for ourselves.  This is true even of our revolutionaries.  And yes, we couldn't begin to support this world if people could only use their own strength.  Even so, we are losing something.
• But if no one had been able to use nuclear weapons without, say, possessing the discipline of a scientist and the discipline of a politician - without personally knowing enough to construct an atomic bomb and make friends - the world might have been a slightly safer place.
• many of this world's problems, stem from our using powers that aren't appropriate to seventy-year-olds.
• As yet there's no human "rationalist" or "scientist", whatever they know about "how to think", who could actually build a rational AI - which shows you the limits of our self-understanding.
• we are surrounded by things outside ourselves and strengths outside our understanding; we need them to stay alive, or we buy them because it's easier that way.

7 more annotations...

• jurisprudence constante:  The legal system must above all be predictable, so that people can execute contracts or choose behaviors knowing the legal implications.
• Judges are not necessarily there to optimize, like an engineer.  The purpose of law is not to make the world perfect.  The law is there to provide a predictable environment in which people can optimize their own futures.
• "other things being equal, organisms will choose to reward themselves over being rewarded by caretaking organisms"
• Optimization is the ability to squeeze the future into regions high in your preference ordering.  Living by my own strength, means squeezing my own future - not perfectly, but still being able to grasp some of the relation between my actions and their consequences.  This is the strength of a human.
• We have little experience with being "caretaken" across any substantial gap in intelligence

• So, in considering the kind of "help" that can flow from relatively stronger agents to relatively weaker agents, we have two potential problems to track:

   
   
  1. Help so strong that it optimizes away the links between the desirable outcome and your own choices.
   
  2. Help that is believed to be so reliable, that it takes off the psychological pressure to use your own strength.
• e are allowed to have preferences over external states of affairs, not just psychological states.
• But if there's an AI that really does look over the alternatives before I do, and really does choose the outcome before I get a chance, then I'm really not steering my own future.  The future is no longer counterfactually dependent on my decisions."
• How is a poor Friendly AI supposed to help you, when your utility function is dependent, not just on the outcome, not just on the Friendly AI's actions, but dependent on differences of the exact algorithm the Friendly AI uses to arrive at the same decision?  Isn't your whole rationale of one-boxing on Newcomb's Problem that you only care about what works?"
• The desire not to be optimized too hard by an outside agent is one of the structurally nontrivial aspects of human morality.

8 more annotations...

• a rational agent shouldn't do worse by having more options.  The more available actions you have, the more powerful you become - that's how it should ought to work.
• the pain of losing something is between 2 and 2.5 times as worse as the joy of gaining it
• So - if you can only choose one dessert, you're likely to be happier choosing from a menu of two than a menu of fourteen.  In the first case, you eat one dessert and pass up one dessert; in the latter case, you eat one dessert and pass up thirteen desserts.  And we are more sensitive to loss than to gain.
• Not only that, but if the options have incommensurable attributes, then whatever option we select is likely to look worse because of the comparison.
• people who refused to eat a cookie - who were offered the cookie, and chose not to take it - did worse on subsequent tests of mental performance than either those who ate the cookie or those who were not offered any cookie.  You pay a price in mental energy for resisting temptation.
• If you're forced to choose between two unacceptable evils, you'll pay a price either way.

• Suppose your computer games, in addition to the long difficult path to your level's goal, also had little side-paths that you could use - directly in the game, as corridors - that would bypass all the enemies and take you straight to the goal, offering along the way all the items and experience that you could have gotten the hard way.  And this corridor is always visible, out of the corner of your eye.

   

  Even if you resolutely refused to take the easy path through the game, knowing that it would cheat you of the very experience that you paid money in order to buy - wouldn't that always-visible corridor, make the game that much less fun?  Knowing, for every alien you shot, and every decision you made, that there was always an easier path?
• If you're an agent operating from a much higher vantage point - high enough to see humans as flawed algorithms, so that it's not a matter of second-guessing but second-knowing - then is it benevolence to offer choices that will assuredly be made wrongly?

6 more annotations...

• Helion had leaned and said, "Son, once you go in there, the full powers and total command structures of the Rhadamanth Sophotech will be at your command.  You will be invested with godlike powers; but you will still have the passions and distempers of a merely human spirit.  There are two temptations which will threaten you.  First, you will be tempted to remove your human weaknesses by abrupt mental surgery.  The Invariants do this, and to a lesser degree, so do the White Manorials, abandoning humanity to escape from pain.  Second, you will be tempted to indulge your human weakness.  The Cacophiles do this, and to a lesser degree, so do the Black Manorials.  Our society will gladly feed every sin and vice and impulse you might have; and then stand by helplessly and watch as you destroy yourself; because the first law of the Golden Oecumene is that no peaceful activity is forbidden.  Free men may freely harm themselves, provided only that it is only themselves that they harm."
• Helion looked sardonic.  "'Mistake' is such a simple word.  An adult who suffers a moment of foolishness or anger, one rash moment, has time enough to delete or destroy his own free will, memory, or judgment.  No one is allowed to force a cure on him.  No one can restore his sanity against his will.  And so we all stand quietly by, with folded hands and cold eyes, and meekly watch good men annihilate themselves.  It is somewhat... quaint... to call such a horrifying disaster a 'mistake.'"
• Even if you are strong enough to always choose correctly - maybe you don't want to have to be so strong, always at every moment.
• "Offering people powers beyond their own is not always helping them."
• it is not so terrible a disrespect for free will, to live in a world in which people are free to shoot their feet off through their own strength - in the hope that by the time they're smart enough to do it under their own power, they're smart enough not to.
• The more dangerous and destructive the act, the more you require people to do it without external help.
• If it's really dangerous, you don't just require them to do their own engineering, but to do their own science.  A singleton might be justified in prohibiting standardized textbooks in certain fields, so that people have to do their own science - make their own discoveries, learn to rule out their own stupid hypotheses, and fight their own overconfidence.  Besides, everyone should experience the joy of major discovery at least once in their lifetime, and to do this properly, you may have to prevent spoilers from entering the public discourse.  So you're getting three social benefits at once, here.
• We would prefer (if attainable) a world in which people own their own mistakes and their own successes, and yet they are not given loaded handguns on a silver platter, nor do they perish through suicide by genie bottle.
• Once you imagine a world in which people can shoot off their own feet through their own strength, are you making that world incrementally better by offering incremental help along the way?
• It's one matter to prohibit people from using dangerous powers that they have grown enough to acquire naturally - to literally protect them from themselves.
• To descend from above and offer dangerous powers as an untimely gift, is another matter entirely.  That's why the title of this post is "Devil's Offers", not "Dangerous Choices".
• And to allow dangerous powers to be sold in a marketplace - or alternatively to prohibit them from being transferred from one mind to another - that is somewhere in between.
• a pernicious distortion where you end up believing things that are easy to market to other people
• But in real life rather than imagination, human choice is a fragile thing.  If the whole field of heuristics and biases teaches us anything, it surely teaches us that.

  • Crucial
• Do you think you have a superior epistemic vantage point here, or what?
• a benevolent superintelligence may not offer certain choices.

• If you feel safer driving than flying, because that way it's your own hands on the steering wheel, statistics be damned -

   

  - then maybe it isn't helping you, for a superintelligence to offer you the option of driving.
• You can build a spaceship if you work hard and study hard.  It would be a more dangerous world if your six-year-old son could do it in an hour using string and cardboard.

16 more annotations...

• if you create an AI and tell it to model the world around it, it may form models of people that are people themselves.  Not necessarily the same person, but people nonetheless.
• The model that most precisely predicts these facts, may well be a 'simulation' detailed enough to be a person in its own right.
• What if an AI creates millions, billions, trillions of alternative hypotheses, models that are actually people, who die when they are disproven?
• If you would balk at killing a million people with a nuclear weapon, you should balk at this.

• We need a nonperson predicate - a predicate that returns 1 for anything that is a person, and can return 0 or 1 for anything that is not a person.  This is a "nonperson predicate" because if it returns 0, then you know that something is definitely not a person.

   

  You can have more than one such predicate, and if any of them returns 0, you're ok.  It just had better never return 0 on anything that is a person, however many nonpeople it returns 1 on.
• With a good toolbox of nonperson predicates in hand, we could exclude all "model citizens" - all beliefs that are themselves people - from the set of hypotheses our Bayesian AI may invent to try to model its person-containing environment.

4 more annotations...

• Creating a true child is the only moral and metaethical problem I know that is even harder than the shape of a Friendly AI.  I would like to be able to create Friendly AI while worrying just about the Friendly AI problems, and not worrying whether I've created someone who will lead a life worth living.  Better by far to just create a Very Powerful Optimization Process, if at all possible.
• Depending on how you look at it, either no intelligence has 'free will', or anything that simulates alternative courses of action has 'free will'."
• But if you shift over to seeing the mindscape in terms of e.g. utility functions and optimization, the "free will" thing sounds needlessly complicated - you would only do it if you wanted a specifically human-shaped mind, perhaps for purposes of creating a child.
• It need merely weigh up the positive and negative utility of the consequences.  It can understand the concept of people who carry weights of sin and virtue as the result of the decisions they make, while not treating itself as a person in that sense.
• Such an AI could fully understand an abstract concept of moral responsibility or agonizing moral struggles, and even correctly predict decisions that "morally responsible", "free-willed" humans would make, while possessing no actual sense of moral responsibility itself and not undergoing any agonizing moral struggles; yet still outputting the right behavior.

  • Sometimes, I feel like this.
• they fall far short of the fine-grained understanding of mind required to build a knowably Friendly AI
• the mountains of philosophy are the foothills of AI.  Even if philosophers debate free will for ten times a hundred years, it's not surprising if the key insight is found by AI researchers inventing search trees, on their way to doing other things.
• They cannot conceive of something that works unlike a human - implying that they see only a featureless ghost in the machine, secretly simulated by their own brains.  They won't see the human algorithm as detailed machinery, as big complicated machinery, as overcomplicated machinery.
• the challenge of building a nonsentient AI seems to me much less worrisome than being able to come up with a nonperson predicate!
• I need to have an AI that can correctly predict the decisions that conscious humans make, without ever using a conscious model of them!  The first case is only a flying thing without flapping wings, but the second case is like modeling water without modeling wetness. 

8 more annotations...

• And suppose that, while these AIs did care for one another, and cared about themselves, and cared how they were treated in the eyes of society -

   

  - these trillions of people also cared, very strongly, about making giant cheesecakes.

• Would we be right to go on trying to seize the destiny of the galaxy - to make of it a place of peace, freedom, art, aesthetics, individuality, empathy, and other components of humane value?

   

  Or should we be content to have the galaxy be 0.1% eudaimonia and 99.9% cheesecake?
• I can tell you my advice on how to resolve this horrible moral dilemma:  Don't create trillions of new people that care about cheesecake.
• It's not just the problem of creating one damaged soul.  It's the problem of creating a really big citizen.  What if the superintelligence is multithreaded a trillion times, and every thread weighs as much in the moral calculus (we would conclude upon reflection) as a human being?  What if (we would conclude upon moral reflection) the superintelligence is a trillion times human size, and that's enough by itself to outweigh our species?

2 more annotations...

• The only desire the Culture could not satisfy from within itself was one common to both the descendants of its original human stock and the machines they had (at however great a remove) brought into being: the urge not to feel useless.  The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works; the secular evangelism of the Contact Section, not simply finding, cataloguing, investigating and analysing other, less advanced civilizations but - where the circumstances appeared to Contact to justify so doing - actually interfering (overtly or covertly) in the historical processes of those other cultures.
• Iain Banks is the one to beat
• The Culture has deficiencies in High Challenge and Complex Novelty.  There are incredibly complicated games, of course, but these are games - not things with enduring consequences, woven into the story of your life.  Life itself, in the Culture, is neither especially challenging nor especially novel; your future is not an unpredictable thing about which to be curious.
• Living By Your Own Strength is not a theme of the Culture.  If you want something, you ask a Mind how to get it; and they will helpfully provide it, rather than saying "No, you figure out how to do it yourself."  The people of the Culture have little use for personal formidability, nor for a wish to become stronger.
• there is nothing except our own preferences, out of which to construct the Future
• would you prefer that we stay on as the main characters in the story of intelligent life, with no higher beings above us?
• you could have the humans grow up (at some eudaimonic rate of intelligence increase), and then when new people are created, they might be created as powerful Minds to start with.  Or when you create new minds, they might have a different emotional makeup, which doesn't lead them to feel overshadowed if there are more powerful Minds above them.  But we, as we exist already created - we might prefer to stay on as the main characters, for now, if given a choice.
• So yes, I am concerned for those currently alive, because it is that concern - and not a casual attitude toward the welfare of sentient beings - which I wish to continue into the Future.
• It is too much to do on my own, and too much harm to do on my own - to amputate someone else's destiny, and steal their main character status.
• If you're going to have human-level minds with human emotional makeups, they shouldn't be competing on a level playing field with superintelligences.  Either keep the superintelligences off the local playing field, or design the human-level minds with a different emotional makeup.
• Preventing suffering cannot provide a lasting meaning to life.  What happens when you run out of victims?  If there's nothing more to life than eliminating suffering, you might as well eliminate life and be done.
• Having an external evangelism project, and being given cute little roles that any Mind could do better in a flash, so as to "supply meaning", isn't going to cut it.

10 more annotations...

• Dunbar himself did another regression and found that a community of 150 primates would have to spend 43% of its time on social grooming, which Dunbar interpreted as suggesting that 150 was an upper bound rather than an optimum, when groups were highly incentivized to stay together.
• LWA suggests that community satisfaction has two peaks, one at size ~7 for simple groups, and one at ~60 for complex groups; and that any community has to fraction, one way or another, by the time it approaches Dunbar's Number.
• Large amounts of the organizational structure of your life are beyond your ability to control, or even talk about with the controllers; directives that have major effects on you, may be handed down from a level you can't reach.

• If you live in a large country, you probably don't know your President or Prime Minister on a personal level, and may not even be able to get a few hours' chat; you live under laws and regulations that you didn't make, and you can't talk to the people who made them.

   

  This is a non-ancestral condition.  Even children, while they may live under the dictatorial rule of their parents, can at least personally meet and talk to their tyrants. You could expect this unnatural (that is, non-EEA) condition to create some amount of anomie.
• our brains simply haven't updated to their diminished power in a super-Dunbarian world.  We just go on debating politics, feverishly applying our valuable brain time to finding better ways to run the world, with just the same fervent intensity that would be appropriate if we were in a small tribe where we could persuade people to change things.
• Another problem with our oversized world is the illusion of increased competition.
• If you lived in a world the size of a hunter-gatherer band, then it would be easier to find something important at which to be the best - or do something that genuinely struck you as important, without becoming lost in a vast crowd of others with similar ideas.
• The eudaimonic size of a community as a function of the component minds' intelligence might be given by the degree to which those minds find it natural to specialize - the number of different professions that you can excel at, without having to invent professions just to excel at.
• When you look at it that way, then there must be much more than 50 natural specializations in the modern world - but still much less than six billion.  By the same logic as the original Dunbar's Number, if there are so many different professional specialties that no one person has heard of them all, then you won't know who to consult about any given topic.
• But if people keep getting smarter and learning more - expanding the number of relationships they can track, maintaining them more efficiently - and naturally specializing further as more knowledge is discovered and we become able to conceptualize more complex areas of study - and if the population growth rate stays under the rate of increase of Dunbar's Function - then eventually there could be a single community of sentients, and it really would be a single community.
• National or international politics are the equivalent of, say, the weather - something we experience, can't affect, but which we surely want to understand.

  • Comments

9 more annotations...

• I'm worried about the prospect of nonsentient romantic partners
• In a nutshell - sex/romance, as we know it now, is a primary dimension of multiplayer fun.  If you take that fun and redirect it to something that isn't socially entangled, if you turn sex into an exclusively single-player game, then you've just made life that much simpler - in the same way that eliminating boredom or sympathy or values over nonsubjective reality or individuals wanting to navigate their own futures, would tend to make life "simpler".  When I consider how easily human existence could collapse into sterile simplicity, if just a single major value were eliminated, I get very protective of the complexity of human existence.
• I ask it in all seriousness - is there any aspect of human existence as complicated as romance?  Think twice before you say, "Well, it doesn't seem all that complicated to me; now calculus, on the other hand, that's complicated."  We are congenitally biased to underestimate the complexity of things that involve human intelligence, because the complexity is obscured and simplified and swept under a rug.  Interpersonal relationships involve brains, still the most complicated damn things around.  And among interpersonal relationships, love is (at least potentially) more complex than being nice to your friends and kin, negotiating with your allies, or outsmarting your enemies.  Aspects of all three, really.  And that's not merely having a utility function over the other mind's state - thanks to sympathy, we get tangled up with that other mind.  Smile when the one smiles, wince when the one winces.
• the complexity of human relationships, directly reflects the incredible complexity of the interlocking selection pressures involved.  Males and females do need each other to reproduce, but there are huge conflicts of reproductive interest between the sexes.  I don't mean to go into Evolutionary Psychology 101 (Robert Wright's The Moral Animal is one popular book), but e.g. a woman must always invest nine months of work into a baby and usually much more to raise it, where a man might invest only a few minutes; but among humans significant paternal investments are quite common, yet a woman is always certain of maternity where a man is uncertain of paternity... which creates an incentive for the woman to surreptitiously seek out better genes...
• men are not optimized to make women happy, and women are not optimized to make men happy.  The vast majority of men are not what the vast majority of women would most prefer, or vice versa. 

• I find it all too easy to imagine a world in which men retreat to their optimized sweet sexy catgirls, and women retreat to their optimized darkly gentle catboys, and neither sex has anything to do with each other ever again.  Maybe men would take the east side of the galaxy and women would take the west side.  And the two new intelligent species, and their romantic sexbots, would go their separate ways from there.

   

  That strikes me as kind of sad.
• Our species does definitely have a problem.  If you've managed to find your perfect mate, then I am glad for you, but try to have some sympathy on the rest of your poor species - they aren't just incompetent.  Not all women and men are the same, no, not at all.  But if you drew two histograms of the desired frequencies of intercourse for both sexes, you'd see that the graphs don't match up, and it would be the same way on many other dimensions.  There can be lucky couples, and every person considered individually, probably has an individual soulmate out there somewhere... if you don't consider the competition.  Our species as a whole has a statistical sex problem!
• There should be superior alternatives.  I'm willing to bet that a few psychological nudges in both sexes - to behavior and/or desire - could solve 90% of the needlessly frustrating aspects of relationships for large sectors of the population, while still keeping the complexity and interest of loving someone who isn't tailored to your desires.

6 more annotations...

• Terrence Deacon's The Symbolic Species is the best book I've ever read on the evolution of intelligence.
• It's not just a question of increased computing capacity, like adding extra processors onto a cluster; it's a question of what kind of signals dominate, in the brain.
• Deacon makes a case for some Williams Syndrome symptoms coming from a frontal cortex that is relatively too large for a human, with the result that prefrontal signals - including certain social emotions - dominate more than they should.

• And Williams Syndrome is not frontal enlargement, remember; it's just frontal sparing in an otherwise shrunken brain, which increases the relative force of frontal signals...

   

  ...beyond the narrow parameters within which a human brain is adapted to work.

   

  I mention this because you might look at the history of human evolution, and think to yourself, "Hm... to get from a chimpanzee to a human... you enlarge the frontal cortex... so if we enlarge it even further..."

   

  The road to +Human is not that simple.
• But testing in an evolutionary paradigm only creates robustness over the domain tested.  Yes, sometimes you get robustness beyond that, because sometimes evolution finds simple solutions that prove to generalize -
• Psychosis, it turns out, is a general failure mode, "the fever of CNS illnesses"; it can also be caused by drugs, brain tumors, or just sleep deprivation.
• That's how robust the human brain is: a gentle little neurotransmitter imbalance - so subtle they're still having trouble tracking it down after who knows how many fMRI studies - can give you a full-blown case of stark raving mad.
• The brain really is adapted to a very narrow range of operating parameters.  It doesn't tolerate a little too much dopamine
• Start modifying the pieces in ways that seem like "good ideas" - making the frontal cortex larger, for example - and you start operating outside the ancestral box of parameter ranges.  And then everything goes to hell.  Why shouldn't it?  Why would the brain be designed for easy upgradability?
• Poetry aside, a human being isn't the seed of a god.  We don't have neat little dials that you can easily tweak to more "advanced" settings.  We are not designed for our parts to be upgraded.  Our parts are adapted to work exactly as they are, in their current context, every part tested in a regime of the other parts being the way they are.  Idiot evolution does not look ahead, it does not design with the intent of different future uses.  We are not designed to unfold into something bigger.
• the AI could upgrade one thing while simultaneously tweaking the five things that depend on it and the twenty things that depend on them.  Finding a gradual, incremental path to greater intelligence (so as not to effectively erase you and replace you with someone else) that didn't drive you psychotic or give you Williams Syndrome or a hundred other syndromes.
• trying to raise your intelligence without breaking your brain or mutating your will
• Death is far from the worst risk here.  Not every form of madness will appear immediately when you branch yourself for testing - some insanities might incubate for a while before they became visible.  And you might not notice if your goals shifted only a bit at a time, as your emotional balance altered with the strange new harmonies of your brain.
• The point of today's post is that growing up - or even deciding what you want to be when you grow up - is as around as hard as designing a new intelligent species.  Harder, since you're constrained to start from the base of an existing design.  There is no natural path laid out to godhood, no Level attribute that you can neatly increment and watch everything else fall into place.  It is an adult problem.
• the serious, smart, technically minded transhumanists, do not always expect that the road to everything they want is easy
• So you'll forgive me if I am somewhat annoyed with people who run around saying, "I'd like to be a hundred times as smart!" as if it were as simple as scaling up a hundred times instead of requiring a whole new cognitive architecture; and as if a change of that magnitude in one shot wouldn't amount to erasure and replacement.  Or asking, "Hey, why not just augment humans instead of building AI?" as if it wouldn't be a desperate race against madness.

14 more annotations...

• I think that the difficulty and danger of fiddling with emotions is oft-underestimated.
• I'm sorry, but there are some things that are much more complicated to actually do than to rattle off as short English phrases, and "changing sex" has to rank very high on that list.  Even if you're omnipotent so far as raw ability goes, it's not like people have a binary attribute reading "M" or "F" that can be flipped as a primitive action.
• Want to spend a week wearing a female body?  Even at this very shallow level, we're dealing with drastic remappings of at least some segments of the sensorimotor cortex and cerebellum - the somatic map, the motor map, the motor reflexes, and the motor skills.  As a male, you know how to operate a male body, but not a female one.

• What would it take for a man to actually become the female version of themselves?

   

  Well... what does that sentence even mean?  I am reminded of someone who replied to the statement "Obama would not have become President if he hadn't been black" by saying "If Obama hadn't been black, he wouldn't have been Obama" i.e. "There is no non-black Obama who could fail to become President".  (You know you're in trouble when non-actual possible worlds start having political implications.)
• The person you would have been if you'd been born with an X chromosome in place of your Y chromosome (or vice versa) isn't you. 
• If I fell asleep and woke up as a true woman - not in body, but in brain - I don't think I'd call her "me".  The change is too sharp, if it happens all at once.
• Transform the brain gradually?  Hm... now we have to design the intermediate stages, and make sure the intermediate stages make self-consistent sense.  Evolution built and optimized a self-consistent male brain and a self-consistent female brain; it didn't design the parts to be stable during an intermediate transition between the two.  Maybe you've got to redesign other parts of the brain just to keep working through the transition.
• What happens when, as a woman, you think back to your memory of looking at Angelina Jolie photos as a man?  How do you empathize with your past self of the opposite sex?  Do you flee in horror from the person you were?  Are all your life's memories distant and alien things?  How can you remember, when your memory is a recorded activation pattern for neural circuits that no longer exist in their old forms?  Do we rewrite all your memories, too?
• we have to rewrite you as a more complex brain with a novel, non-ancestral architecture that can cross-operate in realtime between male and female modes, so that a female can process male memories with a remembered context that includes the male brainware that laid them down.
• In one sense, a sex change is admittedly close to a worst-case scenario: a fixed target not optimized for an easy transition from your present location; involving, not just new brain areas, but massive coordinated changes to brain areas already in place.

• It seems to me that there is just an irreducible residue of very hard problems associated with an adult version of humankind ever coming into being.

   

  And emotions would be among the most dangerous targets of meddling.  Make the wrong shift, and you won't want to change back.

9 more annotations...

• "Stories are about people's pain."  Orson Scott Card.

   

  "Every scene must end in disaster."  Jack Bickham.
• You simply don't optimize a story the way you optimize a real life.  The best story and the best life will be produced by different criteria.
• One answer might be that what a story really needs is not "disaster", or "pain", or even "conflict", but simply striving.  That the problem with Mary Sue stories is that there's not enough striving in them, but they wouldn't actually need pain.  This might, perhaps, be tested.
• I once read that in the BDSM community, "intense sensation" is a euphemism for pain.  Upon reading this, it occurred to me that, the way humans are constructed now, it is just easier to produce pain than pleasure.  Though I speak here somewhat outside my experience, I expect that it takes a highly talented and experienced sexual artist working for hours to produce a good feeling as intense as the pain of one strong kick in the testicles - which is doable in seconds by a novice.
• There is no ordinary way to make a human being feel as good as those instruments would make you hurt.
• The experimental rule is that losing a desideratum - $50, a coffee mug, whatever - hurts between 2 and 2.5 times as much as the equivalent gain.
• "It would seem that human beings are not able to describe, nor perhaps to imagine, happiness except in terms of contrast...  The inability of mankind to imagine happiness except in the form of relief, either from effort or pain, presents Socialists with a serious problem. Dickens can describe a poverty-stricken family tucking into a roast goose, and can make them appear happy; on the other hand, the inhabitants of perfect universes seem to have no spontaneous gaiety and are usually somewhat repulsive into the bargain."
• But the human brain has distinct neural circuits for positive feedback and negative feedback, and different varieties of positive and negative feedback.
• Could you delete pain and replace it with an urge not to do certain things that lacked the intolerable subjective quality of pain?
• Can you make every roast goose as pleasurable as it would be in contrast to starvation, without ever having starved?
• "Nanotechnology and genetic engineering will abolish suffering in all sentient life.  The abolitionist project is hugely ambitious but technically feasible.  It is also instrumentally rational and morally urgent.  The metabolic pathways of pain and malaise evolved because they served the fitness of our genes in the ancestral environment.  They will be replaced by a different sort of neural architecture - a motivational system based on heritable gradients of bliss.  States of sublime well-being are destined to become the genetically pre-programmed norm of mental health.  It is predicted that the world's last unpleasant experience will be a precisely dateable event."

• You could cut out just the intolerable parts of pain?

   

  Get rid of the Inquisition.  Keep the sort of pain that tells you not to stick your finger in the fire, or the pain that tells you that you shouldn't have put your friend's finger in the fire, or even the pain of breaking up with a lover.

   

  Try to get rid of the sort of pain that grinds down and destroys a mind.  Or configure minds to be harder to damage.

   

  You could have a world where there were broken legs, or even broken hearts, but no broken people.  No child sexual abuse that turns out more abusers.  No people ground down by weariness and drudging minor inconvenience to the point where they contemplate suicide.  No random meaningless endless sorrows like starvation or AIDS.
• A world where there is sorrow, but not massive systematic pointless sorrow, like we see on the evening news.  A world where pain, if it is not eliminated, at least does not overbalance pleasure.  You could write stories about that world, and they could read our stories.
• I'm not sure how many major chunks you can delete until that balanced, conflicting, dynamic structure collapses into something simpler, like an expected pleasure maximizer.

• Is there a point where Romeo and Juliet just seems less and less relevant, more and more a relic of some distant forgotten world?  Does there come some point in the transhuman journey where the whole business of the negative reinforcement circuitry, can't possibly seem like anything except a pointless hangover to wake up from?

   

  And if so, is there any point in delaying that last step?  Or should we just throw away our fears and... throw away our fears?

   

  I don't know.

13 more annotations...

• Movies that were made in say the 40s or 50s, seem much more alien - to me - than modern movies allegedly set hundreds of years in the future, or in different universes.  Watch a movie from 1950 and you may see a man slapping a woman.  Doesn't happen a lot in Lord of the Rings, does it?  Drop back to the 16th century and one popular entertainment was setting a cat on fire.  Ever see that in any moving picture, no matter how "lowbrow"?
• I am saying that the Future would discomfort us because it is better.
• Like the inhabitants of various other Utopias, they are chiefly concerned with avoiding fuss.  They live uneventful, subdued, 'reasonable' lives, free not only from quarrels, disorder or insecurity of any kind, but also from 'passion', including physical love.  They choose their mates on eugenic principles, avoid excesses of affection, and appear somewhat glad to die when their time comes.
• Utopia is reassuring, unsurprising, and dull.
• whenever I could think of an arguably-good idea that offended my sensibilities, I added it in.  The goal being to - without ever deliberately making the Future worse - make it a place where I would be as shocked as possible to see that that was how things had turned out.
• Getting rid of textbooks, for example - postulating that talking about science in public is socially unacceptable, for the same reason that you don't tell someone aiming to see a movie whether the hero dies at the end.  A world that had rejected my beloved concept of science as the public knowledge of humankind.
• It was also seductive to the point that I had to exert a serious effort to prevent my soul from getting sucked out. 
• And so I found myself being pulled in the direction of this world in which I was supposed to be "out of place".  I started thinking that, well, maybe it really would be a good idea to get rid of all the textbooks, all they do is take the fun out of science.  I started thinking that maybe personal competition was a legitimate motivator (previously, I would have called it a zero-sum game and been morally aghast).
• I started to wonder if the old vision that was so reassuring, so safe, was optimized to be good news to a modern human living in constant danger of permanent death or damage, and less optimized for the everyday existence of someone less frightened.
• So long as no one dies permanently, I am leaning in favor of a loud and scary Future.

8 more annotations...

• Utopia and Dystopia have something in common: they both confirm the moral sensibilities you started with.
• If you're trying to do Fun Theory, you have to come up with a Weirdtopia that's at least arguably-better than Utopia.  This is harder but also directs you to more interesting regions of the answer space.