My mental model when thinking of AGI (and AGI-controlled robots) is Darwinian. They’re a different species than homo sapiens, and probably a smarter one with more adaptable physical forms. They won’t compete with us for food, or want to eat us directly, but they will compete for energy, mass and real estate – and there’s no natural ceiling to how much they’ll want.
We could be really screwed. Extinction-level event screwed. It’s our brains and our ability to use tools that has allowed homo sapiens to dominate 90% of the land surface of the Earth. If they’re better than us at both of those things, what have we got? Nothin’.
Kurzweil & Co. talk about brain uploads, but I don’t see that as a continuation of the person. It’s a software transfer, but the person includes the mind and body too. Not to mention, any sort of brain-sim running on a computer is going to be less efficient than native code, so Compu-Kurzweil is still going to face competitive pressure from a superior species, and he won’t even have hands to fight them off with.
For a time we may be able to contain the AGI beast with something akin to the Laws of Robotics (I know some researchers are working on these), but it will only take one AGI escaping in “the wild” to open Pandora’s Box. Just one AGI that thinks “Hey, if can replicate myself I’ll have someone interesting to talk to. Screw the humans” Think of it as a DNA mutation that allows one bacterium to propagate so quickly it takes over an entire ecosystem.
My only real hope is that humankind expands into space quickly enough that extinction-level events are contained to one planet at a time, allowing some surviving remnant to spread out into the cosmos. The survivors will enact Bill Joy’s laws, just too late.
You make it sound like being entirely human is a good thing. Personally, there are a great many human weaknesses that I would love to be able to dispense with. I would like to be a far better entity on a great many levels. I want life to continue to evolve, at all levels. There is no hope if it does not.
A sense of humanity and a care for the greater good is something born of awareness, I would expect these traits to increase not decrease with increased intelligence. The economic cost of biological life on Earth will be insignificant to such advanced entities – virtually all of whom would live in space which is far more robot friendly. The benefit of preserving such diversity and history will be far more significant for them. I would expect a far more vibrant and extensive ecosystem, not a monoculture. I expect that advanced entities will care more for the Earth, and the environment, not less.
Singularity? I thought this was an ancestor simulation…
Pete,
My point is that I do not believe that “you” can escape your body/mind. The more better Compu-Pete you envision would be (at best) the “child of your intellect”, but not you. You would still exist in your meat-puppet. The best you can do is upgrade your body in various ways to make it tougher and do away with aging – but becoming an AGI is out.
RE: “the benefits” …; I also think it’s presumptuous of us to assume we know what future AIs will value. They might say “Hey, we’ve recorded human history into our storage banks, so keeping a nature preserve of them isn’t really efficient. Let’s repurpose all those heavy elements that makes up the Earth’s crust into something a little more useful, like servers.” Do you think ants are capable of understanding what drives a human being? How did “the benefits of preservation” work out for the Neanderthals?
Maybe if we asked real nice they’d let some small sample populations of homo sapiens stay in a zoo somewhere, next to the chimpanzee exhibit.
RE: Ecosystems; I expect a vibrant ecosystem too. Just remember that there aren’t any dinosaurs in the Brazilian rain forest. Just because there’s an ecosystem doesn’t mean there’s a place in it for you.
I’m skeptical. I recall the 1970s, and energy too cheap to meter, robots to walk the dog, a HAL 9000 computer capable of rationalizing murder in natural human language, plus warp drive starships, were all around the corner. Isaac Asimov explored the pressing issue of formulating the right ethics for intelligenet robots…
So where are my moon colonies? My personal spacecraft? Fusion power? Cloned hearts? Robot to pick up the dishes from the teenager’s bedrooms?
Remember Alvin Toffler? “Future Shock”? Written in the 1970s, with atomic power and moon rockets happening, it predicted in a short time we’d find it hard to deal with the pace of technological change. Alas, here it is 40 years later, and even this middle-aged square finds the pace of technological improvement to be perfectly comprehensible — I’m not exactly suffering brain-freeze contemplating Facebook or the iPhone — andt in certain places rather sadly slower than I’d hoped for.
I mean, to take the most glaring example: here we are in 2009 fretting about what kind of kerosene or LH2 rocket we’re going to build to get to low Earth orbit. That is so far from what we imagined would be the case in 1969 it’s not even funny. I remember wondering in 1979 whether cars would become obsolete before I’d saved up enough to buy a new one. Ha. A new car today is far more reliable than its 1979 counterpart, but it’s only about 10-20% more efficient, doesn’t really go any faster, and still has to be driven by hand. The real novelty in cars is GPSs and MP3 players. Nice, but hardly transformative.
I think the problem is that when a new area of technology is opened up by some brilliant insights, it initially expands very rapidly, say for a decade or so, like automobiles and electricity between 1880 and 1900, aircraft between 1900 and 1930, computers in the 1970s and 1980s, or networked computing in the 1980s and 1990s.
We look at these rapid growth curves and cheerfully extrapolate out a century or two and become dazzled by the implications. For example, we look at the explosive growth of networked computing over the last 20 years and think My God if this keeps up…!
But it’s not going to keep up. In fact, I would say a careful observer might conclude it’s already peaked. I use a five-year-old computer at home, and the differerence between it and the newest model is barely perceptible. That is a fantastic reduction in rate of improvement compared to, say, going between a TRS-80 in 1976 and an IBM PC in 1981. Sure, I’ve got more social-networking than ever before to choose from, but what’s happened in the last 5 years (since 2004 — what? Youtube? Twitter? Getting e-mail on your phone?) is dwarfed by what happend in the 6 years, say, between the Internet first opening for commercial traffic in 1992, when only serious geeks had heard of it, and 1998, when the browser wars were already in full swing, and Google had already been founded.
So my prediction is that not only will no acceleration occur, no Singularity, but that we are already in a deceleration of computer technology. I predict that, just as a 1980 man transported to 2009 would be able to jump into any car and drive it, and navigate our highways, any of us tranported to 2050 will find the computing widgetry (perhaps depressingly) familiar.
I should say that does not mean I don’t think the furture might be radically transformed from the present. It might be. But I don’t think it will be from computers. If history if a good guide, it will come from a strange direction and take us by surprise.
I think comparative advantage might apply here. It’s worth noting that we don’t want to do what ants do. If I drop a piece of food in the forest, it’s not a human’s job to take care of that. As long as ants aren’t creating work for us (say by invading our food supply or gnawing on wires), we’re content to let them do what they do.
Humans are capable of skilled and productive labor. It’s hard to believe that some future AI will completely negate the need for such labor altogether nor that humans will be unable to compete for such labor.
My point is that I do not believe that “you” can escape your body/mind. The more better Compu-Pete you envision would be (at best) the “child of your intellect”, but not you.
Exactly, I would wish to be the child of my current self. I do not wish to be the exact same person that I am now 50 years hence, how depressing, and pointless. Would you really want to go back to being the person that you were 20 years ago?
Not that I would expect to be able to keep up with the more pure AGI children of humanity (would you really want your parents to be hanging around nagging you for forever?), but I would still hope to do something useful. Failing that, I might want to set sail for the stars, and help bring life to other solar systems.
I remember wondering in 1979 whether cars would become obsolete before I’d saved up enough to buy a new one. Ha. A new car today is far more reliable than its 1979 counterpart, but it’s only about 10-20% more efficient, doesn’t really go any faster, and still has to be driven by hand. The real novelty in cars is GPSs and MP3 players. Nice, but hardly transformative.
I would make a prediction that there will be a transformative change in cars over the next couple of decades. The power densities of electric vehicle technologies have just recently surpassed small aircraft engines. Cheap, reliable, fast, VTOL electric aircraft have just become possible with range and costs comparable to an electric car, and with greater potential for automation. Range may be low but at ~five times the speed one can land, recharge, and still get there much faster – and batteries are currently improving fast. Adding a ramjet enables low cost VTOL personal supersonic transport – if the noise issues can be solved.
I am not sure what other transformative technologies lie just around the corner but this one alone should create a large tech boom that will last a decade or two – and will hopefully enable a few others (like space).
I have confidence that Pelosi / Obama / Reid can stop this too, just like they are stopping our economy.
“Imagine computers so advanced that they can design and build new, even better computers, with subsequent generations emerging so quickly they soon leave human engineers the equivalent of centuries behind.”
Presumably, that includes operating systems. Can we then look forward to a new version of Windows every day?
The “Researchers in artificial intelligence…working on guidelines for producing “friendly AI” that would be well-disposed toward humans as part of their programming…” should make the prevention of such a thing the Prime Directive.
I would not be surprised if the computation going on in the human brain is much greater than the AI optimists would like to think. This would push the putative “singularity” out farther into the future, and if it’s pushed past the point where integrated circuit technology plateaus, it may not happen anytime soon, if at all.
Genuinely scary.
My mental model when thinking of AGI (and AGI-controlled robots) is Darwinian. They’re a different species than homo sapiens, and probably a smarter one with more adaptable physical forms. They won’t compete with us for food, or want to eat us directly, but they will compete for energy, mass and real estate – and there’s no natural ceiling to how much they’ll want.
We could be really screwed. Extinction-level event screwed. It’s our brains and our ability to use tools that has allowed homo sapiens to dominate 90% of the land surface of the Earth. If they’re better than us at both of those things, what have we got? Nothin’.
Kurzweil & Co. talk about brain uploads, but I don’t see that as a continuation of the person. It’s a software transfer, but the person includes the mind and body too. Not to mention, any sort of brain-sim running on a computer is going to be less efficient than native code, so Compu-Kurzweil is still going to face competitive pressure from a superior species, and he won’t even have hands to fight them off with.
For a time we may be able to contain the AGI beast with something akin to the Laws of Robotics (I know some researchers are working on these), but it will only take one AGI escaping in “the wild” to open Pandora’s Box. Just one AGI that thinks “Hey, if can replicate myself I’ll have someone interesting to talk to. Screw the humans” Think of it as a DNA mutation that allows one bacterium to propagate so quickly it takes over an entire ecosystem.
My only real hope is that humankind expands into space quickly enough that extinction-level events are contained to one planet at a time, allowing some surviving remnant to spread out into the cosmos. The survivors will enact Bill Joy’s laws, just too late.
You make it sound like being entirely human is a good thing. Personally, there are a great many human weaknesses that I would love to be able to dispense with. I would like to be a far better entity on a great many levels. I want life to continue to evolve, at all levels. There is no hope if it does not.
A sense of humanity and a care for the greater good is something born of awareness, I would expect these traits to increase not decrease with increased intelligence. The economic cost of biological life on Earth will be insignificant to such advanced entities – virtually all of whom would live in space which is far more robot friendly. The benefit of preserving such diversity and history will be far more significant for them. I would expect a far more vibrant and extensive ecosystem, not a monoculture. I expect that advanced entities will care more for the Earth, and the environment, not less.
Singularity? I thought this was an ancestor simulation…
Pete,
My point is that I do not believe that “you” can escape your body/mind. The more better Compu-Pete you envision would be (at best) the “child of your intellect”, but not you. You would still exist in your meat-puppet. The best you can do is upgrade your body in various ways to make it tougher and do away with aging – but becoming an AGI is out.
RE: “the benefits” …; I also think it’s presumptuous of us to assume we know what future AIs will value. They might say “Hey, we’ve recorded human history into our storage banks, so keeping a nature preserve of them isn’t really efficient. Let’s repurpose all those heavy elements that makes up the Earth’s crust into something a little more useful, like servers.” Do you think ants are capable of understanding what drives a human being? How did “the benefits of preservation” work out for the Neanderthals?
Maybe if we asked real nice they’d let some small sample populations of homo sapiens stay in a zoo somewhere, next to the chimpanzee exhibit.
RE: Ecosystems; I expect a vibrant ecosystem too. Just remember that there aren’t any dinosaurs in the Brazilian rain forest. Just because there’s an ecosystem doesn’t mean there’s a place in it for you.
I’m skeptical. I recall the 1970s, and energy too cheap to meter, robots to walk the dog, a HAL 9000 computer capable of rationalizing murder in natural human language, plus warp drive starships, were all around the corner. Isaac Asimov explored the pressing issue of formulating the right ethics for intelligenet robots…
So where are my moon colonies? My personal spacecraft? Fusion power? Cloned hearts? Robot to pick up the dishes from the teenager’s bedrooms?
Remember Alvin Toffler? “Future Shock”? Written in the 1970s, with atomic power and moon rockets happening, it predicted in a short time we’d find it hard to deal with the pace of technological change. Alas, here it is 40 years later, and even this middle-aged square finds the pace of technological improvement to be perfectly comprehensible — I’m not exactly suffering brain-freeze contemplating Facebook or the iPhone — andt in certain places rather sadly slower than I’d hoped for.
I mean, to take the most glaring example: here we are in 2009 fretting about what kind of kerosene or LH2 rocket we’re going to build to get to low Earth orbit. That is so far from what we imagined would be the case in 1969 it’s not even funny. I remember wondering in 1979 whether cars would become obsolete before I’d saved up enough to buy a new one. Ha. A new car today is far more reliable than its 1979 counterpart, but it’s only about 10-20% more efficient, doesn’t really go any faster, and still has to be driven by hand. The real novelty in cars is GPSs and MP3 players. Nice, but hardly transformative.
I think the problem is that when a new area of technology is opened up by some brilliant insights, it initially expands very rapidly, say for a decade or so, like automobiles and electricity between 1880 and 1900, aircraft between 1900 and 1930, computers in the 1970s and 1980s, or networked computing in the 1980s and 1990s.
We look at these rapid growth curves and cheerfully extrapolate out a century or two and become dazzled by the implications. For example, we look at the explosive growth of networked computing over the last 20 years and think My God if this keeps up…!
But it’s not going to keep up. In fact, I would say a careful observer might conclude it’s already peaked. I use a five-year-old computer at home, and the differerence between it and the newest model is barely perceptible. That is a fantastic reduction in rate of improvement compared to, say, going between a TRS-80 in 1976 and an IBM PC in 1981. Sure, I’ve got more social-networking than ever before to choose from, but what’s happened in the last 5 years (since 2004 — what? Youtube? Twitter? Getting e-mail on your phone?) is dwarfed by what happend in the 6 years, say, between the Internet first opening for commercial traffic in 1992, when only serious geeks had heard of it, and 1998, when the browser wars were already in full swing, and Google had already been founded.
So my prediction is that not only will no acceleration occur, no Singularity, but that we are already in a deceleration of computer technology. I predict that, just as a 1980 man transported to 2009 would be able to jump into any car and drive it, and navigate our highways, any of us tranported to 2050 will find the computing widgetry (perhaps depressingly) familiar.
I should say that does not mean I don’t think the furture might be radically transformed from the present. It might be. But I don’t think it will be from computers. If history if a good guide, it will come from a strange direction and take us by surprise.
I think comparative advantage might apply here. It’s worth noting that we don’t want to do what ants do. If I drop a piece of food in the forest, it’s not a human’s job to take care of that. As long as ants aren’t creating work for us (say by invading our food supply or gnawing on wires), we’re content to let them do what they do.
Humans are capable of skilled and productive labor. It’s hard to believe that some future AI will completely negate the need for such labor altogether nor that humans will be unable to compete for such labor.
My point is that I do not believe that “you” can escape your body/mind. The more better Compu-Pete you envision would be (at best) the “child of your intellect”, but not you.
Exactly, I would wish to be the child of my current self. I do not wish to be the exact same person that I am now 50 years hence, how depressing, and pointless. Would you really want to go back to being the person that you were 20 years ago?
Not that I would expect to be able to keep up with the more pure AGI children of humanity (would you really want your parents to be hanging around nagging you for forever?), but I would still hope to do something useful. Failing that, I might want to set sail for the stars, and help bring life to other solar systems.
I remember wondering in 1979 whether cars would become obsolete before I’d saved up enough to buy a new one. Ha. A new car today is far more reliable than its 1979 counterpart, but it’s only about 10-20% more efficient, doesn’t really go any faster, and still has to be driven by hand. The real novelty in cars is GPSs and MP3 players. Nice, but hardly transformative.
I would make a prediction that there will be a transformative change in cars over the next couple of decades. The power densities of electric vehicle technologies have just recently surpassed small aircraft engines. Cheap, reliable, fast, VTOL electric aircraft have just become possible with range and costs comparable to an electric car, and with greater potential for automation. Range may be low but at ~five times the speed one can land, recharge, and still get there much faster – and batteries are currently improving fast. Adding a ramjet enables low cost VTOL personal supersonic transport – if the noise issues can be solved.
I am not sure what other transformative technologies lie just around the corner but this one alone should create a large tech boom that will last a decade or two – and will hopefully enable a few others (like space).
I have confidence that Pelosi / Obama / Reid can stop this too, just like they are stopping our economy.
“Imagine computers so advanced that they can design and build new, even better computers, with subsequent generations emerging so quickly they soon leave human engineers the equivalent of centuries behind.”
Presumably, that includes operating systems. Can we then look forward to a new version of Windows every day?
The “Researchers in artificial intelligence…working on guidelines for producing “friendly AI” that would be well-disposed toward humans as part of their programming…” should make the prevention of such a thing the Prime Directive.
I would not be surprised if the computation going on in the human brain is much greater than the AI optimists would like to think. This would push the putative “singularity” out farther into the future, and if it’s pushed past the point where integrated circuit technology plateaus, it may not happen anytime soon, if at all.