Among transhumanists, Nick Bostrom is well-known for promoting the idea of \u2018existential risks\u2019, potential harms which, were they come to pass, would annihilate the human condition altogether. Their probability may be relatively small, but the expected magnitude of their effects are so great, so Bostrom claims, that it is rational to devote some significant resources to safeguarding against them. (Indeed, there are now institutes for the study of existential risks on both sides of the Atlantic.) Moreover, because existential risks are intimately tied to the advancement of science and technology, their probability is likely to grow in the coming years.<\/p>\n
Contrary to expectations, Bostrom is much less concerned with ecological suicide from humanity\u2019s excessive carbon emissions than with the emergence of a superior brand of artificial intelligence \u2013 a \u2018superintelligence\u2019. This creature would be a human artefact, or at least descended from one. However, its self-programming capacity would have run amok in positive feedback, resulting in a maniacal, even self-destructive mission to rearrange the world in the image of its objectives. Such a superintelligence may appear to be quite ruthless in its dealings with humans, but that would only reflect the obstacles that we place, perhaps unwittingly, in the way of the realization of its objectives. Thus, this being would not conform to the science fiction stereotype of robots deliberately revolting against creators who are now seen as their inferiors.<\/p>\n
I must confess that I find this conceptualisation of \u2018existential risk\u2019 rather un-transhumanist in spirit. Bostrom treats risk as a threat rather than as an opportunity. His risk horizon is precautionary rather than proactionary: He focuses on preventing the worst consequences rather than considering the prospects that are opened up by whatever radical changes might be inflicted by the superintelligence. This may be because in Bostrom\u2019s key thought experiment, the superintelligence turns out to be the ultimate paper-clip collecting machine that ends up subsuming the entire planet to its task, destroying humanity along the way, almost as an afterthought.<\/p>\n
But is this really a good starting point for thinking about existential risk? Much more likely than total human annihilation is that a substantial portion of humanity \u2013 but not everyone \u2013 is eliminated. (Certainly this captures the worst case scenarios surrounding climate change.) The Cold War remains the gold standard for this line of thought. In the US, the RAND Corporation\u2019s chief analyst, Herman Kahn \u2014 the model for Stanley Kubrick\u2019s Dr Strangelove \u2013 routinely, if not casually, tossed off scenarios of how, say, a US-USSR nuclear confrontation would serve to increase the tolerance for human biological diversity, due to the resulting proliferation of genetic mutations. Put in more general terms, a severe social disruption provides a unique opportunity for pursuing ideals that might otherwise be thwarted by a \u2018business as usual\u2019 policy orientation.<\/p>\n
Here it is worth recalling that the Cold War succeeded on its own terms: None of the worst case scenarios were ever realized, even though many people were mentally prepared to make the most of the projected adversities. This is one way to think about how the internet itself arose, courtesy the US Defense Department\u2019s interest in maintaining scientific communications in the face of attack. In other words, rather than trying to prevent every possible catastrophe, the way to deal with \u2018unknown unknowns\u2019 is to imagine that some of them have already come to pass and redesign the world accordingly so that you can carry on regardless. Thus, Herman Kahn\u2019s projection of a thermonuclear future provided grounds in the 1960s for the promotion of, say, racially mixed marriages, disability-friendly environments, and the \u2018do more with less\u2019 mentality that came to characterize the ecology movement.<\/p>\n
Kahn was a true proactionary thinker. For him, the threat of global nuclear war raised Joseph Schumpeter\u2019s idea of \u2018creative destruction\u2019 to a higher plane, inspiring social innovations that would be otherwise difficult to achieve by conventional politics. Historians have long noted that modern warfare has promoted spikes in innovation that in times of peace are then subject to diffusion, as the relevant industries redeploy for civilian purposes. We might think of this tendency, in mechanical terms, as system \u2018overdesign\u2019 (i.e. preparing for the worst but benefitting even if the worst doesn\u2019t happen) or, more organically, as a vaccine that converts a potential liability into an actual benefit.<\/p>\n
In either case, existential risk is regarded in broadly positive terms, specifically as an unprecedented opportunity to extend the range of human capability, even under radically changed circumstances. This sense of \u2018antifragility\u2019, as the great \u2018black swan\u2019 detector Nicholas Taleb would put it, is the hallmark of our \u2018risk intelligence\u2019, the phrase that the British philosopher Dylan Evans has coined for a demonstrated capacity that people have to make step change improvements in their lives in the face of radical uncertainty. From this standpoint, Bostrom\u2019s superintelligence concept severely underestimates the adaptive capacity of human intelligence.<\/p>\n
Perhaps the best way to see just how much Bostrom shortchanges humanity is to note that his crucial thought experiment requires a strong ontological distinction between humans and superintelligent artefacts. Where are the cyborgs in this doomsday scenario? Reading Bostrom reminds me that science fiction did indeed make progress in the twentieth century, from the world of Karl \u010capek\u2019s Rossum\u2019s Universal Robots<\/em> in 1920 to the much subtler blending of human and computer futures in the works of William Gibson and others in more recent times.<\/p>\n Bostrom\u2019s superintelligence scenario began to be handled in more sophisticated fashion after the end of the First World War, popularly under the guise of \u2018runaway technology\u2019, a topic that received its canonical formulation in Langdon Winner\u2019s 1977 Autonomous Technology: Technics out of Control<\/em>, a classic in the field of science and technology of studies. Back then the main problem with superintelligent machines was that they would \u2018dehumanize\u2019 us, less because they might dominate us but more because we might become like them \u2013 perhaps because we feel that we have invested our best qualities in them, very much like Ludwig Feuerbach\u2019s aetiology of the Judaeo-Christian God. Marxists gave the term \u2018alienation\u2019 a popular spin to capture this sentiment in the 1960s.<\/p>\n Nowadays, of course, matters have been complicated by the prospect of human and machine identities merging together. This goes beyond simply implanting silicon chips in one\u2019s brain. Rather, it involves the complex migration and enhancement of human selves in cyberspace. (Sherry Turkle has been the premier ethnographer of this process in children.) That such developments are even possible points to a prospect that Bostrom refuses to consider, namely, that to be \u2018human\u2019 is to be only contingently located in the body of Homo sapiens<\/em>. The name of our species \u2013 Homo sapiens<\/em> \u2013 already gives away the game, because our distinguishing feature (so claimed Linnaeus) had nothing to do with our physical morphology but with the character of our minds. And might not such a \u2018sapient\u2019 mind better exist somewhere other than in the upright ape from which we have descended?<\/p>\n The prospects for transhumanism hang on the answer to this question. Aubrey de Grey\u2019s indefinite life extension project is about Homo sapiens<\/em> in its normal biological form. In contrast, Ray Kurzweil\u2019s \u2018singularity\u2019 talk of uploading our consciousness into indefinitely powerful computers suggests a complete abandonment of the ordinary human body. The lesson taught by Langdon Winner\u2019s historical account is that our primary existential risk does not come from alien annihilation but from what social psychologists call \u2018adaptive preference formation\u2019. In other words, we come to want the sort of world that we think is most likely, simply because that offers us the greatest sense of security. Thus, the history of technology is full of cases in which humans have radically changed their lives to adjust to an innovation whose benefits they reckon outweigh the costs, even when both remain fundamentally incalculable. Success in the face such \u2018existential risk\u2019 is then largely a matter of whether people \u2013 perhaps of the following generation \u2013 have made the value shifts necessary to see the changes as positive overall. But of course, it does not follow that those who fail to survive the transition or have acquired their values before this transition would draw a similar conclusion.<\/p>\n","protected":false},"excerpt":{"rendered":" Among transhumanists, Nick Bostrom is well-known for promoting the idea of \u2018existential risks\u2019, potential harms which, were they come to pass, would annihilate the human condition altogether. Their probability may be relatively small, but the expected magnitude of their effects are so great, so Bostrom claims, that it is rational to devote some significant resources [\u2026]<\/p>\n","protected":false},"author":299,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1527,3,1499,14,1528,30,12,20,386,37,418,9,225,31,1500,224,64,1501],"tags":[1700,1698,1702,1701,1699],"class_list":["post-12368","post","type-post","status-publish","format-standard","hentry","category-alien-life","category-biological","category-cyborgs","category-defense","category-disruptive-technology","category-ethics","category-existential-risks","category-futurism","category-homo-sapiens","category-human-trajectories","category-internet","category-military","category-philosophy","category-policy","category-posthumanism","category-science","category-singularity","category-transhumanism-2","tag-adaptive-preference","tag-bostrom","tag-cold-war","tag-herman-kahn","tag-superintelligence"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/12368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/299"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=12368"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/12368\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=12368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=12368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=12368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}