Nick Bostrom, Oxford’s Future of Humanity Institute

Professor Nick Bostrom Faculty of Philosophy Director, Future of Humanity Institute James Martin 21 st Century School Oxford University The Big Picture

“ At the very apex of the first stratified societies, dynastic dreams were dreamt and visions of triumph or ruin entertained; but there is no mention in the papyri and cuniform tablets on which these hopes and fears were recorded that they envisaged, in the slightest degree, changes in the material conditions of the great masses, or for that matter, of the ruling class itself.”

Robert Heilbroner

The future of humanity? ? Time 2007 Technological development pre-human condition human condition posthuman condition

Extinction Time 2007 Technological development pre-human condition human condition posthuman condition

Past catastrophes killing >10 million 15 20C First World War 17 19C British India (mainly famine) 17 14C-15C Timur Lenk 18 15C-19C Atlantic Slave Trade 19 7C-19C Mideast Slave Trade 20 20C Stalin (famines and purges) 20 15C-19C Decimation of the American Indians 20 19C Taiping Rebellion (1851–1864) 24 20C Chinese Famine of 1907 25 17C Fall of the Ming Dynasty 27 (20C-21C) HIV/AIDS 36 8C An Shi Rebellion (756–763) 40 13C Mongol conquests 40 20C Spanish flu pandemic (1918-1919) 40 20C Great Leap Forward in China (famine) 55 20C Second World War 75 14C Black death (1347-1350) 100 6C Plague of Justinian 400 20C Smallpox Deaths (millions) Century Catastrophe

Scope Intensity Loss of one hair Congestion from one extra vehicle 0.01 degree global warming Loss of one species of beetle Car is stolen Recession in a country Destruction of the ozone layer Drastic loss of biodiversity Fatal car crash Genocide Ageing? Human extinction Personal Global Trans-generational Local Imperceptible Endurable Terminal (Hellish?) (Cosmic?) Global Catastrophic Risks Existential Risks

Existential risk

One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.

No existential catastrophe

> 99.9% of all species are extinct

Toba eruption 75,000 years ago  ~500 reproducing females

Neanderthals extinct 33,000 – 24,000 years ago

Homo florensis extinct 12,000 years ago?

Opinions…

Existential risks on the horizon…

Some opinions on net existential risk “ significant” Human extinction this century Richard Posner 0.25% Probability of humanity extinct <5,100 yrs Richard Gott ≥ 25% Cumulative existential risk (no time limit) Early Bostrom (2002) 50% End of civilization by 2100 (note: this need not entail extinction or existential catastrophe) Martin Rees 30% Human extinction by 2496 (based partly on the doomsday argument and Leslie’s view about how quantum indeterminacy affects this argument) John Leslie (1996) 0.1% Extinction risk per year Stern Report 19% Median answer to “Overall risk of human extinction prior to 2100” GCR conference poll Probability Claim Source

Natural

Anthropogenic

Other Human infertility Space radiation Other climate change Back-contamination Kinetic impact Non-anthropogenic vaccuum decay Extraterrestrial intelligence Physics experiment Supervolcanic eruption Self-destroying superintelligent AI Nonspecific conflict Emissions-caused global warming Natural pandemic Non-weapons nanotech accident Genetically engineering / synthetic biology Simulation shutdown Nuclear holocaust Nanotech weapons systems HUMAN EXTINCTION RISKS?

Stagnation or Plateau Time 2007 Technological development pre-human condition human condition posthuman condition

Revolutionary technologies?

Artificial intelligence and superintelligence

Machine-phase nanotechnology

Uploading

Information technology (virtual reality; wearable computers, etc.)

Anti-aging medicine

Nootropics

Mood enhancers

Genetic engineering

Collaborative information filtering

Institutions: information markets…

Recurrent collapse Time 2007 Technological development pre-human condition human condition posthuman condition

The longer term Time 2007 Technological development posthuman condition pre-human condition

Posthumanity

“ Posthuman condition”:

Population greater than 1 trillion persons

Life expectancy greater than 500 years

Large fraction of the population has cognitive capacities more than two standard deviations above the current human maximum

Near-complete control over the sensory input, for the majority of people for most of the time

Human psychological suffering becoming rare occurrence

Any change of magnitude or profundity comparable to that of one of the above

Time 2007 Technological development pre-human condition human condition posthuman condition

If Earth had formed one year ago…

Homo sapiens evolved less than 12 minutes ago

Agriculture began a little over 1 minute ago

Industrial revolution took place less than 2 seconds ago

Electronic computer was invented 0.4 seconds ago

Electronic computer was invented 0.4 seconds ago

Internet less than 0.1 seconds ago

Kanzi Great Ape Trust, Des Moines Witten Institute for Advanced Study, Princeton

The singularity hypothesis

Ulam talking with John von Neumann in 1958:

“ One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.”

The singularity hypothesis

I. J. Good in 1965:

“ Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make…”

Computing power

Classical AI Uploading Neuromorphic engineering Genetic algorithms, neural networks, etc.

Space of possible modes of being

Space of possible modes of being ? ? ? ? ? ? ? ?

Space of possible modes of being ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Guardian Activate Summit