SlideShare a Scribd company logo
http://ea-foundation.org
S-risks
Max Daniel
Executive Director, Foundational Research Institute
Why they are the worst existential risks,
and how to prevent them
Shaping the long-term future
2
AI strategy and policy AI safetyBiosecurity
Shaping the long-term future
Bostrom 2002, Beckstead 2013 3
AI strategy and policy AI safetyBiosecurity
Broad vs. narrow interventions (Beckstead 2013) AI forecasting
Maxipok principle (Bostrom 2002) X-risk (Bostrom 2002)
Shaping the long-term future
Bostrom 2002, Beckstead 2013 4
AI strategy and policy AI safetyBiosecurity
Broad vs. narrow interventions (Beckstead 2013) AI forecasting
Maxipok principle (Bostrom 2002) X-risk (Bostrom 2002)
Population ethics??
What is ‘wellbeing’??
Shaping the long-term future
Bostrom 2002, Beckstead 2013 5
AI strategy and policy AI safetyBiosecurity
Broad vs. narrow interventions (Beckstead 2013) AI forecasting
Maxipok principle (Bostrom 2002) X-risk (Bostrom 2002)
Population ethics??
What is ‘wellbeing’??
Based on Bostrom 2013, Figure 2
Scope
Severity
imperceptible endurable crushing (hellish)
Fatal car
crash
personal
local
global
trans-generational
pan-generational
(cosmic)
X-risk
Zooming in on existential risk
Based on Bostrom 2013, Figure 2
cosmic
pan-
generational Extinction
crushing hellish
Zooming in on existential risk
cosmic S-risk
pan-
generational Extinction
crushing hellish
What should we do about s-risks?
Not controversial:
• Reducing s-risk is desirable, all else being equal.

But: Should reducing s-risks be a priority?
• How likely are s-risks?

• What interventions are there to reduce s-risks? Do they differ
from those reducing extinction risk?

• Who is working on such interventions?
Probability?
Technologies that enable …
● Sentient simulations
● Suffering subroutines
● Multiplying wild animal suffering
● Unknown unknowns
WBE roadmap: Sandberg & Bostrom (2008)
What s-risks are there?
By type of intent:
● No intent: accidents, indifference, evolution …
● Evil intent
● Strategic intent: conflicts, threats, bribes, …
Tractability?
Targeted interventions
● (Some work in) AI safety
● (Some work in) AI policy
Broad interventions
● International cooperation, expanding moral circle (?)
● Going meta: research anti-s-risk strategies
Suffering-focused AI safety (Gloor 2016)
On the neglectedness of s-risk
● Not totally neglected, but gets little attention
● Misconception: existential risk = extinction risk
● The Foundational Research Institute is the only organization
that specifically focuses on reducing s-risk
Summary: why and how to prevent s-risk
● Whether to focus on s-risk depends on both empirical and value
judgments
● S-risks not much more unlikely than AI-related extinction risk
● Some, but not all, familiar work on x-risk also reduces s-risk
● Reducing s-risks seems neglected
How shaping the far future does not look like
Presentation title
John Smith | Head of Department 28.06.2016
Subtitle or caption
Thank you.
More on s-risk? Check out
foundational-research.org
max@foundational-research.org | Add me on Facebook! :)
S-risks: Why they are the worst existential risks, and how to prevent them

More Related Content

Similar to S-risks: Why they are the worst existential risks, and how to prevent them

Big Data, Psychografics and Social Media Advertising - Alessandro Sisti
Big Data, Psychografics and Social Media Advertising - Alessandro SistiBig Data, Psychografics and Social Media Advertising - Alessandro Sisti
Big Data, Psychografics and Social Media Advertising - Alessandro Sisti
Data Driven Innovation
 
Big data , Behavioural science, Social Media Adv
Big data , Behavioural science, Social Media AdvBig data , Behavioural science, Social Media Adv
Big data , Behavioural science, Social Media Adv
Alessandro Sisti
 
Big data behavioural science addressable social media adv
Big data behavioural science addressable social media advBig data behavioural science addressable social media adv
Big data behavioural science addressable social media adv
Alessandro Sisti
 
Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand
Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand
Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand
International Institute for Learning
 
Black swan decision making sikich 2014 rev 0
Black swan decision making sikich 2014 rev 0Black swan decision making sikich 2014 rev 0
Black swan decision making sikich 2014 rev 0
Geary Sikich
 
Measurement, Quantitative vs. Qualitative and Other Cool Stuff
Measurement, Quantitative vs. Qualitative and Other Cool StuffMeasurement, Quantitative vs. Qualitative and Other Cool Stuff
Measurement, Quantitative vs. Qualitative and Other Cool Stuff
Jody Keyser
 
How to Secure America
How to Secure AmericaHow to Secure America
How to Secure America
SecurityStudio
 
Journal+Feature-InsiderThreat
Journal+Feature-InsiderThreatJournal+Feature-InsiderThreat
Journal+Feature-InsiderThreat
Anthony Buenger
 
From Risk Transfer to Risk Prevention
From Risk Transfer to Risk PreventionFrom Risk Transfer to Risk Prevention
From Risk Transfer to Risk Prevention
Matteo Carbone
 
1427 Women in Cybersecurity-Taking Charge and Protecting the World
1427 Women in Cybersecurity-Taking Charge and Protecting the World1427 Women in Cybersecurity-Taking Charge and Protecting the World
1427 Women in Cybersecurity-Taking Charge and Protecting the World
Career Communications Group
 
Botnet Research TemplateLearning ContentKey Botnet Notes.docx
Botnet Research TemplateLearning ContentKey Botnet Notes.docxBotnet Research TemplateLearning ContentKey Botnet Notes.docx
Botnet Research TemplateLearning ContentKey Botnet Notes.docx
jasoninnes20
 
Social engineering – posing challenges to the thinking security professional ...
Social engineering – posing challenges to the thinking security professional ...Social engineering – posing challenges to the thinking security professional ...
Social engineering – posing challenges to the thinking security professional ...
Paul Devassy, CPP
 
Research about artificial intelligence (A.I)
Research about artificial intelligence (A.I)Research about artificial intelligence (A.I)
Research about artificial intelligence (A.I)
Alị Ŕỉźvị
 
AI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSE
AI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSEAI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSE
AI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSE
Rajkumar R
 
Foresight: Your Hidden Superpower
Foresight: Your Hidden SuperpowerForesight: Your Hidden Superpower
Foresight: Your Hidden Superpower
John Smart
 
Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?
Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?
Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?
Capgemini
 
IS Project Risk Management: Risk Factors in IS Project Management
IS Project Risk Management:  Risk Factors in IS Project ManagementIS Project Risk Management:  Risk Factors in IS Project Management
IS Project Risk Management: Risk Factors in IS Project Management
Eashani Rodrigo
 
Manajemen Bencana
Manajemen BencanaManajemen Bencana
Manajemen Bencana
azisbustari
 
Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020
Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020
Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020
Artificial Intelligence Institute at UofSC
 
'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...
'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...
'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...
Brittany Ferdinands
 

Similar to S-risks: Why they are the worst existential risks, and how to prevent them (20)

Big Data, Psychografics and Social Media Advertising - Alessandro Sisti
Big Data, Psychografics and Social Media Advertising - Alessandro SistiBig Data, Psychografics and Social Media Advertising - Alessandro Sisti
Big Data, Psychografics and Social Media Advertising - Alessandro Sisti
 
Big data , Behavioural science, Social Media Adv
Big data , Behavioural science, Social Media AdvBig data , Behavioural science, Social Media Adv
Big data , Behavioural science, Social Media Adv
 
Big data behavioural science addressable social media adv
Big data behavioural science addressable social media advBig data behavioural science addressable social media adv
Big data behavioural science addressable social media adv
 
Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand
Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand
Risks - Threats and Opportunities: How Not to Bury Your Head in the Sand
 
Black swan decision making sikich 2014 rev 0
Black swan decision making sikich 2014 rev 0Black swan decision making sikich 2014 rev 0
Black swan decision making sikich 2014 rev 0
 
Measurement, Quantitative vs. Qualitative and Other Cool Stuff
Measurement, Quantitative vs. Qualitative and Other Cool StuffMeasurement, Quantitative vs. Qualitative and Other Cool Stuff
Measurement, Quantitative vs. Qualitative and Other Cool Stuff
 
How to Secure America
How to Secure AmericaHow to Secure America
How to Secure America
 
Journal+Feature-InsiderThreat
Journal+Feature-InsiderThreatJournal+Feature-InsiderThreat
Journal+Feature-InsiderThreat
 
From Risk Transfer to Risk Prevention
From Risk Transfer to Risk PreventionFrom Risk Transfer to Risk Prevention
From Risk Transfer to Risk Prevention
 
1427 Women in Cybersecurity-Taking Charge and Protecting the World
1427 Women in Cybersecurity-Taking Charge and Protecting the World1427 Women in Cybersecurity-Taking Charge and Protecting the World
1427 Women in Cybersecurity-Taking Charge and Protecting the World
 
Botnet Research TemplateLearning ContentKey Botnet Notes.docx
Botnet Research TemplateLearning ContentKey Botnet Notes.docxBotnet Research TemplateLearning ContentKey Botnet Notes.docx
Botnet Research TemplateLearning ContentKey Botnet Notes.docx
 
Social engineering – posing challenges to the thinking security professional ...
Social engineering – posing challenges to the thinking security professional ...Social engineering – posing challenges to the thinking security professional ...
Social engineering – posing challenges to the thinking security professional ...
 
Research about artificial intelligence (A.I)
Research about artificial intelligence (A.I)Research about artificial intelligence (A.I)
Research about artificial intelligence (A.I)
 
AI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSE
AI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSEAI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSE
AI and covid19 | Mr. R. Rajkumar, Assistant Professor, Department of CSE
 
Foresight: Your Hidden Superpower
Foresight: Your Hidden SuperpowerForesight: Your Hidden Superpower
Foresight: Your Hidden Superpower
 
Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?
Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?
Failing and Failing Fast in AppDev – How Do We Keep up in AppSec?
 
IS Project Risk Management: Risk Factors in IS Project Management
IS Project Risk Management:  Risk Factors in IS Project ManagementIS Project Risk Management:  Risk Factors in IS Project Management
IS Project Risk Management: Risk Factors in IS Project Management
 
Manajemen Bencana
Manajemen BencanaManajemen Bencana
Manajemen Bencana
 
Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020
Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020
Artificial intelligence Institute @ UofSC: Brief Overview: 07/2020
 
'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...
'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...
'Unsustainably Sustainable' by Dr Carl Ungerer and Vanessa Liell at Mumbrella...
 

More from Effective Altruism Foundation

Current Status of the EA Movement
Current Status of the EA MovementCurrent Status of the EA Movement
Current Status of the EA Movement
Effective Altruism Foundation
 
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltung
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltungNationale Volksinitiative zur Abschaffung der Massen­tierhaltung
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltung
Effective Altruism Foundation
 
The New Meat
The New MeatThe New Meat
Lessons from Building an EA Charity: New Incentives
Lessons from Building an EA Charity: New IncentivesLessons from Building an EA Charity: New Incentives
Lessons from Building an EA Charity: New Incentives
Effective Altruism Foundation
 
What Does (and Doesn't) AI Mean for Effective Altruism?
What Does (and Doesn't) AI Mean for Effective Altruism?What Does (and Doesn't) AI Mean for Effective Altruism?
What Does (and Doesn't) AI Mean for Effective Altruism?
Effective Altruism Foundation
 
Delivering Development Impact at Scale
Delivering Development Impact at ScaleDelivering Development Impact at Scale
Delivering Development Impact at Scale
Effective Altruism Foundation
 
Using Evidence to Fight Poverty
Using Evidence to Fight PovertyUsing Evidence to Fight Poverty
Using Evidence to Fight Poverty
Effective Altruism Foundation
 
Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...
Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...
Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...
Effective Altruism Foundation
 
Raising the Stakes: When Poker Meets EA
Raising the Stakes: When Poker Meets EARaising the Stakes: When Poker Meets EA
Raising the Stakes: When Poker Meets EA
Effective Altruism Foundation
 
Against Naive Effective Altruism
Against Naive Effective AltruismAgainst Naive Effective Altruism
Against Naive Effective Altruism
Effective Altruism Foundation
 
Decision Theory Research at FRI
Decision Theory Research at FRIDecision Theory Research at FRI
Decision Theory Research at FRI
Effective Altruism Foundation
 
Wild-Animal Suffering Movement Building Through Research
Wild-Animal Suffering Movement Building Through ResearchWild-Animal Suffering Movement Building Through Research
Wild-Animal Suffering Movement Building Through Research
Effective Altruism Foundation
 
Ethische Berufswahl – Mit 80.000 Stunden die Welt retten
Ethische Berufswahl – Mit 80.000 Stunden die Welt rettenEthische Berufswahl – Mit 80.000 Stunden die Welt retten
Ethische Berufswahl – Mit 80.000 Stunden die Welt retten
Effective Altruism Foundation
 
Effektiver Altruismus: Minimaldefinition und Begründung
Effektiver Altruismus: Minimaldefinition und BegründungEffektiver Altruismus: Minimaldefinition und Begründung
Effektiver Altruismus: Minimaldefinition und Begründung
Effective Altruism Foundation
 
Effektiver Altruismus: Einwände und Erwiderungen
Effektiver Altruismus: Einwände und ErwiderungenEffektiver Altruismus: Einwände und Erwiderungen
Effektiver Altruismus: Einwände und Erwiderungen
Effective Altruism Foundation
 
Rationaler Altruismus – eine Begründung moralischen Engagements
Rationaler Altruismus – eine Begründung moralischen EngagementsRationaler Altruismus – eine Begründung moralischen Engagements
Rationaler Altruismus – eine Begründung moralischen Engagements
Effective Altruism Foundation
 
Meta­-Strategien: Investitionen in die EA­-Bewegung
Meta­-Strategien: Investitionen in die EA­-BewegungMeta­-Strategien: Investitionen in die EA­-Bewegung
Meta­-Strategien: Investitionen in die EA­-Bewegung
Effective Altruism Foundation
 
Dysrationalia — The IQ­-RQ gap and what to do about it
Dysrationalia — The IQ­-RQ gap and what to do about itDysrationalia — The IQ­-RQ gap and what to do about it
Dysrationalia — The IQ­-RQ gap and what to do about it
Effective Altruism Foundation
 
Political and legal activism for all sentient beings
Political and legal activism for all sentient beingsPolitical and legal activism for all sentient beings
Political and legal activism for all sentient beings
Effective Altruism Foundation
 
Künstliche Intelligenz und Effektiver Altruismus
Künstliche Intelligenz und Effektiver AltruismusKünstliche Intelligenz und Effektiver Altruismus
Künstliche Intelligenz und Effektiver Altruismus
Effective Altruism Foundation
 

More from Effective Altruism Foundation (20)

Current Status of the EA Movement
Current Status of the EA MovementCurrent Status of the EA Movement
Current Status of the EA Movement
 
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltung
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltungNationale Volksinitiative zur Abschaffung der Massen­tierhaltung
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltung
 
The New Meat
The New MeatThe New Meat
The New Meat
 
Lessons from Building an EA Charity: New Incentives
Lessons from Building an EA Charity: New IncentivesLessons from Building an EA Charity: New Incentives
Lessons from Building an EA Charity: New Incentives
 
What Does (and Doesn't) AI Mean for Effective Altruism?
What Does (and Doesn't) AI Mean for Effective Altruism?What Does (and Doesn't) AI Mean for Effective Altruism?
What Does (and Doesn't) AI Mean for Effective Altruism?
 
Delivering Development Impact at Scale
Delivering Development Impact at ScaleDelivering Development Impact at Scale
Delivering Development Impact at Scale
 
Using Evidence to Fight Poverty
Using Evidence to Fight PovertyUsing Evidence to Fight Poverty
Using Evidence to Fight Poverty
 
Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...
Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...
Toward a #FutureFortified: How Food Fortification Can Help End Micronutrient ...
 
Raising the Stakes: When Poker Meets EA
Raising the Stakes: When Poker Meets EARaising the Stakes: When Poker Meets EA
Raising the Stakes: When Poker Meets EA
 
Against Naive Effective Altruism
Against Naive Effective AltruismAgainst Naive Effective Altruism
Against Naive Effective Altruism
 
Decision Theory Research at FRI
Decision Theory Research at FRIDecision Theory Research at FRI
Decision Theory Research at FRI
 
Wild-Animal Suffering Movement Building Through Research
Wild-Animal Suffering Movement Building Through ResearchWild-Animal Suffering Movement Building Through Research
Wild-Animal Suffering Movement Building Through Research
 
Ethische Berufswahl – Mit 80.000 Stunden die Welt retten
Ethische Berufswahl – Mit 80.000 Stunden die Welt rettenEthische Berufswahl – Mit 80.000 Stunden die Welt retten
Ethische Berufswahl – Mit 80.000 Stunden die Welt retten
 
Effektiver Altruismus: Minimaldefinition und Begründung
Effektiver Altruismus: Minimaldefinition und BegründungEffektiver Altruismus: Minimaldefinition und Begründung
Effektiver Altruismus: Minimaldefinition und Begründung
 
Effektiver Altruismus: Einwände und Erwiderungen
Effektiver Altruismus: Einwände und ErwiderungenEffektiver Altruismus: Einwände und Erwiderungen
Effektiver Altruismus: Einwände und Erwiderungen
 
Rationaler Altruismus – eine Begründung moralischen Engagements
Rationaler Altruismus – eine Begründung moralischen EngagementsRationaler Altruismus – eine Begründung moralischen Engagements
Rationaler Altruismus – eine Begründung moralischen Engagements
 
Meta­-Strategien: Investitionen in die EA­-Bewegung
Meta­-Strategien: Investitionen in die EA­-BewegungMeta­-Strategien: Investitionen in die EA­-Bewegung
Meta­-Strategien: Investitionen in die EA­-Bewegung
 
Dysrationalia — The IQ­-RQ gap and what to do about it
Dysrationalia — The IQ­-RQ gap and what to do about itDysrationalia — The IQ­-RQ gap and what to do about it
Dysrationalia — The IQ­-RQ gap and what to do about it
 
Political and legal activism for all sentient beings
Political and legal activism for all sentient beingsPolitical and legal activism for all sentient beings
Political and legal activism for all sentient beings
 
Künstliche Intelligenz und Effektiver Altruismus
Künstliche Intelligenz und Effektiver AltruismusKünstliche Intelligenz und Effektiver Altruismus
Künstliche Intelligenz und Effektiver Altruismus
 

Recently uploaded

2024: The FAR - Federal Acquisition Regulations, Part 38
2024: The FAR - Federal Acquisition Regulations, Part 382024: The FAR - Federal Acquisition Regulations, Part 38
2024: The FAR - Federal Acquisition Regulations, Part 38
JSchaus & Associates
 
PUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptx
PUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptxPUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptx
PUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptx
Marked12
 
加急办理华威大学毕业证硕士文凭证书原版一模一样
加急办理华威大学毕业证硕士文凭证书原版一模一样加急办理华威大学毕业证硕士文凭证书原版一模一样
加急办理华威大学毕业证硕士文凭证书原版一模一样
uu1psyf6
 
Border towns and spaces of (in)visibility.pdf
Border towns and spaces of (in)visibility.pdfBorder towns and spaces of (in)visibility.pdf
Border towns and spaces of (in)visibility.pdf
Scalabrini Institute for Human Mobility in Africa
 
Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...
Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...
Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...
OECDregions
 
About Potato, The scientific name of the plant is Solanum tuberosum (L).
About Potato, The scientific name of the plant is Solanum tuberosum (L).About Potato, The scientific name of the plant is Solanum tuberosum (L).
About Potato, The scientific name of the plant is Solanum tuberosum (L).
Christina Parmionova
 
快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样
快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样
快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样
yemqpj
 
PPT Item # 5 - 318 Tuxedo Ave. (sign. review)
PPT Item # 5 - 318 Tuxedo Ave. (sign. review)PPT Item # 5 - 318 Tuxedo Ave. (sign. review)
PPT Item # 5 - 318 Tuxedo Ave. (sign. review)
ahcitycouncil
 
原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样
原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样
原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样
3woawyyl
 
Item # 10 -- Historical Presv. Districts
Item # 10 -- Historical Presv. DistrictsItem # 10 -- Historical Presv. Districts
Item # 10 -- Historical Presv. Districts
ahcitycouncil
 
Milton Keynes Hospital Charity - A guide to leaving a gift in your Will
Milton Keynes Hospital Charity - A guide to leaving a gift in your WillMilton Keynes Hospital Charity - A guide to leaving a gift in your Will
Milton Keynes Hospital Charity - A guide to leaving a gift in your Will
fundraising4
 
Donate to charity during this holiday season
Donate to charity during this holiday seasonDonate to charity during this holiday season
Donate to charity during this holiday season
SERUDS INDIA
 
World Food Safety Day 2024- Communication-toolkit.
World Food Safety Day 2024- Communication-toolkit.World Food Safety Day 2024- Communication-toolkit.
World Food Safety Day 2024- Communication-toolkit.
Christina Parmionova
 
PPT Item # 4 - 434 College Blvd. (sign. review)
PPT Item # 4 - 434 College Blvd. (sign. review)PPT Item # 4 - 434 College Blvd. (sign. review)
PPT Item # 4 - 434 College Blvd. (sign. review)
ahcitycouncil
 
AHMR volume 10 number 1 January-April 2024
AHMR volume 10 number 1 January-April 2024AHMR volume 10 number 1 January-April 2024
AHMR volume 10 number 1 January-April 2024
Scalabrini Institute for Human Mobility in Africa
 
Transit-Oriented Development Study Working Group Meeting
Transit-Oriented Development Study Working Group MeetingTransit-Oriented Development Study Working Group Meeting
Transit-Oriented Development Study Working Group Meeting
Cuyahoga County Planning Commission
 
State crafting: Changes and challenges for managing the public finances
State crafting: Changes and challenges for managing the public financesState crafting: Changes and challenges for managing the public finances
State crafting: Changes and challenges for managing the public finances
ResolutionFoundation
 
Monitoring Health for the SDGs - Global Health Statistics 2024 - WHO
Monitoring Health for the SDGs - Global Health Statistics 2024 - WHOMonitoring Health for the SDGs - Global Health Statistics 2024 - WHO
Monitoring Health for the SDGs - Global Health Statistics 2024 - WHO
Christina Parmionova
 
IEA World Energy Investment June 2024- Statistics
IEA World Energy Investment June 2024- StatisticsIEA World Energy Investment June 2024- Statistics
IEA World Energy Investment June 2024- Statistics
Energy for One World
 
CFYT Rolling Ads Dawson City Yukon Canada
CFYT Rolling Ads Dawson City Yukon CanadaCFYT Rolling Ads Dawson City Yukon Canada
CFYT Rolling Ads Dawson City Yukon Canada
pmenzies
 

Recently uploaded (20)

2024: The FAR - Federal Acquisition Regulations, Part 38
2024: The FAR - Federal Acquisition Regulations, Part 382024: The FAR - Federal Acquisition Regulations, Part 38
2024: The FAR - Federal Acquisition Regulations, Part 38
 
PUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptx
PUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptxPUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptx
PUBLIC FINANCIAL MANAGEMENT SYSTEM (PFMS) and DBT.pptx
 
加急办理华威大学毕业证硕士文凭证书原版一模一样
加急办理华威大学毕业证硕士文凭证书原版一模一样加急办理华威大学毕业证硕士文凭证书原版一模一样
加急办理华威大学毕业证硕士文凭证书原版一模一样
 
Border towns and spaces of (in)visibility.pdf
Border towns and spaces of (in)visibility.pdfBorder towns and spaces of (in)visibility.pdf
Border towns and spaces of (in)visibility.pdf
 
Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...
Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...
Preliminary findings _OECD field visits to ten regions in the TSI EU mining r...
 
About Potato, The scientific name of the plant is Solanum tuberosum (L).
About Potato, The scientific name of the plant is Solanum tuberosum (L).About Potato, The scientific name of the plant is Solanum tuberosum (L).
About Potato, The scientific name of the plant is Solanum tuberosum (L).
 
快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样
快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样
快速办理(UVM毕业证书)佛蒙特大学毕业证学位证一模一样
 
PPT Item # 5 - 318 Tuxedo Ave. (sign. review)
PPT Item # 5 - 318 Tuxedo Ave. (sign. review)PPT Item # 5 - 318 Tuxedo Ave. (sign. review)
PPT Item # 5 - 318 Tuxedo Ave. (sign. review)
 
原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样
原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样
原版制作(英国Southampton毕业证书)南安普顿大学毕业证录取通知书一模一样
 
Item # 10 -- Historical Presv. Districts
Item # 10 -- Historical Presv. DistrictsItem # 10 -- Historical Presv. Districts
Item # 10 -- Historical Presv. Districts
 
Milton Keynes Hospital Charity - A guide to leaving a gift in your Will
Milton Keynes Hospital Charity - A guide to leaving a gift in your WillMilton Keynes Hospital Charity - A guide to leaving a gift in your Will
Milton Keynes Hospital Charity - A guide to leaving a gift in your Will
 
Donate to charity during this holiday season
Donate to charity during this holiday seasonDonate to charity during this holiday season
Donate to charity during this holiday season
 
World Food Safety Day 2024- Communication-toolkit.
World Food Safety Day 2024- Communication-toolkit.World Food Safety Day 2024- Communication-toolkit.
World Food Safety Day 2024- Communication-toolkit.
 
PPT Item # 4 - 434 College Blvd. (sign. review)
PPT Item # 4 - 434 College Blvd. (sign. review)PPT Item # 4 - 434 College Blvd. (sign. review)
PPT Item # 4 - 434 College Blvd. (sign. review)
 
AHMR volume 10 number 1 January-April 2024
AHMR volume 10 number 1 January-April 2024AHMR volume 10 number 1 January-April 2024
AHMR volume 10 number 1 January-April 2024
 
Transit-Oriented Development Study Working Group Meeting
Transit-Oriented Development Study Working Group MeetingTransit-Oriented Development Study Working Group Meeting
Transit-Oriented Development Study Working Group Meeting
 
State crafting: Changes and challenges for managing the public finances
State crafting: Changes and challenges for managing the public financesState crafting: Changes and challenges for managing the public finances
State crafting: Changes and challenges for managing the public finances
 
Monitoring Health for the SDGs - Global Health Statistics 2024 - WHO
Monitoring Health for the SDGs - Global Health Statistics 2024 - WHOMonitoring Health for the SDGs - Global Health Statistics 2024 - WHO
Monitoring Health for the SDGs - Global Health Statistics 2024 - WHO
 
IEA World Energy Investment June 2024- Statistics
IEA World Energy Investment June 2024- StatisticsIEA World Energy Investment June 2024- Statistics
IEA World Energy Investment June 2024- Statistics
 
CFYT Rolling Ads Dawson City Yukon Canada
CFYT Rolling Ads Dawson City Yukon CanadaCFYT Rolling Ads Dawson City Yukon Canada
CFYT Rolling Ads Dawson City Yukon Canada
 

S-risks: Why they are the worst existential risks, and how to prevent them

  • 1. http://ea-foundation.org S-risks Max Daniel Executive Director, Foundational Research Institute Why they are the worst existential risks, and how to prevent them
  • 2. Shaping the long-term future 2 AI strategy and policy AI safetyBiosecurity
  • 3. Shaping the long-term future Bostrom 2002, Beckstead 2013 3 AI strategy and policy AI safetyBiosecurity Broad vs. narrow interventions (Beckstead 2013) AI forecasting Maxipok principle (Bostrom 2002) X-risk (Bostrom 2002)
  • 4. Shaping the long-term future Bostrom 2002, Beckstead 2013 4 AI strategy and policy AI safetyBiosecurity Broad vs. narrow interventions (Beckstead 2013) AI forecasting Maxipok principle (Bostrom 2002) X-risk (Bostrom 2002) Population ethics?? What is ‘wellbeing’??
  • 5. Shaping the long-term future Bostrom 2002, Beckstead 2013 5 AI strategy and policy AI safetyBiosecurity Broad vs. narrow interventions (Beckstead 2013) AI forecasting Maxipok principle (Bostrom 2002) X-risk (Bostrom 2002) Population ethics?? What is ‘wellbeing’??
  • 6. Based on Bostrom 2013, Figure 2 Scope Severity imperceptible endurable crushing (hellish) Fatal car crash personal local global trans-generational pan-generational (cosmic) X-risk
  • 7. Zooming in on existential risk Based on Bostrom 2013, Figure 2 cosmic pan- generational Extinction crushing hellish
  • 8. Zooming in on existential risk cosmic S-risk pan- generational Extinction crushing hellish
  • 9. What should we do about s-risks? Not controversial: • Reducing s-risk is desirable, all else being equal.
 But: Should reducing s-risks be a priority? • How likely are s-risks?
 • What interventions are there to reduce s-risks? Do they differ from those reducing extinction risk?
 • Who is working on such interventions?
  • 10. Probability? Technologies that enable … ● Sentient simulations ● Suffering subroutines ● Multiplying wild animal suffering ● Unknown unknowns WBE roadmap: Sandberg & Bostrom (2008)
  • 11. What s-risks are there? By type of intent: ● No intent: accidents, indifference, evolution … ● Evil intent ● Strategic intent: conflicts, threats, bribes, …
  • 12. Tractability? Targeted interventions ● (Some work in) AI safety ● (Some work in) AI policy Broad interventions ● International cooperation, expanding moral circle (?) ● Going meta: research anti-s-risk strategies Suffering-focused AI safety (Gloor 2016)
  • 13.
  • 14. On the neglectedness of s-risk ● Not totally neglected, but gets little attention ● Misconception: existential risk = extinction risk ● The Foundational Research Institute is the only organization that specifically focuses on reducing s-risk
  • 15. Summary: why and how to prevent s-risk ● Whether to focus on s-risk depends on both empirical and value judgments ● S-risks not much more unlikely than AI-related extinction risk ● Some, but not all, familiar work on x-risk also reduces s-risk ● Reducing s-risks seems neglected
  • 16. How shaping the far future does not look like
  • 17.
  • 18. Presentation title John Smith | Head of Department 28.06.2016 Subtitle or caption Thank you. More on s-risk? Check out foundational-research.org max@foundational-research.org | Add me on Facebook! :)