Decision Theory Research at FRI

Effective Altruism Foundation
Effective Altruism FoundationEffective Altruism Foundation
Johannes Treutlein
Foundational Research Institute
Decision theory research
at FRI
Johannes Treutlein
Foundational Research Institute
A wager for evidential decision
theory
Altruistic Newcomb problem
3
Ω
?
one
wish
predicts one-boxing:

two wishes
predicts two-boxing:

nothing
Altruistic Newcomb problem
4
S1 S2
A1 2 0
A2 3 1
● A1: One-box; A2: Two-box
● S1: opaque box contains two wishes; S2: opaque box empty
Evidential decision theory
5
Causal decision theory
6
Meta decision theory
7
(Nozick 1993; MacAskill 2016)
8
Altruistic Newcomb problem in a large
universe
Ω
Ω
Ω
Ω
Ω
Ω
Ω
Altruistic Newcomb problem in a large
universe
9
EDT Wager
10
● Large universe
● Caring about the gains of our copies
● Non-zero credence in EDT
● Meta decision theory
Wager for evidential decision theory (and all other theories that
take impact of copies into account)
Relevance
11
● AI Safety
● Macrostrategy
● Multiverse-wide superrationality (Oesterheld 2017a)
Caspar Oesterheld 

Foundational Research Institute
Decision theory and approval-
directed agents
Implementing decision theories in AIs
13
• Two problems of decision theory in AI safety:
• What is the right decision theory for an AI?
• How do we implement decision theories in AI?
• Decision theory not explicit in AI architecture
• Example: Doing what has worked well in the past (Oesterheld
2017b)
• Exception: Gödel machine (Schmidhuber 2006)
Approval-directed agency
14
(Christiano 2014)
Two decision theories
15
Two decision theories
16
Example
17
Two decision theories
18
Example
19
20
In the paper…
If overseer only looks at the world, the agent’s DT is
decisive.
If overseer only looks at the agent’s action, the
overseer’s DT is decisive.
Presentation title
John Smith | Head of Department 28.06.2016
Subtitle or caption
Thank you.
{johannes,caspar}@foundational-research.org
References
22
• Ahmed, A. (2014): Evidence, Decision and Causality. Cambridge University Press.
• Almond, P. (2010): On Causation and Correlation. Part 2: Implications of Evidential
Decision Theory. https://casparoesterheld.files.wordpress.com/2017/03/
correlation2.pdf
• Bostrom, N. (2014b): Superintelligence: Paths, Dangers, Strategies. Oxford
University Press.
• Christiano, P. (2014): Model-free decisions. https://ai-alignment.com/model-free-
decisions-6e6609f5d99e
• MacAskill, W. (2016): Smokers, Psychos, and Decision-Theoretic Uncertainty. The
Journal of Philosophy
• Nozick, R. (1993): The Nature of Rationality. Princeton: Princeton University Press
References
23
• Oesterheld, C. (2017b): Doing what has worked well in the past leads to evidential
decision theory. https://casparoesterheld.files.wordpress.com/2017/09/learningdt.pdf
• Oesterheld, C. (2017a): Multiverse-wide Cooperation via Correlated Decision
Making. https://foundational-research.org/files/Multiverse-wide-Cooperation-via-
Correlated-Decision-Making.pdf
• Schmidhuber, J. (2006): Gödel Machines: Self-Referential Universal Problem Solvers
Making Provably Optimal Self-Improvements. ftp://ftp.idsia.ch/pub/juergen/gm6.pdf
• Soares, N. and Fallenstein, B. (2014a): Aligning Superintelligence with Human
Interests: A Technical Research Agenda. MIRI Tech. rep. 2014-8. https://
intelligence.org/files/TechnicalAgenda.pdf
• Soares, N. and Fallenstein, B. (2014b): Toward Idealized Decision Theory. MIRI
Tech. rep. 2014-7. https://arxiv.org/abs/1507.01986
• Soares and Levinstein (2017): Cheating Death in Damascus. https://intelligence.org/
files/DeathInDamascus.pdf
1 of 23

Recommended

APS GDS data science talk by Trevor Rhone by
APS GDS data science talk by Trevor RhoneAPS GDS data science talk by Trevor Rhone
APS GDS data science talk by Trevor RhoneTrevorDavidRhone
223 views35 slides
5. Workshop Responsible Data Science - Discussion on Transparency in data sci... by
5. Workshop Responsible Data Science - Discussion on Transparency in data sci...5. Workshop Responsible Data Science - Discussion on Transparency in data sci...
5. Workshop Responsible Data Science - Discussion on Transparency in data sci...Jheronimus Academy of Data Science
391 views10 slides
3. Workshop Responsible Data Science - Discussion on Accuracy in data science... by
3. Workshop Responsible Data Science - Discussion on Accuracy in data science...3. Workshop Responsible Data Science - Discussion on Accuracy in data science...
3. Workshop Responsible Data Science - Discussion on Accuracy in data science...Jheronimus Academy of Data Science
312 views5 slides
BanditProblems_final by
BanditProblems_finalBanditProblems_final
BanditProblems_finalShweta Gupte
563 views29 slides
Connecting Research Questions and Research Method by
Connecting Research Questions and Research MethodConnecting Research Questions and Research Method
Connecting Research Questions and Research MethodUniversity of Stuttgart
1K views55 slides
META-SPACE: Psycho-physiologically Adaptive and Personalized Virtual Reality ... by
META-SPACE: Psycho-physiologically Adaptive and Personalized Virtual Reality ...META-SPACE: Psycho-physiologically Adaptive and Personalized Virtual Reality ...
META-SPACE: Psycho-physiologically Adaptive and Personalized Virtual Reality ...Advanced-Concepts-Team
72 views26 slides

More Related Content

Similar to Decision Theory Research at FRI

Whether simulation models that fall under the information systems category ad... by
Whether simulation models that fall under the information systems category ad...Whether simulation models that fall under the information systems category ad...
Whether simulation models that fall under the information systems category ad...Elisavet Andrikopoulou
371 views24 slides
Bias and the Data Lifecycle by
Bias and the Data LifecycleBias and the Data Lifecycle
Bias and the Data LifecycleRichard Ferrers
488 views24 slides
PO 397 Introduction to Social Science Research by
PO 397 Introduction to Social Science Research PO 397 Introduction to Social Science Research
PO 397 Introduction to Social Science Research atrantham
67 views24 slides
R m 101 by
R m 101R m 101
R m 101Magdy Mahdy
916 views135 slides
2022_Fried_Workshop_theory_measurement.pptx by
2022_Fried_Workshop_theory_measurement.pptx2022_Fried_Workshop_theory_measurement.pptx
2022_Fried_Workshop_theory_measurement.pptxROBERTOENRIQUEGARCAA1
6 views122 slides
chapter-3.pptx by
chapter-3.pptxchapter-3.pptx
chapter-3.pptxAsmaRauf5
16 views60 slides

Similar to Decision Theory Research at FRI(20)

Whether simulation models that fall under the information systems category ad... by Elisavet Andrikopoulou
Whether simulation models that fall under the information systems category ad...Whether simulation models that fall under the information systems category ad...
Whether simulation models that fall under the information systems category ad...
PO 397 Introduction to Social Science Research by atrantham
PO 397 Introduction to Social Science Research PO 397 Introduction to Social Science Research
PO 397 Introduction to Social Science Research
atrantham67 views
chapter-3.pptx by AsmaRauf5
chapter-3.pptxchapter-3.pptx
chapter-3.pptx
AsmaRauf516 views
Leibniz: A Digital Scientific Notation by khinsen
Leibniz: A Digital Scientific NotationLeibniz: A Digital Scientific Notation
Leibniz: A Digital Scientific Notation
khinsen512 views
Engineering design of an environmental management system: A trans-disciplinar... by Henk (Jan) Roodt
Engineering design of an environmental management system: A trans-disciplinar...Engineering design of an environmental management system: A trans-disciplinar...
Engineering design of an environmental management system: A trans-disciplinar...
Henk (Jan) Roodt403 views
West-Vanderbilt-Talk--Revised-22March2017.ppt by kait23
West-Vanderbilt-Talk--Revised-22March2017.pptWest-Vanderbilt-Talk--Revised-22March2017.ppt
West-Vanderbilt-Talk--Revised-22March2017.ppt
kait231 view
Presentation on-Resarch-paradigms.pptx by BenjaminKumi
Presentation on-Resarch-paradigms.pptxPresentation on-Resarch-paradigms.pptx
Presentation on-Resarch-paradigms.pptx
BenjaminKumi15 views
ISWC2015 Opening Session by Steffen Staab
ISWC2015 Opening SessionISWC2015 Opening Session
ISWC2015 Opening Session
Steffen Staab1.4K views
Modelling Innovation – some options from probabilistic to radical by Bruce Edmonds
Modelling Innovation – some options from probabilistic to radicalModelling Innovation – some options from probabilistic to radical
Modelling Innovation – some options from probabilistic to radical
Bruce Edmonds352 views
Scientific mind (nov. 2016-feb.2017) by scientificmind
Scientific mind (nov. 2016-feb.2017)Scientific mind (nov. 2016-feb.2017)
Scientific mind (nov. 2016-feb.2017)
scientificmind531 views
Borner - Modelling science technology and innovation by innovationoecd
Borner - Modelling science technology and innovationBorner - Modelling science technology and innovation
Borner - Modelling science technology and innovation
innovationoecd487 views
Learning Local Lessons in Software Engineering by CS, NcState
Learning Local Lessons in Software EngineeringLearning Local Lessons in Software Engineering
Learning Local Lessons in Software Engineering
CS, NcState464 views

More from Effective Altruism Foundation

Current Status of the EA Movement by
Current Status of the EA MovementCurrent Status of the EA Movement
Current Status of the EA MovementEffective Altruism Foundation
268 views26 slides
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltung by
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltungNationale Volksinitiative zur Abschaffung der Massen­tierhaltung
Nationale Volksinitiative zur Abschaffung der Massen­tierhaltungEffective Altruism Foundation
116 views17 slides
The New Meat by
The New MeatThe New Meat
The New MeatEffective Altruism Foundation
114 views24 slides
Lessons from Building an EA Charity: New Incentives by
Lessons from Building an EA Charity: New IncentivesLessons from Building an EA Charity: New Incentives
Lessons from Building an EA Charity: New IncentivesEffective Altruism Foundation
137 views25 slides
What Does (and Doesn't) AI Mean for Effective Altruism? by
What Does (and Doesn't) AI Mean for Effective Altruism?What Does (and Doesn't) AI Mean for Effective Altruism?
What Does (and Doesn't) AI Mean for Effective Altruism?Effective Altruism Foundation
167 views52 slides
Delivering Development Impact at Scale by
Delivering Development Impact at ScaleDelivering Development Impact at Scale
Delivering Development Impact at ScaleEffective Altruism Foundation
86 views16 slides

More from Effective Altruism Foundation(20)

Recently uploaded

2023 Q1-Q2 Newsletter - First Tee Puerto Rico by
2023 Q1-Q2 Newsletter - First Tee Puerto Rico2023 Q1-Q2 Newsletter - First Tee Puerto Rico
2023 Q1-Q2 Newsletter - First Tee Puerto RicoFirst Tee Puerto Rico
35 views15 slides
UNiTE- Invest to Prevent Violence against Women & Girls! by
UNiTE- Invest to Prevent Violence against Women & Girls!UNiTE- Invest to Prevent Violence against Women & Girls!
UNiTE- Invest to Prevent Violence against Women & Girls!Christina Parmionova
13 views4 slides
AABS project overview by
AABS project overviewAABS project overview
AABS project overviewWorldFish
23 views17 slides
How to Find Contractors and Architects for Your Historic Home Renovation by
How to Find Contractors and Architects for Your Historic Home RenovationHow to Find Contractors and Architects for Your Historic Home Renovation
How to Find Contractors and Architects for Your Historic Home RenovationNational Trust for Historic Preservation
100 views8 slides
Strategic Planning & Managment by
Strategic Planning & ManagmentStrategic Planning & Managment
Strategic Planning & ManagmentJo Balucanag - Bitonio
5 views31 slides
Support Girl students with Education by
Support Girl students with EducationSupport Girl students with Education
Support Girl students with EducationSERUDS INDIA
6 views6 slides

Recently uploaded(20)

UNiTE- Invest to Prevent Violence against Women & Girls! by Christina Parmionova
UNiTE- Invest to Prevent Violence against Women & Girls!UNiTE- Invest to Prevent Violence against Women & Girls!
UNiTE- Invest to Prevent Violence against Women & Girls!
AABS project overview by WorldFish
AABS project overviewAABS project overview
AABS project overview
WorldFish23 views
Support Girl students with Education by SERUDS INDIA
Support Girl students with EducationSupport Girl students with Education
Support Girl students with Education
SERUDS INDIA6 views
Mukhya Mantri Gramin Peyjal Nishchay Yojana (MGPNY) – Bihar_Pankaj Kumar_AKRS... by India Water Portal
Mukhya Mantri Gramin Peyjal Nishchay Yojana (MGPNY) – Bihar_Pankaj Kumar_AKRS...Mukhya Mantri Gramin Peyjal Nishchay Yojana (MGPNY) – Bihar_Pankaj Kumar_AKRS...
Mukhya Mantri Gramin Peyjal Nishchay Yojana (MGPNY) – Bihar_Pankaj Kumar_AKRS...
Dr. Ousmane Badiane - 2023 ReSAKSS Conference.pptx by AKADEMIYA2063
Dr. Ousmane Badiane - 2023 ReSAKSS Conference.pptxDr. Ousmane Badiane - 2023 ReSAKSS Conference.pptx
Dr. Ousmane Badiane - 2023 ReSAKSS Conference.pptx
AKADEMIYA20637 views
COP 28 GHANA DELEGATES.docx by Kweku Zurek
COP 28 GHANA DELEGATES.docxCOP 28 GHANA DELEGATES.docx
COP 28 GHANA DELEGATES.docx
Kweku Zurek640 views
Managing drinking water infrastructure in West Bengal Gram Panchayats_Sujata ... by India Water Portal
Managing drinking water infrastructure in West Bengal Gram Panchayats_Sujata ...Managing drinking water infrastructure in West Bengal Gram Panchayats_Sujata ...
Managing drinking water infrastructure in West Bengal Gram Panchayats_Sujata ...
Social behavioural change to drive community ownership_ Divyang Waghela_Tata ... by India Water Portal
Social behavioural change to drive community ownership_ Divyang Waghela_Tata ...Social behavioural change to drive community ownership_ Divyang Waghela_Tata ...
Social behavioural change to drive community ownership_ Divyang Waghela_Tata ...
Arrow Adoption Training for Kinship Families by ArrowMarketing
Arrow Adoption Training for Kinship FamiliesArrow Adoption Training for Kinship Families
Arrow Adoption Training for Kinship Families
ArrowMarketing40 views
Ms. Julie Collins - 2023 ReSAKSS Conference.pptx by AKADEMIYA2063
Ms. Julie Collins - 2023 ReSAKSS Conference.pptxMs. Julie Collins - 2023 ReSAKSS Conference.pptx
Ms. Julie Collins - 2023 ReSAKSS Conference.pptx
AKADEMIYA206310 views
Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference by AKADEMIYA2063
Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference
Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference
AKADEMIYA20635 views
ΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑ by ssuser9e6212
ΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑ
ΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑ
ssuser9e6212170 views
Case study of Gokarna Multi-village scheme, Kumta, Karnataka_IIM-B_2023.pdf by India Water Portal
Case study of Gokarna Multi-village scheme, Kumta, Karnataka_IIM-B_2023.pdfCase study of Gokarna Multi-village scheme, Kumta, Karnataka_IIM-B_2023.pdf
Case study of Gokarna Multi-village scheme, Kumta, Karnataka_IIM-B_2023.pdf

Decision Theory Research at FRI

  • 1. Johannes Treutlein Foundational Research Institute Decision theory research at FRI
  • 2. Johannes Treutlein Foundational Research Institute A wager for evidential decision theory
  • 3. Altruistic Newcomb problem 3 Ω ? one wish predicts one-boxing:
 two wishes predicts two-boxing:
 nothing
  • 4. Altruistic Newcomb problem 4 S1 S2 A1 2 0 A2 3 1 ● A1: One-box; A2: Two-box ● S1: opaque box contains two wishes; S2: opaque box empty
  • 7. Meta decision theory 7 (Nozick 1993; MacAskill 2016)
  • 8. 8 Altruistic Newcomb problem in a large universe Ω Ω Ω Ω Ω Ω Ω
  • 9. Altruistic Newcomb problem in a large universe 9
  • 10. EDT Wager 10 ● Large universe ● Caring about the gains of our copies ● Non-zero credence in EDT ● Meta decision theory Wager for evidential decision theory (and all other theories that take impact of copies into account)
  • 11. Relevance 11 ● AI Safety ● Macrostrategy ● Multiverse-wide superrationality (Oesterheld 2017a)
  • 12. Caspar Oesterheld 
 Foundational Research Institute Decision theory and approval- directed agents
  • 13. Implementing decision theories in AIs 13 • Two problems of decision theory in AI safety: • What is the right decision theory for an AI? • How do we implement decision theories in AI? • Decision theory not explicit in AI architecture • Example: Doing what has worked well in the past (Oesterheld 2017b) • Exception: Gödel machine (Schmidhuber 2006)
  • 20. 20 In the paper… If overseer only looks at the world, the agent’s DT is decisive. If overseer only looks at the agent’s action, the overseer’s DT is decisive.
  • 21. Presentation title John Smith | Head of Department 28.06.2016 Subtitle or caption Thank you. {johannes,caspar}@foundational-research.org
  • 22. References 22 • Ahmed, A. (2014): Evidence, Decision and Causality. Cambridge University Press. • Almond, P. (2010): On Causation and Correlation. Part 2: Implications of Evidential Decision Theory. https://casparoesterheld.files.wordpress.com/2017/03/ correlation2.pdf • Bostrom, N. (2014b): Superintelligence: Paths, Dangers, Strategies. Oxford University Press. • Christiano, P. (2014): Model-free decisions. https://ai-alignment.com/model-free- decisions-6e6609f5d99e • MacAskill, W. (2016): Smokers, Psychos, and Decision-Theoretic Uncertainty. The Journal of Philosophy • Nozick, R. (1993): The Nature of Rationality. Princeton: Princeton University Press
  • 23. References 23 • Oesterheld, C. (2017b): Doing what has worked well in the past leads to evidential decision theory. https://casparoesterheld.files.wordpress.com/2017/09/learningdt.pdf • Oesterheld, C. (2017a): Multiverse-wide Cooperation via Correlated Decision Making. https://foundational-research.org/files/Multiverse-wide-Cooperation-via- Correlated-Decision-Making.pdf • Schmidhuber, J. (2006): Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements. ftp://ftp.idsia.ch/pub/juergen/gm6.pdf • Soares, N. and Fallenstein, B. (2014a): Aligning Superintelligence with Human Interests: A Technical Research Agenda. MIRI Tech. rep. 2014-8. https:// intelligence.org/files/TechnicalAgenda.pdf • Soares, N. and Fallenstein, B. (2014b): Toward Idealized Decision Theory. MIRI Tech. rep. 2014-7. https://arxiv.org/abs/1507.01986 • Soares and Levinstein (2017): Cheating Death in Damascus. https://intelligence.org/ files/DeathInDamascus.pdf