T11. Normative multi-agent systems

Tutorial Normative

Reasoning and Decision Making

Guido Boella & Leon van der Torre

Agreement Technologies

• WG1: “Semantics”
• WG2: “Norms”
• WG3: “Organizations”
• WG4: “Argumentation and Negotiation”
• WG5: “Trust”

2

Some Norms
• You are obliged to return the book in 2 weeks
• Role X is permitted to access resource Y
• Actions of agent X count as actions of agent Y
• Advocates should act in the interest of their clients
• Violations should be sanctioned
• Don’t do to them what you don’t want them to do to you
• People should know the law
• Every adult has the right to vote
• Make your homework!
• We start at 09:00

3

Merriam-Webster Online Dictionary
• ‘normative’: ‘conforming to or based on norms’,
– as in normative behavior, normative judgments
– not: ‘of, relating to, or determining norms or standards’, as in
normative tests, or ‘prescribing norms’, as in normative rules
of ethics or normative grammar.
• ‘norm’: ‘a principle of right action binding upon the
members of a group and serving to guide, control, or
regulate proper and acceptable behavior’.
– Not: ‘an authoritative standard or model’, ‘an average like a
standard, typical pattern, widespread practice or rule in a
group’, and various definitions used in mathematics.

4

Normative Systems (1971)
“When a deductive correlation is such that the
first sentence of the ordered pair is a case and
the second is a solution, it will be called
normative. If among the deductive correlations of
the set there is at least one normative
correlation, we shall say that the set has
normative consequences. A system of sentences
which has some normative consequences will be
called a normative system.”
C. Alchourron and E. Bulygin. Normative Systems. Springer, 1971.

5

Normative Systems (1993)
• Normative systems are “systems in the behavior
of which norms play a role and which need
normative concepts in order to be described or
specified”
J.-J. Meyer and R. Wieringa. Deontic Logic in Computer Science: Normative System Specification. John Wiley & Sons, 1993.

• Many distinct notions of “normative systems”
– Social expectation, legal law, linguistic imperative…
• Role of norms in computer science is changing
– Solutions based on multiagent systems increasing

6

Layout
• Challenges due to uncertainty and imprecision
in normative reasoning and decision making

1. Normative reasoning in computer science
2. Imprecision & uncertainty normative systems
3. Decision making in normative systems

2. Uncertainty and imprecision in normative reasoning
3. Decision making in normative systems 7

AgentLink RoadMap (2003)

8

2005: Normative MultiAgent Systems
• are multiagent systems with normative systems
• in which agents can decide whether to follow the
explicitly represented norms, and
• the normative systems specify how and in which
extent the agents can modify the norms.’’
G. Boella, L. van der Torre, H. Verhagen, Introduction to normative multiagent
systems. Computational and Mathematical Organization Theory, double
special issue on normative multiagent systems, 2006.

“The normchange definition”


2007: Normative MultiAgent System
• is a multiagent system organized by means of
• mechanisms to represent, communicate,
distribute, detect, create, modify, and enforce
norms, and
• mechanisms to deliberate about norms and
detect norm violation and fulﬁllment.”
G. Boella, L. van der Torre, and H. Verhagen, Introduction to the special issue
on normative multiagent systems. Autonomous Agents and MultiAgent
Systems, Aug. 2008.
“The mechanism design definition”


Layout



Norms: Certain and Precise?
• Norms are rules to distinguish right from wrong.

• Dilemmas: when may we shoot down a hijacked plane?

Violations: Certain and Precise?
• Norms describe what counts as a violation.

• Budget deficit > 3 % of GDP is a violation?

Design: Uncertain and Imprecise
• Norms are rules to guide, control or regulate behavior.

• What are the effects of new norms on behavior?

Guide, Control, Regulate Behavior

What’s new with respect to constraints?


Guide, Control, Regulate Behavior
Ways to deal with violations, representation of permissive norms,
the evolution of norms over time (in deontic logic), the relation
between the cognitive abilities of agents and the global properties
of norms, how agents can acquire norms, how agents can violate
norms, how an agent can be autonomous (in normative agent
architectures and decision making), how norms are created by
legislator, emerge spontaneously or are negotiated among agents,
how norms are enforced, how constitutive or counts-as norms are
used to describe institutions, how norms are related to social and
legal concepts, how norms structure organizations, how norms
coordinate groups and societies, how contracts are related to
contract frames & contract law, how legal courts are related, how
normative systems interact, …

Normative Reasoning

Norms are rules to guide, control or regulate behavior.

Infrastructure for open communities.
Reasoning in open environments.
Trust and reputation.

More than distinguishing right from wrong.
More than describing what counts as a violation.
More than constraints.
Uncertain and imprecise.

Security and Reliability
• Norms make systems more secure & reliable.

• How to prevent over-regulation?

Trust
• Norms build up trust in international trade.

ESCROW

• Which mechanisms for electronic commerce?

Example: ESCROW
• Norms describing coordination
– Agent communication: sales agreement
– Agent transaction: payment for shipment
• Norms describing fees and compensation
– ESCROW gets percentage (from seller or buyer)
– Return merchandize within 5 days
• Norms ensure fulfillment of some obligations
– Seller is paid for goods, or goods will be returned
– (under assumption of trustworthiness of ESCROW)
• Some obligations can still be violated:
– Seller does not ship goods, shipper unreliable, …

4. Norm change 20

Virtual Communities
• Social norms are emerging in Second Life.

• How to prevent exclusion from communities?

Autonomous Systems
• aut- + nomos: making ones own norms.

• How to define global policies about local ones?

Applications Normative Reasoning

Norms are rules used to control or regulate behavior.

Risk management for computer security.
Designing trust mechanisms for electronic commerce.
Recognition of emerging social norms in Second Life.
Local and global policies for autonomous systems.


Layout



Examples: How to decide …
• … whether to fulfill or violate a norm?
• … whether behavior counts as a violation?
• … whether an excuse for a violation is acceptable?
• … whether a violation should be sanctioned?
• … whether to accept a norm / contract / … ?
• ... whether to accept sanctions of a norm?
• … which contract to propose / norm to create?
• … whether an agent conforms to the norm?
• … whether a procedure is compliant to the norm?
• … on a constitution for Europe?

Basis: Decision Making With Norms
• Introducing obligations in agent architectures
•
tl O tl
tl al cl
hl act
obs B I P
D
obligations as normative goals
• Introducing obligations in agent programming


Exercise: When Violate a Norm?
• When you are not able to fulfill it
• When the sanction is too low
• When the probability to be caught is too low
• …


Exercise: When Violate a Norm?
• When you are not able to fulfill it
• When the sanction is too low
• When the probability to be caught is too low
• When you don’t know the norm
• When there is another more important desire
• When there is another more important obligation
• When you have a good excuse
• When you like to take risks
• When other agents violate the norm
• When the norm does not make sense

When to Violate/Fulfill: Challenge
• Multiagent structure of a normative system
– Many agents involved in agent interaction
– Interaction is highly structured

• Normative system regulates their roles
– Powers of roles described by counts-as norms, and
– Their behavior is regulated by procedural norms.

Norm Compliance
• Attacker = agent trying to profit from violation
– Violation undetected, no sanction, or less than profit

normative system Attacker / adversary

$$

• Anticipate rather than regiment (mechanism design)

Profile NS of Attacker A (for Op)
1. NS desires p (“your wish is my command”).
2. Absence of p is considered as violation of A.
! Anderson’s reduction of deontic logic to modal logic.
3. NS desires that there are no violations.
4. If violation, then NS is motivated to sanction.
5. NS does not like to sanction.
6. A does not like being sanctioned.
7. NS has the power to count absence of p as violation.
8. NS has the power to enforce sanction.
Example: You are obliged to return the book in 2 weeks
Question: Why do we need all of them? Is it complete?

When to Accept a Contract?

normative system trustee

$$
trustor
• Decision making of trustor taking profiles of
trustee and normative system into account


Challenge: When to Accept a Norm
•Immanuel
Kant,
Metaphysics
of Morals,
1785

• “Act only according to that
maxim whereby you can at
the same time will that it
should become a universal
law.”


Emergence of Norms
• Don’t do to them, what you don’t want them to
do to you
• Social delegation cycle
– Agent desires
– Merging: Social goal
– Planning: Norm
– Acceptance: Agent desires
• I accept if you fulfill the norm on the
assumption that all others fulfill it

Autonomous Systems

• Legal institutions in context of legal institutions
– Norm dynamics described by counts-as norms
• Strong permissions in norm creation (Bulygin)
• Global policies about local policies
– Von Wright: transmission of will by nested norms


Hierarchical Normative Systems

Global authority Norm. 2

$$
Norm. 1
• Decision making of norm. system 1 taking profiles of
norm. system 2 and global authority into account


Decision Making in NMAS

Norms are rules used to control or regulate behavior.

Norm conformance and compliance.
Multiagent structure of normative systems.
Norm acceptance for emergence of norms.
Hierarchical normative systems for autonomy.

37

Concluding Remarks


38

http:deonticlogic.org

39

Further Reading
• Normative multiagent systems proceedings:
– Computational and Mathematical Organization Theory, double special
issue on normative multiagent systems, 2006.
– Normative Multiagent Systems, 18.03. - 23.03.2007, Schloss Dagstuhl,
Germany, 2007.
– Autonomous Agents and Multiagent Systems Journal, special issue on
normative multiagent systems, August 2008.
– 4rd International Workshop on Normative Multiagent Systems
(NORMAS2009), Dagstuhl, March 2009
• Introductions to the area and its challenges
– Ten philosophical problems in deontic logic
– Ten challenges for normative multiagent systems
– Ten guidelines for norms in computer science

40

Ten Philosophical Problems
1. How can deontic logic be reconstructed in accord with the philosophical position that
norms are neither true nor false?
2. When is a set of norms to be termed `coherent'?
3. How can deontic logic accommodate possible conflicts of norms? How can the
resolution of apparent conflicts be semantically modeled?
4. How do we reason with contrary-to-duty obligations which are in force only in case of
norm violations?
5. How to define dyadic deontic operators with regard to given sets of norms and facts?
6. How to distinguish various kinds of permissions and relate them to obligations?
7. How can meaning postulates and intermediate terms be modeled in semantics for
deontic logic reasoning?
8. How to define counts-as conditionals and relate them to obligations and permissions?
9. How to revise a set of regulations or obligations? Does belief revision offer a
satisfactory framework for norm revision?
10. Can the belief merging framework deal with the problem of merging sets of norms?

J. Hansen, G. Pigozzi and L. van der Torre, Ten philosophical problems in deontic logic, Dagstuhl DROPS proceedings 07122, 2007.

41

Ten Challenges for NorMAS
1. Tools for agents supporting communities in their task of recognizing, creating, and
communicating norms to agents.
2. Tools for agents to simplify normative systems, recognize when norms have
become redundant, and to remove norms.
3. Tools for agents to enforce norms. In a distributed approach, roles should be
defined for agents in charge of monitoring and sanctioning.
4. Tools for agents to preserve their autonomy.
5. Tools for agents to construct organizations.
6. Tools for agents to create intermediate concepts and normative ontology, for
example to decide about normative gaps.
7. Tools for agents to decide about norm conflicts.
8. Tools for agents to voluntarily give up some norm autonomy by allowing automated
norm processing in agent acting and decision making
9. Tools for conviviality.
10. Tools for legal responsibility of the agents and their principals.

G. Boella, L. van der Torre, H. Verhagen, Introduction to the special issue on normative
multiagent systems. Autonomous Agents and MultiAgent Systems, Aug 2008.
42

Guideline 1
1. Motivate which definition of normative
multiagent system is used.

• Norms explicitly represented in system
– (the ‘strong’ interpretation: too strict?)
• Norms explicitly represented in specification
– (the ‘weak’ interpretation: too general?)
– Norm compliance, norm implementation, ...
• Something else

44

Guideline 2
2. Make explicit why norms are a kind of (soft)
constraints deserving special analysis.

• Ways to deal with violations, representation of permissive norms, the
evolution of norms over time (in deontic logic), the relation between the
cognitive abilities of agents and the global properties of norms, how agents
can acquire norms, how agents can violate norms, how an agent can be
autonomous (in normative agent architectures and decision making), how
norms are created by legislator, emerge spontaneously or are negotiated
among agents, how norms are enforced, how constitutive or counts-as norms
are used to describe institutions, how norms are related to social and legal
concepts, how norms structure organizations, how norms coordinate groups
and societies, how contracts are related to contract frames & contract law,
how legal courts are related, how normative systems interact?

45

Guideline 3
3. Explain why and how norms can be changed at
runtime.

• E.g., legislators and voting on acceptance,
observe behavior and violations to modify…

46

Guideline 4
4. Discuss the use and role of norms as a
mechanism in a game-theoretic setting.

• D. Lewis “master and slave” game
• E. Bulygin “rex, minister and subject” game
• G. Boella c.s.: violation games, institutionalized
games, negotiation games, norm creation
games, control games Norms are rules specifying violation games.

47

Guideline 5
5. Clarify role of norms in the multiagent system.

• Norms guide (…) desired system behavior

• Norms are incentives to motivate agents
– Gneezy and Rustichini’s daycare example
• Norms organize systems
– Modularity, abstractions

48

Guideline 6
6. Relate the notion of “norm” to the legal,
social, or moral literature.

Five phases in normas design:
1.off-line norm design
2.norm representation
3.norm manipulation
4.social reality
5.moral reality
49

Guideline 7
7. Use norms not only to distinguish right from
wrong, but also to resolve dilemmas, and use
norms not only describe violations, but in
general to coordinate, organize, guide, regulate
or control interaction among agents.

50

Guideline 8
8. Distinguish norms from obligations, prohibitions
and permissions.

• Deontic logic: logical relations obligations, etc
– normative system is typically left implicit
• Two distinct philosophical traditions
– Von Wright: norms and normative propositions
– Alchourron: prescriptive and descriptive obligations

J. Hansen. Imperatives and Deontic Logic. PhD thesis, University of Leipzig, 2008.

51

Guideline 9
9. Use the deontic paradoxes only to illustrate the
normative multiagent system.

1. A certain man should go to the assistance of his
neighbors,
2. If he goes, he should tell them he is coming
3. If he does not go, he should not tell them that he is
coming
4. He does not go.
{Oa,O(a ! t),¬a ! O(¬t),¬a} {Oa, a ! O(t),¬a ! O(¬t),¬a}

52

Guideline 10
10. Consider regulative norms in relation to other
kinds of norms and concepts.

• in relation to permissive norms, constitutive
norms, procedural norms, agents, roles, groups,
societies, rights, duties, obligations, time,
beliefs, desires, intentions, goals, roles, and
other kinds of norms and other social-cognitive
computer science concepts.

53

The Question
• Could (or should) “norms” play a similar role in
computer science like “service”, “contract” or “trust”?
– Since the use of norms is a key element of human social
intelligence, norms may be essential too for artificial agents
that collaborate with humans, or that are to display behavior
comparable to human intelligent behavior.
– Norms are thought to ensure efficiency at the level of the
multiagent system whilst respecting individual autonomy.
• We have to build more flexible normative multiagent
systems, and test them in practice, before we know
where they can be used best.

54

Objects, relationships, and roles

Each modelling language has its own reference ontology.

The duality of objects and relationships is present in all mod-
elling languages. E.g., variables and constants vs predicates
and relations.

But there is another basic ontological distinction: roles.

Roles, ﬁrst appearance

Bachman and Daya 1977’s role DB data model:

“Records and n-tuples are role oriented. These deal with
employees, customers, patients, or students: role types. This
is in contrast with DB theory: each record should represent
all aspects of some one entity in the real world. The reason
for confusion is that neither the roles of the real world nor
the entities of the real world are a subset of the other”

“If members of a set are records of various types, large pieces
of redundant code had to be written in order to address
the semantically equivalent, but syntactically distinct fields
(items) of the members when iterating through a set.

By letting all the entity types involved play the same role and
by defining the fields as fields of the role, not of the types, it
became possible to process all members with the same piece
of code.”

Natural types vs roles

Sowa 1984’s conceptual structures distinguish between natu-
ral types “that relate to the essence of the entities” and role
types “that depend on an accidental relationship to some
other entity”: child depends on parent, pet on owner, etc.

Sowa: role types are subtypes of natural types. Child, Pet,
Quotient and Food are subtypes of the natural types Person,
Animal, Number and Physical-Substance.

Properties of roles in Object Orientation

from Steimann 2000

1. A role comes with its own properties and behavior. Hence,
it is a type. E.g., a director of a department commands
other members and makes buy-orders.

2. Roles depend on relationships: e.g., an employee is re-
lated to an employer, a student to a school, etc.

3. Objects of unrelated types can play the same role. E.g.,
both a person and an organization are customers.

4. An object may play diﬀerent roles. E.g., a person can be
a student and an employee.

5. An object may play the same role several times. E.g., a
person can hold several employments.

6. An object may acquire and abandon roles dynamically.
E.g., a person can acquire the role of student or of em-
ployee.

7. The sequence in which roles may be acquired and relin-
quished can be subject to restrictions. E.g., a person
becomes a teaching assistant only if it is a student.

8. Roles can play roles. E.g., an employee can be a project
leader, a role of the employee.

9. A role can be transferred from one object to another.
E.g., the salary of an open position may be speciﬁed
independently of the person that will be employed.

10. The state of an object can vary depending on the role in
which it is being addressed: this should be viewed as a
separate instance of the object. E.g., an employee has
an address per job and also a private one.

11. If an object plays several of roles simultaneously, it re-
sponds according to the role in which it is being ad-
dressed. E.g., an person gives the address of the em-
ployee it is playing.

12. Roles restrict access. This corresponds to an object hav-
ing different perspectives, facets, or aspects. E.g., the
private phone number of an employee can be invisible
when the person is playing the employee role.

13. Different roles may share structure and behavior: role
definitions inherit from each other. E.g., the role first
year student can inherit the behavior of giving exams from
the role student.

14. An object and its roles share identity. Since roles do not
exist by themselves they cannot have an identity.

15. An object and its roles have different identities. E.g., the
number of passengers of an airline can be greater than
the number of person who travelled with it.

Possible views of roles

1. Roles as named places in relationships.

2. Roles as speciﬁcations or generalizations of natural types.

3. Roles as adjunct instances.

Role as a named place in a relationship

If two places of a relationship are of the same type, then
the type would not suﬃce to distinguish them. Role names
should serve to identify the places.

Every place of a relationship has a role name in the ER model
(and UML).

For example, the is-parent-of (or is-child-of ) relationship is
suﬃciently characterized by the role names parent and child.

However, it fails to account for the fact that roles come with
their own properties and behavior.

Roles as specializations

If roles are types, how are the roles and the natural types
related?

A role type is more speciﬁc than the natural type of its role
players: if Customer and Supplier are subtypes of Person.
But, not only persons, but also organizations like companies,
etc., can be customers and suppliers. However, declaring
Customer and Supplier as subtypes of both Person and Or-
ganization is a problem.

Roles as generalizations

If roles are not subtypes, could they be supertypes?

No, if roles are more speciﬁc than the types of their players.

But this is false, anyway: a person has many properties not
required of a customer or supplier. Rather, being a customer
or supplier imposes its required properties on persons and
organizations, making the former supertypes of the latter.

From the extensional point of view, roles are supertypes stat-
ically, while dynamically they are subtypes.

Roles as adjunct instances

Roles as independent types the instances of which are carriers
of role speciﬁc state and behavior, but not identity.

An object and its roles are related by a special played-by rela-
tion, and an object and its roles form an aggregate (subject).

Picking up a role corresponds to the creation of a new in-
stance of the corresponding role type and its integration into
the compound.

Ontological properties of roles

Following Sowa, Guarino and Welty 2002 presented an onto-
logical distinction between role and natural types.

A role is a concept that is:

• founded: the individuals of a role stand in relation to
other individuals,

• lacks semantic rigidity: they can enter and leave the ex-
tent of the concept without losing their identity.

A natural type, on the other hand, is characterized by:

• semantic rigidity: an individual of a natural type cannot
drop this type without losing its identity,

• lack of foundation: for an individual to belong to that
type it is not required that it stands in relationship to
others.

E.g., Person is a natural type, Student is a role.

Adolescent and Adult are states or phases, lacking both se-
mantic rigidity and foundation.

Dependence of roles

Fine 1995’s notion of dependence: “to say that an object x
depends upon an F is to say that an F will be ineliminably
involved in any definition of x”.

Masolo et al. 2004’s definitional dependence: e.g., the def-
inition of the concept of student makes reference not to a
specific school but to the concept of school, etc.

Handbook Norms Change Permission Institution Agency

Deontic Logic as Normative Reasoning

Leon van der Torre

Individual and Collective Reasoning Group
Computer Science and Communication
University of Luxembourg

EASSS 2012 (Valencia)

May 2012

Leon van der Torre Deontic Logic as Normative Reasoning


1 Handbook

2 Problem 1 - In what sense are obligations diﬀerent from norms?

3 Problem 2 - How do norms change?

4 Problem 3 - How to relate various kinds of permissions?

5 Problem 4 - What is the role of constitutive norms?

6 Problem 5 - Where does deontic logic meet game theory?



Volume 1: Deontic logic and imperatives

A. BACKGROUND
1. R. Hilpinen and P. McNamara, Deontic logic: history, overview.
2. J. Hansen, Imperative logic: history, overview.
B. DEONTIC PROBLEMS
3. S. Hansson, Varieties of permission.
4. M. Sergot, Contrary-to-duty norms.
5. L. Goble, Prima facie norms, normative conﬂicts and dilemmas.
6. M. Sergot, The theory of normative positions.
7. A.J.I. Jones and D. Grossi, Constitutive norms and counts as conditionals.
8. P. McNamara, Logics for supererogation and allied normative concepts.
C. CANDIDATES FOR A NEW STANDARD
9. S. Hansson, Alternative semantics for deontic logic.
10. X. Parent and L. van der Torre, Input/output logic.
11. J. Hansen, Imperativist approach to deontic logic.
12. L. Lindahl and J. Odelstad, TJS. Normative systems with intermediaries.



Volume 2: Norms in systems, and interactions
D. DYNAMICS
1 ?, Time and deontic logic. 2. J. Broersen, J. Horty and J.-J. Ch. Meyer,
Actions and deontic logic. 3. G. Boella, G. Pigozzi and L. van der Torre,
Normative system change. 4. J. van Benthem and F. Liu, Deontic logic and
changing preferences. 5. D. Gabbay, Reactive approaches.
E. INTERACTIONS
6 R. Thomason, The formalization of practical reasoning: problems and
prospects. 7 B. Kooi and A. Tamminga, Deontic logic and game theory. 8 K.
van Fintel and T. Gillies, Deontic logic and natural language. 9 G. Sartor, G.
Governatori, and A. Rotolo. Deontic logic and legal reasoning.
F. INTEGRATION INTO SYSTEMS
10. D. Garg, Modalities for access control. 11. J. Lobo, E. Lupu and A. Russo,
Distributed systems management. 12. F. Cuppens and N. Cuppens-Boulahia,
Obligations for usage control. 13. G. Aucher and G. Boella, Privacy and
epistemic obligations. 14. R. Parikh, L. Oldeloohuisl, and C. Baskent,
Epistemic norms. 15. G. Governatori, Rule, Norms and compliance.



In what sense are obligations diﬀerent from norms?

BQ Can norms have truth values? (Jorgensen, 1938) Imperatives,
promises, legal statutes, and moral standards are usually not
viewed as being true or false. E.g.: “John, leave the room!”
BQ Is a logic of norms possible? Without truth, norms cannot:
be premise or conclusion in an inference,
be termed consistent or contradictory, or
be compounded by truth-functional operators.
BQ Can deontic logic be built on descriptive obligations only?
(Makinson, 1998) turns the Bad Questions into a Good Question:
Problem 1. How can deontic logic be reconstructed in accord with
the philosophical position that norms are neither true nor false?

Explained in following slides in three steps.



Step 1: Traditional Deontic Logic, axiomatic (1951)

SDL = normal modal logic of type KD
extends the propositional tautologies with the axioms
K : O(x → y ) → (Ox → Oy ) and D : ¬(Ox ∧ O¬x), closed
under modus ponens x, x → y /y and Necessitation x/Ox
Prohibition & permission deﬁned Fx = O¬x and Px = ¬O¬x.
BQ Which representation for conditional norms O(x|a)?
O(x|a) = O(a → x), or
O(x|a) = a → Ox, or something else
BQ Factual or deontic detachment?
Factual detachment: (O(x|a) ∧ a) → Ox
Deontic detachment: (O(x|a) ∧ Oa) → Ox



Step 2: Traditional Deontic Logic, possible worlds (1969)

DSDL3 = dyadic modal logic
Kripke models W , ≥ where ≥ is total pre-order
O(x|a) iﬀ x holds in all the most preferred a worlds
Axiomatized by Lewis (counterfactuals, 1973) and Spohn
BQ Which logical relations hold among conditional norms?
BQ Factual or deontic detachment?
Factual detachment: (O(x|a) ∧ a) → Ox
Deontic detachment: (O(x|a) ∧ Oa) → Ox
BQ How are logical relations and detachment related?
For example, transitivity and deontic detachment



Step 3: Modern Deontic Logic, iterative (1998)

Makinson’s iterative detachment approach
Normative code N is pairs of propositional formulas (a, x)
Gross output: out(N, A) = apply N as rules on A
E.g., E0 = ∅,En+1 = En ∪ {x|(a, x) ∈ N, En ∪ A implies a}
Net output: consistency constraint for output
E.g., En+1 = En ∪ {x|(a, x) ∈ N, En ∪ A implies a} if cons.
En otherwise
GQ How to detach obligations from conditional norms?
Factual detachment: ((a, x) ∧ a) → Ox
Deontic detachment: ((a, x) ∧ Oa) → Ox
BQ How are logical relations and detachment related?



Candidates for a New Standard
GQ Which approaches do not ﬁt into this general framework?
GQ What are the alternatives to this iterative framework?

Algebraic Programming

Non-Monotonic
a:b/Oc

Reactive
Iterative
a in out(C,b)
Diagnostic
q / ¬V(n) ! p Labeled Imperativistic
Op!p,Oq!q ! O(p/q)!p,!q !p,!q ! O(p/q)
Dynamic
Input/Output

K O



Problem 1 - Some Good Questions

GQ When is a normative system coherent?
GQ When is a norm redundant?
GQ When are two normative systems equivalent?
GQ How are the three questions above related?
GQ Which meta-logical properties can or should hold?
Law2case principle
Strong case2law principle
GQ How to prevent the pragmatic oddity
without creating the drowning problem?



Problem 2 - How do norms change?




Problem 1 - In what sense are obligations diﬀerent from norms?
GQ How is obligation change diﬀerent from norm change?
How to formalize the relation between them?

Obligations can change while normative system remains same
Detachment of obligation, delegation, etc
A relatively clear and well studied subject (in 70s and 80s)
Norm change: the new challenge (workshops 2007, 2010)
Norm revision and contraction, e.g. change of legal code
Norm evolution, e.g. change of social norms
Merging normative systems, e.g. merger of companies




Problem 2.1. How to revise a set of norms?

Alch´urron Makinson [1981] study changes of legal code:
o
addition
amendment: revision
derogation: contraction
Legal code is a non-empty and ﬁnite set of propositions.
Starting point of belief revision (with G¨rdenfors), a.k.a. AGM
a
AGM 1985: minimality represented by recovery postulate
More advanced variants, e.g. for iterated belief revision
Also some issues with prioritized norms we do not consider
Defeasible deontic logic derived from non-monotonic logic



AGM 85

C. E. Alchourrón, P. Gärdenfors, and D. Makinson. On the logic of theory change: Partial
meet contraction and revision Leon van der Torre Log., 50(2):510–530, 1985. Reasoning
functions. J. Symb. Deontic Logic as Normative


AGM 85



Problem 2.1 - How to revise a set of norms?
BQ Do the AGM postulates hold for norm revision?
E.g., success postulate criticized in belief revision
GQ Does AGM oﬀer a satisfactory framework for norm revision?
Role of minimal change in norm change
Revision of conditional norms
Example
If we have {(, a), (a, b)} and we have that c is an exception to
the obligation to do b, then we need to have b ∈ out(N, c). Two
solutions seem to be {(¬c, a), (a, b)} or {(, a), (a ∧ ¬c, b)}.
GQ What triggers the change of a norm?
AGM says a new or removed norm
GQ Do general patterns in the revision of norms exist?
If so, how to formalize them?
GQ Derogation is contraction? How about annulment?



GQ How to formalize the evolution of (social) norms?
E.g., norm about phone calls during meetings
No persons who change the norm like in legal code
Change of social norm triggered by norm violations
Social delegation cycle: relates agent desires and social goals
GQ How to merge sets of norms?
Problem only recently addressed in the literature.
GQ What is norm merging?
1 Merging norms that belong to the same normative system
2 Merging norms that belong to diﬀerent systems



Merging norms: same normative system

Example (Cholvy Cuppens 1999)
An organization that works with secret documents has two rules.
R1 = It is obligatory that any document containing some secret
information is kept in a safe, when nobody is using this document.
R2 = If nobody has used a given document for ﬁve years, then it is
obligatory to destroy this document by burning it.

To derive a contradiction we need to add an IC that keeping a
document and destroying it are contradictory actions
⇒ The notion of coherence between norms can involve
information other than norms.

Is this a genuine problem of merging norms?



Merging norms: diﬀerent systems

Example (Gr´goire, 2004)
e
The Belgian-French bilateral agreement preventing double taxation:
Resident in B ∧¬ France State Worker ⇒ Taxable in B
Resident in B ∧¬ Work in France ⇒ Taxable in B
Belief merging would simply give a set containing the two initial
rules. But if someone is not a State worker in France (but works in
France), we would get contradictory requirements from the
Belgium and the France parties.



Problem 2.3 - How to merge sets of norms?

Belief merging (a.k.a. belief fusion) is a ‘generalization’ of belief
revision. Belief merging dedicated to the combination of
information coming from different (conflicting) sources.

GQ Can belief merging offer a satisfactory framework for norms?
GQ Do distinctions in belief merging correspond to something?
Belief merging distinguishes majoritarian and egalitarian
operators.
They try to capture the intuitions that often guide the
aggregation of individual preferences into a social one.
These intuitions seem to have nothing to say when we try to
model the merging of sets of norms.
GQ What are alternative frameworks for norm merging?



Problem 3 - Permissive norms

BQ How to solve the following problem with permissions:
Example (Ross’ paradox)
Px implies P(x ∨ y ), e.g., if a person is permitted to smoke, he is
also permitted to smoke or kill.

BQ How to formalize free choice permissions:
Example (Free choice permission)
P(x ∨ y ) implies Px ∧ Py , e.g., “You may either sleep on the
sofa-bed or sleep on the guest room bed” implies “You may sleep
on the sofa-bed and you may sleep on the guest room bed”.




BQ Is permission unilateral (Op → Pp) or bilateral (Pp → ¬Op)?
BQ What is diﬀerence between weak (negative, tacit) and strong
(positive, implicit and explicit) permission?
von Wright (1951): Permission is dual of obligation
Px =def ¬O¬x (weak permission)
Negative permission deﬁned without permissive norms
Px =def ¬x ∈ out(N, A) (weak permission)

BQ Are permissive norms necessary for normative systems?
Example (Bulygin 1986)
Permissive norms are needed when there multiple authorities. If a
higher authority permits you, a lower authority cannot prohibit it.




GQ What is the diﬀerence between permission and authorization?
Example (Makinson1986)
A priest may be authorized to marry people, in the sense that his
actions may have that eﬀect, without being permitted to do so.

GQ What is a right?
Example
If you have the right to enter, then a guard must let you in.

Concept of permission is multi-faceted.

Problem 3. How to relate various kinds of permissions and relate
them to obligations?



GQ How to formalize relation permissions and permissive norms?
Something is positively permitted iﬀ the code explicitly
presents it. However, permissions follow from explicit ones.
GQ How do permissive norms inﬂuence obligations?
Obligations imply permissions, but how about the other way?
GQ What is role of permissive norms in norm change / merging?



Example
Consider a set G consisting of the norm (work, tax), and a set P
consisting of (18y, vote). Does it follow from our mini-code that
voting is permitted on condition of being employed?

In one sense yes: there is nothing in the code that forbids it
(negative permission).
In another sense no: a person may be employed at the age of
17 and not covered by the explicit permission (positive static
permission).
But in another sense, yes again. For if we were to forbid a
person to vote on the condition of being employed we would
be creating an incoherence in the code, in its application to
people who are both employed and 18 or over (dynamic
positive permission).



Kinds of permissions used in different circumstances:
Negative permissions seem to answer to the needs of the
citizen, who needs to now that his action is not forbidden.
Since it’s difficult to establish these kinds of permissions (also
because of ambiguities in the code), positive static
permissions seem to suit better the need to know what is
explicitly permitted.
Lastly, dynamic permissions are the answers to the need of the
legislator, who need to anticipate the effect of changing an
existing corpus of norms by adding a prohibition.



Problem 4 - What is the role of constitutive norms?

Normative system contains definitions, expressions of purposes,
conceptual rules, etc.: we call them constitutive norms.

Example
“An act of theft is punished by a prison sentence not exceeding 5
years or a fine.”

GQ What is the role of constitutive norms? (What can’t we do if
we have only regulative and permissive norms)
Meaning postulates and intermediate concepts
Creation of social reality, counts-as conditionals
Legal and institutional powers of agents
Defining the way normative systems can change
Interpretation issues
...


Example meaning postulates and intermediate concepts
“An act of theft is punished by a prison sentence not exceeding 5
years or a ﬁne.”
“Someone commits an act of theft if that person has taken a
movable object from the possession of another person into his
own possession with the intention to own it, and if the act
occurred without the consent of the other person or some
other legal authorization.”
A person in the sense of the law is a human being that has
been born.
A movable object is any physical object that is not a person or
a piece of land.
A movable object is in the possession of a person if that
person is able to control its uses and location.
The owner of an object is within the limits of the law entitled
to do with it whatever he wants, namely keep it, use it,
transfer possession or ownership of the object to another
person, and destroy or abandon it.


Meaning postulates and intermediate concepts

Intermediate concepts link legal terms to terms that describe
natural facts.

GQ Why are there intermediate concepts?
Wedberg Ross (1950s): a legal term like ’ownership’ (x is
the owner of y at time t) serves the purpose of economy of
expression of a set of legal rules.
To regulate normative change and legal interpretation by judge
To resolve conﬂicts



Problem 4 - Meaning postulates and intermediate concepts

Example (Intermediates to resolve conflicts)
Suppose that (, ¬dog) forbids dogs on the premises
There is also a higher order principle that no blind person may be
required to give up her guide dog. The judge cannot change the
statute. What he can do is to conclude that the statute does not
apply to guide dogs. So the statute must be re-interpreted as
reading (, ¬tdog) with the additional intermediate
(dog ∧ ¬guidedog, tdog) ∈ T , and thus no conflict arises for the
case of blind persons that want to keep their guide dog.

GQ What is the exact process of creating and modifying
theoretical terms in order to resolve conflicts?



Problem 4 - Constitutive norms

GQ How do regulative, permissive and constitutive norms interact?



Problem 4 - Constitutive norms

There is no consensus on the logic of counts-as conditionals,
probably due to the fact that the concept is not studied in
depth yet.
For example, the adoption of the transitivity rule T for Jones
Sergot’s logic is criticized by Artosi, Rotolo Vida: They
claim that non-monotonicity is a crucial feature of count-as
conditionals:
Example: Suppose that in an auction if the agent x raises one
hand, this may count as making a bid. It is clear that this
does not hold if x raises one hand and scratches his own head.



Problem 5 - Where does deontic logic meet game theory?


Where Does Deontic Logic Meet
Game Theory?
Leon van der Torre
EASSS 2012

1

Why do we need deontic detachment?
•  Thomason 1980:
–  Context of justification (e.g. a judge, looking back)
–  Context of deliberation (normative decision making)
•  Consider Chisholm’s paradox without DD
–  Day after tomorrow you must help neighbors
–  Tomorrow you must tell
•  S.O. Hansson: If you don’t tell you help, then
you never received “moral cue for action” to tell

R. Thomason, “Deontic logic as founded on tense logic. In New Studies in 2
Deontic Logic, R. Hilpinen, ed., D. Reidel, Dordrecht, 1981, pp. 165-176.

Quick History Practical Reasoning
•  Aristotle: theoretical practical reasoning
•  40s Practical Reas. splits into two branches
–  Classical decision and game theory in economics
–  Bounded rationality and means-end reasoning in AI
•  90s unification in multi-agent systems
–  Russell and Norvig’s “modern approach” to AI
–  Shoham’s DTGT foundations of MAS
•  Deontic logic: obligations treated as goals in AI
–  Distinguished from theory of normative systems
3

NORMAS
•  Deontic logic and normative reasoning are one
•  They should be studied both from a bounded
rationality or AI perspective, and from a DTGT
or economic perspective: they should be
studied from a multi-agent perspective
–  The DL handbook: “a modern approach”
•  Consequences:
–  Action and time: DTGT model
–  Normative decision theory: BOID

4

NORMAS
•  State of art in action and time Deontic Logic
•  Same situation as in preference-based DL:
–  Semantics is relatively clear and intuitive
–  Languages and proof systems are to be improved
•  All you need to know action and time in DL is in
Horty’s “Agency and Deontic Logic” (2001)

J. Broersen, L. van der Torre: What an Agent Ought To Do. Artificial Intelligence 5
and Law 11(1): 45-61 (2003)

DTGT vs Deontic Logic
•  Models are the same, choices over branching
time, with only one distinction:
–  DTGT: each agent has its own utility
–  DL: there is a single global utility

•  Savage’s sure thing principle plays central role
•  There exist more detailed DTGT models and
we can use them as more detailed DL models

6

The Gambling Problem
•  Obligation to be and obligation to do
–  Not interdefinable
•  Granularity of indeterministic actions

7

The Driving Example
•  Assume a collision
•  Dominance act utilitarianism
–  There was no dominant action
•  Orthodox perspective on the agent's ought
–  You should have changed lanes
•  The issue of moral luck (Horty, 2001, p.121).
–  Orthodox ought: agent looks back in time
–  Dominant: agent may regret, but not be blamed
•  What if utility of 5 and 3? (Whiff and Poof)
8

Procrastinate's Choice
•  Field = agent’s reasoning is limited to subtree
–  “Intention” in BDI agent theory?
•  Possibilist interpretation
–  It is possible to write the review, so he accept it
•  Strategic interpretation
–  He will not write it, so he should not accept it

9

A Challenge
•  The bystander effect:
–  One person watches a child in water and saves it
–  One hundred people watch child and it drowns
•  How to reason about this in deontic logic?
–  If deontic logic is going to play a role in ethics, it has
to say something about examples like this

10

A Challenge
•  The bystander effect:
–  One person watches a child in water and saves it
–  One hundred people watch child and it drowns
•  How to reason about this in deontic logic?
–  If deontic logic is going to play a role in ethics, it has
to say something about examples like this
•  Introduce normative systems in DTGT model
–  Needs an explicit mechanism of norm creation

11

Summary
•  How to combine deontic logic game theory?
–  Mechanism design, norm is a mechanism
–  Introduce norms in Horty’s model

•  Computational methods for deontic logic
–  Algorithms, complexity, proof systems

12

T11. Normative multi-agent systems

Recommended

Recommended

More Related Content

Similar to T11. Normative multi-agent systems

Similar to T11. Normative multi-agent systems (20)

More from EASSS 2012

More from EASSS 2012 (8)

Recently uploaded

Recently uploaded (20)

T11. Normative multi-agent systems