Kenneth D. Forbus, Walter P. Murphy Professor of Computer Science and Professor of Education at Northwestern University, gave a wonderful presentation “Software Social Organisms: Implications for measuring AI progress” as part of our Cognitive Systems Institute Speaker Series.

1. Software Social Organisms:
Implications for Measuring AI Progress
Kenneth D. Forbus
Northwestern University

2. Big Picture
• Achieving human-level AI = learning how to
create sufficiently smart software social
organisms
• No single test will be sufficient to measure
progress
• Evaluation should be ability to participate in our
culture
– Apprentices
– Provides multiple dimensions within which progress
can be measured
• Modalities, range of domains, levels of knowledge…

3. Today’s AI
systems
can be fast and
effective.
But they requires
careful design for
narrow
circumstances,
maintenance by
highly trained
personnel
What if AI
systems
were as robust,
trainable, and
taskable as
mammals?

4. Companion Cognitive Architecture
• Approach: Understand how to build Software
social organisms that
– Work with people using natural modalities
– Learn and adapt over extended periods of time
– Maintain themselves
– Capable of relating to people as apprentices, not tools
• Companion architecture incorporates other
hypotheses as well
– Centrality of analogy, qualitative representations
• But the idea of software social organisms is more
general, and is a path to human-level AI

5. Why Organisms?
• Autonomy
– We do not have direct access to the internal
representations of children or our collaborators
– We communicate via language, sketching, …
• Adaptation
– Incremental learning in everyday life
– Building up knowledge over weeks, months, years
• New interest in interactive task learning
Kirk & Laird,
2013
Hinrichs & Forbus
2013

6. Why Social?
• Being social is a driver of intelligence
– e.g. Tomasello’s research
• Cultural knowledge is central in human life
– We can all think about things we never experienced
• Plate tectonics, the Revolutionary War, …
– Stories are an important component of human
learning
• e.g. Winston’s research
• Smart organisms want to cooperate and
collaborate as well as compete
– AI has started to model this, e.g. Grosz’s research

7. Human-level AI
• Will end up being social organisms
• Sufficiently smart software social organisms will
be human-level AIs
– Sufficiently smart = capable of learning to perform a
sufficiently broad range of tasks that people do, with
the same or better levels of performance

8. Apprenticeship
• Natural trajectory for bringing people into a role
– Also provides one trajectory for evaluations
• Start as a student
– Lots of “book learning”, interaction
– Tests to gauge learning
– Legitimate peripheral participation
• Expand range of responsibilities
– Joint work
– Negotiated roles
• Autonomous operation with a community
– Performs well on its own, or with others at their level
– Can ask for help, and help others

9. Substrate Capacities Needed for Human-like Social
Organism
• Autonomy
– Its own needs, drives, and capabilities for acting and learning
• Operates in environments that support shared focus
– Each participant has some information about what others can sense
– Participants can make focus of attention known to each other easily
• Language sufficient to express beliefs, hypotheticals
• Ability to build models of the intent of others
– Implies learning the types of goals they can have, and how available
actions feed into those goals
– Models of needs, drives as wellsprings of particular goals
• Strong interest in interactions with conspecifics
– Helping and teaching

10. Implications for Evaluation
• No single test will work
– Won’t measure adaptability and breadth
– Can always be hacked
• Need to evaluate via ability to participate in our
culture
– Do real work
– Interact with people appropriately
– Learn and adapt about its work, others, and itself
– Without AI experts constantly fiddling with its internals

11. Scalable Evaluation Dimensions
• Modalities
– Text, sketching, speech, vision, mobility, …
• Starting point
– Nothing versus some prior knowledge versus lots
• Level of domain knowledge
– Elementary school  Professional life
• Range of tasks responsible for
– Single task versus multiple
– Interactive task learning
• Learning ability
– About its job, its people, and itself
– Amount and quantity of data required

12. Summary
• Human level AI = sufficiently smart software
social organisms
• Apprenticeship trajectory provides framework for
many kinds of evaluation
– Initially a student, learning from instructors through
carefully designed exercises
– Working as an assistant to a mentor, with increasing
responsibility as it learns
– Working autonomously, reporting back periodically for
debriefing, review, and retasking
Check out QRGLab channel on
YouTube