A keynote talk at a Ubicomp 2014 workshop. This talk looks at the opportunities for social science due to ubiquitous computing and offers some techniques for problem finding, problem formulation and problem reframing.

1. Mobile Systems for Computational Social Science:
A Perfect Storm
John Charles Thomas!
!Problem Solving International!
UbiComp, Seattle WA!
13 September, 2014
1

2. Interacting Factors for Perfect Storm
• Smaller, Cheaper, Faster, Lower Power Computing!
• Smaller, Cheaper Sensors and Effectors!
• Smart Phone Growth!
• Globalization!
• Shorter Cycles and Productivity Press —>
Continuous Measurement !
• “Classical” Statistics —> Big Data Analytics;
Imputation; Monte Carlo Simulations; Random
Forest; AI Techniques. !
• Laboratory Studies —> Field Observations and
Measures!
• “Simple” Theories —> Complex Theories!
• These interact in positive feedback loops; e.g.,
complex theories + faster computing + more data
—> theories can be more quickly refined.
2

3. What are the limiting factors in mobile computing for social
science?
❖ Not compute speed!
❖ Not sensors!
❖ Not effectors!
❖ Not cost!
❖ Not power requirements!
❖ Only Imagination….
3

4. Normative Model of Development: All these “arts” can be
instrumented, studied, and improved.
❖ Problem Finding!
❖ Problem Formulation!
❖ Component Solution!
❖ Integration!
❖ Reality Check — Reframing.!
❖ Design!
❖ Development!
❖ Deployment!
❖ Post-Mortem on Process and Product
4

5. Google Hits
❖ “Problem Solving” - 63M!
❖ “Problem Solving Techniques” - 470K!
❖ “Problem Finding” - 3M!
❖ “Problem Finding Techniques” - 14K!
❖ “Problem Formulation” - 730K!
❖ “Problem Formulation Techniques” - 39K!
❖ “Problem Reframing” - 8K!
❖ “Problem Reframing Techniques” - 6
5

6. Early Studies of Query Languages (1974)
❖ Query By Example showed great improvement over IQF in a user’s ability to
translate questions from English into formal query language: !
❖ IQF 4-24 hours training; QBE < 3 hours training!
❖ Ave. T/Query in IQF 5-12 min; QBE 1.6 min.!
❖ % correct IQF: 35%; QBE 67%!
❖ BUT: When given a series of problems and a DB description and asked to write
their own relevant query and translate into QBE, users could not do it.!
❖ Answered question (without being able to look at any actual data!).!
❖ Wrote (and translated) irrelevant queries.!
❖ Wrote “HAL+” queries.
6

7. Some Methods of Community Knowledge Generation and Sharing
(besides math models at one extreme and opinion at the other extreme)
❖ From General to Particular:
Story!
❖ From Particular to General:
Patterns and Pattern Language!
❖ Reframing: Context Generation
7

8. Problem Finding by Observation of Patterns of Behavior that
Violate your Expectations (often non-linearities)
❖ Random Drops Become a Lake!
❖ Table Tennis Club Destruction
8

9. Abstraction: Random Drops Become a Lake
❖ Main tendency but with variation !
❖ Extreme outliers have qualitatively different
behavior!
❖ The behavior of the extreme outliers changes
the field; in particular, makes the probability
of other extreme outliers increase!
❖ Another example from The Power of Positive
Deviance: How Unlikely Innovators Solve the
World’s Toughest Problems. Childhood hunger
in Vietnam.!
❖ In this case, the positive feedback loop did
not exist without intervention. !
❖ Ubiquity could be used to help find such
“positive deviance”
9

10. Two Tables (Blue) supports stable community of 20-30 people every noon
What happens with One Table?
❖ H1: Community will stay at about 20-30 people (the
interesting in table tennis trumps facility).!
❖ H2: Community will diminish to about 15-20 people
(the facility will not support so many people).

11. 30
22.5
15
7.5
0
Two Tables (Blue) supports stable community
One Table (Green) does not support community (feedback disruption)
Monday Tuesday Wednesday Thursday Friday Monday Tuesday
11

12. Problem Finding from Story
❖ Stories deal with the “edges” of human
experience!
❖ Stories thrive on conflict !
❖ Stories thrive on emotion!
❖ Follow the Anger back to source of
frustration: A problem to be solved.!
❖ In stories, typically it is the determination,
cleverness, or bravery of the hero that
saves the day.!
❖ However, they often have a special power
or gift: Make that a reality. !
❖ Or, “re-write” the story so that the
problem(s) can still be solved, but by
“ordinary” people.
12

13. Stories tend to focus on the “edges”
of human experience
(Note similarity to Patterns of
Behavior that Violate Expectations)
13

14. Stories can be viewed as three-dimensional:
14

15. 15

16. Problem Finding Examples:
❖ Pets do not always do what they should. Reinforcement works, but owners
are busy and away. S: Remote monitoring and delivery of reinforcement. !
❖ Home objects have instructions that are illegible. S: Mobile phone could
“read” what the device is and display legible instructions. !
❖ Plant signs are ambiguous. S: Photo sent to service which returns four
similar pictures with names and links. !
❖ New inventions promise wonders but lack convincing experiential
evidence. S: !
❖ Waiting turn for haircut is a pain. Plus, hard to describe how short you
want your hair to be cut. S: While waiting, iteratively choose haircut view
on based on your photo.
16

17. ❖ Pets do not always do what they should. Reinforcement works, but owners
are busy and away. S: Remote monitoring and delivery of reinforcement.
❖ Planning the next !
“Catastrophe”
17

18. Home objects have instructions that are illegible. S: Mobile phone
could “read” what the device is and display legible instructions.
❖ Top view: Bose DVD player!
❖ Bottom view: Home thermostat!
❖ The “real” objects are just this!
difficult to read.!
Mobile device also allows a UX!
“intervention point” for updates,!
different languages, large print, etc
18

19. Example: Computerizing a Chair by Story-izing
❖ Components of a Chair: Back, Seat,
Legs!
❖ Material of a Chair: Fabric, Wood,
Metal, Rubber!
❖ Purpose to Which Chairs are Put:
Relaxation, Socialization, Work.!
❖ History of the Chair: Desires,
Acquiring information, Designing,
Raw Materials, Component
Construction, Assembly,
Transportation, Preparation of
Materials, Packaging, Sales, Wear
(what fails? under what conditions?),
Disposal?
19

20. Playing with the Character Dimension of Story
❖ Person —> Group, Team, Friends, Family,
Clan, State, Nation, World, All Life…!
❖ Person —> Role, Mood, Age, Job, Hobby,
As Relation, Time of Day, Time of Year…!
❖ Special Needs —> Sight, Hearing, Touch,
Coordination, Germ Free….!
❖ Sight —> Lack of glare, slow changes in
illumination, large type, slow change of
focus…!
❖ Situations —> Going on a family trip;
attending a sporting event; shopping for a
house; choosing a restaurant….!
❖ Values —> Theoretical, Religious,
Practical, Experiential, Social…
20

21. ❖ New inventions promise wonders but lack convincing
experiential evidence. S: ??
❖ Grill cleaner, new skates,!
mosquito hood and jacket,!
rain barrel !
❖ What do these feel like?!
❖ How long do they last?!
❖ What are maintenance issues?!
❖ Will this still seem cool when !
I am not at 40,000 feet and have!
just had 3 martinis?!
❖ What if all these inventions were
instrumented BOTH for continuous
improvement AND so potential buyers could
see how they actually performed?
21

22. Plantar Fasciitis
❖ Which one of these products
do I buy to fix my plantar
fasciitis? !
❖ Cheapest? !
❖ Most expensive?!
❖ Most stars?!
❖ Doctor prescribed high
powered anti-inflammatory
and stop exercising!
❖ Solution?
22

23. Remove pebble from shoe
23

24. According to Judy Mod, founder of Social Executive Council
❖ Companies who produce and
sell focus most of their energy
on “beating the competition”
on price, performance,
features, etc.!
❖ For IT system decisions,
10-20% of lost sales prospects
are to competition.!
❖ 80-90% are lost to “no
decision”
24

25. Companies do “Market Research” but …
❖ Largely constrain the nature of the presumed problem up
front.!
❖ Study with ecologically invalid methods (e.g., “New Coke”).!
❖ Focus on beating the competition. !
❖ Focus on selling the product…but cannot see what it “looks
like” from the customer’s viewpoint.!
❖ “It’s a clown. It is smiling. It has big eyes. It has all the
features that our research shows are correlated with
cuteness. It has to be cute!”
25

26. 26

27. Ubiquitous Computing Allows:
❖ Studying in situ both physically (in the
small and in the large) and “socially” !
❖ Caveat: Still subject to interpretation!
❖ Pattern: Reality Check!
❖ Which one is the “real” desk?
27

28. Pattern: Reality Check
❖ Often something easy to
measure is highly correlated
with what you really want to
measure.!
❖ You measure this “ersatz”
measure.!
❖ But, the correlation may change
over time. (e.g., programming
skill and speed).!
❖ Therefore, you need to
periodically do a reality check.
28

29. Solving a Problem; Reframing a Problem
❖ TRIZ!
❖ Subtracting a Constraint!
❖ Solving Successive
Subproblems!
❖ Work from and Transcend
Apparent Contradiction!
❖ Adding a Constraint!
❖ Reframing by Adding !
Context (story technique)!
❖ Iroquois “Rule of Six”
29

30. Where’s Jonathan?! Supposed to be here at 8:00; now 8:15!
❖ He doesn’t care about the project!!
❖ OR….Your appointment book
has the wrong time.!
❖ OR…Your watch is wrong.!
❖ OR…Jonathan comes from a
culture where 8:15 is not late.!
❖ OR…Jonathan was waylaid in
the hall by the CEO to talk about
the project. !
❖ OR…You are in the wrong room.
30

31. Generalizing the Solution
❖ Social Pattern: “Who Speaks
for Wolf?”!
❖ Spatial Pattern: Context-Setting
Entrance!
❖ Information Pattern:
Clarification Graffiti!
❖ Temporal Pattern: Small
Successes Early
31

32. Social Pattern: “Who Speaks for Wolf?”!
❖A lot of effort and thought goes into
decision making and design.
❖Nonetheless, it is often the case that bad
decisions are made and bad designs
conceived and implemented primarily
because some critical and relevant
perspective has not been brought to bear.
❖ This is especially often true if the
relevant perspective is that of a
stakeholder in the outcome.
❖ Therefore, make sure that every relevant
stakeholder’s perspective is brought to
bear early.
32

33. Spatial Pattern: Context-Setting Entrance!
❖ Because people function in many different contexts
and come from many different backgrounds and
cultures, there are a wide variety of behaviors that
are considered “appropriate” in various
circumstances.
❖ Sometimes, we are expected to compete with each
other vigorously. Other times, we are expected to be
highly cooperative.
❖ When our own expectations are violated, we may
feel resentful, angry, or afraid. When we violate what
we later find to be the expectations of others, we may
feel embarrassed or resentful.
❖ We don’t want to be the only person at a party to
show up in a tux while everyone else is in blue jeans
--- or vice versa.
❖ Therefore, provide a context-setting entrance so that
people know what is appropriate.
33

34. Information Pattern: Clarification Graffiti
❖ Often people design formal
information systems without an
adequate understanding of what the
world is like to the end user.!
❖ When a user comes upon a puzzling
situation, they sometimes find a
solution. !
❖ Often, when this happens, the user
wants to share what they learned
with others.!
❖ When possible, this leads to informal
annotations that help clarify what is
really meant for other users.
34

35. Temporal Pattern: Small Successes Early
❖ Some problems require large teams of relative
strangers to work together cooperatively in order
to solve the overall problem.
❖ Yet, people generally take time to learn to trust
one another as well as to learn another's
strengths and weaknesses and preferred styles.
❖ Plunging a large group of strangers immediately
into a complex task often results in non-productive
jockeying for position, failure,
blaming, finger-pointing, etc.
❖ Therefore, insure that the team or community
first undertakes a task that is likely to bring some
small success before engaging in a complex
effort.
35

36. Major Challenges: Scientific and Ethical
❖ Technology keeps changing; people
keep learning; tasks and goals and
contexts keep changing and
expanding —> How can we cumulate
science?!
❖ Query Study!
❖ www.ibm.com!
❖ We may be able to accurately
(statistically) predict “bad behavior”
before it occurs.!
❖ Who decides when, how, and
whether to intervene?!
❖ Minority Report; The Circle
36

37. Mastering the Opportunity Offered by “The Perfect Storm”
❖ Find Problems!
❖ In Daily Life!
❖ In Stories!
❖ Note and Store Patterns!
❖ Use Ubiquity to Find Problems!
❖ Formulate Problems (Rule of Six)!
❖ Generate many Possible Solutions !
❖ The “Real” Competition may be NO
DECISION = NO SALE!
❖ Test in situ!
❖ Reality Check!
❖ Learn to Improve Over Time
37

38. Three different Disciplines are Converging:
Science
Invention
Operations
Hypothesis: The “perfect storm” allows
on-going measurement, refinement,
improvement, reframing, reinvention,
and scientific discovery all at the same
time from using the same data and using
various combinations of the same methods.
38

39. Science
❖ “Triple Blind” experiments:
people do not even know they
are in a study. Ethical? !
❖ Contingent Experiments:
Rather than “pre-plan” the
entire experiment, conditions
evolve and multiply as
evidence accumulates. !
❖ In Situ experiments: As more of
the real world conditions can
be monitored and dealt with,
less need to perform in lab.
39

40. Invention
❖ More scientific studies of the
invention processes will
snowball number and breadth
of inventions. !
❖ “Brute Force” exploration will
happen more quickly. (e.g.,
light bulb, lead storage battery,
scrabble). !
❖ The instrumentation of reality
will lead to finding a great
number of problems to be
solved.
40

41. Operations
❖ Manufacturing is already heavily
instrumented; that trend will
continue.!
❖ Now, the entire value chain will be
instrumented: problem
identification, design,
development, deployment, sales,
maintenance, disposal. !
❖ Feedback from later stages can
alter decisions earlier in the
process changing problem as
defined, design, manufacturing
process, transportation, etc.
41

42. Key to Making this All Happen is You and Your Approach
❖ Using your knowledge, skill,
and a variety of sophisticated
techniques while inside…!
❖ Still being the inquisitive child.!
❖ To boldly go ….
42

43. References:
!
❖ Alexander, C. Ishikawa S., Silverstein, M. Jacobson, M. , Fikshdahl-King, I., Angel, S. (1977), A Pattern Language. New York: Oxford
University Press.
❖ Srivastava, S., Rajput, N, Dhanesha, K., Basson, S., and Thomas, J. (2013) Community-oriented spoken web browser for low literate users.
Accepted for CSCW Paper, San Antonio, TX, 2013.
❖ Pan, Y., Roedl, D., Blevis, E. and Thomas, J. (2012), Re-conceptualizing Fashion in Sustainable HCI. Designing Interactive Systems
conference. New Castle, UK, June 2012.
❖ Thomas, J. C. (2012). Patterns for emergent global intelligence. In Creativity and Rationale: Enhancing Human Experience By Design J.
Carroll (Ed.), New York: Springer.
❖ Thomas, J. C. & Richards, J. T. (2012). Achieving psychological simplicity: Measures and methods to reduce cognitive complexity. In
The Human-Computer Interaction Handbook. J. Jacko (Ed.) Boca Raton, FL: CRC Press.
❖ Trewin, S., Richards, J., Hanson, V., Sloan, D., John, B., Swart, C., Thomas, J. (2012). Understanding the role of age and fluid intelligence
in information search. Presented at the ASSETS Conference, Boulder CO.
❖ Thomas, J., Diament,J., Martino, J. and Bellamy, R., (2012) Using “Physics” of Notations to Analyze a Visual Representation of Business
Decision Modeling. Presented at VL/HCC 2012 conference in Salsburg, Austria.
❖ Srivastava, S., Dhanesh, K., Basson, S., Rajput, N., Thomas, J., Srivastava, K. (2012) Voice user interface and growth markets. India HCI
conference.
❖ Trewin, S., John, B.E., Richards, J., Swart, C., Brezin, J. and Thomas, J. C. (2010). Towards a Tool for Keystroke Level Modeling of
Skilled Screen Reading, ASSETS 2010.
❖ Thomas, J. C. and Gould, J. D. (1974). A psychological study of Query By Example. IBM Research Report, RC 5124. Armonk NY: IBM.
❖ Thomas, J. C. (1983), Psychological issues in the design of database query languages. In Designing for Human-Computer Communication.
M.E. Sime and M. J. Coombs (Eds.), London: Academic Press.
❖ Thomas, J.C. (1983). Studies in office systems I: The effect of communication medium on person perception. Office Systems Research
Journal, 1 (2), pp. 75-88.
!
43

44. References
❖ Sternin, J. and Sternin, J. (2010). The Power of Positive Deviance: How Unlikely Innovators Solve the World’s Toughest
Problems. Harvard Business Review Press.
❖ Green, S., Jones, L. Matchen, P. & Thomas, J. (2003). Iterative development in the field. IBM Sysems Journal, 42 (2).
❖ Thomas, J. C., Kellogg, W.A., and Erickson, T. (2001) The Knowledge Management puzzle: Human and social factors in
knowledge management. IBM Systems Journal, 40(4), 863-884.
❖ Thomas, J. C. (2001). An HCI Agenda for the Next Millennium: Emergent Global Intelligence. In R. Earnshaw, R. Guedj, A.
van Dam, and J. Vince (Eds.), Frontiers of human-centered computing, online communities, and virtual environments. London:
Springer-Verlag.
❖ Thomas, J. C. (1999) Narrative technology and the new millennium. Knowledge Management Journal, 2(9), 14-17.
❖ Desurvire, H. & Thomas, J.C. (1993). Enhancing performance of interface evaluators using non-empirical usability methods.
In Proceedings of the Human Factors 37th Annual Meeting, 2, 1132-1136. Seattle, WA: October 11-15. Santa Monica, CA:
Human Factors and Ergonomics Society. !
❖ Thomas, J.C. and Kellogg, W.A. (1989). Minimizing ecological gaps in interface design, IEEE Software, January 1989.