1. ©2015CarnegieMellonUniversity:1
Making Sense of
Cyberspace
Cyber Intel Research Consortium
May 5, 2015
Jason Hong
Computer
Human
Interaction:
Mobility
Privacy
Security

2. ©2015CarnegieMellonUniversity:2
•Bandwidth
Time
Computing Trends

3. ©2015CarnegieMellonUniversity:3
•Bandwidth
•Storage
Time
Computing Trends

4. ©2015CarnegieMellonUniversity:4
Time
Computing Trends
•Bandwidth
•Storage
•Computing Power

5. ©2015CarnegieMellonUniversity:5
Time
Computing Trends
•Bandwidth
•Storage
•Computing Power
•Information

6. ©2015CarnegieMellonUniversity:6

7. ©2015CarnegieMellonUniversity:7
Time
Human Capabilities
Cognitive Processing

8. ©2015CarnegieMellonUniversity:8
Cognitive Processing
Visual acuity
Communication bandwidth
Attention
…
Time
Human Capabilities

9. ©2015CarnegieMellonUniversity:9
7  2

10. ©2015CarnegieMellonUniversity:10
Evidence suggests it’s more like 4

11. ©2015CarnegieMellonUniversity:11
How to Scale Up Our Ability to
Understand and Analyze?
• Research in mobility, privacy, security
– Anti-phishing
– Smartphone apps, Internet of Things
• Key themes today
– Information visualization
– Machine learning
– Wisdom of crowds

12. ©2015CarnegieMellonUniversity:12

13. ©2015CarnegieMellonUniversity:13

14. ©2015CarnegieMellonUniversity:14
•Which state has highest income?
•Is there a relationship between
income and education?
•Are there any outliers?

15. ©2015CarnegieMellonUniversity:15
Visualization for Eliciting
Knowledge from Data

16. ©2015CarnegieMellonUniversity:16
How You Represent Info Matters
• What is VIII x XCI = ???
• Another example:
– Game with two players
– Alternately choose numbers between 1-15
– First to get three numbers to 15 wins
– Each number can be taken only once

17. ©2015CarnegieMellonUniversity:17
How You Represent Info Matters
• It’s actually Tic-Tac-Toe
– Trivial in this representation
8 1 6
3 5 7
4 9 2

18. ©2015CarnegieMellonUniversity:18
Other Design Principles?
1. Start out going Southwest on ELLSWORTH AVE
Towards BROADWAY by turning right.
2: Turn RIGHT onto BROADWAY.
3. Turn RIGHT onto QUINCY ST.
4. Turn LEFT onto CAMBRIDGE ST.
5. Turn SLIGHT RIGHT onto MASSACHUSETTS AVE.
6. Turn RIGHT onto RUSSELL ST.

19. ©2015CarnegieMellonUniversity:19

20. ©2015CarnegieMellonUniversity:20

21. ©2015CarnegieMellonUniversity:21
Some Lessons
1. How you represent info matters
2. InfoViz is what you do show as well
as what you don’t

22. ©2015CarnegieMellonUniversity:22
Focus + Context
• More pixels to stuff that matters,
fewer pixels to supporting context

23. ©2015CarnegieMellonUniversity:23
Focus + Context

24. ©2015CarnegieMellonUniversity:24
Appropriate Use of Color

25. ©2015CarnegieMellonUniversity:25
Appropriate Use of Color

26. ©2015CarnegieMellonUniversity:26
Some Lessons
1. How you represent info matters
2. InfoViz is what you do show as well
as what you don’t
3. Focus + Context
4. Appropriate use of colors (perception)

27. ©2015CarnegieMellonUniversity:27
US Election 2004

28. ©2015CarnegieMellonUniversity:28
InfoViz’s Can Show and Hide
Info

29. ©2015CarnegieMellonUniversity:29
All Viz’s Show and Hide Info

30. ©2015CarnegieMellonUniversity:30
InfoViz’s Can Show and Hide
Info

31. ©2015CarnegieMellonUniversity:31
All Viz’s Show and Hide Info

32. ©2015CarnegieMellonUniversity:32
Some Lessons
1. How you represent info matters
2. InfoViz is what you do show as well
as what you don’t
3. Focus + Context
4. Appropriate use of colors (perception)
5. All visualizations have inherent biases

33. ©2015CarnegieMellonUniversity:33

34. ©2015CarnegieMellonUniversity:34
Visualizations Don’t Have to
Be Complex to be Effective

35. ©2015CarnegieMellonUniversity:35
Example from Jeff Heer

36. ©2015CarnegieMellonUniversity:36

37. ©2015CarnegieMellonUniversity:37

38. ©2015CarnegieMellonUniversity:38
Work by Jeff Heer

39. ©2015CarnegieMellonUniversity:39

40. ©2015CarnegieMellonUniversity:40
Collaborative Analysis
• Many Eyes (by IBM)

41. ©2015CarnegieMellonUniversity:41

42. ©2015CarnegieMellonUniversity:42

43. ©2015CarnegieMellonUniversity:43
Can We Have Crowds of
People Help Analyze Data?
• What are better ways of getting
groups of people working together?
• One approach is to use lots of crowd
workers to improve accuracy and time
– Common structure in your problem?
– Can you break it down into small tasks?

44. ©2015CarnegieMellonUniversity:44

45. ©2015CarnegieMellonUniversity:45

46. ©2015CarnegieMellonUniversity:46

47. ©2015CarnegieMellonUniversity:47
Wisdom of Crowds Approach
• Mechanics of PhishTank
– Submissions require at least 4 votes
and 70% agreement
– Some votes weighted more
• Total stats (Oct2006 – Feb2011)
– 1.1M URL submissions from volunteers
– 4.3M votes
– resulting in about 646k identified phish
• Why so many votes for only 646k phish?

48. ©2015CarnegieMellonUniversity:48
PhishTank Statistics
Jan 2011
Submissions 16019
Total Votes 69648
Valid Phish 12789
Invalid Phish 549
Median Time 2hrs 23min
• 69648 votes  max of 17412 labels
– But only 12789 phish and 549 legitimate identified
– 2681 URLs not identified at all
• Median delay of 2+ hours still has room
for improvement (Jan 2015 is 14+ hours)

49. ©2015CarnegieMellonUniversity:49
Ways of Smartening the Crowd
• Change the order URLs are shown
– Ex. most recent vs closest to completion
• Change how submissions are shown
– Ex. show one at a time or in groups
• Adjust threshold for labels
– PhishTank is 4 votes and 70%
– Ex. vote weights, algorithm also votes
• Motivating people / allocating work
– Filtering by brand, competitions,
teams of voters, leaderboards

50. ©2015CarnegieMellonUniversity:50
Clustering Phish
• Motivations
– Most phish are generated by toolkits
and thus similar
– Labeling single sites as phish can be
hard, easier if multiple examples
– Reduce labor by labeling suspicious
sites in bulk

51. ©2015CarnegieMellonUniversity:51
Clustering Phish
• Motivations
– Most phish are generated by toolkits
and thus similar
– Labeling single sites as phish can be
hard, easier if multiple examples
– Reduce labor by labeling suspicious
sites in bulk

52. ©2015CarnegieMellonUniversity:52
Most Phish Can be Clustered
• With all data over two weeks, 3180 of
3973 web pages can be grouped (80%)
– Used shingling and DBSCAN (see paper)
– 392 clusters, size from 2 to 153 URLs

53. ©2015CarnegieMellonUniversity:53
Comparing Coverage and Time

54. ©2015CarnegieMellonUniversity:54
CrowdVerify
• This is the New York Times Privacy Policy
• Can we use crowd to help summarize?
Flag surprises?
• Can we build predictive language models?

55. ©2015CarnegieMellonUniversity:55

56. ©2015CarnegieMellonUniversity:56

57. ©2015CarnegieMellonUniversity:57

58. ©2015CarnegieMellonUniversity:58

59. ©2015CarnegieMellonUniversity:59

60. ©2015CarnegieMellonUniversity:60
Which statement is more important?
Privacy and Security
CrowdVerify
We receive data whenever
you visit a game,
application, or website that
uses Facebook Platform …
When you sign up for
Facebook, you are required
to provide information such
as your name, email
address, birthday, and
gender.
A B

61. ©2015CarnegieMellonUniversity:61
Privacy and Security
CrowdVerify on Google’s Policies
0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
Scores
Statement ID
Frequency Experiment Sum of Scores
Google 08
Google 13
Google 07
Google 30
Google 10
Google 12
Google 04
Google 16
Google 03
Google 11
Google 05
Google 09
Google 28
Google 02
Google 06
Google 23
Google 14

62. ©2015CarnegieMellonUniversity:62
Top Statement - Google
• Google 08: When you use our services
or view content provided by Google, we
may automatically collect and store
certain information in server logs. This
may include: telephony log information
like your phone number, calling-party
number, forwarding numbers, time
and date of calls, duration of calls, SMS
routing information and types of calls.

63. ©2015CarnegieMellonUniversity:63
Bottom 3 Statements - Google
• Google 29: Whenever you use our services,
we aim to provide you with access to your
personal information.
• Google 26: For example, you can: Take
information out of many of our services.
• Google 24: People have different privacy
concerns. Our goal is to be clear about what
information we collect, so that you can make
meaningful choices about how it is used.

64. ©2015CarnegieMellonUniversity:64
Designing Crowd Systems
• Motivation for contributing?
– Money, altruism, fun?
• Quality control?
– How to ensure good quality? Prevent bozos?
• How is crowd wisdom aggregated?
– How are people and computers organized?
– How are decisions made?
• Skill level required?
– Novice, intermediate, expert?

65. ©2015CarnegieMellonUniversity:65
One Real-World Success
Clickworkers
• 100k+ volunteers identified 2M Mars
craters from space photographs (14k
hours of free work!!)
• Aggregate results “virtually
indistinguishable” from experts

66. ©2015CarnegieMellonUniversity:66
Another Real-World Success
Galaxy Zoo
• Problem: you are an astronomer looking
at the Sloan Digital Sky survey, which has
images of over 1M galaxies. You want to
classify these by galaxy type.

67. ©2015CarnegieMellonUniversity:67
DARPA Red Balloon
• 2009 challenge to find 10 red weather
balloons at unknown locations in the USA
• Winner gets $40k
• Surprise: top team wins
in < 9 hours
– Other teams did well too
– 9 balloons in 9 hours (1)
– 8 balloons (2 teams)
– 7 balloons (5 teams)

68. ©2015CarnegieMellonUniversity:68
DARPA Red Balloon

69. ©2015CarnegieMellonUniversity:69
DARPA Red Balloon
• MIT Team (10 balloons in less than 9 hours)
• Overall team strategy
– Get as many people as possible (over 5k people)
– Use multiple approaches to verify
• How people were organized
– Core team / recruit / spotters
• “Social engineering”
– Recursive financial structure to get more people
– Incentivizes people who might not be able to help
• Technologies used
– Surprisingly little, one centralized web site + google maps

70. ©2015CarnegieMellonUniversity:70
MIT Reward Structure

71. ©2015CarnegieMellonUniversity:71
Not Quite as Successful
Search for Steve Fossett

72. ©2015CarnegieMellonUniversity:72
DARPA Network Challenge

73. ©2015CarnegieMellonUniversity:73
Summary
• Collecting info is easy,
understanding it is hard
• How to scale?
– Information visualization
– Machine learning
– Wisdom of crowds

74. ©2015CarnegieMellonUniversity:74
More Resources
• A Tour through the Visualization Zoo
– https://queue.acm.org/detail.cfm?id=1805128
• Interactive Dynamics for Visual Analysis
– https://queue.acm.org/detail.cfm?id=2146416
• Tableau Software
– http://www.tableau.com/
• Many Eyes
– http://www-01.ibm.com/software/analytics/many-eyes/
•

75. ©2015CarnegieMellonUniversity:75
More Resources
• DARPA Red Balloon Challenge
– http://cacm.acm.org/magazines/2011/4/106587-reflecting-
on-the-darpa-red-balloon-challenge/fulltext
• Pirolli and Card, The Sensemaking Process
and Leverage Points for Analyst Technology
as Identified Through Cognitive Task Analysis
• Apolo, combining data mining with infoviz
– https://www.youtube.com/watch?v=EFxYVWkj1aE

76. ©2015CarnegieMellonUniversity:76

77. ©2015CarnegieMellonUniversity:77
Patterns in Crowd Analysis
Unit of Work
Unit of Work
Unit
Unit
Unit
…
Final
Voting
(Ex. PhishTank)
MapReduce
(Ex. CrowdVerify)
Unit of Work
Funnel
Unit of Work
Unit of WorkDecision

78. ©2015CarnegieMellonUniversity:78
Visualization of the Internet

79. ©2015CarnegieMellonUniversity:79
Combine Data Mining + Viz
User can specify exemplars of a group
Belief Propagation to find more nodes

80. ©2015CarnegieMellonUniversity:80