Discusses the challenges that a strong privacy stance poses for the Wikimedia Foundation, including how it affects data collection, aggregation, and preservation practices. Presentation details some creative workarounds that allow WMF to calculate metrics in a privacy-conscious way.

1. Data and Privacy At Scale
nuria@wikimedia.org

3. Part 1. Philosophy: Free
Knowledge and Privacy

4. Calculating Web Metrics
Identifiability
Part 2. The Technical Challenge
of Privacy

6. “Imagine a world in which
every single human being
can freely share in the sum
of all knowledge”.

7. What does Wikipedia
know about you?

8. What does Wikipedia
know about you?
Very little

9. Privacy is a Core
Value of the Free
Knowledge
Movement.

10. How many people
use wikipedia every
day?

11. Part 1. Philosophy: Free
Knowledge and Privacy

12. By Lane Hartwell - Photographed by Lane Hartwell (http://fetching.net/) on behalf of the Wikimedia Foundation, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=8927361
Free as in “free” access

13. No account
needed
@dimitryarov 's installation Visualizing Knowledge

14. Free as in “free” of corporate
influence
By Michael Mandiberg - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=53190022

15. https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2018-06-29/Opinion

16. Very
Incorrect
assumption
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2018-06-29/Opinion

17. Free Knowledge
Core Beliefs around
Privacy

18. Should not have to provide any
information to participate in free
knowledge movement.
There cannot be access to free
knowledge without a strong
guarantee of privacy.

19. Free Knowledge
Privacy Policy

20. Data and Privacy At Scale

21. Build the wiki
way
By Helpameout [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia
Commons

22. 195,000
words
By Helpameout [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia
Commons

24. Free knowledge
Privacy Policy Highlights

25. No data is sent to third parties or sold for
marketing purposes.
We only retain personal data for 90 days.
IPs and user agents (in raw format) are
deleted after 90 days.
Articles browsed by readers are retained at
most 90 days, if retained at all, then only in
aggregate form

26. Photo by Mikito Tateisi on Unsplash
The Technical Challenge of
Privacy

27. No data is sent to third parties or sold
for marketing purposes.

28. No data is sent to third parties or sold
for marketing purposes.
Commercial solutions to gather
user data are not an option.
Translation:

29. We only retain personal data and session
data for 90 days. IPs and user agents (in raw
format) are deleted after 90 days.

30. We only retain personal data and session
data for 90 days. IPs and user agents (in raw
format) are deleted after 90 days.
We need to build an ETL system in which is
easy to selectively sanitize and aggregate
data at scale.
Translation:

31. ~150,000 per second at peak
HTTP Requests

32. How many people
use wikipedia every
day?

33. The Technical Challenge of Privacy
Web Metrics and
Identifiability

34. How many people use
wikipedia every day?
(DAU= Daily Active Users)

35. Daily Active Users
Traditional solution:
count logged in sessions
that day.

36. Daily Active Devices
Traditional solution:
Session tokens.

37. somewebsite.com/bike

38. somewebsite.com/bike
Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -

39. somewebsite.com/bike
Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -

40. somewebsite.com/bike
Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
Set-Cookie:Session=1011

41. somewebsite.com/airplane
Timestamp IP Page Cookies
2018-03-01 343.4.* Airplane Session=1011

42. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011

43. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011
2018-03-01 143.4.* Sushi Session=5055
2018-03-01 343.4.* Tacos Session=6066

44. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011
2018-03-01 143.4.* Sushi Session=5055
2018-03-01 443.4.* Tacos Session=6066
2018-03-01 747.4.* Ronaldo Session=5055
2018-03-01 747.4.* Milan Session=1011
2018-03-01 345.2.* Rome Session=1011

45. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011
2018-03-01 143.4.* Sushi Session=5055
2018-03-01 443.4.* Tacos Session=6066
2018-03-01 747.4.* Ronaldo Session=5055
2018-03-01 747.4.* Milan Session=1011
2018-03-01 345.2.* Rome Session=1011
How many uniques?

46. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011
2018-03-01 143.4.* Sushi Session=5055
2018-03-01 443.4.* Tacos Session=6066
2018-03-01 747.4.* Ronaldo Session=5055
2018-03-01 747.4.* Milan Session=1011
2018-03-01 345.2.* Rome Session=1011
How many uniques?

47. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011
2018-03-01 143.4.* Sushi Session=5055
2018-03-01 443.4.* Tacos Session=6066
2018-03-01 747.4.* Ronaldo Session=5055
2018-03-01 747.4.* Milan Session=1011
2018-03-01 345.2.* Rome Session=1011
How many uniques? 3 devices

48. Identifiability

49. Timestamp IP Page Cookies
2018-03-01 343.4.* Bike -
2018-03-01 343.4.* Airplane Session=1011
2018-03-01 747.4.* Milan Session=1011
2018-03-01 345.2.* Rome Session=1011

50. Timestamp IP Page Cookies
2018-03-01 USA Bike -
2018-03-01 USA Airplane Session=1011
2018-03-01 Italy Milan Session=1011
2018-03-01 Italy Rome Session=1011

51. http://transparency.wikimedia.org

52. The Technical Challenge of Privacy
Reducing
Identifiability

53. How many people use
wikipedia every day?

54. March 1st, Kim comes to Wikipedia

55. March 1st, Kim comes to Wikipedia
Timestamp IP Page Cookies
2018-03-01 123.4.* Bike -

56. March 1st, Kim comes to Wikipedia
Last-Access:
2018-03-01

57. Last-Access:
2018-03-01
March 1st, Kim comes to Wikipedia

58. March 15th, Kim visits again
Last-Access:
2018-03-01

59. Timestamp IP Page Cookies
2018-03-15 776.9.* Airplane Last-Access=2018-03-01
Last-Access:
2018-03-01
March 15th, Kim visits again

60. Last-Access:
2018-03-15
March 15th, Kim visits again

61. Timestamp IP Page Cookies
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 776.9.* Milan Last-Access=2018-03-15

62. March 15th, More visits...
Timestamp IP Page Cookies
2018-03-15 113.4.* Tacos Last-Access=2018-03-15
2018-03-15 113.4.* Sushi Last-Access=2018-01-01

63. Timestamp IP Page Cookies
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 776.9.* Milan Last-Access=2018-03-01

64. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
Timestamp IP Page Cookies
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
2018-03-15 776.9.* Milan Last-Access=2018-03-01

65. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
Timestamp IP Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
2018-03-15 776.9.* Milan Last-Access=2018-03-15

66. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
Timestamp IP Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
2018-03-15 776.9.* Milan Last-Access=2018-03-15
How many uniques?

67. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
2018-03-15 776.9.* Milan Last-Access=2018-03-15
Timestamp IP Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
How many uniques?

68. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
2018-03-15 776.9.* Milan Last-Access=2018-03-15
Timestamp IP Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
How many uniques?

69. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
Timestamp IP Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
2018-03-15 776.9.* Milan Last-Access=2018-03-15
Lessens Identifiability

70. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
Timestamp IP Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
2018-03-15 776.9.* Milan Last-Access=2018-03-15
Not able to reconstruct all sessions

71. 2018-03-15 223.2.* Sushi Last-Access=2018-01-01
Timestamp UA + IP? Page Cookies
2018-03-15 133.1.* Ronaldo Last-Access=2018-02-25
2018-03-01 123.4.* Bike -
2018-03-15 123.4.* Airplane Last-Access=2018-03-01
2018-03-15 223.2.* Tacos Last-Access=2018-03-15
2018-03-15 776.9.* Milan Last-Access=2018-03-15
Not able to reconstruct all sessions

72. Caveats of last access solution are
pretty much the same as token
solution but lessens identifiability
significantly
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices/Last_access_solution

74. Privacy as a Feature

75. You can compute key
metrics in a privacy
conscious manner.
It might require some
extra effort.

76. Questions?
https://xkcd.com/285
nuria@wikimedia.org

77. Data Sanitization
The Technical Challenge of Privacy

80. {
"webHost": "en.wikipedia.org",
"wiki": "enwiki",
"location": "Barcelona",
"destination": "2018_FIFA_Worldcup",
"category": "Sports",
"mode": "stable",
"username": "kim",
"userAgent": {
"browser_family": "Mobile Safari",
"browser_major": "11",
"browser_minor": "0",
"device_family": "iPhone", "
"os_family": "iOS",
"os_major": "11",
"os_minor": "4"
}
}
PagePreviewTracking

81. {
"webHost": "en.wikipedia.org",
"wiki": "enwiki",
"location": "Barcelona",
"destination": "2018_FIFA_Worldcup",
"category": "Sports",
"mode": "stable",
"username": "kim",
"userAgent": {
"browser_family": "Mobile Safari",
"browser_major": "11",
"browser_minor": "0",
"device_family": "iPhone",
"os_family": "iOS",
"os_major": "11",
"os_minor": "4"
}
}
PagePreviewTracking

82. {
"webHost": "keep",
"wiki": "keep",
"category": "keep",
"destination": "2018_FIFA_Worldcup",
"userAgent": {
"browser_family": "keep",
"os_family": "keep",
“wmf_app_version”: "keep"
}
}
https://meta.wikimedia.org/wiki/Research:Schemas
https://github.com/wikimedia/analytics-refinery/blob/master/static_data/eventlogging/whitelist.yam
l
PagePreviewTracking - Whitelist

83. {
"webHost": "en.wikipedia.org",
"wiki": "enwiki",
"location": "Barcelona",
"category": "Sports",
"destination": "2018_FIFA_Worldcup",
"mode": "stable",
"username": "kim",
"userAgent": {
"browser_family": "Mobile Safari",
"browser_major": "11",
"browser_minor": "0",
"device_family": "iPhone",
"os_family": "iOS",
"os_major": "11",
"os_minor": "4"
}
}
PagePreviewTracking

84. {
"webHost": "en.wikipedia.org",
"wiki": "enwiki",
"category": "Sports",
"mode": "stable",
"userAgent": {
"browser_family": "Mobile Safari",
"device_family": "iPhone",
}
}
PagePreviewTracking

85. Event
Clients

86. Event
Clients
HTTP Beacon
Endpoint
Varnish
{JSON}

87. Event
Clients
HTTP Beacon
Endpoint
Varnishkafka
Kafka
Varnish

88. Event
Clients
Event
Processor
(Spark)
HTTP Beacon
Endpoint
Varnishkafka
Kafka
Varnish HDFS

89. Event
Clients
Event
Processor
(Spark)
Clean
Events
Events
<90 days
HTTP Beacon
Endpoint
Varnishkafka
Kafka
Varnish
Whitelist
HDFS
https://github.com/wikimedia/analytics-refinery-source/tree/master/refinery-job

91. March 15th, more visits
Timestamp IP Page Cookies
2018-03-15 133.1.* Cake Last-Access=2018-02-25