These are the slides from a talk I gave at dropbox this month (Feb 2012). It was a narrative about the evolution of bitly and a technical presentation about algorithms and infrastructure. The live demo portion is not represented in the slides (and each of the visuals has an accompanying story).

1. Short URLs, Big Fun:
Understanding the World in Realtime
Hilary Mason
Chief Scientist, bitly
@hmason
h@bit.ly

3. http://www.pcworld.com/article/223409/move_over_dr_soong_
girls_can_build_android_apps_too.html
http://bit.ly/hOnbWg

11. [fireplace]

13. How do we change the world?

14. Can we understand the world, first?

15. Big Data

16. Data

17. 10s of millions of URLs per day
100s of millions of clicks per day
10s of billions of URLs

18. encodes
{"g": "zalAU0",
"i": "173.213.X.X",
"h": "zalAU0",
"l": "bitly",
"u": "http://www.amazon.com/Country-Life-Cal-Mag-
Potassium-Target-
Tablets/dp/B0001VUZ3A?SubscriptionId=AKIAJGA7
AAB6QE7WENSQ&tag=mycellrevi-
20&linkCode=sp1&camp=2025&creative=165953&cr
eativeASIN=B0001VUZ3A", "t": 1328266799,
"_id": "4f2bbe2f-0035d-063a1-3d1cf10a"}

19. decode
{"a": "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)",
"c": "US",
"nk": 1,
"tz": "America/New_York",
"gr": "NY",
"g": "xNaZ9h",
"i": "98.118.X.X",
"h": "wXxuKW",
"k": "4eefe4be-003e4-X-X",
"l": "moma",
"al": "en-US", "hh":
"bit.ly",
"r":
"http://www.facebook.com/l.php?u=http%3A%2F%2Fbit.ly%2FwXxuKW&h=rAQG_ZZ2G
AQGQth1IOej-9_KHmbpEvh6FlllZjsAqg6A7Rw",
"u": "http://www.brainpickings.org/index.php/2012/02/02/jackson-pollock-father-letter/",
"t": 1328272481,
"hc": 1328232072,
"cy": "East Amherst",
"ll": [43.044101715087891, -78.694900512695312]}

20. a link
• URL
• Content
• Ref distribution
• Geo distribution
• Language
• Key phrases
• Topic

21. Data Science?
Analytics Science

22. Data Science?
Things you can Things you
just count. can’t.

23. Data scientists?
engineering
math
nerds
nerds nerds
nerds
comp sci
hacking
awesome nerds

24. bitly science team!

25. What can we learn from a lot of
people talking to each other?

29. A few things that we can count...

30. How do people use different
devices?

33. What happens on the internet when
society isn’t stable?

37. Revolution.

38. (Silly Things on the Internet)

40. the cutest kitten

41. A few things that we can count...
cleverly.

42. What spoken languages are in a
page?

43. raw data
"es"
"en-us,en;q=0.5"
"pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4"
"en-gb,en;q=0.5"
"en-US,en;q=0.5"
"es-es,es;q=0.8,en-us;q=0.5,en;q=0.3”
"de, en-gb;q=0.9, en;q=0.8"

45. entropy calculation
def ghash2lang(g, Ri, min_count=3, max_entropy=0.2):
""”
returns the majority vote of a langauge for a given hash
""”
lang = R.zrevrange(g,0,0)[0]
# let's calculate the entropy!
# possible languages
x = R.zrange(g,0,-1)
# distribution over those languages
p = np.array([R.zscore(g,langi) for langi in x])
p /= p.sum()
# info content
I = [pi*np.log(pi) for pi in p]
# entropy: smaller the more certain we are! - i.e. the lower our surprise
H = -sum(I)/len(I) #in nats!
# note that this will give a perfect zero for a single count in one language
# or for 5K counts in one language. So we also need the count..
count = R.zscore(g,lang)
if count < min_count and H > max_entropy:
return lang, count
else:
return None, 1

46. http://4sq.com/96kc1O

47. What’s the context
around a link?

50. [demo]

51. Things we have to think about.
(science)

52. What’s a human?

54. normal click distributions

59. abnormal click distributions

63. Organic vs Inorganic?

66. AT SCALE

67. 1. Research offline
2. Do fancy math – find the shortcuts
3. Design infrastructure
4. Re-design to run at scale and speed

68. Realtime Search

69. Realtime Search
Attributes calculated either at index time or
query time.
Rankings can vary by second.

70. [demo]

71. What are people paying
attention to right now?

72. actual rate of clicks on phrases
vs
expected rate of clicks on phrases

73. Dragoneye
We calculate clickrate with a sort of moving
average:
where

74. Dragoneye
We represent as a sum of delta spikes.
This simplifies to:

75. Dragoneye
Choosing is important.
It must be interpretable, and smooth (but not
too smooth).
We use a distribution for that is a function
that sums to 1. The function is 0 at the
origin.

76. [demo]

77. philosophy

78. simple math > fancy math

80. How do we know
when we’ve won?

81. How do we communicate what
we’ve learned effectively?

82. Ask the crazy questions.

83. Thank you!
h@bit.ly
@hmason