1. The human side of services
John Billings, Joseph Lynch
{billings,jlynch}@yelp.com

2. Yelp’s Mission:
Connecting people with great
local businesses.

3. Yelp Stats:
As of Q3 2015
89M 3271%90M

4. What’s Important?

5. What’s Important?

6. What’s Really Important?

7. What’s Really Important?

8. Getting Started

10. 2005 2016
yelp-main
0
3,000,000
LoC

11. 2011: A service is born

12. 2011 - 2016: A Cambrian
Explosion of services

13. 2011 - 2016: New practices

14. Education

15. Why?
“There are only two hard problems in distributed
systems:
2. Exactly-once delivery
1. Guaranteed order of messages
2. Exactly once delivery”
- Mathias Varraes

16. How?

17. How?

18. How?

19. Consistency

20. Tooling

21. "/pets": {
"get": {
"description": "Returns all pets from the system",
"responses": {
"200": {
"description": "A list of pets.",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/Pet"
}
}
}
}
}
}
Interface design

22. Organizational
Objectives

23. Org Objectives
Distributed System Objectives
● Performance
● Security
● Stability
● Cost
These are cross cutting objectives that are crucial in a
SOA.

24. Org Objectives
No problem …
● Performance is important!
● Security is important!
● Stability is important!
● Cost is important!
… all done right?

25. Org Objectives
What gets measured gets managed
- Drucker

26. Good Objective Metric: Performance
-

27. -
Good Objective Metric: Security

28. Good Objective Metric: Reliability
-

29. Good Objective Metric: Cost
-

30. Service level objectives
● Contracts are cool
● Performance is cool
● Uptime is cool
● Keeping cost low is cool
… So write these things down in a Contract and page
owners when we violate them?

31. Operating Services

32. Ownership

33. Dealing with Failure

34. Microservice Pitfalls
(from those that have fallen)

35. Lots of Flowers
“Let 1000 flowers
bloom. Then rip 999 of them out
by the roots”
- Peter Seibel

36. Append Only Services
Service Service Service ...
Service Service Service ...
Service Service Service ...

37. Ditching Libraries
● Libraries can be pretty terrible
○ Break tests
○ Deploy 20 versions
○ Function calls work only in ${language}
○ Bugs can take weeks to fix
○ Tests can take a long time

38. Service
Library
Ditching Libraries

39. Service
Library
That’s not a SOA
… that’s just RPC
Ditching Libraries

40. ● Libraries can be pretty awesome
○ Break tests not websites
○ Deploy 20 versions
○ Function calls are wicked fast
○ Have weeks to fix a bug
○ Unit tests are fast
Ditching Libraries

41. Everybody does Ops?
Dev vs Ops = Dev frustration, Ops burnout
Dev + Ops = ?

42. ● Not all Devs can (want to) do Ops
● Not all Ops can (want to) do Dev
● What about?
○ DBAs
○ Security Engineers
○ Designers
Everybody does Ops?

43. What to Aim For Instead?
1. Encourage cooperation
2. Acknowledge your engineers have varied
skills: {Ops, Dev, Security, Databases, Design,
Frontend, API design, Performance, etc …}
3. Try to build teams that have a wide range of
skills

44. Any questions?

45. Image Citations
● Sailing Ship: https://en.wikipedia.org/wiki/Sailing_ship
● Programmer with Laptop: https://commons.wikimedia.org/wiki/File:
Typing_computer_screen_reflection.jpg
● Moore’s Law: https://en.wikipedia.org/wiki/Moore%27s_law#/media/File:
Transistor_Count_and_Moore%27s_Law_-_2011.svg
● Cambrian explosion: https://en.wikipedia.org/wiki/Devonian#/media/File:Fish_evolution.png
● Bus queue: https://en.wikipedia.org/wiki/File:Bus_Queue.jpg
● Deputy badge: https://commons.wikimedia.org/wiki/File:
Badge_of_the_San_Diego_County_Sheriff%27s_Department.png

46. Image Citations
● Deep dive: https://en.wikipedia.org/wiki/Deep_diving#/media/File:
Trevor_Jackson_returns_from_SS_Kyogle.jpg
● Map: https://commons.wikimedia.org/wiki/File:Carta_Marina_AB_stitched.jpg
● AWS Total Cost of Onwership: https://aws.amazon.com/blogs/aws/the-new-aws-tco-
calculator/
● Sharing milkshake: https://commons.wikimedia.org/wiki/File:Children_sharing_a_milkshake.
jpg
● Field of flowers: TODO