Moving away from a monolith to a microservices architecture is a process fraught with hidden challenges. There's legacy code, infrastructure, and organizational processes that all need to change, in order to make the switch successful. But microservices come with a huge increase in infrastructure complexity. We'll see how distributed traces empower developers to work with greater autonomy, in increasingly complex deployment environments.

1. Moving to
Microservices with
the Help of
Distributed Traces
Kirk Kaiser
@burningion

2. Software
Development
is Changing

3. More Container Adoption

4. Especially in Larger
Environments

5. And Containers Usually Means
Orchestration

6. But there is a
tradeoff...

7. – etcd, default database to
manage kubernetes state,
recommends 5 (!) server
instances for durability
– Growing list of subpackages
to control networking layers,
load balancing (Istio, Envoy)
– Completely rethink
development, testing,
deployment as kubernetes
grows beyond single dev
machine
Kubernetes
introduces
complexity

8. etcd
Docker
Kubernetes
Envoy
Minikube
Istio
containerd
Spinnaker

9. Initial response: I can get to market quicker
with literally worst practices
and “pet” servers in less time
than it’ll take to learn
Kubernetes and how it fails in
practice.
No thanks!

10. I can no longer run my entire
application locally?
Local permanent filesystems
don’t really exist?
I have to build and push
images to test?
The network is separate for
each container?
Where are the
tools?!

11. Why are so many high
growth, great software
companies using
microservices despite the
added complexity??

12. What are they
optimizing for with microservices
anyway?

13. ●Scalability
●Team language preferences
●Separation of concerns

15. Management Theory
And You!

16. Agrarian,
independent
Industrial,
dependent

17. Frederick Winslow Taylor
We should be scientific about the ways we
work!
Let’s measure exactly how big a shovel
should be.
Let’s have people who observe the workers
and design the work to make sure the
people doing the work are efficient.

18. The methods of work are
designed from above for
repetition and simplicity
Given a set of inputs,
generate high quality outputs
at lowest possible cost
"Ideal” work week is 40
hours
Scientific
Management

20. Created the idea of a
“knowledge worker”
“Companies work best when
they are decentralized”
“A company’s primary
responsibility is to serve its
customers.”
“Management by objectives”
Peter Drucker

21. The things you need to know
change every day.
Not enough bandwidth for a
single person to absorb
information and dictate back
down to the “subordinates”
Work is largely autonomous.
Many non-software
companies still struggle to
understand how knowledge
work is different
Knowledge Work

22. Software projects are becoming too big
to be designed and spec’d at the
beginning by a single person.

23. Humans work best when they have
agency over the work they do.

24. Autonomy
You have choices and
can pick what to work
on.
Relatedness
You and your teams are
in it together.
People can rely on you,
and you can rely on
them.
Competence
You feel you are good at
what you do.
You are consistently
challenged and
improving.
Self Determination Theory

25. Microservices are fundamentally a
human technology, not a computer
technology

26. Moving to microservices is a move away
from massively coordinated releases,
and to smaller scale, personal
accountability

27. Microservices and the modern
company

28. UBER
NETFLIX
AIRBNB

29. A bunch of unicorns,
growing
unreasonably quick!

30. BLITZSCALING
Reid Hoffman’s term for scaling unsafely fast to grow a market share moat
before anyone else can catch up.
High spending, heavy potential short term loss.
Do or die, take the whole market or default to gone.

31. But my company can’t spend 10 years
being unprofitable!
Microservices are still coming for you too!
The slowing of Moore’s law means we are losing the ability to scale up
Many of our systems are already distributed anyways

32. Attempting to prove this book wrong.
Microservices Promise:

33. … and making this approach possible.

34. Microservices require re-
thinking tools
for Developer Autonomy and
Independence at scale

35. MetricsTracesLogs
Observability
Synthetics

36. Observability empowers
individuals to drill down on
problems in complex, rapidly
changing systems.

37. Distributed
Traces
A single view for
units of work as requests
pass through our systems.

38. Distributed Traces
(not just in one place) (breadcrumbs of what’s
happening)

39. Distributed System

40. Example Trace

41. See complete units of work as they
pass through your entire system,
especially useful to diagnose ever
changing dependencies.
Add tags to be able to see specific
customers, organizations, and their
direct request paths within your
constantly changing systems.
Compress Distributed System
Complexity with Traces

42. TracesLogs
Unlike logs, traces have
structure.

43. Let’s See How Traces Help
When Developing

44. City Water System

45. System Architecture

46. Add Trace
Instrumentation
– Install Agent once per host
– Install language specific library
– Set environment variables
– Wrap call to start service

47. System Architecture (Instrumented)

48. For each incoming request,
the trace library checks to see
if there’s a trace attached. If
not, it creates a trace.

49. We then add spans to each
trace as it passes through the
downstream sub-systems.

50. Optionally, we can add tags
to spans, adding metadata
for analysis later.

51. Service Map

52. Service List

53. Individual Trace

54. Trace Analytics

55. Classic case for manual
tagging

56. Our largest customer’s CEO
says the order page doesn’t
work for them at all on their
Razer Flip!

57. Devs dive in, everything looks
fine on their end.
No way to reproduce.

58. #wontfix

59. Let’s instrument by adding a
customer ID to the trace at
endpoint in question.

60. Grab span context,
set sampling priority,
add tag

62. Traces are a fundamental shift
from monitoring infrastructure
to software services.

63. Again, a way for you to
compress complexity and
enable developer autonomy
while diagnosing distributed
systems.

64. So….

65. Where do I begin with my
legacy infrastructure?

66. Move monolith to containers
Insert observability
Distribute ownership of services

67. Further Info:
Distributed Tracing
with APM Workshop
Example IoT App
instrumented with Traces
https://dtdg.co/iot-example

68. If you want to build tools to
enable developer autonomy,
Datadog is hiring!

69. Thank you!
Kirk Kaiser
@burningion