This document discusses running Kafka Streams applications on Docker. It outlines some lessons learned around tuning Java Virtual Machines to work well within Docker container resource constraints, dealing with networking complexity when using containers, and ensuring there is a good use case for Docker before adopting it. The key takeaways are that Kafka Streams applications need to be tuned as regular Java applications; networking adds complexity; and Docker should only be used if it meets real infrastructure or deployment needs rather than just for the sake of using the technology.

1. 1
R U N N I N G
S T R E A M I N G
A P P S O N
D O C K E R
O N E C O U L D . . . B U T S H O U L D O N E ?

2. 2

3. 3
LEARN YOU
A DOCKER
GAMEDAY
SURPRISE
EXPERIMENT

4. 4
LEARN
YOU A
DOCKER
New Paradigms

5. 5
HOW (NOT?) TO SET UP A STREAMING PLATFORM

6. 6
K A F K A S T R E A M S
“ H e y , C o n f l u e n t c r e a t e d
t h i s r e a l l y c o o l n e w
s t r e a m i n g d a t a
f r a m ewo r k w e s h o u l d t r y
o u t . ”
HOW (NOT?) TO SET UP A STREAMING PLATFORM

7. 7
“But what if
someone else’s app
eats up all of my
resources?”

8. 8
Image credit: DesertIslandBrooklyn.com

9. 9

10. 10
K A F K A S T R E A M S D O C K E R
“ H e y , C o n f l u e n t c r e a t e d
t h i s r e a l l y c o o l n e w
s t r e a m i n g d a t a
f r a m ewo r k w e s h o u l d t r y
o u t . ”
“ Yo u k n o w w h a t w e
s h o u l d d o ? R U N O U R
S T R E A M I N G A P P S O N
D O C K E R ! ”
HOW (NOT?) TO SET UP A STREAMING PLATFORM

11. 11
Docker images
made it easy for us
to deploy anywhere
very quickly.
N O M O R E C H E F !
( W E L L , M O S T L Y . )

12. 12
Isolated containers
freed us from
systemwide
requirements.

13. 13
Build-time arguments
made our deploy
setup faster and
more flexible.

14. 14
Docker also helped
with a Kafka Connect
issue: rebalancing.

15. 15
Adding a new
connector to a KC
cluster causes a
rebalance of all
workers.

16. 16

17. 17

18. 18
K A F K A S T R E A M S D O C K E R C O N F U S I O N
“ H e y , C o n f l u e n t c r e a t e d
t h i s r e a l l y c o o l n e w
s t r e a m i n g d a t a
f r a m ewo r k w e s h o u l d t r y
o u t . ”
“ Yo u k n o w w h a t w e
s h o u l d d o ? R U N O U R
S T R E A M I N G A P P S O N
D O C K E R ! ”
“ H o w d o e s K a f k a S t r e a m s
w o r k ? W h a t ’s h a p p e n i n g
i n s i d e m y a p p ? H ow d o
yo u eve n D o c ke r ? ”
HOW (NOT?) TO SET UP A STREAMING PLATFORM

19. 19
GAMEDAY
Docker and the JVM

20. 20
GAMEDAY (n):
All your coworkers get
together to try and
make your brand new
system fall over.

21. 21
It’s great for testing
the resilience of your
system...

22. 22
It’s great for testing
the resilience of your
system...
But it also becomes a
test of its viability.

23. 23
High latency /
Slow recovery
=
NOT VIABLE

24. 24

25. 25
KA FKA STRE AMS

26. 26
KA FKA STRE AMS DO CKER SWAR M

27. 27
8 replicas
Multithreaded
> 1 billion records /
day

28. 28

29. 29
STEP 1:
Kill all but the
Swarm manager.

30. 30
STEP 1:
Kill all but the
Swarm manager.
STEP 2:
Wait for Swarm to
restart replicas.

31. 31
1 replica
+
Lots of messages
+
Lots of processing

32. 32
“It is better to fail
in originality than
to succeed in
imitation.”
H e r m a n M e l v i l l e

33. 33
After we restarted
all the replicas, lag
continued to
stagnate.

34. 34
Record lag now
numbered in the
hundreds of
millions.

35. 35
“It is better to fail
in originality than
to succeed in
imitation.”
H e r m a n M e l v i l l e

36. 36
“It is better to fail
in originality than
to succeed in
imitation.”
H e r m a n M e l v i l l e

37. 37
“It is better to fail
in originality than
to succeed in
imitation.”
H e r m a n M e l v i l l e

38. 38
In the end,
a simple
`docker stats`
revealed the
problem:

39. 39
MEMORY
CONSTRAINTS.

40. 40
Why do we want to
run JVMs inside
Docker containers?

41. 41
So we can pack
lots of them
onto one server ...
Image credit: datadoghq.com

42. 42
So we can pack
lots of them
onto one server ...
and take advantage
of Docker’s resource
constraints.
Image credit: datadoghq.com

43. 43
The JVM doesn’t
care that you’ve
set resource
constraints.

44. 44
It’s not cgroup aware.
P R O J E C T D E S C R I P T I O N
Image credit: mairin.wordpress.com

45. 45
JVM ERGONOMICS
Th e p r o ce s s by w h i c h t h e J V M t u n e s i t s e l f d e p e n d i n g o n i t s e nv i r o n m e n t .
G A R B A G E C O L L E C T I O N
I N I T I A L H E A P S I Z E
M A X I M U M H E A P S I Z E
R U N T I M E C O M P I L E R

46. 46
It can be easy to
run a single JVM
on a server-class
machine with
minimal tuning.

47. 47
But how does the
JVM behave when
you combine
ergonomics with
cgroups?

48. 48
We’ve established
that my apps were
suffering due to
memory
constraints.

49. 49
But they weren’t
being reaped
by the OOM killer…
Image credit: turnoff.us

50. 50
But they weren’t
being reaped
by the OOM killer…
They were just
running
incredibly
slowly.
Image credit: turnoff.us

51. 51
This is
generally not
something
you want to
see.

52. 52
Limiting memory
like this allows you
2x as much swap...

53. 53
... while this
constrains your swap
limit to 300M.
Limiting memory
like this allows you
2x as much swap...

54. 54
Tune your JVMs or
you might break
your system by
deploying to a
different server.

55. 55
WHAT HAVE WE LEARNED SO FAR?
L E S S O N O N E
Your Kafka Streams app is
just a regular Java app.
Tune settings carefully
and intentionally so you
can take full advantage of
containerization.

56. 56
SURPRISE
EXPERIMENT
Adventures in Networking

57. 57
My manager came
to me with an
urgent proposal.

58. 58
T R E N D I N G S E A R C H E S O N E T S Y

59. 59
RECREATE
THIS... USING
STREAMING
DATA.
T R E N D I N G S E A R C H E S O N E T S Y

60. 60
TRENDING
SEARCHES:
Aggregate query
counts, get top K.

61. 61
A company is an association or
collection of individuals, whether
natural persons, legal persons,.
Y O U R T I T L E H E R E
A company is an association or
collection of individuals, whether
natural persons, legal persons,.
Y O U R T I T L E H E R E
A company is an association or
collection of individuals, whether
natural persons, legal persons,.
Y O U R T I T L E H E R E
VALUE
NAME 1
Y O U C A N W R I T E H E R E
E n t r e p r e n e u r i a l a c t i v i t i e s d i f f e r
s u b s t a n t i a l l y d e p e n d i n g o n t h e t y p e
o f o r g a n i z a t i o n

62. 62
The aggregation
worked ... but the
interactive queries
piece didn’t.

63. 63
Sometimes I would
see data…

64. 64
... and sometimes
I would see this.
Sometimes I would
see data…
*Etsy’s 404 image.

65. 65
It didn’t seem to be
following any sort
of pattern.

66. 66
Because reloading
worked ...
Sometimes.

67. 67
At this point I was
starting to tear my
hair out...

68. 68
…and then I decided
to RTFM.

69. 69
This was the code for
the RPC layer.
P R O J E C T D E S C R I P T I O N

70. 70
This was the Docker
ser vice definition.
P R O J E C T D E S C R I P T I O N

71. 71
Can you guess what
we did wrong?

72. 72

73. 73

74. 74

75. 75
Image credit: thenewstack.io

76. 76
Overlay network:
More overhead,
higher latency

77. 77
Host network:
Lower latency,
less reusability

78. 78

79. 79
MACVLAN:
Less overhead,
more configuration

80. 80
WHAT HAVE WE LEARNED SO FAR?
L E S S O N O N E
Your Kafka Streams app is
just a regular Java app.
Tune settings carefully
and intentionally so you
can take full advantage of
containerization.
L E S S O N T W O
Networking in containers
is not always
straightforward. It adds
an extra layer of
complexity to your
application deployment.

81. 81
K A F K A S T R E A M S D O C K E R C O N F U S I O N
“ H e y , C o n f l u e n t c r e a t e d
t h i s r e a l l y c o o l n e w
s t r e a m i n g d a t a
f r a m ewo r k w e s h o u l d t r y
o u t . ”
“ Yo u k n o w w h a t w e
s h o u l d d o ? R U N O U R
S T R E A M I N G A P P S O N
D O C K E R ! ”
“ H o w d o e s K a f k a S t r e a m s
w o r k ? W h a t ’s h a p p e n i n g
i n s i d e m y a p p ? H ow d o
yo u eve n D o c ke r ? ”
HOW (NOT?) TO SET UP A STREAMING PLATFORM

82. 82
K A F K A S T R E A M S D O C K E R C O N F U S I O N
“ H e y , C o n f l u e n t c r e a t e d
t h i s r e a l l y c o o l n e w
s t r e a m i n g d a t a
f r a m ewo r k w e s h o u l d t r y
o u t . ”
“ Yo u k n o w w h a t w e
s h o u l d d o ? R U N O U R
S T R E A M I N G A P P S O N
D O C K E R ! ”
“ H o w d o e s K a f k a S t r e a m s
w o r k ? W h a t ’s h a p p e n i n g
i n s i d e m y a p p ? H ow d o
yo u eve n D o c ke r ? ”
HOW (NOT?) TO SET UP A STREAMING PLATFORM
L E A R N A L L
T H E T H I N G S !

83. 83
WHAT HAVE WE LEARNED SO FAR?
L E S S O N O N E L E S S O N T H R E E
Learning two things at
the same time is hard!
Docker will change how
you think about
deployment and failure
recovery.
Your Kafka Streams app is
just a regular Java app.
Tune settings carefully
and intentionally so you
can take full advantage of
containerization.
L E S S O N T W O
Networking in containers
is not always
straightforward. It adds
an extra layer of
complexity to your
application deployment.

84. 84
In the end,
running Kafka
Streams on Docker
turned out just fine.

85. 85
Working out the
best way to use
Docker took by far
the most time.

86. 86
WHAT HAVE WE LEARNED SO FAR?
L E S S O N O N E L E S S O N T H R E E L E S S O N F O U R
Learning two things at
the same time is hard!
Docker will change how
you think about
deployment and failure
recovery.
Your Kafka Streams app is
just a regular Java app.
Tune settings carefully
and intentionally so you
can take full advantage of
containerization.
Make sure you’re using
Docker for the right
reasons!
Otherwise, it’s not worth
the time.
L E S S O N T W O
Networking in containers
is not always
straightforward. It adds
an extra layer of
complexity to your
application deployment.

87. 87
If you’re thinking
about running your
Streams apps in
containers...

88. 88
Ask yourself these
questions.

89. 89
Do you have a good use case for
Docker?

90. 90
M U LT I P L E
D E P L O Y
S E T U P S
( F L E X I B L E
I M A G E S )

91. 91
M U LT I P L E
D E P L O Y
S E T U P S
( F L E X I B L E
I M A G E S )
R U N N I N G
AT L A R G E
S C A L E
( R E S O U R C E
C O N S T R A I N T S )

92. 92
Do you have a good use case for
Docker?
Do you have an existing Docker
setup?

93. 93
M U LT I P L E
D E P L O Y
S E T U P S
( F L E X I B L E
I M A G E S )
R U N N I N G
AT L A R G E
S C A L E
( R E S O U R C E
C O N S T R A I N T S )
I N T E G R AT I N G
W I T H E X I S T I N G
D O C K E R
S E T U P F O R
M I C R O S E R V I C E S

94. 94
Do you have a good use case for
Docker?
Do you have an existing Docker
setup?
Do you have Docker expertise on
your team?

95. 95
Do you have a good use case for
Docker?
Do you have an existing Docker
setup?
Do you have Docker expertise on
your team?
Or were you already running Kafka
Streams in production?

96. 96
Even if the answer
to all these
questions is no...

97. 97
Don’t let me
stop you.

98. 98
“It is better to fail
in originality than
to succeed in
imitation.”
H e r m a n M e l v i l l e

99. 99
FURTHER
QUESTIONS?
nikki.thean@gmail.com
E M A I L
@NikkiThean
T W I T T E R
http://bit.ly/2weFcJz
L I N K E D I N