How Coinbase manages, moves and derives realtime insight from production containers.

1. R O B W I T O F F , D I R E C T O R
C O N T A I N E R M A N A G E M E N T
M O V E M E N T A N D I N S I G H T

2. coinbase.com
Our Container Environment

3. coinbase.com
Container Management

4. coinbase.com
Kinesis
Container Insight

6. coinbase.com

7. coinbase.com

8. coinbase.com
2012

9. coinbase.com
2015

10. coinbase.com

11. coinbase.com
We care about security,
reliability and moving fast
with the best team.

12. coinbase.com

13. coinbase.com
Our Container Environment
Container Management
Container Insight

14. coinbase.com
Our Container Environment
Container Management
Container Insight

16. Engineers
Dev Tool Prod

17. Engineers
Dev Tool Prod

18. dev
prod prod prod

19. dev
prod prod prod

20. prod prod prod

21. docker run nginx
www
docker run cache

22. www api admin

23. www api admin

24. coinbase.com

25. coinbase.com

26. coinbase.com
AMIs

27. coinbase.com

28. coinbase.com

29. coinbase.com
ish

30. coinbase.com

31. coinbase.com

32. coinbase.com
ONE Configurable Pillar

33. coinbase.com
Many Containers

34. coinbase.com

35. coinbase.com

36. coinbase.com
{
"builders": [
{
"source_ami": "ami-de0d9eb7",
"ami_name": “My CoreOS"
}
],
"provisioners": [
{
"type": "shell",
"script": "setup_things.sh"
}
]
}

37. coinbase.com
"provisioners": [
{
"type": "shell",
"inline": [
“docker pull debian”,
“docker pull ruby”,
“docker pull python”
]
}
]

38. coinbase.com

39. coinbase.com

40. coinbase.com
Our Container Environment
Container Management
Container Insight

41. coinbase.com
docker run redis
docker run ubuntu
docker run nginx

42. coinbase.com
docker run redis
docker run ubuntu
docker run nginx
docker run oops

43. coinbase.com

44. coinbase.com
registry.coinbase.com/redis
registry.coinbase.com/ubuntu
registry.coinbase.com/nginx

45. coinbase.com
coinbase/debian
latest stable
non-root users
baked into base

46. coinbase.com
coinbase/debian

47. coinbase.com
rails
ruby node nginx
coinbase/debian

48. coinbase.com
rails
ruby
web1
trader cache
node nginx
coinbase/debian

49. coinbase.com
immutable patch versions
regularly patched minor + majors
docker run node:0.10.36
docker run node:0.10

50. coinbase.com
Engineers
Dev Tool

51. coinbase.com
Engineers
Dev Tool Prod

52. coinbase.com
Engineers
Dev Tool Prod

53. coinbase.com
Engineers
Dev Tool Prod

54. coinbase.com

55. coinbase.com

56. coinbase.com

57. coinbase.com

58. coinbase.com

59. coinbase.com
Our Container Environment
Container Management
Container Insight

60. coinbase.com

61. coinbase.com

62. coinbase.com
Know. Decide. Now.

63. coinbase.com
Cloud Streams
Host Logs
Host Stats
Container Logs
Container Stats
App Logs
App Stats
Stat Processing
Log Processing
Key Event
Processing
Anomalous
Events
Archival
Elasticsearch

64. coinbase.com
Cloud Streams
Host Logs
Host Stats
Container Logs
Container Stats
App Logs
App Stats
Stat Processing
Log Processing
Key Event
Processing
Anomalous
Events
Archival
Elasticsearch

65. coinbase.com
Stat Processing
Log Processing
Key Event
Processing
Anomalous
Events
Archival
Elasticsearch

66. coinbase.com
Kinesis
→ →

67. coinbase.com
Kinesis
→ →

68. coinbase.com
syslog logspout stats

69. coinbase.com
ExecStart=
/bin/sh -c
”journalctl -f |
docker run syslog
syslog.service
syslog logspout stats

70. coinbase.com
ExecStart=docker run
-v=/var/run/docker.sock:/docker.sock
-m 20m -c 50
gliderlabs/logspout
logspout.service
syslog logspout stats
logspout-collector.service
ExecStart=docker run
—link logspout:logspout
-m 20m -c 50
logspout-collector

71. coinbase.com
ExecStart=docker run
-v=/var/run/docker.sock:/docker.sock
-v /proc:/proc
stats
stats.service
syslog logspout stats

72. coinbase.com
syslog logspout stats

73. coinbase.com
Kinesis
→ →

74. coinbase.com
Kinesis
→

75. coinbase.com
Kinesis
→

76. coinbase.com
Kinesis
→

77. coinbase.com
Kinesis
→ →

78. coinbase.com
→

79. coinbase.com
→

80. coinbase.com
→

81. coinbase.com
Kinesis
→ →

82. coinbase.com
Kinesis
→ →

83. coinbase.com
Kinesis
→ →

84. coinbase.com
Kinesis
→ →

85. coinbase.com
Own Your Images
Ship Across Environments
Give Engineers RT Insight

86. coinbase.com
@rwitoff |
Thanks!jobs@coinbase.com