This document outlines New Relic's approach to containerizing databases through a project named Megabase, addressing challenges such as supply chain delays, increasing demand, and cost efficiency. It details the marketing strategies, technological options, project requirements, and the results of implementing a containerized database solution. The presentation includes insights on managing database deployments, achieving rapid deployment times, and ensuring consistent performance across multiple databases.

1. ©2008–17 New Relic, Inc. All rights reserved.
Joshua Galbraith, Marty Matheny / March 28, 2018
Megabase: How We Containerized
Databases at New Relic

2. Confidential ©2008–17 New Relic, Inc. All rights reserved
Safe Harbor
2
This presentation and the information herein (including any information that may be incorporated by reference) is provided for informational
purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell
securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and
may not be replicated or disclosed without New Relic’s express written permission.
Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact
or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking
statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as
“believes,” “anticipates,” “expects” or words of similar import.
Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are
subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with
New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by
such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks,
uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially
from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking
statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting
New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.
New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes
no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided.

3. Confidential ©2008–17 New Relic, Inc. All rights reserved
Building Megabase
3
Context - Why we built it
Marketing - Building support
Options - Available technologies
Requirements - Choosing scope
Results - How we built it, how it works

4. Confidential ©2008–17 New Relic, Inc. All rights reserved
Building Megabase
4
Context - Why we built it
Marketing - Building support
Options - Available Technologies
Requirements - Choosing Scope
Results - How we built it, how it works

5. Confidential ©2008–17 New Relic, Inc. All rights reserved
Building Megabase
5
Context - Why we built it
Marketing - Building support
Options - Available technologies
Requirements - Choosing scope
Results - How we built it, how it works

6. Confidential ©2008–17 New Relic, Inc. All rights reserved
Building Megabase
6
Context - Why we built it
Marketing - Building support
Options - Available technologies
Requirements - Choosing scope
Results - How we built it, how it works

7. Confidential ©2008–17 New Relic, Inc. All rights reserved
Building Megabase
7
Context - Why we built it
Marketing - Building support
Options - Available Technologies
Requirements - Choosing Scope
Results - How we built it, how it works

8. Confidential ©2008–17 New Relic, Inc. All rights reserved 8
01 Context - Why we built it

9. Confidential ©2008–17 New Relic, Inc. All rights reserved
Fall 2016
🍁🍂
9

10. Confidential ©2008–17 New Relic, Inc. All rights reserved
Need a database? 4 months!!
• Quarterly hardware order
• Dell ships hardware
• Rack & stack hardware
• Provision database
Problem #1: Supply Chain
10
(3 months)
(6 weeks)
(1 week)
(2 days)

11. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Reorg → autonomous teams
• Rise of microservices
• Agility = unpredictable demand
Problem #2: Increasing Demand
11

12. Confidential ©2008–17 New Relic, Inc. All rights reserved
• 💰 $50 database, $15,000 server
• 🚫 VMware not supported
• 💵 RDS cost + latency
Problem #3: Cost Efficiency
12

13. Confidential ©2008–17 New Relic, Inc. All rights reserved
What do you think about
running database on Docker?
Because everything needs to
in our new data centers.
-- Principal Engineer
13
“
”
Nerd Sniped

14. Confidential ©2008–17 New Relic, Inc. All rights reserved
Is running databases on Docker crazy?
14

15. Confidential ©2008–17 New Relic, Inc. All rights reserved 15

16. Confidential ©2008–17 New Relic, Inc. All rights reserved 16
02 Marketing - Building support

17. Confidential ©2008–17 New Relic, Inc. All rights reserved 17
Because branding works.
Why marketing??

18. Confidential ©2008–17 New Relic, Inc. All rights reserved
Branding Matters!
18

19. Confidential ©2008–17 New Relic, Inc. All rights reserved 19

20. Confidential ©2008–17 New Relic, Inc. All rights reserved 20
Good idea != Project approval
● Projects require investment
● Teams are busy
● Competing initiatives

21. Confidential ©2008–17 New Relic, Inc. All rights reserved
• 📅 2 months for MVP
• 👤 +1 senior engineer
• ✅ Permission (leadership approval)
What We Needed
21

22. Confidential ©2008–17 New Relic, Inc. All rights reserved
Document and Socialize Plan
22

23. Confidential ©2008–17 New Relic, Inc. All rights reserved
Build Brand Awareness
23

24. Confidential ©2008–17 New Relic, Inc. All rights reserved
Sign Up Internal Customers
24

25. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Brand awareness via repetition
• Internal “press release”
• Publish adoption stats
• Broadcast customer wins
Internal Project Marketing Advice
25

26. Confidential ©2008–17 New Relic, Inc. All rights reserved 26
03 Options - Available technologies

27. Confidential ©2008–17 New Relic, Inc. All rights reserved
Open-Source Orchestration Options
27

28. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Dynamic
Provisioning
• Reservation
Labels
• Local
Persistent
Volumes
• External
Volumes
Stateful Service Support - Mesos and Marathon
28

29. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Stateful Sets
• Pods
• Headless
Service
• Persistent
Volumes
• Operators
Stateful Service Support - Kubernetes
29

30. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Jobs
• Task Groups
• Allocations
• Sticky
Volumes
• Volume
Plugins
• FS Drivers
Stateful Service Support - Nomad
30

31. Confidential ©2008–17 New Relic, Inc. All rights reserved
Joyent’s Autopilot Pattern for Databases
31

32. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Application-aware
orchestration
• Autopilot pattern for
lifecycle management
• Storage fabrics and
networked storage 😞
Containerizing Databases - Emergent Patterns
32

33. Confidential ©2008–17 New Relic, Inc. All rights reserved
• 💾 Database Software
– system stability and state distribution
• 🙇 Database Operators
– configuration and deployment
• 🙋 Database Developers
– availability and consistency
Containerizing Databases - Challenging Assumptions
33

34. Confidential ©2008–17 New Relic, Inc. All rights reserved
Stateful Service Support - Summary
34

35. Confidential ©2008–17 New Relic, Inc. All rights reserved 35
04 Requirements - Choosing scope

36. Confidential ©2008–17 New Relic, Inc. All rights reserved
📦 Build a consistent,
deployable unit for databases
🚚 Reduce database delivery
time from months to minutes
💸 Reduce wasted resources
($50 database on $15K server)
Revisiting Goals
36

37. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Our team is busy with
existing databases
• Need to ship an MVP
with limited dev time
• Devs blocked on
database delivery
• Adoption → Survival
A Compressed Timeline
37

38. Confidential ©2008–17 New Relic, Inc. All rights reserved
❏ Scheduler vs. fixed
container placement
❏ App-specific orchestration
❏ Distributed consensus
❏ Local object storage
❏ Container Pilot process
❏ Supervisor process
Making Trade-Offs
38

39. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Build on upstream images
• Run containers on our
own separate hardware
• Write an API server to
wrap Docker and LVM
• Write a management CLI
MVP Specification
39

40. Confidential ©2008–17 New Relic, Inc. All rights reserved 40
05 Results - How we built it, how it works

41. Confidential ©2008–17 New Relic, Inc. All rights reserved
1. Use upstream base image
1. Add custom labels
1. Install dependencies
1. Add custom binaries for:
○ backup sync
○ configuration sync
○ replication bootstrap
1. Entrypoint script
Docker Image Building
41

42. Confidential ©2008–17 New Relic, Inc. All rights reserved
1. Validate data mount
1. SSH client for git pull
1. Sync configs from git
1. Sync state from primary
1. Initialize database and
start database server
Docker Entrypoint
42

43. Confidential ©2008–17 New Relic, Inc. All rights reserved
• Environment variables
– via deploy config
• Configuration file
– via version control
• Docker Image
– via image registry
Injecting Configuration
43

44. Confidential ©2008–17 New Relic, Inc. All rights reserved
Megabase API Client - Deployment Manifest
44
● Container name
● Image path and tag
● Resource requirements
● VIP for LB and DNS
● Owning team
● Max severity of outage
● Environment variables

45. Confidential ©2008–17 New Relic, Inc. All rights reserved
Megabase Architecture
45

46. Confidential ©2008–17 New Relic, Inc. All rights reserved
Megabase API Server - Endpoints
46

47. Confidential ©2008–17 New Relic, Inc. All rights reserved
Megabase CLI - Deployment Process
47
● Target an environment
and a host cluster
● Specify a config file
● Generate passwords
● Send API requests
● Check responses
● Validate deployment

48. Confidential ©2008–17 New Relic, Inc. All rights reserved
• 📦 Database deployments
are consistent, repeatable
• 🚚 Database requests can
be fulfilled within minutes
• 💰 We collocate dozens of
databases per host group
Outcomes
48

49. Confidential ©2008–17 New Relic, Inc. All rights reserved
Megabase Adoption - First Year
49

50. Confidential ©2008–17 New Relic, Inc. All rights reserved
• https://mesosphere.github.io/marathon/docs/persistent-volumes.html
• https://docs.mesosphere.com/1.11/tutorials/stateful-services/
• https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
• https://youtu.be/J-Ke0TxGUSg (Kubernetes StatefulSet)
• https://coreos.com/blog/introducing-operators.html
• https://youtu.be/faUQcd5_MUc (Towards Running Stateful Applications on Nomad)
• https://github.com/hashicorp/nomad/issues/150
• https://twitter.com/kelseyhightower/status/963415653930553345
• https://twitter.com/kelseyhightower/status/963418681148502016
• https://www.joyent.com/blog/dbaas-simplicity-no-lock-in
• https://www.joyent.com/blog/persistent-storage-patterns
• https://thenewstack.io/methods-dealing-container-storage/
• https://techcrunch.com/2015/11/21/i-want-to-run-stateful-containers-too/
References
50

51. ©2008–17 New Relic, Inc. All rights reserved.
Thank you
Marty Matheny - @martymatheny | http://martymatheny.com
Joshua Galbraith - @jtgalbraith