In a typical day, New Relic receives half a trillion data points from our customers. All of these need to be processed, stored and made available for charts, queries and alerts as soon as possible. Getting from humble roots to querying more than a billion events a second has meant radically reinventing and re-architecting the whole platform multiple times. In this session, Matthew Flaming and Nic Benders from New Relic will take you on a deep dive into some of the lessons we've learned along the way and where we go from here. They'll touch on designing scalable services, optimizing for predictability, creating software architecture through team structure, and what it takes to (continuously) rebuild a system that can never stop.

1. Confidential ©2008-15 New Relic, Inc. All rights reserved.
Building a System
that Never Stops:
New Relic at Scale
1
Matthew Flaming Nic Benders

2. Confidential ©2008-15 New Relic, Inc. All rights reserved.

4. EVERY MINUTE
requestsaccepts
over 16M
stores
over
analytic
events2M
aggregates
over 600M metrics
3Bqueries
over
data
points
▪

5. ▪

6. different
services
contains
over
200
maintained/
built by
25+ engineering
teams
▪
2.5more
than
SSD
storage
PETABYTES

7. 1. NOTHING Lasts Forever
2. Run EXPERIMENTS
3. SYNCHRONOUS CALLS are going to be a problem
4. Master the ROLLOUT
5. NEW TECH = NEW CHALLENGES
6. Use the right WORKLOAD DISTRIBUTION
7. Technology enables CULTURE
8. Software architecture: THE BIG PICTURE
The New Relic lesson plan

8. Collector
New Relic
App Agents

9. Confidential ©2008-15 New Relic, Inc. All rights reserved.
▪ Double-click to edit

10. Collector
New Relic
App Agents
Browser R.U.M.
Beacon
New Relic
Browser Agents
Mobile Proxy
New Relic
Mobile Agents

11. Collector
Browser R.U.M.
Beacon
Mobile Proxy
Zzz
New Relic
App Agents
New Relic
Browser Agents
New Relic
Mobile Agents

12. Confidential ©2008-15 New Relic, Inc. All rights reserved.

13. NOTHING
Lasts Forever
Lesson 1

14. Run
EXPERIMENTS
Lesson 2

15. Confidential ©2008-15 New Relic, Inc. All rights reserved.

16. Confidential ©2008-15 New Relic, Inc. All rights reserved.

17. EXPERIMENTS
don’t always work
Corollary to 2

18. SYNCHRONOUS CALLS
are going to be a problem
Lesson 3

19. ▪
Kafka
Kafka
Browser Data
Consumer
Browser Data
Router
New Relic
App Agents
New Relic
Browser Agents
Collector

20. ▪
Kafka
Kafka
Browser Data
Consumer
New Relic
App Agents
Collector
Zzz
Browser Data
Router
New Relic
Browser Agents

21. Confidential ©2008-15 New Relic, Inc. All rights reserved.

22. Data Routers
Kafka Topics
Data Consumers
Data Stores/DBs
Retrieval Services
Ingest
Queue
Process/Aggregate
Store
Query

23. Data Routers
Kafka Topics
Data Consumers
Data Stores/DBs
Retrieval Services

24. Master the
ROLLOUT
Lesson 4

25. Rollout
Techniques
Kafka
Load Balancer Cluster
Kafka
Browser Data
Consumer
Browser Data
Router
Browser
Data
Consumer
V2
Browser
R.U.M.
Beacon
%- %+
New Relic Browser Agents
Collector
Old
New

26. NEW TECH
NEW CHALLENGES
Lesson 5
=

27. Monitoring Tooling Operations Runbooks

28. Confidential ©2008-15 New Relic, Inc. All rights reserved.

29. Lesson 6
Use the right
WORKLOAD
DISTRIBUTION

30. ©2008-15 New Relic, Inc. All rights reserved.
▪
Workload Distribution
A
C
B
D
D
D
D
D
B
C B BB
B
D D
A
A
A
A
A DD
Active Management Random
C B BBD DAA D C
B BD
B C
AA D
D
C
C
A B

31. Technology enables
CULTURE
Lesson 7

32. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166

33. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166
�����

34. Software Architecture:
THE BIG PICTURE
Lesson 8

35. Software Architecture: The Big Picture
▪
▪
▪
▪
▪
Organic
growth
Paradigm
shift
Breaking
point

36. New Paradigm
Collector
New Relic
App Agents

37. Organic Growth
Collector
New Relic
App Agents
Browser R.U.M.
Beacon
New Relic
Browser Agents
Mobile Proxy
New Relic
Mobile Agents

38. Breaking Point
Collector
Browser R.U.M.
Beacon
Mobile Proxy
Zzz
New Relic
App Agents
New Relic
Browser Agents
New Relic
Mobile Agents

39. ▪
Kafka
Kafka
Browser Data
Consumer
Browser Data
Router
New Relic
App Agents
New Relic
Browser Agents
Collector

40. Confidential ©2008-15 New Relic, Inc. All rights reserved.
▪ Double-click to edit

41. Confidential ©2008-15 New Relic, Inc. All rights reserved.
▪ Double-click to edit

43. Confidential ©2008-15 New Relic, Inc. All rights reserved.
43
Thank you
Matthew Flaming Nic Benders