This document summarizes lessons learned from setting up New Relic's infrastructure. It discusses issues like underestimating traffic levels, not properly separating site operations from IT work, performing maintenance at off-peak hours, using private IP addresses, choosing appropriate hardware, managing databases, avoiding long-running queries, and thanking operations staff for fixing mistakes. Key lessons include having enough IP addresses, separating duties, performing work during low usage times, using proper hardware, addressing database limitations, and appreciating operations teams.

1. Setting up New Relic's Infrastructure:
A bunch of things we did wrong and a few we
did right.
DAVE PETERSON & NIC BENDERS
2013-10-25

2. David Peterson
(dave@newrelic.com)
Nic Benders
(nic@newrelic.com)

3. [“star” graph of code base at launch]

5. 100,000req/sec
2Gbps
200TB

6. Who Does What?

7. Site Operations != IT

8. You are not Batman
Do your work during the day

9. Lowest hour is still 60% of peak hour!

10. Put your servers
on the moon

11. Never open the box

12. Domain names are cheap

13. 10.0.0.0/31
10.0.0.0/30
10.0.0.0/29
10.0.0.0/28
10.0.0.0/27
10.0.0.0/26
10.0.0.0/25
10.0.0.0/24
10.0.0.0/23
10.0.0.0/22
10.0.0.0/21
10.0.0.0/20
10.0.0.0/19
10.0.0.0/18
10.0.0.0/17
10.0.0.0/16
10.0.0.0-10.0.0.1
10.0.0.0-10.0.0.3
10.0.0.0-10.0.0.7
10.0.0.0-10.0.0.15
10.0.0.0-10.0.0.31
10.0.0.0-10.0.0.63
10.0.0.0-10.0.0.127
10.0.0.0-10.0.0.255
10.0.0.0-10.0.1.255
10.0.0.0-10.0.3.255
10.0.0.0-10.0.7.255
10.0.0.0-10.0.15.255
10.0.0.0-10.0.31.255
10.0.0.0-10.0.63.255
10.0.0.0-10.0.127.255
10.0.0.0-10.0.255.255
255.255.255.254
255.255.255.252
255.255.255.248
255.255.255.240
255.255.255.224
255.255.255.192
255.255.255.128
255.255.255.0
255.255.254.0
255.255.252.0
255.255.248.0
255.255.240.0
255.255.224.0
255.255.192.0
255.255.128.0
255.255.0.0
Private IPs
are FREE
2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
32768
65536

14. Hardware

15. System Management

16. 189 ms
167 ms
158 ms
137 ms
Average response time
X6550
E5-2450
E5-2643
E5-2690

17. 25% performance improvement
2x 6-Core X6550
@ 2.67 GHz
2x 8-Core E5-2690
@ 2.90 GHz

18. Storage
√

19. Databases

20. 32-bits isn't enough
class CreateTribbles < ActiveRecord::Migration
def change
create_table :tribbles do |t|
t.boolean :hungry
t.string :color
t.references :parent, index: true
end
end
end

21. 32-bits isn't enough
`id` int(11) NOT NULL AUTO_INCREMENT
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT
`id` SERIAL

22. Your database is not a queue

23. Long running queries
will kill you

24. And now, a thanks to the people who are out
there every day fixing the mistakes that we
made:

25. Jonathan Owens
Sean Kane
Jonathan Thurman
Paul Showalter
Aaron Bento
Karl Matthias