Setting up New Relic's Infrastructure:
A bunch of things we did wrong and a few we
did right.
DAVE PETERSON & NIC BENDERS
...
David Peterson
(dave@newrelic.com)

Wednesday, November 6, 13

Nic Benders
(nic@newrelic.com)
[“star” graph of code base at launch]

Wednesday, November 6, 13
Wednesday, November 6, 13
100,000req/sec
2Gbps
200TB
Wednesday, November 6, 13
Who Does What?
Wednesday, November 6, 13
Site Operations != IT

Wednesday, November 6, 13
Wednesday, November 6, 13
You are not Batman

Do your work during the day
Wednesday, November 6, 13
Lowest hour is still 60% of peak hour!

Wednesday, November 6, 13
Put your servers
on the moon
Wednesday, November 6, 13
Put your servers
on the moon
Wednesday, November 6, 13
Never open the box

Wednesday, November 6, 13
Domain names are cheap

Wednesday, November 6, 13
10.0.0.0/31
10.0.0.0/30
10.0.0.0/29
10.0.0.0/28
10.0.0.0/27
10.0.0.0/26
10.0.0.0/25
10.0.0.0/24
10.0.0.0/23
10.0.0.0/22
10...
Hardware
Wednesday, November 6, 13
System Management

Wednesday, November 6, 13
189 ms

158 ms

137 ms

X6550

Wednesday, November 6, 13

167 ms

Average response time

E5-2450

E5-2643

E5-2690
25% performance improvement

2x 6-Core X6550
@ 2.67 GHz

Wednesday, November 6, 13

2x 8-Core E5-2690
@ 2.90 GHz
Storage

√

Wednesday, November 6, 13
Databases
Wednesday, November 6, 13
32-bits isn't enough
class CreateTribbles < ActiveRecord::Migration
def change
create_table :tribbles do |t|
t.boolean :hu...
32-bits isn't enough
`id` int(11) NOT NULL AUTO_INCREMENT

`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT

`id` SERIAL

...
Your database is not a queue

Wednesday, November 6, 13
Long running queries
will kill you

Wednesday, November 6, 13
And now, a thanks to the people who are out
there every day fixing the mistakes that we
made:

Wednesday, November 6, 13
Jonathan Owens

Jonathan Thurman

Paul Showalter

Aaron Bento

Wednesday, November 6, 13

Sean Kane

Karl Matthias
Upcoming SlideShare
Loading in …5
×

FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

2,928 views

Published on

Benefit from our mistakes and hindsight! There's a lot that goes into running a SaaS company - from continuous deployment to DNS to who gets to play with production. We've been running New Relic for over five years now, but if we had to do it over again there are a few things we'd do differently. But there's also a couple things we'd repeat.
Come with your questions to this interactive session where we dissect some of the "big decisions" we made in the world of technical operations and how they effected the company and culture.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,928
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

FUTURESTACK13: Setting up New Relic's Infrastructure from Nic Benders & Dave Peterson, Site Engineering Manager & VP of Operations at New Relic

  1. 1. Setting up New Relic's Infrastructure: A bunch of things we did wrong and a few we did right. DAVE PETERSON & NIC BENDERS 2013-10-25 Wednesday, November 6, 13
  2. 2. David Peterson (dave@newrelic.com) Wednesday, November 6, 13 Nic Benders (nic@newrelic.com)
  3. 3. [“star” graph of code base at launch] Wednesday, November 6, 13
  4. 4. Wednesday, November 6, 13
  5. 5. 100,000req/sec 2Gbps 200TB Wednesday, November 6, 13
  6. 6. Who Does What? Wednesday, November 6, 13
  7. 7. Site Operations != IT Wednesday, November 6, 13
  8. 8. Wednesday, November 6, 13
  9. 9. You are not Batman Do your work during the day Wednesday, November 6, 13
  10. 10. Lowest hour is still 60% of peak hour! Wednesday, November 6, 13
  11. 11. Put your servers on the moon Wednesday, November 6, 13
  12. 12. Put your servers on the moon Wednesday, November 6, 13
  13. 13. Never open the box Wednesday, November 6, 13
  14. 14. Domain names are cheap Wednesday, November 6, 13
  15. 15. 10.0.0.0/31 10.0.0.0/30 10.0.0.0/29 10.0.0.0/28 10.0.0.0/27 10.0.0.0/26 10.0.0.0/25 10.0.0.0/24 10.0.0.0/23 10.0.0.0/22 10.0.0.0/21 10.0.0.0/20 10.0.0.0/19 10.0.0.0/18 10.0.0.0/17 10.0.0.0/16 Wednesday, November 6, 13 10.0.0.0-10.0.0.1 10.0.0.0-10.0.0.3 10.0.0.0-10.0.0.7 10.0.0.0-10.0.0.15 10.0.0.0-10.0.0.31 10.0.0.0-10.0.0.63 10.0.0.0-10.0.0.127 10.0.0.0-10.0.0.255 10.0.0.0-10.0.1.255 10.0.0.0-10.0.3.255 10.0.0.0-10.0.7.255 10.0.0.0-10.0.15.255 10.0.0.0-10.0.31.255 10.0.0.0-10.0.63.255 10.0.0.0-10.0.127.255 10.0.0.0-10.0.255.255 255.255.255.254 255.255.255.252 255.255.255.248 255.255.255.240 255.255.255.224 255.255.255.192 255.255.255.128 255.255.255.0 255.255.254.0 255.255.252.0 255.255.248.0 255.255.240.0 255.255.224.0 255.255.192.0 255.255.128.0 255.255.0.0 Private IPs are FREE 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536
  16. 16. Hardware Wednesday, November 6, 13
  17. 17. System Management Wednesday, November 6, 13
  18. 18. 189 ms 158 ms 137 ms X6550 Wednesday, November 6, 13 167 ms Average response time E5-2450 E5-2643 E5-2690
  19. 19. 25% performance improvement 2x 6-Core X6550 @ 2.67 GHz Wednesday, November 6, 13 2x 8-Core E5-2690 @ 2.90 GHz
  20. 20. Storage √ Wednesday, November 6, 13
  21. 21. Databases Wednesday, November 6, 13
  22. 22. 32-bits isn't enough class CreateTribbles < ActiveRecord::Migration def change create_table :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end end end Wednesday, November 6, 13
  23. 23. 32-bits isn't enough `id` int(11) NOT NULL AUTO_INCREMENT `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT `id` SERIAL Wednesday, November 6, 13
  24. 24. Your database is not a queue Wednesday, November 6, 13
  25. 25. Long running queries will kill you Wednesday, November 6, 13
  26. 26. And now, a thanks to the people who are out there every day fixing the mistakes that we made: Wednesday, November 6, 13
  27. 27. Jonathan Owens Jonathan Thurman Paul Showalter Aaron Bento Wednesday, November 6, 13 Sean Kane Karl Matthias

×