Download as PDF, PPTX
Uploaded byServer Density
High performance Infrastructure Oct 2013
The document discusses high performance infrastructure for Server Density which includes 150 servers that have been running since June 2009 and migrated from MySQL to MongoDB. It stores 25TB of data per month. Key aspects of performance discussed are using fast networks like 10 Gigabit Ethernet on AWS, ensuring high memory, using SSDs over spinning disks for performance, and factors like replication lag based on location. The document also compares options like using cloud, dedicated servers, or colocation and discusses monitoring, backups, dealing with outages, and other operational aspects.
More Related Content
Similar to High performance Infrastructure Oct 2013
![Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]](https://cdn.slidesharecdn.com/ss_thumbnails/thursday115ionfs-141107125307-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds)
High performance Infrastructure Oct 2013
- 1.
- 2.
- 4.
- 5.
- 6. Server Density Infrastructure •150servers • June 2009 - 4yrs
- 7. Server Density Infrastructure •150servers • June 2009 - 4yrs •MySQL -> MongoDB
- 8. Server Density Infrastructure •150servers • June 2009 - 4yrs •MySQL -> MongoDB •25TB data per month
- 9. Performance • Fast network Pictureis unrelated! Mmm, ice cream.
- 10. Performance • Fast network EC210 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage - Network cards - VLAN separation
- 11. Performance • Fast network Workload:Read/Write? What is being stored? Result set size - Read / write: adds to replication oplog - Images? Web pages? Tiny documents? - What is being returned? Optimised to return certain fields?
- 12. Performance • Fast network Use NetworkThroughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog
- 13. Performance • Fast network Inter-DCLAN - Latency
- 14. Performance • Fast network Inter-DCLAN Cross USA Washington, DC - San Jose, CA
- 15. Performance • Fast network Location PingRTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms Ping - low overhead Important for replication
- 16.
- 17. Failover •Replication •Master/slave - One masteraccepts all writes - Many slaves staying up to date with master - Can read from slaves
- 18. Failover •Replication •Master/slave •Min 3 nodes Minimumof 3 nodes to form a majority in case one goes down. All store data. Odd number otherwise != majority Arbiter
- 19. Failover •Replication •Master/slave •Min 3 nodes •Automaticfailover Drivers handle automatic failover. First query after a failure will fail which will trigger a reconnect. Need to handle retries
- 20. Performance •Replication lag Location Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe- Japan - Replication lag Ping RTT Latency 300ms
- 21. Replication Lag 1. Reads:eventual consistency
- 22. Replication Lag 1. Reads:eventual consistency 2. Failover: slave behind
- 23. Eventual Consistency Stale data Notwhat the user submitted?
- 24. Eventual Consistency Stale data Inconsistentdata Doesn’t reflect the truth
- 25. Eventual Consistency Stale data Inconsistentdata Changing data Could change on every page refresh
- 26. Eventual Consistency Use Case Needsconsistency? Graphs No User profile Yes Statistics Depends Alert config Yes Statistics - depends on when they’re updated
- 27. Replication Lag 1. Reads:eventual consistency 2. Failover: slave behind
- 28. Slave behind Failover: outof date master Old data Rollback
- 29. MongoDB WriteConcern • Safeby default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value 0 1 2 3 wtimeout - wait for write before raising an exception Meaning Unsafe Primary Primary + x1 secondary Primary + x2 secondaries
- 31. Performance • Fast network •MoreRAM Picture is unrelated! Mmm, ice cream.
- 32. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs No 32 bit NoHigh CPU RAM RAM RAM.
- 33.
- 34.
- 35. Performance More RAM =expensive x2 4GB RAM 12 month Prices
- 36.
- 37.
- 38.
- 39. Performance SSD vs Spinning SSDsare better at buffered disk reads, sequential input and random i/o.
- 40. Performance SSD vs Spinning However,CPU usage for SSDs is higher. This may be a driver issue so worth testing your own hardware. Tests done using Bonnie.
- 41.
- 42.
- 43.
- 44. •Elastic workloads •Demand spikes •Unknownrequirements Cloud?
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52. •Hardware spec/value •Total cost •Internalskills? Colo?
- 53. •Hardware spec/value •Total cost •Internalskills? •More fun?! Colo?
- 55. Colo experiment • Buildmaster (buildbot): VM x2 CPU 2.0Ghz, 2GB RAM – $89/m • Build slave (buildbot): VM x1 CPU 2.0Ghz, 1GB RAM – $40/m • Staging load balancer: VM x1 CPU 2.0Ghz, 1GB RAM – $40/m • Staging server 1: VM x2 CPU 2.0Ghz, 8GB RAM – $165/m • Staging server 2: VM x1 CPU 2.0Ghz, 2GB RAM – $50/m • Puppet master: VM x2 CPU 2.0Ghz, 2GB RAM – $89/m Total: $473/m
- 56. Colo experiment •Dell 1UR415 •x2 8C AMD 2.8Ghz •32GB RAM
- 57. Colo experiment •Dell 1UR415 •x2 8C AMD 2.8Ghz •32GB RAM •Dual PSU, NIC
- 58. Colo experiment •Dell 1UR415 •x2 8C AMD 2.8Ghz •32GB RAM •Dual PSU, NIC •x4 1TB SATA hot swappable
- 60. Colo: Networking •10-50Mbps: £20-25/Mbps/m •51-100Mbps:£15/Mbps/m •100+Mbps: £13/Mbps/m
- 61.
- 62. •£300-350/kWh/m •4.5A = £520/m •9A= £900/m Colo: Power
- 64. Tips: rand() •Field names -Fieldnames take up space
- 65. Tips: rand() •Field names •Coveredindexes - Get everything from the index
- 66. Tips: rand() •Field names •Coveredindexes •Collections / databases - Dropping collections faster than remove() - Split use cases across databases to avoid locking - Put databases onto different disks / types e.g. SSDs
- 67. Backups What is theuse case?
- 68. Backups What is theuse case? Fixing user errors? Point in time restore? Disaster recovery?
- 69. Backups •Disaster recovery Offsite - Whatkind of disaster? - Store backups offsite
- 70. Backups •Disaster recovery Offsite Age How logdo you keep the backups for? How far do they go back? How recent are they?
- 71. Backups •Disaster recovery Offsite Age Restore time Latencyissue - further away geographically, slower the transfer time Partition backups to get critical data restored first
- 72. david@asriel ~: scpdavid@stelmaria:~/local/local.11 . local.11 100% 2047MB 6.8MB/s 05:01 Restore time - Needed to resync a database server across the US - Take too long; oplog not large enough - Fast internal network but slow internet
- 73.
- 74. Backups Frequency Consistency Verification - How often? -Backing up cluster at the same time - data moving around - Can the backups be restored?
- 75. Monitoring •System Disk i/o Disk use www.flickr.com/photos/daddo83/3406962115/ Diski/o % util Disk space usage
- 76. david@pan ~: df-a Filesystem /dev/sda1 proc none none none binfmt_misc david@pan ~: df -ah Filesystem /dev/sda1 proc none none none 1K-blocks Used Available Use% Mounted on 156882796 148489776 423964 100% / 0 0 0 - /proc 0 0 0 - /dev/pts 2097260 0 2097260 0% /dev/shm 0 0 0 - /proc/sys/fs/ Size 150G 0 0 2.1G 0 - Needed to upgrade a machine - Resize = downtime - Resyncing finished just in time Used Avail Use% Mounted on 142G 415M 100% / 0 0 - /proc 0 0 - /dev/pts 0 2.1G 0% /dev/shm 0 0 - /proc/sys/fs/binfmt_
- 77. Monitoring •System Disk i/o Disk use Swap www.flickr.com/photos/daddo83/3406962115/ Diski/o % util Disk space usage
- 78.
- 79. Monitoring tools Run yourself Ganglia SoServer Density is the tool my company produces but if you don’t like it, want to run your own tools locally or just want to try some others, then that’s fine.
- 80.
- 81. Dealing with humans On-call - Sharingout the responsibility Determining level of response: 24/7 real monitoring or first responder 24/7 real monitoring for HA environments, real people at a screen at all times First responder: people at the end of a phone
- 82. Dealing with humans On-call 1)Ops engineer - During working hours our dedicated ops engineers take the first level - Avoids interrupting product engineers for initial fire fighting
- 83. Dealing with humans On-call 1)Ops engineer 2) All engineers - Out of hours we rotate every engineer, product and ops - Rotation every 7 days on a Tuesday
- 84. Dealing with humans On-call 1)Ops engineer 2) All engineers 3) Ops engineer - Always have a secondary - This is always an ops engineer - Thinking is if the issue needs to be escalated then it’s likely a bigger problem that needs additional systems expertise
- 85. Dealing with humans On-call 1)Ops engineer 2) All engineers 3) Ops engineer 4) Others - Support from design / frontend engineering - Have to press a button to get them involved
- 86. Dealing with humans Off-call -Responders to an incident get next 24 hours off-call - Social issues to deal with
- 87. Dealing with humans On-callCEO - I receive push notifications + e-mails for all outages
- 88. Dealing with humans Uptimereporting - Weekly internal report on G+ - Gives visibility to entire company about any incidents - Allows us to discuss incidents to get to that 100% uptime
- 89. Dealing with humans Socialissues - How quickly can you get to a computer? Are they out drinking on a Friday? What happens if someone is ill? What if there’s a sudden emergency: accident? family emergency? Do they have enough phone battery? Can you hear the ringtone?
- 90. Dealing with humans Backupresponder - Backup responder Time out the initial responder Escalate difficult problems Essentially human redundancy: phone provider, geographic area, internet connectivity
- 91. Dealing with outages Expected -Outages are going to happen, especially at the beginning - Costs money for redundancy - How you deal with them
- 92. Dealing with outages Communication Externally -Telling people what is happening - Frequently - Dependent on audience - we can go into more detail because our customers are techies - Github do a good job of providing incident writeups but don’t provide a good idea of what is happening right now - Generally Amazon and Heroku are good and go into more detail
- 93. Dealing with outages Communication Internally -Open Skype conferences between the responders - Usually mostly silence or the sound of the keyboard, but simulates being in the situation room - Faster than typing
- 94. Dealing with outages Reallytest your vendors - Shows up flaws in vendor support processes Frustrating when waiting on someone else You want as much information as possible Major outage? Everyone will be calling them
- 95. Dealing with outages Simulations -Try and avoid unncessary problems - Do servers come back up from boot? - Can hot spares handle the load? - Test failover: databases, HA firewalls - Regularly reboot servers - Wargames can happen at another stage: startups are usually too focused on building things first
- 96.