What I Know I Don't Know about Large Scale Production Engineering


Published on

Chuma Nnaji's lightning talk from http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/calendar/15464663/

  • Be the first to comment

What I Know I Don't Know about Large Scale Production Engineering

  1. 1. What I Know I Don’t Know About Large Scale Production Engineering<br />By<br />Chuma Nnaji<br />
  2. 2. Server Roles<br />Specialized server infrastructure.<br />What types of servers work well for databases, what types work well for applications?<br />What are the different roles that a server might be specialized for?<br />
  3. 3. Application Frameworks<br />Are there any application frameworks built for scaling?<br />How much of the application should be pre-built components and how much should be hand-tweaked?<br />
  4. 4. Multiple Servers?<br />I can use multiple server to satisfy the same roles.<br />How can I implement this duplication securely?<br />What hardware, software and support staff would be needed.<br />
  5. 5. Backup<br />What should I back up?<br />How often should I backup?<br />On-site or off-site? Or both?<br />In-house or out-sourced?<br />Wait, does restore() actually work?<br />
  6. 6. Testing…<br />How do I test all the different possible situation that could come up?<br />What sort of testing frameworks should be used?<br />Benchmarking? How often?<br />Remember Schrödinger’s principle.<br />
  7. 7. Testing…Staging…<br />Testing servers.<br />Staging servers.<br />Production servers.<br />What’s the differences?<br />How powerful should these computers be?<br />How much replication should they have of each other?<br />
  8. 8. Esoteric Languages<br />When should I stop using PHP and start using Erlang?<br />Wait, what is Erlang?<br />What are the benefits of these languages and where should I be looking to apply them?<br />How do I integrate them into my existing architecture?<br />
  9. 9. Wait, What’s C Doing Here?<br />When does my web application start to look like a server extension?<br />What parts of my application should be done in low level languages and what parts should not be?<br />
  10. 10. Localization<br />Apparently, not everyone speaks English on the internet! <br />Do I need to write multiple versions of my site to work on different languages?<br />I don’t think I can just run everything through Google Translate…<br />
  11. 11. Distributed Computing<br />Distributed computing could make my application really fast and reduce the hits on a specific server cluster.<br />Involves me with potentially multiple backbones and ISPs.<br />
  12. 12. Distributed Parallel Nightmare<br />What happens to my server variables?<br />Could my application switch servers within?<br />How do I set up a distributed network?<br />
  13. 13. What is a CDN?<br />High speed broadband is not enough.<br />Sometimes we need to replace fatter pipes for more of them.<br />CDN distribute content, but to who?<br />How do you put your content up there, and how do you make sure its duplicated?<br />How best can you “cheat” your CDN?<br />
  14. 14. Legal Issues<br />Does it matter what countries I host my site in?<br />Does it matter what states I host my site’s nodes in?<br />Legal issues in countries where you’re application is hosted could impact your service.<br />
  15. 15. Encryption and HTTPS<br />Who is responsible for that?<br />Do I have to manage all of this across all my servers?<br />How does it integrate this into the architecture?<br />
  16. 16. People<br />How do I hire people to scale my product?<br />Who should I get?<br />And where do interns fit into the picture?<br />
  17. 17. Data Integrity and Security<br />Who should have access to customer records?<br />Everyone logging into the database is fine for 3 people.<br />What about 3, 000?<br />How do I control access to my database?<br />
  18. 18. Conclusion<br />There’s a lot involved in becoming large scale.<br />A lot of it has to be learned at the time.<br />But maybe it helps to know about them before-hand?<br />