Web 2.0 Presentation Sept 2008

1,012 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,012
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Web 2.0 Presentation Sept 2008

  1. 1. Building a Kickass Network Dynamic Network Services Tom Daly, CTO [email_address]
  2. 2. About Me <ul><li>With Dyn Inc since 2001 </li></ul><ul><li>Helped scale the operations from 100k users to 2.5M users, over 100k domain names, 8 global datacenters </li></ul><ul><ul><li>O(B) queries per day </li></ul></ul><ul><ul><li>O(M) updates processed per day </li></ul></ul><ul><li>Previously worked for a New Hampshire telco, deploying DSL to residences </li></ul>
  3. 3. What is DNS? 192.168.15.102 Comcast User 199.192.8.2 (cnn.com server) www.cnn.com? DNS Server 199.192.8.2 DNS Query Web Page Delivery
  4. 4. What can Dyn Inc do? <ul><li>DynDNS – Consumer Dynamic DNS Service </li></ul><ul><ul><li>Use our domain or register your own domain </li></ul></ul><ul><ul><li>E-mail services, SSL certificates, VPS hosting </li></ul></ul><ul><li>Dynect – Enterprise DNS Service </li></ul><ul><ul><li>Built for businesses whose web site presence is critical, failover, load balancing, global traffic management </li></ul></ul><ul><ul><li>Global anycast network with 8 locations online </li></ul></ul>
  5. 5. DynDNS – our talk’s focus <ul><li>We started the service in 1998 so we could remotely access our PCs on Dynamic IP addresses. </li></ul><ul><li>Instead of knowing what our current IP address was, we just remember a hostname (tomdaly.dyndns.org) and the DynDNS system keeps it updated with the right IP. </li></ul>
  6. 6. What can you do with DynDNS? <ul><li>Share web pages, photos, legal music, downloads to your friends </li></ul><ul><li>Access remote desktop, VNC, your security system, cameras, thermostats </li></ul><ul><li>Laptop lo-jack </li></ul><ul><li>Gather data from remote weather stations </li></ul><ul><li>Update Amber Alert signs </li></ul>
  7. 7. Starting Small - 1998 <ul><li>Website Form updates BIND </li></ul><ul><li>CGI script and text file DB </li></ul><ul><li>Functions: </li></ul><ul><ul><li>Log in, log out, get a hostname, set an IP </li></ul></ul><ul><li>Open Source Software: </li></ul><ul><ul><li>Perl with Net::DNS module </li></ul></ul><ul><ul><li>Apache 1.2 with CGI </li></ul></ul>
  8. 8. Next <ul><li>Designed /nic/update </li></ul><ul><ul><li>RESTful update interface </li></ul></ul><ul><ul><li>/nic/update?hostname=test.dyndns.org&myip=1.2.3.4 </li></ul></ul><ul><li>Does the same job as the website CGI </li></ul><ul><li>5000 users on a 56K dialup modem </li></ul>
  9. 9. CGI got slow - 1999 <ul><li>Multiple users didn’t work – locking problem </li></ul><ul><li>Solutions: </li></ul><ul><ul><li>MySQL 3.23 </li></ul></ul><ul><ul><li>mod_perl </li></ul></ul><ul><ul><li>Apache::Registry </li></ul></ul><ul><li>Problems: </li></ul><ul><ul><li>CPU </li></ul></ul><ul><ul><li>RAM </li></ul></ul><ul><li>Basic colo in Ithaca and 20,000 users </li></ul>
  10. 10. Users demand more - 2001 <ul><li>Users have: “username.dyndns.org” </li></ul><ul><li>Users want: “www.username.com” </li></ul><ul><li>Problems: </li></ul><ul><ul><li>More metadata to store </li></ul></ul><ul><ul><li>More interaction with BIND (setting up zones) </li></ul></ul><ul><li>Solution: </li></ul><ul><ul><li>Lots of shell scripting glue </li></ul></ul><ul><ul><li>High-end collocation facility, 5 servers and a switch </li></ul></ul>
  11. 11. Infrastructure Layout <ul><li>Internet </li></ul>Firewall Switch DNS/Web DNS/Web DNS/Web Database
  12. 12. Basic Scaling <ul><li>DNS round robin for web servers </li></ul><ul><ul><li>www.dyndns.org 300 A 10.0.0.11 </li></ul></ul><ul><ul><li>www.dyndns.org 300 A 10.0.0.12 </li></ul></ul><ul><ul><li>www.dyndns.org 300 A 10.0.0.13 </li></ul></ul><ul><li>frontend and backend Apache servers </li></ul><ul><ul><li>Frontend to spoon feed slow web clients </li></ul></ul><ul><ul><li>Backend to pool MySQL database connections </li></ul></ul><ul><li>Tweak my.cnf for MySQL </li></ul>
  13. 13. Redundancy <ul><li>DNS cannot go down, ever. </li></ul><ul><li>Ithaca colo for secondary DNS service </li></ul><ul><li>Managed servers for secondary DNS </li></ul><ul><ul><li>Consider companies like: </li></ul></ul><ul><ul><ul><li>Rackspace </li></ul></ul></ul><ul><ul><ul><li>Hurricane Electric </li></ul></ul></ul><ul><ul><ul><li>Server Beach </li></ul></ul></ul><ul><ul><ul><li>The Planet </li></ul></ul></ul><ul><ul><ul><li>Many many others </li></ul></ul></ul>
  14. 14. Continued Growth - 2002 <ul><li>Rewrote the CGI running members.dyndns.org </li></ul><ul><li>Used a perl based templating system </li></ul><ul><li>Created chunks of reusable code </li></ul><ul><li>Very hard to maintain, hard to involve new developers, hard to debug </li></ul><ul><li>Website – migrated over to PHP </li></ul><ul><ul><li>Mistake for us, spread our competencies out </li></ul></ul>
  15. 15. Rewrite - 2003 <ul><li>Intergraded www.dyndns.org and members.dyndns.org into one site </li></ul><ul><li>Database and business logic abstraction layer </li></ul><ul><ul><li>Perl </li></ul></ul><ul><ul><li>Heavy lifting goes here. </li></ul></ul><ul><li>HTML::Mason – embed perl code in HTML </li></ul><ul><li>Great for synchronous job processing </li></ul>
  16. 16. Scaling and Redundancy <ul><li>MySQL Replication </li></ul><ul><ul><li>Problem: Long running queries block </li></ul></ul><ul><ul><li>Solution: Replicate data, do queries there </li></ul></ul><ul><li>Load Balancing </li></ul><ul><ul><li>Problem: Web servers break </li></ul></ul><ul><ul><li>Solution: Buy a load balancer ($$$) </li></ul></ul><ul><ul><li>Today: Use Dynect (more later) </li></ul></ul><ul><li>Other single points of failure </li></ul><ul><ul><li>Firewalls, Dual PSU in DB server, A+B Power </li></ul></ul>
  17. 17. More Scaling Tips <ul><li>Performance is slow: </li></ul><ul><ul><li>Disk vs. CPU vs. Network bound? </li></ul></ul><ul><ul><li>Use tools: ps, top, netstat, vmstat </li></ul></ul><ul><ul><li>man(3) pages are extremely helpful here </li></ul></ul><ul><li>Know your hardware: </li></ul><ul><ul><li>RAID Levels – different performance benefits </li></ul></ul><ul><ul><li>32-bit CPU vs. 64-bit CPU </li></ul></ul><ul><ul><li>RAM, RAM, RAM </li></ul></ul><ul><ul><li>Fast Ethernet? Gigabit Ethernet? </li></ul></ul>
  18. 18. Monitoring <ul><li>Is your site up? Use Nagios </li></ul><ul><li>Is it fast? Use Smokeping </li></ul><ul><li>Look at the internals: Cacti, Munin </li></ul><ul><li>Know where your weak spots are </li></ul><ul><ul><li>Logging – what are people doing? </li></ul></ul><ul><ul><li>MySQL slow query log – Is the DB suffering? </li></ul></ul><ul><ul><li>Exception handlers – e-mail to yourself </li></ul></ul><ul><ul><li>Profiling – print time(); stuff(); print time(); </li></ul></ul>
  19. 19. Nagios
  20. 20. Smokeping
  21. 21. Cacti
  22. 22. Network is down? <ul><li>Console servers </li></ul><ul><li>POTS lines and modems </li></ul><ul><li>Views </li></ul><ul><ul><li>Outside to inside </li></ul></ul><ul><ul><li>Inside to outside </li></ul></ul><ul><ul><li>Third party monitoring </li></ul></ul>
  23. 23. DDoS <ul><li>We’ve had a lot of ‘em </li></ul><ul><li>Stay calm, get your ISP involved right away </li></ul><ul><li>Know your network </li></ul><ul><li>tcpdump on firewalls </li></ul><ul><li>Block destination to recover your network </li></ul><ul><li>Block sources at your ISP to make it stop </li></ul><ul><li>Don’t make yourself a target </li></ul>
  24. 24. 2003 through 2006 <ul><li>Colocation </li></ul><ul><ul><li>Dedicated racks, no more managed servers </li></ul></ul><ul><ul><li>Added Chicago, Palo Alto, Hong Kong </li></ul></ul><ul><li>New hardware </li></ul><ul><ul><li>BGP routing, multiple carriers everywhere </li></ul></ul><ul><ul><li>High end routing gear, switches </li></ul></ul><ul><li>Add MailHop </li></ul><ul><li>Add DNS SLA </li></ul>
  25. 25. Scaling /nic/update <ul><li>2.5MM update clients hitting this API. </li></ul><ul><li>Apache/mod_perl/mysql </li></ul><ul><li>SQLThrottle – distribute R and RW DB connections to different servers </li></ul><ul><li>Same DB queries – use memcached </li></ul>
  26. 26. memcached rocks! <ul><li>Memory cache for database calls </li></ul><ul><li>Update, insert, delete – invalidate the cache </li></ul><ul><li>Authentication caching, hammers </li></ul><ul><li>Consider UDFs </li></ul>
  27. 27. In-house vs. Outsourced Development <ul><li>When should you outsource? </li></ul><ul><ul><li>Don’t have the talent or knowledge </li></ul></ul><ul><ul><li>Need more human staff </li></ul></ul><ul><li>When shouldn’t you outsource? </li></ul><ul><ul><li>Complex needs or variables </li></ul></ul><ul><ul><li>Takes longer to spec out than just do yourself </li></ul></ul>
  28. 28. Cloud Computing <ul><li>We’re not using it yet, but we’re looking at it </li></ul><ul><ul><li>On demand usage </li></ul></ul><ul><ul><ul><li>Crunch logs </li></ul></ul></ul><ul><ul><ul><li>Analyze statistics </li></ul></ul></ul><ul><ul><ul><li>Occasional jobs </li></ul></ul></ul><ul><li>We won’t use the cloud for services </li></ul><ul><ul><li>We need control of everything </li></ul></ul><ul><ul><li>Not guaranteed for speed or stability </li></ul></ul>
  29. 29. Dynect <ul><li>Enterprise DNS service for business customers </li></ul><ul><ul><li>Multi User Support </li></ul></ul><ul><ul><li>Anycast DNS Network (8 facilities globally) </li></ul></ul><ul><ul><li>Logging and Reporting </li></ul></ul><ul><ul><li>Failover </li></ul></ul><ul><ul><li>Load Balancing </li></ul></ul><ul><ul><li>/nic/update and SOAP APIs </li></ul></ul>
  30. 30. Dynect (con’t) <ul><li>Rewrote our business logic and database abstraction to be more coherent </li></ul><ul><li>Web code never makes decisions, ever, all in the BLL </li></ul><ul><li>Huge amounts of unit and system testing every night </li></ul><ul><li>Uses MySQL UDFs to talk to BIND, rather than Perl – less work for the developers ongoing </li></ul>
  31. 31. Anycast <ul><li>Distributes the same IP address globally for DNS queries </li></ul><ul><li>Queries come to local servers, not randomly picked </li></ul><ul><li>Speeds DNS query latency from ~50 – 300ms to a guaranteed 100ms, typ. 50ms </li></ul><ul><li>You have to know BGP, get your ISPs to work with you, much secret sauce </li></ul><ul><li>Completely redundant, hard to monitor </li></ul>
  32. 32. Summary <ul><li>Start small, grow smart, get big </li></ul><ul><li>Get what you need, don’t over build out </li></ul><ul><li>Invest into open source software </li></ul><ul><li>Determine the right infrastructure: </li></ul><ul><ul><li>Hosted, Managed, Your Own, Cloud </li></ul></ul><ul><li>Plan for scaling (scaling != more boxes) </li></ul><ul><li>Listen to what your user base wants </li></ul>
  33. 33. Thank You Q & A

×