Practice and challenges from building IaaS


Published on

It is an invited presentation for NCSC2012 (China National Conference on Social Computing) on cloud computing from industry.
It summarized what we learn on developing and operating an Infrastructure as a Service in a highly scalable manner. The service described inside the corporation is kind of dogfood that engineers work with in their daily work.

Published in: Technology
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Practice and challenges from building IaaS

  1. 1. Practice and Challenges fromBuilding Infrastracture-as-a-Service 朱可
  2. 2. Disclaimer● Representing personal opinion only
  3. 3. IaaS in Our Development Lab● Virtual machine● Block storage● Virtual machine template● VLAN● Static ip address● Virtual Desktop $ ./iaas-deploy-vms -i centos63 -n 100 $ ./iaas-deploy-vms -i centos63 -n 100
  4. 4. The Machinery Node: 16 Cores 192GB RAM 1,6TBRack: 20+ nodes, 2 rack switches
  5. 5. Quick Stats● 5,800 VMs provisioned in 2 months● 700+ individual visitors per month● 50,000+ requests to web services per single day – Less than 40% requests are sent by human
  6. 6. Design for Failure● “Failure is not an option, its a requirement.”● Things will crash – Linux kernel panic – Defunct process – File system becomes read only suddenly● HW just doesnt work in every week – Broken disk – Flaws in CPU – Network adapter varies among 10/100/1000 Mbps
  7. 7. Event In Red: Failure
  8. 8. FlakinessNov 14 00:39:27 r007x072 kernel: bonding: bond1: link status definitely down for interface eth3, disabling itNov 14 00:39:35 r007x072 kernel: e1000e: eth3 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TXNov 14 00:39:35 r007x072 kernel: e1000e 0000:1a:00.1: eth3: 10/100 speed: disabling TSONov 14 00:39:35 r007x072 kernel: bonding: bond1: link status definitely up for interface eth3.Nov 14 00:39:36 r007x072 kernel: e1000e: eth3 NIC Link is DownNov 14 00:39:36 r007x072 kernel: bonding: bond1: link status definitely down for interface eth3, disabling it Analysis Analysis Unqualified Network Cables Unqualified Network Cables
  9. 9. [root@r007x072 ~]# cat /proc/net/bonding/bond1Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)Bonding Mode: adaptive load balancingPrimary Slave: NoneCurrently Active Slave: eth2MII Status: upMII Polling Interval (ms): 100Up Delay (ms): 0Down Delay (ms): 0Slave Interface: eth2MII Status: upLink Failure Count: 0Permanent HW addr: 00:1b:21:98:2a:4cSlave queue ID: 0Slave Interface: eth3MII Status: upLink Failure Count: 1627Permanent HW addr: 00:1b:21:98:2a:4dSlave queue ID: 0
  10. 10. Keep Simple and Robust● “I have 4 letters for you: KISS (Keep it simple and stupid)”● Complex system === hazardous system● Just enough fault-tolerance – Reboot machine if it goes wrong – Logout iSCSI session and login again – Mini toolkit to fix broken DM (device mapper) table
  11. 11. Example: Stateless OS ● Mount root partition in RAM – Think about how you install Ubuntu or Fedora ● Fix problem by reboot only[root@r009x090 ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/live-rw 7,9G 1,5G 6,4G 19% /tmpfs 71G 4,0K 71G 1% /dev/shm/dev/sda2 7,9G 1,4G 6,2G 18% /var/log/dev/sda4 1,6T 183G 1,4T 12% /iaas/local-storage
  12. 12. P2P based Socialized Communication● Bots “talk” to each other● Anyone can be re-run in seconds when things go wrong
  13. 13. Robust Application ● A number of roles in distributed system do there own jobs – Bot, manager, watch dog, zookeeper, agent, hbase, hadoop, etc ( HBase Region Region Region Server Server Server zookeeper Data Data Data node node node HDFSRegular bot Watch dog Manager bot
  14. 14. Dedicated Network-accessible Services● NTP (controversial in VM but good enough)● ZooKeeper – Node presence – Configuration data – Leader election● HBase: store schema-less data● Rsyslog: centralize logs● Web Service: accept HTTP requests only
  15. 15. Scale-out Architecture For Growth● Single namespace for global infrastructure –● Multi-region for Geo-distribution● Use cache when possible● Share nothing by autonomy● Leader election (elect new manager if former dies)● Collect metrics
  16. 16. Requirement grows/decreases faster than purchasing HW● “I need 200 large VMs this afternoon and will terminate all of them tomorrow.”
  17. 17. Storage is Always Not Enough● Walk-around: recycle unused files – Move low hit virtual images out of hot zone – Setup SLA to limit availability (provide redundancy only when necessary)
  18. 18. Metrics Collection is Critical● “Gathering, storing, and displaying metrics should be considered a mission-critical part of your infrastructure.”*● Measurement for performance boost (or downgrade) (* comes from chapter 3 of the book “web operations”)
  19. 19. Example #1: Fix Side Effect of the Leap Second ● The latest leap second occurred on the end of June 2012/var/log/messages grows too much It take 10 times long in job distribution between botstgtd: work_timer_evt_handler(89) failed to read from timerfd,Resource temporarily unavailable # service ntpd stop; date -s “`date`”; service ntpd start
  20. 20. Example #2: Recycle Unused Resources@zhukecdl Our analysis of your VM instance(s) shows thatCPU utilization and network traffic in the past 48 hourshave dropped below 2% and 10 MB.Instance ID CPU Time (s) CPU Rate (%) TX (MB)r007.x072.17897.u51393 337.3 0.20 0We would strongly urge you to consider recycling yourinstance(s) so that others can make use of these resources.If you didnt contact the administrator before 2011-08-1617:00+8000, the instance r007.x072.17897.u51393 will be recycledRegards,
  21. 21. Automated Operation (and More)● Goals – Daily upgrade all components – One administrator for 1k systems – No working overtime● Tool – Ruby chef – SmartCloud portfolio● Process – Run benchmark to the system every week – Stay in office until build break is fixed
  22. 22. Run Benchmark to the System Often● Measurement to your performance tweaks● Tools – Netperf – Apache JMeter Benchmarking network infrastructure # netserver # netperf -H -l 43200 TCP_CRR & # netperf -H -l 43200 TCP_CRR &
  23. 23. Infrastructure as Code● Building network accessible services● Integration these services [root@beijing-mn03 ~]# virsh list Id Name State ---------------------------------- 1 hbm1 running 2 bj-jenkins running 3 hjt running 4 webservice-1 running 5 hnn2 running 6 bugzilla running 8 hslave07 running 9 hslave08 running 10 hslave09 running 11 hslave10 running 12 hslave11 running 13 ScannerSlackware running
  24. 24. Real time Feedback by Tracing Logs● Manager X: “I need daily success rate report on deploy VM from department Y today.”
  25. 25. Visualize Traces Via Timeline
  26. 26. Summary● Keep it simple and robust● Scale-out architecture● Automated operation