Testing Salesforce at Cloud Scale

2,982 views
2,764 views

Published on

Greg Wester - Salesforce.com
Thomas Kim - Salesforce.com

Learn about Salesforce's diversified test automation strategy, which spans both the public cloud and a private VM farm. Salesforce engineers will present an overview of how Jenkins is augmenting their existing CI runner and decreasing cycle time.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,982
On SlideShare
0
From Embeds
0
Number of Embeds
69
Actions
Shares
0
Downloads
81
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Testing Salesforce at Cloud Scale

  1. 1. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Testing Salesforce at Cloud Scale Thomas Kim @tksfz Greg Wester @gwestr
  2. 2. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Agenda •  History of CI at Salesforce •  Adding elasticity with public clouds •  Reducing cycle time with Jenkins-CI •  Jenkins Everywhere
  3. 3. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Continuous Testing at Salesforce •  Multi-million line code base •  Several major branches •  200,000 tests o  Mostly in-appserver functional tests •  ~3000 hours serialized runtime •  Autobuilds - home-grown system •  2000 VMs •  Hundreds of changelists per day (Perforce)
  4. 4. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Luna
  5. 5. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Autobuild System Design Autobuild Runner Autobuild Runner luna Autobuild Runner luna db Autobuild Runner web Assignment Runner p4
  6. 6. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Test Failure Correlation - Changelists are batched together - Tests that fail are re-executed against the skipped changelists
  7. 7. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf VM Provisioning VMware infrastructure Autobuild Runnerluna Autobuild Runner Autobuild Runnerweb luna db Autobuild Runner VMware API Assignment Runner VM Pool Manager p4
  8. 8. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Autobuilds in the Cloud EC2 Autobuild Runner luna Autobuild Runner Autobuild Runner web luna db Autobuild Runner AWS API Assignment Runner EC2 Poolboy p4
  9. 9. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf VM Provisioning Part II •  Static allocation •  Dynamic allocation •  Predictive allocation •  Elastic allocation
  10. 10. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Selenium •  Spawns a 2nd VM running Windows •  RemoteWebDriver -> IE6,7,8,9; Firefox; Chrome; Safari •  Selenium Browser Compatibility suite •  35,000 UI tests; some Selenium1 debt •  Tricky trifecta: o  Chrome releases every 6 weeks o  Firefox releases every 6 weeks; 3 week stagger o  Selenium library updates •  Selenium committer on staff (Jim Evans)
  11. 11. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Great coverage at great cost •  Multimillion dollar HA test infrastructure •  Cycle time is 6 hours per changelist •  [Automated] Continuous delivery not possible •  Some manual "sign off" at release boundary •  "Butterfly effect" changelists o  which tests to run before pushing code?
  12. 12. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Salesforce + Jenkins =
  13. 13. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Jenkins Satellite Builds •  Automated installer for automation •  Shelve a changelist -> queue for CI run •  Run static analysis, coverage reports •  Faster cycle time, "release ready" mainline
  14. 14. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf
  15. 15. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf
  16. 16. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Linking src/main and src/test
  17. 17. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Linking src/main and src/test (2)
  18. 18. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Future: Jenkins everywhere •  Code base is rapidly modularizing o  Jenkins CI continuously delivering jars •  Jenkins as a Service o  Option A: Cloudbees hosted o  Option B: ad hoc "swarm" o  Option C: Salesforce R&D hosted •  Jenkins for metrics:
  19. 19. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf We invented Testing at Scale
  20. 20. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Were not close to done •  Smarter VM provisioning •  Build-anything platforms hard to test •  Pixel perfect UI test framework •  Headless browser testing (PhantomJS) •  In memory computing for tests (high cost) •  Massively parallel testing on Intel Atom HW o  low cost, low power, natural isolation
  21. 21. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Q&A •  Follow @gwestr on Twitter •  Follow @tksfz on Twitter
  22. 22. Jenkins User Conference San Francisco, Sept 30 2012 #jenkinsconf Thank You To Our Sponsors

×