This talk will dig deep into the process of continuous integration (CI) and the key processes that make up the overall CI process. We will discuss the relationships and process flows between change management, configuration management, and release/build management and how the CI process, when coupled with a solid performance engineering discipline across the product lifecycle, can result in a better user experience for your web & mobile application. We will speak about the entire lifecycle, the "conveyor belt" of the application lifecycle, with concentration on the "Big 3" processes that support the overall CI strategy. We will include a real-world example of how SOASTA uses Jenkins and other open source solutions for its "conveyor belt" and how this process enables SOASTA to complete over 100 product releases in 2014 and still maintain its customer SLA's for its Saas product offerings.

1. May 28, 2015
South Florida Agile Association – Fort Lauderdale, FL
“It’s not called Continuous Integration for Nothing!”
Dan Boutin – Senior Product Evangelist
dboutin@soasta.com
Mobile (404) 304-9529
@DanBoutinSOASTA

2. © 2014 SOASTA. All rights reserved. 2
Agenda:
• SOASTA Introduction
• The need for “Continuous”
• The “Big 3”
• CI with mobile and performance testing

3. © 2014 SOASTA. All rights reserved. 3
100 Billion
User Experiences Tested
10 Million
Tests Performed
5 Million
Failures Avoided
Actual CloudTest
view

4. © 2014 SOASTA. All rights reserved. 4
Agenda:
• SOASTA Introduction
• The need for “Continuous”
• The “Big 3”
• CI with mobile and performance testing

5. © 2014 SOASTA. All rights reserved. 5

6. © 2014 SOASTA. All rights reserved. 6
o Small Batch Sizes

7. © 2014 SOASTA. All rights reserved. 7
o Small Batch Sizes
o Comprehensive Version Control

8. © 2014 SOASTA. All rights reserved. 8
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies

9. © 2014 SOASTA. All rights reserved. 9
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation

10. © 2014 SOASTA. All rights reserved. 10
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback

11. © 2014 SOASTA. All rights reserved. 11
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds

12. © 2014 SOASTA. All rights reserved. 12
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments

13. © 2014 SOASTA. All rights reserved. 13
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer – Tester Collaboration

14. © 2014 SOASTA. All rights reserved. 14
Agenda:
• SOASTA Introduction
• The need for “Continuous”
• The “Big 3”
• CI with mobile and performance testing

15. © 2014 SOASTA. All rights reserved. 15

16. © 2014 SOASTA. All rights reserved. 16
o It’s not what’s within the silos
o It’s about the relationships

17. A typical silo’d organization lacks end-to-end integration
Performance Engineering & Continuous Integration-Full Lifecycle
Operational Configuration
Continuous Integration touchpoints
ADLC
QA/Test Pre-
Production
Production
QA/Test
Environment
Operational
Production
Environment
Application
Development
Environment

18. Development Test
Configuration management
Release management
Change management
Performance management
CI Focus Areas – The assembly line Hand-offs
Key Processes

19. © 2014 SOASTA. All rights reserved. 19
o Bill of Materials (BOM!)
o The “Big 3” => Change, Configuration & Release
• That is your software assembly line conveyor belt

20. © 2014 SOASTA. All rights reserved. 20
• We can only test as fast as the conveyor belt allows.
• If the conveyor belt is moving slow, testing is delayed.
• The Big 3 drives testing cycle.
o Why is CI critical?
• Speed
• New features, functions, etc. need to get to the customer, ASAP!

21. © 2014 SOASTA. All rights reserved. 21
o CI is a key component of Performance Engineering
o Performance Engineering spans the silos
BIG 3
Testing
Application
Development
Life Cycle
IT
Operations
Capacity
Management
Performance
Management
Performance
Engineering

22. © 2014 SOASTA. All rights reserved. 22
Does it work as
designed?
Does it perform
under extreme
circumstances?
Approve materials Real Drivers
What happens
in actual
conditions?
Assembly Line

23. © 2014 SOASTA. All rights reserved. 23
Functional validation
Performance,
reliability,
scalability tests
Real Users
Build
What happens
in actual
conditions?
Unit-tested code

24. © 2014 SOASTA. All rights reserved. 24
Agenda:
• SOASTA Introduction
• The need for “Continuous”
• The “Big 3”
• CI with mobile and performance testing

25. © 2014 SOASTA. All rights reserved. 25

26. © 2014 SOASTA. All rights reserved. 26
Jenkins runs any
unit tests as usual

27. © 2014 SOASTA. All rights reserved. 27
Run iterative load
tests via Jenkins
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual

28. © 2014 SOASTA. All rights reserved. 28
Run iterative load
tests via Jenkins
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual

29. © 2014 SOASTA. All rights reserved. 29
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual

30. © 2014 SOASTA. All rights reserved. 30
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Manage to a
performance
baseline
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual

31. © 2014 SOASTA. All rights reserved. 31
Use real user data to
iterate dev & test plans
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Manage to a
performance
baseline
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual

33. SOASTA – Branch Build from CloudTest

34. SOASTA – Test Results

35. SOASTA – Drill Down (Test Failures)

36. SOASTA – UI Testing Bundle Results

37. SOASTA – UI Testing Failure Drill down into
SOASTA CloudTest from Jenkins

38. SOASTA – Successful regression test results

39. SOASTA – Release Promotion

40. © 2014 SOASTA. All rights reserved. 40
o SOASTA’s Process Metrics
• 3 Main Products
• (2) Builds per day of each product (Dev branch + Customer branch)
• 8,000 tests executed per build
• For Mobile (TouchTest), Over 300 fully-automated tests each for iOS & Android

41. © 2014 SOASTA. All rights reserved. 41
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information

42. © 2014 SOASTA. All rights reserved. 42
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information

43. © 2014 SOASTA. All rights reserved. 43
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information

44. © 2014 SOASTA. All rights reserved. 44
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information

45. © 2014 SOASTA. All rights reserved. 45
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information

46. © 2014 SOASTA. All rights reserved. 46CONFIDENTIAL – Not for Distribution
Revenue
Brand
Competitive advantage

47. © 2014 SOASTA. All rights reserved. 47
We will close with some Mobile Trivia
@DanBoutinSOASTA

48. © 2014 SOASTA. All rights reserved. 48
Questions?
Dan Boutin – Senior Product Evangelist
dboutin@soasta.com
Mobile (404) 304-9529
@DanBoutinSOASTA
o SOASTA TouchTest
• FREE DownLoad: http://goo.gl/6CErMN
o Visit our Blog: http://goo.gl/TA2HKP
o Questions? @DanBoutinSOASTA

49. May 28, 2015
South Florida Agile Association – Fort Lauderdale, FL
“It’s not called Continuous Integration for Nothing!”
Dan Boutin – Senior Product Evangelist
dboutin@soasta.com
Mobile (404) 304-9529
@DanBoutinSOASTA