Keynote presentation at DevOps Summit 2016 in Taipei. http://devopssummit.ithome.com.tw/ I explained the importance of Test Automation and Continuous Integration for cultural change in DevOps context. DevOpsのコンテキストでの「文化の変化」に対する、テスト自動化とCIの重要性を説明した資料です。

1. Shake up the Culture
with Automation!
Jul/06/2016
Hiroyuki Ito / 伊藤 宏幸

2. 2
Hiroyuki Ito (The Hiro)
@hageyahhoo
Yahoo Japan Corporation
 Automation Coach
 Agile Coach
Who I Am

3. 3
Speaker of Agile2014 

4. 4
Japanese #1 Internet Provider
Web Services -> Smartphone Apps
Aiming to become a Tech Giant
About Yahoo Japan Corporation

5. 5
What is
DevOps?

6. 6
CALMS / CLAMS
General Definitions of DevOps
Culture
Automation
Lean
Measurement
Sharing
http://itrevolution.com/devops-culture-part-1/

7. 7
"DevOps" means:
The Hiro Says
Total Optimization
Whole Process Improvement
Based on Technical Foundation

8. 8
Successful DevOps requires
the cultural change
in your organization.

9. 9
Version
Control
Commit
Stage
Acceptance
Test Stage
Release
Stage
Manual
Test Stage
Performance
Testing Stage
Artifact
Repository
Humble, Jez, and Farley David, Continuous Delivery, Addison-Wesley, 2010
Check-in
The Hiro’s Scope

10. 10
Version
Control
Commit
Stage
Acceptance
Test Stage
Release
Stage
Manual
Test Stage
Performance
Testing Stage
Artifact
Repository
Humble, Jez, and Farley David, Continuous Delivery, Addison-Wesley, 2010
Check-in
The Hiro’s Scope
Cultural Change with
Test Automation & CI

11. 11
4 Teams with 3 Products
Our Target
We coaches supported them
Lots of Immature Engineers
Few Automated Tests
Lots of Integration Troubles

12. 12
Agenda
1. Test Automation
3. Evaluation of Measures
2. Continuous Integration
4. Conclusion

13. 13
1. Test Automation
3. Evaluation of Measures
2. Continuous Integration
4. Conclusion

14. 14
Make product teams
write test scripts voluntarily
Self-running Team
Cross-functional Team
Higher Quality
Higher Productivity
Our Goal

15. 15
Background

16. 16
Prerequisite
PHP
Cloud-based Infrastructure
Brownfield

17. 17
Legacy Code: No test scripts!
Brownfield: Our Common Challenges
Test Automation
as a Technical Foundation
Improve design & program
Refactoring
Prevent regressions

18. 18
Lots of Immature Engineers
Engineers
Test Scripts
How to Write
Test Scripts
The Importance of
Test Automation
How to Test
Programs & Products
What should I do?

19. 19
Passive Culture
Front-end
Engineers
Database
Engineers
API
Engineers
Waiting for
Instructions
Silos Based on
Component TeamsLeader

20. 20
Concrete Actions

21. 21
Scope of Test Automation
Unit Testing
Acceptance Testing
Performance Testing
Security Testing
Other ility Testing

22. 22
Stages of Adapting to Unit Testing
1. Write simple test scripts
2. Adapt to TDD
3. Use Test Double (Mock)
4. Write characterization test if necessary
5. Refactor production code & architecture

23. 23
Schedule & Actions
Jan Feb Mar
Preparation
• Investigate products
• Create sample scripts
Execution
• Lectures
• Workshops
• Pair Programming
Execution
Remove architectural
impediments

24. 24
A powerful tool
to learn
Unit Testing & TDD!
Cyber Dojo: Our Weapon
http://cyber-dojo.org/

25. 25
1. Test Automation
3. Evaluation of Measures
2. Continuous Integration
4. Conclusion

26. 26
Make product teams set up CI
processes
Our Goal
Nurture product teams
Avoid integration/deployment troubles
Achieve the target from CTO
Improve product teams

27. 27
Background

28. 28
e.g.) During integration & deployment process
Tons of Failures were There
Needed to avoid these troubles fundamentally
Staging
ENV
Production
ENV
CONFIG File
for Staging
Troubles!

29. 29
Voice of CTO
Let’s do CI Companywide!
Top-down Approach
Bottom-up Approach
CTO

30. 30
Concrete Actions

31. 31
Accustomed to CI
Working
Samples
Team Members Coach
Worked with Samples
Configuration
of CI Server
CONFIG File
Did Pair Setting
(like Pair Programming)
Team Members Coach

32. 32
Always added Automated Tests
Test! Test! Test!
Pull the real power of CI
Adapt to Test Automation more
Nurture product teams

33. 33
Simplified CI Processes
Version
Control
AP
Server
CONFIG
File
DB
Server
Web
Server
Version
Control
AP
Server
CONFIG
File
DB
Server
Web
Server

34. 34
1. Test Automation
3. Evaluation of Measures
2. Continuous Integration
4. Conclusion

35. 35
1. Results

36. 36
Updated on Mar/31/2016
(during about 2 months)
Metrics
Item Quantity
Unit Tests Added/Updated 27
Software Bugs Detected 10
Infrastructure Bugs Detected 24
Prevented Troubles 3
It may be insufficient, but it’s REAL.

37. 37
Self-running Actions by Product Team Members
PO added “writing unit tests” as DoD
Started to write unit tests voluntarily
Established their own CI/CD strategy
Established their own testing rules

38. 38
2. Lessons Learned

39. 39
Coaches should NOT do everything
Role of Coaches
For retaining learning opportunity
For nurturing voluntary actions
Giving hints is sometimes appropriate
Total

40. 40
Should Create Unavoidable Situation
CI:
Unavoidable from Test Automation
Top-down Approach:
Unavoidable to achieve objectives
Bottom-up Approach:
Unavoidable to scale improvements
Total

41. 41
Based on the production code
Sample Test Scripts Are Very Useful
Easy to adapt to Test Automation
Can use them as initial test scripts
Can learn their products & problems
Team
Members
Coaches
Test Automation

42. 42
Need to Solve Architectural Bottlenecks
Unable to apply mocks
Side effects
by static mocks
Hard to identify
web components
(Selenium WebDriver)
Modified interfaces
for injecting mocks
Introduced Phake
(Mock FW for PHP)
Defined locators
to each component
(like “id” or “name”)
Test Automation

43. 43
/**
* @test
*/
public function exampleOfTest() {
$sut = new Foo();
$result = $sut->bar('abc');
$this->assertEquals('huga', $result);
}
Pair Programming Clarifies Real SkillsTest Automation

44. 44
/**
* @test
*/
public function exampleOfTest() {
$sut = Foo::new();
$result = …? /* Cannot write logics  */
$result = Foo::bar('abc');
}
Pair Programming Clarifies Real SkillsTest Automation

45. 45
Detected failures
up-front with Automated Tests
Prevented Troubles
Unexpected Environmental Changes
Failures of Merging Branches
Leakages of Merging Branches
CI

46. 46
Detected Misconfigurations on Servers
Only PHP
Package Hell 
Don’t need to compileInterpreter
Unused Packages Leakages of Packages
CI

47. 47
Detected Misconfigurations on Servers
Can solve Package Hell 
PHP + Automated Test + CI
CI

48. 48
Made team members
learn a lot based on
automated working processes
Nurtured Product Teams
Fast Feedback
Learn from Failures
Lots of Confidence to Go ahead
CI

49. 49
Technical Foundation
Technology-Driven Development
-> Presentation at Agile2014
CI/CDTest Automation
Teams
Members
Organization
Products

50. 50
3. Next Actions

51. 51
Next Challenges
Skills of other languages
Acceptance Test Automation
Gather other useful metrics

52. 52
Example of Additional Metrics
Test &
Test Automation
Code Coverage
Cyclomatic Complexity
# of Bugs Detected
# of Members Who Can Write Tests
Update Times of Test Scripts
(Lots of Updates = Valuable Test)
CI/CD Cycle Time
MTBF
MTTR
# of Releases per Week
# of Members
Who Can Operate CI/CD Servers

53. 53
1. Test Automation
3. Evaluation of Measures
2. Continuous Integration
4. Conclusion

54. 54
What is
DevOps?

55. 55
"DevOps" means:
The Hiro Says Again
Total Optimization
Whole Process Improvement
Based on Technical Foundation

56. 56
We are challenging
the cultural change
in our organization.

57. 57
Test Automation with CI
is sufficiently valuable
Adapting to Test Automation & CI in DevOps Context
Iterative & Incremental Improvement
Do Kaizen / 改善

58. 58
Adapting to Automation
is a long way
It’s Not Easy to Continue…
It takes 6 month
to do the same thing
for 1 team
in Facebook
Photo by Improve It - Kent Beck no Workshop Mapping XP.(2006) / CC BY-SA 2.0

59. 59
But necessary to continue
for achieving
total optimization

60. 60
Shake up
• product teams
• working processes
• ourselves
for
• total optimization
• huge success
Always Improve, Always Advance

61. 61
With DevOps!