Yet Another Continuous
Integration Story
Who I Am
● Anton Serdyuk
● Technical Leader at Startup Labs, Inc.
● anton.serdyuk@gmail.com
● anton.serdyuk@itstartuplabs....
Disclaimer
Disclaimer

Projects are different
● Technical requirements
● Business requirements
● Length
● Budget
● Load
● Data size
●...
Disclaimer

Teams are different
● Level
● Size
● Structure
● etc
Disclaimer

There is no silver bullet
● Worked for us != will work for you
● Adopt patterns and principles rather than the...
Who We Are
Who We Are

Project
● b2b system
● Unexpected downtime is very expensive
● Bugs in production are also very expensive beca...
Who We Are

Technologies
● PHP 5.4
● Java 7
● PostgreSQL
● RabbitMQ
Who We Are

Frameworks
● Hibernate
● JUnit
● Symfony2
● Silex
● Behat
● PHPUnit
Goals and Principles
Goals and Principles

Agile
● Small changes
● Frequent releases
● Early feedback
What Is NOT Our Way
What Is NOT Our Way

Git Flow
What Is NOT Our Way

Feature Branches

feature1

master

feature2
What Is NOT Our Way

Feature Branches
Active feature branches usage example
What Is NOT Our Way

Release Branches
1.1.x
1.0.x

master
What Is NOT Our Way

master/staging/develop branches
develop

master
What Is NOT Our Way

Release Tags

v1.1.1

v1.1.2

master
What IS Our Way
What IS Our Way

Continuous Integration
● Push every day
● Build every commit
● Run all tests on every build
● Fix red bui...
What IS Our Way

One Branch

master
What IS Our Way

Branch by Abstraction
● Commit switch-off for this feature
● Develop, integrate
● … your code in producti...
What IS Our Way

Autotests
● QAs write behat scenarios
● Developers implement
● Every feature covered by behat scenarios
●...
What IS Our Way

Every Build Is Release Candidate
● Do not push if state after commit is not suitable for
production
What IS Our Way

Some Numbers
● ~1000 behat scenarios (unit tests are not counted)
● ~15 minutes build duration
● 2 build ...
Build Process
Build Process

Overview

build

automatic tests

manual tests

load and prod
migrate tests

production
Build Process

Overview
bamboo

pull
build

git
repository

push

developers

build
build
agent
agents

test
test
test
tes...
Build Process

Use The Same Artifacts

build
build
agent
agents

artifacts

apt
repository

test
test
test
agents
test
age...
Build Process

Autoconfigure All Environments
test
test
test
agents
test
agents
agents
agents

QA
server

puppet
master

p...
Build Process

Deploy by Pressing a Button

bamboo

test
test
test
agents
test
agents
agents
agents

QA
server

pre
prod

...
Build Phase
Build Phase

Goals
● Create artifacts which will be used in all phases
● Run unit tests
Build Phase

Technologies
● Phing
● Maven
● deb-packages (dpkg-buildpackage)
● PHPUnit
● JUnit
Build Phase

Example
Automatic Test Phase
Automatic Test Phase

Goals
● Run functional tests
Automatic Test Phase

Deploy to Completely Clean System
Automatic Test Phase

Run Tests in Parallel
agent 1
behat process 1

application 1

agent 2
behat process 2

application 2...
Automatic Test Phase

Run Tests in Parallel
feature 1
feature 2

job 01

agent 1

feature 3
feature 4
feature 5

job 02

f...
Automatic Test Phase

Run Tests in Parallel
github: StartupLabs/behat-partial-runner
● bin/behat --count 15 --current 1
● ...
Automatic Test Phase

Write Fast Tests
● Continuously speed-up your tests
● Use curl instead of Selenium
● Put initial dat...
Automatic Test Phase

Write Fast Tests
Merge scenarios which do not modify data
Feature: Posts pagination

Feature: Posts ...
Automatic Test Phase

Write Fast Tests
Translate outline to single scenarios with TableNode
Feature: Post validation

Feat...
Automatic Test Phase

Write Fast Tests
Parallel test execution
@parallel
Feature: Login

server
behat process 1

Backgroun...
Manual Test Phase
Manual Test Phase

Goals
● Run manual tests
● Place where QA see new functionality
Manual Test Phase
● Deploy to completely clean system
Load and Prod Migrate
Tests
Load and Prod Migrate Tests

Load Tests Goals
● Catch race conditions
● Catch serious performance degradation when databas...
Load and Prod Migrate Tests

Prod Migrate Tests Goals
● Test migrations on real data
● Test migrations time on real data
Load and Prod Migrate Tests
● Migrate load test env to new version
● Rollback migrate test env to production snapshot
Deploy to Production
Deploy to Production
● nb-prod - small production copy used by other teams for
integration purposes
Problems
Problems

Red Builds for a Long Time
● QA can not deploy new version to staging
● New commits make new tests red like a sn...
Problems

Long Builds
● Nobody wants to wait green build
● Nobody cares about green build after commit
● Red builds for a ...
Problems

Blinking Failures
● Failures are trustless - “it is not my fault - it is blinking,
just do rebuild”
● Time consu...
Key Points, Again
Key Points, Again

Autotests
● There is no sense in CI without tests
● Easy refactoring
● Less bugs
Key Points, Again

Fast Builds
● Long builds => More changes => Red builds

● Fast tests
● Parallel run
● Hacks in tests
●...
Key Points, Again

One branch
● There are no builds from other branches
● There are no hotfixes
● Easy deploy and build pi...
Todo
Todo
● Rollback strategy
● Use Bamboo deployment
Inspired By
● Continuous Delivery: Reliable Software Releases
through Build, Test, and Deployment Automation (Jez
Humble, ...
Questions?
Upcoming SlideShare
Loading in...5
×

Yet Another Continuous Integration Story

1,955

Published on

PHP User Group Minsk #4

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,955
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Yet Another Continuous Integration Story

  1. 1. Yet Another Continuous Integration Story
  2. 2. Who I Am ● Anton Serdyuk ● Technical Leader at Startup Labs, Inc. ● anton.serdyuk@gmail.com ● anton.serdyuk@itstartuplabs.com ● @anton_serdyuk ● Skype: serdyuk.anton
  3. 3. Disclaimer
  4. 4. Disclaimer Projects are different ● Technical requirements ● Business requirements ● Length ● Budget ● Load ● Data size ● etc
  5. 5. Disclaimer Teams are different ● Level ● Size ● Structure ● etc
  6. 6. Disclaimer There is no silver bullet ● Worked for us != will work for you ● Adopt patterns and principles rather than the full process
  7. 7. Who We Are
  8. 8. Who We Are Project ● b2b system ● Unexpected downtime is very expensive ● Bugs in production are also very expensive because of the reputation loss ● Small amount of traffic (But expensive) ● 1.5 years ● 4-15 developers (+ layout, + qa) ● 15 Gb Database (Grows not very fast)
  9. 9. Who We Are Technologies ● PHP 5.4 ● Java 7 ● PostgreSQL ● RabbitMQ
  10. 10. Who We Are Frameworks ● Hibernate ● JUnit ● Symfony2 ● Silex ● Behat ● PHPUnit
  11. 11. Goals and Principles
  12. 12. Goals and Principles Agile ● Small changes ● Frequent releases ● Early feedback
  13. 13. What Is NOT Our Way
  14. 14. What Is NOT Our Way Git Flow
  15. 15. What Is NOT Our Way Feature Branches feature1 master feature2
  16. 16. What Is NOT Our Way Feature Branches Active feature branches usage example
  17. 17. What Is NOT Our Way Release Branches 1.1.x 1.0.x master
  18. 18. What Is NOT Our Way master/staging/develop branches develop master
  19. 19. What Is NOT Our Way Release Tags v1.1.1 v1.1.2 master
  20. 20. What IS Our Way
  21. 21. What IS Our Way Continuous Integration ● Push every day ● Build every commit ● Run all tests on every build ● Fix red builds quickly
  22. 22. What IS Our Way One Branch master
  23. 23. What IS Our Way Branch by Abstraction ● Commit switch-off for this feature ● Develop, integrate ● … your code in production now, but switched-off ● Develop, integrate ● Test ● Switch-on this feature in production
  24. 24. What IS Our Way Autotests ● QAs write behat scenarios ● Developers implement ● Every feature covered by behat scenarios ● Great regression tests suite - easy refactoring, less bugs ● + ~100% development cost
  25. 25. What IS Our Way Every Build Is Release Candidate ● Do not push if state after commit is not suitable for production
  26. 26. What IS Our Way Some Numbers ● ~1000 behat scenarios (unit tests are not counted) ● ~15 minutes build duration ● 2 build servers (Intel Xeon, 32 Gb RAM, SSD) ● 5-6 build agents for tests (VMs) ● 3 build agents for build phase (VMs) ● ~15 builds per day ● 2 releases per week
  27. 27. Build Process
  28. 28. Build Process Overview build automatic tests manual tests load and prod migrate tests production
  29. 29. Build Process Overview bamboo pull build git repository push developers build build agent agents test test test test agents test agents agents agents QA server pre prod prod artifacts apt repository puppet master tune release engineer (architect/admin)
  30. 30. Build Process Use The Same Artifacts build build agent agents artifacts apt repository test test test agents test agents agents agents QA server pre prod prod
  31. 31. Build Process Autoconfigure All Environments test test test agents test agents agents agents QA server puppet master pre prod prod tune release engineer (architect/admin)
  32. 32. Build Process Deploy by Pressing a Button bamboo test test test agents test agents agents agents QA server pre prod prod
  33. 33. Build Phase
  34. 34. Build Phase Goals ● Create artifacts which will be used in all phases ● Run unit tests
  35. 35. Build Phase Technologies ● Phing ● Maven ● deb-packages (dpkg-buildpackage) ● PHPUnit ● JUnit
  36. 36. Build Phase Example
  37. 37. Automatic Test Phase
  38. 38. Automatic Test Phase Goals ● Run functional tests
  39. 39. Automatic Test Phase Deploy to Completely Clean System
  40. 40. Automatic Test Phase Run Tests in Parallel agent 1 behat process 1 application 1 agent 2 behat process 2 application 2 ... agent 5 behat process 5 application 5
  41. 41. Automatic Test Phase Run Tests in Parallel feature 1 feature 2 job 01 agent 1 feature 3 feature 4 feature 5 job 02 feature 6 agent 2 feature 7 feature 8 job 03 feature 9 ... ... feature 50 feature 51 Job 15 agent 3
  42. 42. Automatic Test Phase Run Tests in Parallel github: StartupLabs/behat-partial-runner ● bin/behat --count 15 --current 1 ● bin/behat --count 15 --current 2 ● bin/behat --count 15 --current 3 ● bin/behat --count 15 --current 4 ● bin/behat --count 15 --current 5 ● … ● bin/behat --count 15 --current 15
  43. 43. Automatic Test Phase Write Fast Tests ● Continuously speed-up your tests ● Use curl instead of Selenium ● Put initial data directly to DB ● etc
  44. 44. Automatic Test Phase Write Fast Tests Merge scenarios which do not modify data Feature: Posts pagination Feature: Posts pagination Background: Background: Given I am logged in as admin Given I am logged in as admin And there is many posts And there is many posts Scenario: Pagination works Scenario: All pagination tests Given I am at the posts list # Scenario: Pagination works Then I should see pagination Given I am at the posts list Then I should see pagination Scenario: Link “last” works Given I am at the posts list # Scenario: Link “last” works When I click at the “last” link When I click at the “last” link Then I should see last page Then I should see last page
  45. 45. Automatic Test Phase Write Fast Tests Translate outline to single scenarios with TableNode Feature: Post validation Feature: Post validation Background: Background: Given I am logged in as admin Given I am logged in as admin And I am on post add page And I am on post add page Scenario Outline: Validation works Scenario: Validation works When I enter <value> into <field> Then I should see such errors with such input Then I should see error <error> | field | value | error Examples: | field | value | error | | title | | Can not be empty | | tags | Should be letters | | %%%% | | title | | | Can not be empty | tags | %%%% | Should be letters |
  46. 46. Automatic Test Phase Write Fast Tests Parallel test execution @parallel Feature: Login server behat process 1 Background: Given I am registered user alex = alex1 alex = alex2 application behat process 2 | username | alex | | password | 123 | Scenario: I can login Given I am at the login page When I fill form with ● Many problems with race conditions | username | alex | | password | 123 | Then I should be logged in ● Not worked for us :(
  47. 47. Manual Test Phase
  48. 48. Manual Test Phase Goals ● Run manual tests ● Place where QA see new functionality
  49. 49. Manual Test Phase ● Deploy to completely clean system
  50. 50. Load and Prod Migrate Tests
  51. 51. Load and Prod Migrate Tests Load Tests Goals ● Catch race conditions ● Catch serious performance degradation when database grows or after adding new functionality
  52. 52. Load and Prod Migrate Tests Prod Migrate Tests Goals ● Test migrations on real data ● Test migrations time on real data
  53. 53. Load and Prod Migrate Tests ● Migrate load test env to new version ● Rollback migrate test env to production snapshot
  54. 54. Deploy to Production
  55. 55. Deploy to Production ● nb-prod - small production copy used by other teams for integration purposes
  56. 56. Problems
  57. 57. Problems Red Builds for a Long Time ● QA can not deploy new version to staging ● New commits make new tests red like a snowball ● Nobody knows who are responsible for all those failed tests Solutions: ● If build can not be fixed in 15 minutes - revert commit ● Autoreverts on red builds (?)
  58. 58. Problems Long Builds ● Nobody wants to wait green build ● Nobody cares about green build after commit ● Red builds for a long time Solutions: ● Speedup tests ● Parallel build ● Add more build servers
  59. 59. Problems Blinking Failures ● Failures are trustless - “it is not my fault - it is blinking, just do rebuild” ● Time consuming Solutions: ● Eliminate them ASAP ● Do not just run “Rebuild failed jobs”
  60. 60. Key Points, Again
  61. 61. Key Points, Again Autotests ● There is no sense in CI without tests ● Easy refactoring ● Less bugs
  62. 62. Key Points, Again Fast Builds ● Long builds => More changes => Red builds ● Fast tests ● Parallel run ● Hacks in tests ● etc
  63. 63. Key Points, Again One branch ● There are no builds from other branches ● There are no hotfixes ● Easy deploy and build pipeline
  64. 64. Todo
  65. 65. Todo ● Rollback strategy ● Use Bamboo deployment
  66. 66. Inspired By ● Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Jez Humble, David Farley) ● http://timothyfitz.com/
  67. 67. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×