Core Principles of CI
STPcon
1

Intro
2

Who Am I?
• Principal of OpenSource
Connections
• Contributer to CruiseControl
and Continuum CI projects
• Member of Apache Software
Foundation
• Presenter at conferences
(OSCON, ApacheCON, jTDS,
ExpoQA, STPcon)
• Fascinated by the art of
software development
3

Author
4

Father
5

Agilista
6

Schedule
Core Principles of
9 to 10:15
CI
Break 15 minutes
Hudson to the
10:30 to 11:15
Rescue
Agile and CI 11:15 to 12
7

Rules of the Road
• Cell phones on Silent.
• Ask Questions!
• Feel free to get up and
move around the room.
8

What is CI?
9

Conceptual
10

Definition
a fully automated and reproducible build, including testing, that
runs many times a day - Martin Fowler
http://martinfowler.com/articles/continuousIntegration.html
11

Martin Fowler
• Coined the term
Continuous Integration
• Chief Scientist for
ThoughtWorks
• Speaks and Blogs on
software:
• http://martinfowler.com/
12

Mapping
diagram credit to Brian Di Croce @ http://blog.briandicroce.com/2008/03/17/setting-up-a-continuous-
integration-environment-preface/
13

Activity
Turn to the person next to you, and see how many
non-obvious things you have in common.
14

Arm Bands!
• Elisabeth Hendrickson is
TestObsessed.com
• Focuses on Testing in the Agile
community
• Really great blogger: Specialized
Test Management Systems are an
Agile Impediment and Adventures
with Auto-Generated Tests and
RSpec are typical posts
15

CI in a nutshell
16

Demo
17

People
18

Version Control
Repository
19

CI Server
20

Build Script
• Automated Script is Key!
• Needs to be robust and simple
21

Feedback Mechanism
From:
cruisecontrol@opensourceconnections.com
ubject:
[CruiseControl] agenda_direcciones build 370 fixed
S
Date:
October 7, 2008 2:22:50 PM EDT
To:
epugh@opensourceconnections.com
The build has been fixed.
CHANGES
-------
New revision 370 detected
Revision 370 committed by epugh on 2008-10-07 18:22:03
mysql esta mejor que sqlite
M /branches/ERIC_EXPO_08_DEMO/agenda_direcciones/config/database.yml
See http://cruise.opensourceconnections.com:3333/builds/agenda_direcciones/370 for details.
22

photo credit to Jay Grieves @ www.bigswingingdeveloper.com
23

Aspects of CI
24

Source Code
Compilation
• Eclipse “Export to War”
is NOT the solution
25

Database Integration
• Database is integral to • Don’t require a multi gig
your code database file to run
tests!
• DDL, stored
procedures are • Look at techniques like
treated as source fixtures or mock objects
to reduce database
• Involve your DBA in coupling
your project
• Developers need
• DBUnit for Java database sandbox
projects!
26

Testing
with out testing, CI is just a glorified compiler- Eric Pugh
• Testing proves the intent
of the programmer was
met.
• Typically provides the
“checkability” that James
Bach talks about
• Multiple Levels of testing
27

Inspection
• Code Coverage
• Code Complexity
• Copy’n’pasting
• Architectural Standards
• Lint Checking
• Static Analysis
28

Deployment
• Turns what is normally a difficult process
into a non event...
• ant deploy
• Simplifies standup new environments
29

Documentation
30

Feedback
• Email
• IRC
• Devices
• Blog Posts
• Twitter Notifications
• Tray Icons
31

So What is the Value?
32

Life with out CI
• Unstable code, integration is difficult
• Many build errors reported
• Only one person on the team can build
the entire project
• Getting “test ready” product is hard
• Feedback cycle is long
• Same bug keeps coming back
Every day is a struggle to be productive
33

Life with CI
• The build process is simple and
repeatable
• Elimination of Human Error
• Bugs stay fixed!
• Feedback cycle is very rapid
Every day you know you can produce better software!
34

Risk Reduction
• What is in the build?
• What has changed
between builds?
• How do I verify
functionality?
35

Reduce Human Error
36

My Ah-hah! moment
37

Reduce Repetitive
Processes
• DRY: Don’t Repeat
Yourself
• Script Everything!
• “ant deploy”
38

Make Painful Things
Painless!
photo credit to Paul Duvall @ http://www.integratebutton.com/
39

Produce Deployable
Software
• It’s what we are here to do!
• Always in a “Demo Ready” state
• No last minute administrivia tasks to get
the product ready
• Installer is built
• Documentation is written
• Changelog is written
40

Enable Project Visibility
41

42

Increase Project
Confidence
43

44

photo credit to ThoughtWorks
45

Facilitate
Communication
46

photo credit to Eric Pugh @ www.opensourceconnections.com
47

48

49

Activity
Exquisite corpse (also known as "exquisite
cadaver" or "rotating corpse") is a method
by which a collection of words or images is
collectively assembled, the result being
known as the exquisite corpse or cadavre
exquis in French.
50

What Prevents Teams
from Using CI?
51

Cultural Challenges
52

Fear of Commitment
• Developers and Testers
need to commit changes
frequently!
• Incremental small changes
make finding integration
issues simpler
drawing credit to Tom Armitage @ http://infovore.org/archives/2007/03/14/the-continuous-integration-
barometer/
53

This isn’t 1984
54

Fear of Blame
55

• CI needs a champion who acts as the
ambassador to the leaders of the
organization.
• Thought leaders in the organization who
can help encourage team to accept the
process changes that CI brings
• A very successful test case
• A greenfield project if possible
56

57

Structural Challenges
58

Testability
• Modularity? • Perform code
complexity analysis to
• Coupling to underlying see if some parts are
more amenable to
platform?
testing.
• Requires external
services to be up and
available for testing?
59

Tight Coupling
photo credit to Jay Grieves @ www.bigswingingdeveloper.com
60

CI Tool Stability
Challenges
61

Invest in CI Server
• The CI system must be as well
X
maintained as the Source Control
system.
• The CI system performs builds very
✓
quickly.
• Who has responsibility for CI? It’s very
important to have a “throat to choke”.
• No false alarms. If there are false
alarms then the team loses confidence
in the CI system.
62

Monkey CI
photo credit to Eric Pugh @ www.opensourceconnections.com
63

When Should You
Adopt CI?
64

From
good CI
to
great CI!
65

10 minute rule
photo credit to Jay Grieves @ www.bigswingingdeveloper.com
66

Multi Stage CI
• Reduce the amount of
integration required
• Teams integrate as
modules are marked as
ready.
• Maven2 was built with
this in mind!
Damon Poole @ http://damonpoole.blogspot.com/2008/01/advanced-multi-stage-continous.html
67

Break up Tests
• “preflight” or “smoke” tests
• longer integration tests
• periodic full site generation and report
generation
68

Predictive Testing
• Some commercial tools support predicting
which tests will fail based on previous
behavior, and running those first!
69

Cup of Coffee Metric
What happens in the time it takes to get a cup of coffee can
tell you a lot about the quality of your continuous integration
system. - Paul Julius
• Builds shouldn’t take any longer!
• Getting a cup of coffee or drink of
water breaks up your day
• Gives you a chance to review your
work
• Requires a moduler fast build
http://pauljulius.com/blog/2009/09/14/cup-of-coffee-metric-for-continuous-integration/
70

Paul Julius
• co-founder of CITCON
• Original developer of
CruiseControl
• Speaks and Blogs on CI
extensively
• http://pauljulius.com/blog/
71

Virtuous Circle
• Metrics are hard to get
• Teams (Developers?)
rebel against on high
management
• Need good processes
that support and
encourage other good
processes
72

HackyStat
73

CI isn’t one specific
method
74

!F!2!89.+;D+!89!:G:7.!+67
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#$%#&'
!"#$%&'"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B'3C+ ,-'&*+D54E+
,-'&*+.#)/" ,-'&*+.#)/" :)/4"'#45A !"#$%&!'(#%)* >2+.3$/"(4#"
0)*'1$#)*+ ,-'&*+6&-"#)5 .)1-5)+
,-'&*+2$1('3) !"#$%&!'(#%)*
643#'3-4-"+
.4&'*+ ,-'&*"
.4-51)+643#54&
Enterprise CI
!"#$%&'()
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
H)&/)5+.15'/#" D-&&A+.15'/#)* .)&IJ.)5K'1)+ G-"(J,-##43+ ?-#4E$#'1+
9)"#+0)/&4A" G54*+0)/&4AE)3# G54*-1#'43+
0)/&4A"
?-#4+0)/&4A+#4+ N$#)*O+?-#4E$#'1+
;3)+!3KL G54E4#'43"
Maturity Model
24"#&A+ D-&&A+.#$3*$5*'M)*+
.#$3*$5*'M)* 0)/&4A"
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
Came out of discussions at 2009
!"#!$%&
.4E)++ :)C5)""'43+ .4E)+.#$#'1+ !@#)3"'K)+ QRRS+
9)"#+?-#4E$#'43 9)"#"+:-3+P'#(+ ?3$&A"'"+<")* <3'#+9)"#" 9)"#+64K)5$C)
,-'&*
CITcon. Developed by Eric ?-#4E$#)*+
D-31#'43$&+9)"#"
!@#)3"'K)+
?-#4E$#)*+
D-31#'43+9)"#"
Minick of UrbanCode :'"BJ,$")*+
2$3-$&+9)"#"
.)1-5'#A+.1$3"
!"#$!%&'(
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
944&J,$")*+ .'&4)*+:)/45#'3C H'"#45'1$&+ 95)3*'3C G5)*'1#'K)
:)/45#" :)/45#" 24*)&'3C
G-%&'"(+ 654""J.'&4
>'"'%'&'#AT+:)/45#+ F$#)"#+:)/45#" 654""+D-31#'43+ ?3$&A"'"
:-33)5 :)/45#+?11)""
,$"'1+
:)/45#+?3$&A"'"
'!()*+,"$-!).$/ !"#$%&"!'
!"#$%&'()*&)+(&,%-$(.(/0$&12&3"-+(/-4&3)5$6&78/-$.".$(
!""#$%%&&&'()"!*++#,-'.-/%!"/+%,01-2,.01%&!*"03#(#0,1
75

789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#$%#&'
!"#$%&'"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B'3C+ ,-'&*+D54E+
,-'&*+.#)/" ,-'&*+.#)/" :)/4"'#45A !"#$%&!'(#%)* >2+.3$/"(4#"
0)*'1$#)*+ ,-'&*+6&-"#)5 .)1-5)+
,-'&*+2$1('3) !"#$%&!'(#%)*
643#'3-4-"+
.4&'*+ ,-'&*"
.4-51)+643#54&
76

!"#$%&'()
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
H)&/)5+.15'/#" D-&&A+.15'/#)* .)&IJ.)5K'1)+ G-"(J,-##43+ ?-#4E$#'1+
9)"#+0)/&4A" G54*+0)/&4AE)3# G54*-1#'43+
0)/&4A"
?-#4+0)/&4A+#4+ N$#)*O+?-#4E$#'1+
;3)+!3KL G54E4#'43"
24"#&A+ D-&&A+.#$3*$5*'M)*+
.#$3*$5*'M)* 0)/&4A"
77

789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#!$%&
.4E)++ :)C5)""'43+ .4E)+.#$#'1+ !@#)3"'K)+ QRRS+
9)"#+?-#4E$#'43 9)"#"+:-3+P'#(+ ?3$&A"'"+<")* <3'#+9)"#" 9)"#+64K)5$C)
,-'&*
?-#4E$#)*+ !@#)3"'K)+
D-31#'43$&+9)"#" ?-#4E$#)*+
D-31#'43+9)"#"
:'"BJ,$")*+
2$3-$&+9)"#"
.)1-5'#A+.1$3"
78

!"#$!%&'(
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
944&J,$")*+ .'&4)*+:)/45#'3C H'"#45'1$&+ 95)3*'3C G5)*'1#'K)
:)/45#" :)/45#" 24*)&'3C
G-%&'"(+ 654""J.'&4
>'"'%'&'#AT+:)/45#+ F$#)"#+:)/45#" 654""+D-31#'43+ ?3$&A"'"
:-33)5 :)/45#+?11)""
,$"'1+
:)/45#+?3$&A"'"
79

What about People?
80

CI on a Dollar a Day
"From now on, our code in revision control will always build
successfully and pass its tests.". - James Shore
• Build Computer
• Ridiculous Toy
• Annoying Bell
• Automated Build
• Group Agreement
http://jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html
81

credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/
enterprise_ci_cultural_maturity.html
82

Lots of
advanced
techniques!
credit to Christine Mitterbauer @ Microdoc
83

Perils of CI
84

Getting too focused on
Green Bar
Glare of the Green Bar in CI can
obscure issues with your product-
James Bach
85

Making CI the core of too
many systems
86

Making decisions based
on your available
tooling
87

Selecting a CI System
88

Approaches
• Big Hand Rolled System
• Cloud Based Solution
• Completely Hosted
89

Hand Rolled System
Pros Cons
Complete control of Rolling your own can take
configuration more work
Data and code inside Dealing with disparate
firewall platform elements is harder
Works with any type of Multiple kinds of projects can
system be hard to manage
Must provide Sys Admin
Cheapest
work
90

CI Matrix
Matrix of differences between CI tools: http://tinyurl.com/39hol7
91

Cloud Based Solution
Pros Cons
Data and Code outside
Very Scalable
firewall
Use CPU resources only
Cloud computing still new
when needed
Deal well with multiple
GUI apps hard to test
kinds of projects
Must provide Sys Admin
?
work
92

CI in a Box
• Amazon EC2 Hosted
instance
• http://www.ciinabox.com/
93

Hosted Solution
Pros Cons
Typically limited in language
Very Scalable
support
data and code outside
No Sys Admin tasks
firewall
Pricing is fairly
GUI apps hard to test
straightforward
? vendors are still new
94

RunCodeRun.com
95

MikeCI.com
96

Where to Next?
97

CITCON
• Open Spaces Style
• Twice a year, US and International
• Raleigh-Durham, NC, April 2010
• http://www.citconf.com/
98

99

CI Yahoo Group
http://tech.groups.yahoo.com/group/citcon/
100

The Book
Continuous Integration:
Improving Software Quality and
Reducing Risk
101

The CI guys
http://ci-guys.com/
Free online consulting
102