Extending Continuous Integration

About the speaker Johannes Brodwall Lame duck Lead software architect at BBS Organizer Oslo XP meetup Organizer Smidig 2008 Blog: http://brodwall.com/johannes/blog .meetup.com/13

Johannes Brodwall

Lame duck Lead software architect at BBS

Organizer Oslo XP meetup

Organizer Smidig 2008

Blog: http://brodwall.com/johannes/blog

What’s the point of Continuous Integration?

Pay now to insure against defects Pay less to deploy to production Release whenever you want

Pay now to insure against defects

Pay less to deploy to production

Release whenever you want

What is Continuous Integration?

Unit test Functional tests System run in test harness

Unit test

Functional tests

System run in test harness

Result:

Every check in is is now tested

Will this find all our defects?

No!

Limited to our test harness Limited to our imagination And then there’s performance

Limited to our test harness

Limited to our imagination

And then there’s performance

So what should we do?

(Hint: CI is about paying now to avoid paying later)

Pay more now!

Our approach: Automated system test

Realistic setup Realistic load Realistic variation

Realistic setup

Realistic load

Realistic variation

How to automate system tests Automated build (and test) Scrap old data Install latest snapshot Download production data Replay production data Verify result Send results and logs via email

Automated build (and test)

Scrap old data

Install latest snapshot

Download production data

Replay production data

Verify result

Send results and logs via email

” Did you meet any problems?”

No

No, Only challenges!

The hard bits Installation Integration Simulation Verification

Installation

Integration

Simulation

Verification

Installation

How to automate installation Easy, scripted, always working install Simplify (Replace WebSphere with Jetty) (Combine components) Reduce integration All nodes look the same

Easy, scripted, always working install

Simplify

(Replace WebSphere with Jetty)

(Combine components)

Reduce integration

All nodes look the same

(Side effects) Easy installation Simpler design Simpler architecture

Easy installation

Simpler design

Simpler architecture

Integration

Integration Dealing with dependencies

Dealing with integration Don’t integrate - Do it yourself! Simulate other system Fake responses Keep canned data (data centric) Integrate with test version

Don’t integrate - Do it yourself!

Simulate other system

Fake responses

Keep canned data (data centric)

Integrate with test version

Simulation

Simulation Put realistic load on the system

Simulation Depends on what you system does

How to simulate production In our case: Files Crawler (Dyrkorn & Watne) Load generator (D&W) Record HTTP requests

In our case: Files

Crawler (Dyrkorn & Watne)

Load generator (D&W)

Record HTTP requests

Verification

Verification Finding out if it worked

How to verify results Compare with production Verify integrity Check logs

Compare with production

Verify integrity

Check logs

Comparing Store test result in database Store production result in database Full outer join on key fields Find missing or mismatched status Filter out know deviations

Store test result in database

Store production result in database

Full outer join on key fields

Find missing or mismatched status

Filter out know deviations

Date

Number of files

Okay

Missing

Extra

Extra Mismatched

Extra Extra Extra Known deviations

Consistency checks SQL expressions that pick out things that are weird

SQL expressions that pick out things that are weird

Logging It matters! Error logs should be empty if everything is okay

It matters!

Error logs should be empty if everything is okay

Result:

Every build is now system tested

Will this find all our defects?

No!

Limited integration Limited stability

Limited integration

Limited stability

No! Automated staging

No! Automated staging

Automated staging ” Next” version Lock-step with production Promote after a week Monitor 9:00-16:00

” Next” version

Lock-step with production

Promote after a week

Monitor 9:00-16:00

Only when you can think as an operator, can you master your system

Result:

Every release is hardened

Will this find all our defects?

No!

Wrong requirements Poor solutions New user behavior exposes bugs Bugs we didn’t care enough about

Wrong requirements

Poor solutions

New user behavior exposes bugs

Bugs we didn’t care enough about

The goal: Release after every iteration

The sad reality: Pilot release after every iteration

Pilot production

Why releases every iteration?

Find more bugs Try the easy solution first Find new requirement faster Exploit opportunity

Find more bugs

Try the easy solution first

Find new requirement faster

Result:

Find all the bugs cheaply!

Make sure it always works

Pay more now to pay less later

The goal: Release after every iteration

(And throw away the bugtracker)

Thank you!