CI in the Enterprise: Lessons Learned

CI in the Enterprise: Lessons Learned Justin Little

Continuous Integration in the Enterprise: original title was: How to bang out 1000 builds a day without losing your mind and without making dev/ops/ network/storage people hate you

Who is this guy?

This meetup is about scale... we're talking bout scaling ci... thinking about ci as a "ﬁrst class Going citizen" in the sense that it's the path to production, and critical resource. BIG scale? Not so much page views, unique visits… more about # of contributors/ commits, builds, info. straight metrics of interest: - # of commits - # of builds - # build times - # of tests executed some more ops type stuff: - GBs of artifacts - proc load - network trafﬁc more of a concern when scaling. as of this am, artifact repo at WS has 46,238 artifacts

Compare and Contrast Small/Med Large • 2-3 project teams, 7-10 • 10-12 project teams, 9 - people each 12 people each (~15-20 committers) (~60-90 committers) • dozens of builds/day • hundreds of builds/day • gigabytes of artifacts • terabytes of artifacts • geographically close • geographically dispersed

notiﬁcation/info sharing strategies - make them useful, avoid spam - granular control of who gets what - targeted, relevent, timely allow for a subscribe/ unsubscribe model well... build spam anyway

build pruning strategy - what to do with all these artifacts? logs, artifacts, metadata (test results, etc). - describe possible strategy for managing stuff - most tools have some capabilities for managing build cleanup ("expiration"), but in general, keep your builds "self cleaning" "pruning" vs "cleanup" because we're consciously choosing what to keep and what to toss. not everything can just be tossed. in a enter;rise enviro, you may have more stringent auditing/ compliance rules... Pruning

Gloriﬁed CRON + - What is CI really? - challenge of scaling ci is mostly about what to do with all this info? how to manage all these artifacts? logging and - interesting to think about this as considering tool adoption. point is that in the enterprise, tool selection and adoption can be time consuming and sometimes constrained. What is it for? What do we hope to get from CI? rapid feedback to developers. it's gotta be fast to be useful notiﬁcation

Commit with Conﬁdence! a security net of sorts. "commit with conﬁdence" - as a developer, i can commit to some potentially scary code, and feel a little better about it. "if i didn't break the build, i likely didn't break the app". wishful thinking? maybe. the point isn't so much about removing/shifting responsibility - dev still need to be conscientious, more of a mechanism to help cope with the volume.

What’s in this thing qualitative info to help inform development/ potential standards enforcement anyway? -code complexity -test coverage -checkstyle rules -ﬁndbugs -simian less about "using" the tools necessarily, we should all be using them. establishing baselines over time, leveraging CI to (and is it any good?) gain insight into potential risks/opportunities programmatic enforcement of standards/ coding conventions? really tuff to get 50 devs to agree to and adhere to standards (consider turnover, etc, etc)

Hudson - lots of tools, with lots of features - at scale, build logic should be in Tools abound scripts anyway, so it's slightly less relevant - distributed builds - will it allow for a grid? - consider other (typical) things like: availability of support, cost, integration ease, support for your existing tools, etc - ease of management - you may need to manage dozens, maybe hundreds of build conﬁgurations. plan accordingly. does the tool allow for bulk editing of build conﬁgs?

Spread the news! - visibility! team spread out across different buildings/countries, vs a handful of devs in a room... - a little trickier, but not impossible. - "info radiators"

Merge hell revisited please, no more proj branches - What to build continuously? everything that changes. concerns about cross-project integrations

Metrics everywhere! - use CI to inform team about opportunities. - Make a special "metrics" build that runs every now and then to pull qualitative info from build. rich set of data is being created. don't let it bog you down, but don't necessarily dismiss it either. use it to make things better

Questions? justin@justinlittle.com

CI in the Enterprise: Lessons Learned

by Chris Westin on Nov 22, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

CI in the Enterprise: Lessons Learned — Presentation Transcript