Project size over 100 people total Large development activities in Vilnius, Suzhou, MinskProject management located in San Francisco
In the Component View, Business Processes and Functions (Claims, Billing, Policy) are composed of Common services which themselves adhere to a set of defined interface contracts.Services are implemented internally as JavaBeans and orchestrated through IoC via Spring configuration.All interface implementations rely on a common set of operational facilities for configuration, security, document generation, content management, rules and BAM.
If you succeed and you product gets many customer, with money paying for news features. You will need to expand your teams and here problems start. If you succeed and start growing, sometimes you get into multi-location, multi-team, multi product owners situation…
This is view Jenkins for one team CI environment, in most cases all is good, we have green build. Even if build fails it’s easy to fix it as people who broke it are near and can do it quickly.
CI is always red, one issues is find, while fixing it, new issues is created and this is eternal cycle. Teams get annoyed by the fact that CI is never working, they are not confident to take new source code from source control, just because it’s never stable and nobody knows then it will be fixed.
Fight – go investigate each incident, find who can be blamed, request fix, wait for fix and later find out that previous problem get fixed, but meanwhile new is introduced by other developer. After some time you will understand that fighting is not the answer, you will waist all you time and effort for it. Fail fast -
Each team has it’s own CI environment + central CI environment exist as well.
Each team has it’s own branch or repository close there they do all the coding. Only after Feature is done and team decided that code is good enough for integration it gets pushed to central. Push is done by one of the team members who does the “gatekeeper” role, he has to ensure the pushed code will go through.
Team CIPeople are less annoyed by CI failures. It’s easier to fix as people are co-located. It can be tested much earlier in the team scope.GatekeeperOne person who does the push will do merge and adequate testing before push. Less pushes, less builds, less failures.
People, even seniors do mistakes or sometimes are lazy to do adequate testing.
Environment stability: this is most important! If you environment produces unpredictable errors, this approach will not work.Test: If you have large number of failing test, it better to @ignore all of them and keep only stable/working test… Later you can revisit failing test and fix them one by one. CI: you will need huge number of hardware to run such setup.
It is generally accepted that the cost of a software bug goes up dramatically, perhaps exponentially, as the time between introduction and correction goes up.Typically customers don’t pay for code review as such, but if you need to achieve high quality this unavoidable and you should plan in ahead.Decision on code review should be made by project, there are many decision points – you need to know you project quality requirements (it doesn’t make sense to invest into short time project), project size (it doesn’t make much sense to do code review for POC’s or demos). Code review give benefits for larger, longer time to be supported and high quality projects.
Pair programming practically speaking does code review due the fact that two people are working on same code. But on the other end this is not true code review as it involves two developers who develop same thing for same deadline. Less than 100% 30% this is bare minimum to make sure developers care about their code and know that it might happen that their code will be reviewed. It’s Best Cost-Effect ration.This is level then developers are aware about possible code review and write better code. You can chose to review only selected review items – only complex changes, only juniors code, only new features, etc. Or 100%just to make sure nothing is missed.sounds quite much as it accounts small fixes as well, but this is practically this is the only approach ensuring highest quality.
Work as a team – everybody should feel part of team and understand that happens in other parts of code. Review all code changes – even small changes can create big problems… Code review minimizes this.Do not effect others– make sure code is compatible with others people work.Assure – most important is that this system has to assure code review is done. Don’t leave open windows to small hacks.
Coverage is calculate by testing code lines touched (executed by Unittests). It allows you to understand where you have good coverage and where you need add some work.
Chasing code quality in huge multi-location team project
CHASING CODE QUALITY IN HUGE MULTI-LOCATION TEAM PROJECT October 15, 2012 www.ExigenServices.com
GLOBAL EXIGEN SERVICES Riga St. Petersburg Nizhniy Novgorod Kazan R&D Center R&D Center R&D Center R&D Center San Francisco New York Headquarters Europe Eastern Europe Wuxi US London Asia-Pacific R&D Center Minsk Suzhou R&D Center R&D Center Frankfurt Odessa R&D Center Dnepropetrovsk• Founded in 2000 R&D Center Adelaide• 1’500 professionals Vilnius Johannesburg• Revenue approx. $70M San Paulo R&D Center Christchurch 2 www.ExigenServices.com
EXIGEN SUITE COMPONENTS Sales and Customer Service DISTRIBUTIONCORE Distribution ManagementBusiness Invoice & Payment New New Policy Policy Product FNOL Adjudicaton Business ServicingProcesses Collection Processes Business Servicing Management & Settlement&Core BILLINGCORE POLICYCORE CLAIMCOREFunctions Bill Plan Payment PRODUCT Under- Claim Claim Claims Claim FACTORY Management Management Rating Writing Processing Financials Processing FinancialsCommon Accounts Reporting & Tasks & 3rd Party Correspondence Sub-LedgerServices & CRM Compliance Decisions Mgmt ManagementData & Sys Billing Claims Commission Self-Service Reinsurance Analytics 3rd PartyIntegration Interface Interface Interface Portal Interface Interface Interface InterfaceOperations Configuration Document Content Business Process & Business Security ActivityManagement & Scheduling Generation Management Rules Task Mgmt Monitoring 3 www.ExigenServices.com
WHEN YOU START FAILINGAgile works says: One location, team, scrum master and product owner Everybody are universal and senior; But in sometimes we have Multi location, teams, scrum masters and PO’s And mixed level; And here problems arise. 4 www.ExigenServices.com
CI WITH MULTIPLE TEAMS 9 www.ExigenServices.com
WHAT TO DO?Options Fight with continuous integration failures every day Fail fast. Setup your continuous integration to no accept failing code. 10 www.ExigenServices.com
FAILING FAST “Failing fast” for development means getting response about your code quickly, from: compiler deployment unit and integration test UI tests static quality parameters And all those answers should be given before code gets to source control. 11 www.ExigenServices.com
MULTIPLE TEAMS – MULTIPLECONTINUOUS ENVIRONMENTS 12 www.ExigenServices.com
BENEFITS Team continuous integration environment is effected only by team changes. Only finished and tested features will get to central. Gatekeeper would be responsible to merge. 14 www.ExigenServices.com
STILL TO BE IMPROVED There is still possibility that gatekeeper will push bad code We don’t reach our ultimate goal - STABILITY of central. 15 www.ExigenServices.com
ONE STEP FURTHER - TEAM-MERGE To achieve our goal we created CI job that doesnt allow bad code. It does automatic code merge, compile, test and push. – rights write to central are revoked for anyone and only "special" "team-merge" user can do merge. – only possible way to commit code from team-branch to central is through "team-merge" script. 16 www.ExigenServices.com
FINAL TIPS Your environment should be stable enough to do this approach. Better have small number of working test rather than huge number of failing test. Each team has own CI plans – hardware is required 19 www.ExigenServices.com
CODE REVIEW OR NOT CODE REVIEW Code review, if: – You plan long life cycle for your product – Your customer insist on high quality and you have maintenance contract. Don’t code review, if: – Your product has short life cycle (demos, POC, etc.) – You don’t care about maintenance after it’s done 21 www.ExigenServices.com
CODE REVIEW There are various proposals on code review – Pair Programming Review about do less than 100%, with rules, like Review only Selected commits. Review only commits from juniors Random reviews Review 100% of code We chose 100% review. 22 www.ExigenServices.com
CODE REVIEW AFTER COMMIT Guarantee that reviewed code is committed No special actions required if reviewed code passesIssues: In case code doesn’t pass you have bad code in repository until it’s fixed. Bad code effects others 23 www.ExigenServices.com
CODE REVIEW BEFORE COMMIT Review patches or review branch. Bad code doesn’t effect others.Issues: You have no guarantee that good code reviewed will get to repository. As somebody needs to merge it.What we needed is: Automated merge on code review pass. 24 www.ExigenServices.com
100% CODE REVIEWWhat we want is1. Work as a team.2. Review all code changes3. Do not effect othersTherefore we need code review systems what would assure 100%review.Other requirements: DVCS-backed review system Acts as a front end for to central DVCS-repo When review is complete auto merge with default branch 25 www.ExigenServices.com
CODE REVIEW TOOLS For git users there is good tool called gerrit http://code.google.com/p/gerrit/ If you use mercurial (hg), it’s bit more complicated – as where is no analogue for gerrit for hg. Our approach was to use reviewboard.org and mercurial enhanced with python scripts what are mimicking gerrit approach. To initiate review you should call “hg postreview” 26 www.ExigenServices.com
USEFUL STATIC CODE CONTROL 29 www.ExigenServices.com
CONTROL CODE QUALITY We want to control basic parameters of code quality Static code violation Code unit and integration test coverage Other code parameters We can do it effectively with tool called Sonar 30 www.ExigenServices.com
METRICS What you want to measure? – Test Coverage? – Static code violations? – Code metrics? Answer to this is SONAR Automates metrics and creates reports. Allows to analyze code to the code line details. sonarsource.org 31 www.ExigenServices.com
WHAT’S NEXT WITH SONAR Good is that we have tool to control Bad is that what people are still people and tend to do crappy code. number of violations is increasing. still repeating same errors. code is unmaintainable 37 www.ExigenServices.com
OUR APPROACH Decide that violations are critical for you and what you can live with. What we do: Using multistage commit we don’t accept code with blocker violations. Commit is declined and developer should rework their code before next commit. Use personal violations view to work with particular developers coding style. 38 www.ExigenServices.com
PERSONAL VIOLATIONS DRILLDOWN 39 www.ExigenServices.com