Agile:MK            December 2012                                        Journal                                        Ja...
Agile:MK Journal                                   January 2013 Agile:MK User Group                                   Happ...
December 2012The State of the ArtThe adoption of Agile methods and practices within the UK hasbeen on-going for close to 2...
Agile:MK Journalthe project was important. They hired good people,             comply with the new Solvency II regulations...
December 2012The cards consisted of MMFs (Minimum Marketable             •	 If it can go wrong, (with thousands of cores r...
Agile:MK Journal	 o	 Chaos Monkey testing – random server                     Companies		 outage, network failure, full me...
Agile: MK Bookshop                                                                                     December 2012      ...
ripplerock offers a set of services and tools that enable our customersto dramatically improve their software development c...
Upcoming SlideShare
Loading in …5

Agile mk-journal-issue-002


Published on

An Agile Cloud Based Project -
Milliman MG-ALFA Compute for Azure
by Ian Shimmings, Associate at Endjin
This article is based on a short presentation and
discussion to the Agile:MK user group, which was,
in turn, based on nearly 2 years as a senior developer
on the project.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Agile mk-journal-issue-002

  1. 1. Agile:MK December 2012 Journal January 2013 ISSUE 02 A special interest group for professionals from the Milton Keynes area who are interested in learning, sharing and encouragingthe use of agile methodologies such as Scrum, Extreme Programming & Lean Thinking. Agile: “Getting the Business Involved” Open Space Session, facilitated by Iain McKenna, Certified Scrum Trainer Monday 25th February 2013 6pm – 8pm Sponsored by
  2. 2. Agile:MK Journal January 2013 Agile:MK User Group Happy New Year and welcome to the second edition of the Agile:MK Journal. The Agile User Group in Milton Keynes meets once a month to share and learn about agile methods through real-world experiences. Each month Steve Garnett we’ll publish a brief overview of the discussions Founder Agile:MK and insights of the group in this journal. To register your interest, If you’d like to join the group so you can please join the attend, the sessions, and be involved in the LinkedIn Group at discussions face to face, to learn or share agile-mk-liug your experiences, please join us on LinkedIn. Agile: “Getting the Business Involved” Open space discussion workshop Facilitator: Ian McKenna Monday 25th February 2013 Profile Facilitator: Iain McKenna, CST Jurys Inn Hotel, Midsummer Boulevard, Milton Keynes, MK9 2HP A Certified Scrum Trainer with over 30 years’ experience in software engineering and over 10 years’ experience of helping teams use Scrum, XP and other agile and lean frameworks to deliver amazing software products in organisations including United Health, Cisco, AOL, BskyB, The Open University and the BBC. Agile:MK Journal, design & produced by endjin
  3. 3. December 2012The State of the ArtThe adoption of Agile methods and practices within the UK hasbeen on-going for close to 20 years. Scrum has become a morepredominant method over the last decade with a high growthcurve over the past 5 years. However, this high growth of adoptionhas not come without cost.Agile has suffered from a lack of understanding, some governance structure, then it is likely that strongpoor implementations, and a proliferation of terms oversight will be enforced. The creation of steeringand complexity that means that Agile is increasingly groups, status reports, detailed business cases, andbecoming polarised from one of its core other artefacts to facilitate distanced decision-makingprinciples: Simplicity. may be required to see a project to completion. This can lead to waiting for decisions and delays in progress.When a mature company or organisation adopts agile Can this governance be distributed? I.e. Can more ofprinciples, values and methods, it necessarily has to the control of how a project is run, who works on it,undergo significant changes in structures, processes, and its decision making be given to the project teamgovernance, culture, and technologies. without recourse to a governing body?For a company to value its individuals and personal Another example. There are issues with the team’sinteractions OVER processes and tools, working understanding of the business domain required tosoftware OVER comprehensive documentation, create software because they are located off-shorecustomer collaboration OVER contract negotiation away from business subject matter experts? So, canand responding to change OVER following a plan, the software development be brought on-shore tothe mind-set, governance and culture of the the business? Can the business people be secondedorganisation has to change. to the development team? Or is there little movement for change and therefore we may have to implementAt this fundamental level of change lies the crux of Proxy Product Owners or a team of Product Ownersthe problem. (the ‘middle-men’) to represent the business subject matter experts?What is the company’s propensity for change? Because agile teams are constantly trying to improveHow UK companies answer this question is shaping cycle times and remove impediments, they are oftenhow the UK Agile community is developing. looking at problems at their root cause, and finding ways to help the team move on. If the company has a high propensity for change then it is possible to get to the root cause and remove it. However, more commonly, the root cause problem is embedded in the organisation at a level that is very difficult for a team to remove. Work-arounds and sub-optimal solutions are adopted. These sub-optimal solutions have led to a proliferation of “agile” patterns and solutions that are making Agile a complex beast when its essence is simplicity.Propensity for Change & Complexity of Method The Case Study below was the main subject of theWhen adopting agile, we are essentially embarking January session and we learned through ouron a continuous improvement activity and the more discussions that an essential part of adopting agilea company is able to change, the closer to root cause methods effectively and creating hyper-productiveremoval an agile team can reach. teams is making organisational change.For example, if a company has a rigid or centralised MG-Alfa (you’ll hear more about them later) recognised
  4. 4. Agile:MK Journalthe project was important. They hired good people, comply with the new Solvency II regulations, but wasco-located the team, gave the team autonomy over also seen as a valuable exercise in its own right. Astheir working practices, environments, processes part of the wider actuarial project, an on-demand,and architecture and let the team grow. The team cloud-based grid computing platform was also to bemaintained agile principles and values and flourished developed to cost-effectively execute the complexbecause of the key decisions made earlier on about models being to run an agile project. After some proof of concepts work during 2010, anI believe this foresight was key in this project’s success initial team of 2 senior developers began developing the grid and were joined in early 2011 by 2 more senior developers, including myself. With just oneAn Agile Cloud Based Project - change of personnel in late 2011, we became a highlyMilliman MG-ALFA Compute for Azure effective Ian Shimmings, Associate at Endjin The problem we had to solve was how to executeThis article is based on a short presentation and the large number of jobs generated by the actuarialdiscussion to the Agile:MK user group, which was, models in a timely, reliable manner. As a rough guide,in turn, based on nearly 2 years as a senior developer a quarterly complete model recalculation, as requiredon the project. for Solvency II, may consist of thousands of jobs, each with typically 1,000 tasks which may each take betweenSummary 20 minutes and 4 hours on a single core machine.A computing grid platform for Windows Azure, called This equates to many hundreds of years computingMG-ALFA Compute for Azure, was developed and which needed to be complete within 3 to 4 days. It alsolaunched at the end of 2011 followed by multiple needed to provide for the needs of ad-hoc job runs.enhancement releases. It is used to execute thecomplex actuarial models created by actuaries using High level requirements of the system included:the MG-ALFA core model generation tool. It has beentested with more than 45,000 compute nodes in a • Jobs must never be lostsingle grid, across multiple Microsoft data centres. Itis in highly successful production use with Phoenix • Jobs must complete if possibleLife who frequently run multiple grids with manythousands of cores. It is also now being used by • Problems that occur in productionother existing MG-ALFA customers as a Software should be traceableas a Service to simply and cost-effectively run theiractuarial models. MG-ALFA Compute for Azure • Independent of model changes and versioninghas become one of the largest Azure installationsoutside Microsoft. • Be able to control and scale the grid to maximise cost-efficiencyThe grid platform was developed by a highly focussedteam of just 4 co-located senior developers who took The chosen solution was to build the computingfull responsibility for all analysis, testing and life-cycle platform based on the Microsoft Azure Computemanagement, as well as actual development. We used a platform using worker/web roles, storage and thesimple, but very effective, agile process allied with good management API. Azure queues are used todevelopment practices, that evolved with the project. transmit control and status information with blob storage holding data. As well as the main job executionThis team at times reached what may be described and grid control systems, there are custom diagnosticsas ‘hyper-productivity’ where significant functionality and an event driven information service used to drivewas created extremely quickly to very high levels of web, mobile web and Windows 8/RT user interfaces.functional and software design quality. Agile ProcessIntroduction The development process we started with was a simpleIn September 2010, Milliman MG-ALFA was chosen by agile one involving weekly iterations. At the end ofPhoenix Group to simplify, rationalise and streamline each week we would review what was done, have atheir actuarial projection systems and models (http:// retrospective to improve how we worked, and plan the following week. Our tracking consisted solely ofphoenix-group.pdf). This was triggered by the need to cards and a simple three column Kanban style board.
  5. 5. December 2012The cards consisted of MMFs (Minimum Marketable • If it can go wrong, (with thousands of cores running)Features), Stories and Tasks. Where possible we would it will go wrong!only work on a single MMF at a time. o Processes must be resilient and recoverableAs the months went by the process began to evolveuntil we moved to a continuous development style o Persist data securely and often to aid recoverabilitywhere we no longer worked in weekly units but simplyselected MMFs from the frequently re-prioritised and o Write lots of diagnostics, even when running live,modified list of MMFs originally chosen during release as repeating error conditions can be difficult and/ planning. We only created Story cards where a large or expensiveMMF warranted it and rarely wrote Task cards as thatwas left to the developers working on the feature. A • The internet is slow and unreliableretrospective or re-planning session would be heldoccasionally if considered necessary, but changes to o Retry (almost) everythingthe required MMFs and the way we worked typicallyhappened by agreement as and when one of us felt o Expect out of order and repeated eventsthere was a need. • Keep in mind how the system will be managedThis process evolution helped us be, as efficient as in productionwe could be while keeping the process itself as littlemore than a reminder of what we should be doing, o Any tool useful during development and testing developing the application. We specifically avoided may be useful during support and should be keptnaming the process we used (Scrum or Kanban oranything else) as that makes it a thing in its own right, o Consider alerting on error, but reserve such errorsfrequently a hindrance, rather than a help. for worst case scenariosOther process features key to maintaining application Testing large, distributed applications thoroughly forquality included: the cloud can be challenging due to its remoteness, the number of processes and the time and cost• Being rigorous about developing the minimum involved. This makes development in the first place features required at the time of high quality code even more important than some other scenarios.• Frequently holding team design sessions whenever required, at least one for most MMFs • Rapid feedback on quality particularly important• Mandatory TDD (Test Driven Development) o Good unit test coverage• Mandatory pair programming o Effective local ‘implementations’ of the cloud environment important• A complete CI (Continuous Integration) environment, including deployment to an integration environment o Automated local end-to-end tests are vital for quick verification of the ‘wiring up’ of the• A fully automated build, test, package, deployment application developed an maintained by the development team o Automated deployment and in cloud end-to-Developing & Testing Large, end testsDistributed Systems in the CloudAll the best development practices, as well as high o Key integration and UI testsquality, well structured code, are as important whendeveloping for the cloud as for any other kind of • Carefully scheduled full scale tests due to thedevelopment. However, like developing for any time and expense involvedspecific environment, it is important to understandthe features presented and constraints imposed o Automated job submissionwhen working in the cloud and/or developing largescale distributed applications. o Manual UI testing
  6. 6. Agile:MK Journal o Chaos Monkey testing – random server Companies outage, network failure, full memory or disk,etc. Milliman is among the world’s largest independent actuarial and consulting firms. Founded in Seattle,• Continuous Integration test runs that exercise all Washington in 1947, they are among the world’s tests in order of time taken largest providers of actuarial and related products and services. Consulting practices are owned andLearnings managed by principals. MG-ALFA is both a practiceI enjoyed my time developing the Milliman MG-ALFA within Milliman and the name of the powerfulCompute for Azure application. There were many financial modelling tool they produce.challenges and I certainly learned a lot. Below are justa few highlights: Phoenix Group Holdings has a Premium Listing on the London Stock Exchange and is a member of the• Agile is not about named processes, the arguments FTSE 250 index. The Group is a closed life assurance between proponents and sticking to the ‘rules’. It fund consolidator that specialises in the management is about using any simple template and constantly and acquisition of closed life and pension funds, and evolving as the team grows in experience operates primarily in the United Kingdom. Measured by and confidence total assets, the Group is the largest UK consolidator of closed life assurance funds. The Group has over• High quality software developers are required 6 million policyholders and assets of £71.6 billion as and are one of the most important factors for at 30th June 2012. a successful software development project Links• A small, highly skilled, agile team can • Milliman – deliver massively • Phoenix Group –• Each team is unique, construct and modify as required • Endjin – • Microsoft case study –• Once a good team has formed do what you can casestudies/Case_Study_Detail.aspx?casestudy to keep it together id=710000001018 • Good developers can (and are willing to) analyse, • Milliman cloud computing – test, configure builds and deployments, as well expertise/life-financial/products-tools/mg-alfa/ as develop cloud-computing • A good developer without specific skills is generally • Arapsys – better than vice-versa • Developing for the cloud at Netflix –• Only build what is required now: com/2010/12/5-lessons-weve-learned-using-aws. html o Though don’t procrastinate and use as an excuse to ‘bite the bullet’ and make the big changes when needed• Making a practice mandatory (TDD, pairing, etc.) is the best way to REALLY learn it o Prevents reverting to type when under pressure o Once learned make optional to get the best results• Building and testing for large, distributed applications in the cloud is different…
  7. 7. Agile: MK Bookshop December 2012 The Machine that Changed the World By Womack, Jones & Roos At the time of its release, this book threw a bright spotlight on how the Japanese automobile industry was out-stripping the American automobile industry through the use of what is now termed “lean thinking”. The prelude to Lean Thinking, we are told the story of the automobile industry from Henry Ford through to modern day, highlighting the varying philosophies and methods of manufacturing. In taking the historical view, the book provides a clear illustration of the difference between traditional and lean practices. The parallels and connotations with our current software industry leap out from the page, and illustrate how and why some companies are repeating the mistakes of the past in their approach to adopting agile and lean practices. Crucial Conversations Fearless Change The Lean Startup by Patterson, Grenny, McMillan, Switzler by Manns and Rising by Eric Ries Organizational Patterns of Agile Essential Scrum Agile Software Development: Software Development by Kenny Rubin The Cooperative Game by Coplien & Harrison by Alistair Cockburn
  8. 8. ripplerock offers a set of services and tools that enable our customersto dramatically improve their software development capabilityRippleRock formed in 2009 with the mission to assist customers drive dramatic improvements in their software developmentcapability. We apply our expertise across the full lifecycle; facilitating organisational transformation to enable Agile practices,all the way down to improving engineering practices within the teams.The team at RippleRock have a strong track record in the Agile space, some of this through experience gained while at thecentre of Conchango’s Agile Practice and Scrum for Team System tools group. As a specialist in this area we are able to offeraccess to the most experienced Agile coaches, trainers and consultants with the particular mix of skills required to work withpeople, process, organisations and tools.Ripple Rock Ltd is registered in England and Wales No. 0708916Ripple Rock LLC is registered in the United States of America. No. 27-1180168