Maturity Models and agile chap 01


Published on

This is the second upload of the book "The Story of Tahini-Tahini: Software Process Improvement with Agile Methods and Maturity Models". We are seeking help to find mistakes and perfect the book.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Maturity Models and agile chap 01

  1. 1. Software Process Improvement with Agile Methods and Maturity Models The Story of Tahini-Tahini: Software Process Improvement with Agile Methods and Maturity Models Page 1/13
  2. 2. Software Process Improvement with Agile Methods and Maturity Models PART I – Introduction Chapter 1 Introduction Who is this book for This book was written with the following people in mind (in decreasing order of interest):  software process improvement consultants that want to better understand agile methods to install them in software organizations;  project managers that have an interest in understanding agile methods for software development, be it to adopt them or evaluate their adoption;  software engineers attempting to work in an agile environment;  undergraduate computer science professors;  undergraduate computer science students;  graduate software engineering professors;  graduate software engineering students. In as much as agile methods and maturity models have been considered opposites in the disciplines of development and maintenance of software, it is difficult to pinpoint a textbook that tries to build a bridge across the (in our view, inexistent) chasm. It was done, once, and with great success, in the modern classic [BOEHM e TURNER, 2003]. Their reference model is the CMMI (now CMMI – DEV) but it is easy to translate their insights to the MR-MPS-SW given the intentional compatibility of the models. Our reference model in the original Portuguese version, and its subsequent Spanish translation, was the MR-MPS-BR1. In this new English version we will consider, compare and contrast both models, with the intention of making more fans of the MPS because of the insights it adds, while at the same time hoping that the respect and admiration we have for its inspiration, the CMMI, comes through clearly. 1 MR – MPS – BR are initials for Modelo de Referencia – Melhoria de Processos de Software – Brasil, or in English: Reference Model – Software Process Improvement – Brazil. We will refer to it as the MPS for short, even when this is technically wrong, being that two more MPS models have been built. Page 2/13
  3. 3. Software Process Improvement with Agile Methods and Maturity Models Definition of agile method for this book This book focus on four agile methods: Kanban, Scrum, XP and FDD (Feature Driven Development). This choice is not random. These four together account for the vast majority of agile implementations in the world. Besides, they cover most, if not all, the needs for applying agile. Each one of them will be dutifully explained in Chapter 3, when they will be introduced to the reader. This will obviously follow an increasing order of complexity: First the simplest one with the largest return for investment, Kanban, is shown. Kanban has a high return because it organizes the team tasks and helps identify problems fast it allows its users to increase their free time to further improve their processes, by freeing them from rework. Next comes Scrum, which is so frequently chosen as the initial method to enter the agile word that it is regularly conflated with agile itself. However, we choose to introduce Scrum in our story only once our example company is sufficiently stable to hold Scrum meetings with regularity and sufficient respect for the process, something we have seen lacking so many times in our consulting. XP is probably the most quoted and misquoted of all agile methods. We have included it because of its profound contributions to software engineering (we are thinking test driven development, pair programming and refactoring as the very top ones) and its similarities and at the same time differences with Scrum, often overlooked by practitioners. Our final choice, Feature driven Development, is totally personal. We believe that FDD should be the tool of choice when a project is large, and we will attempt to justify it. This said, it departs in many ways from classic agile methods, but it does so in a way that makes its adoption by traditional organizations very easy. If software process improvement is the answer, what is the question? The main enemy of a software development enterprise is poor quality. So far, no one has come with a silver bullet to kill the poor quality monster, other than continuous process improvement. Only organizations that have followed this path have achieved incredible feats of quality. Process improvement, therefore, becomes the focus. It can be argued that people and tools (such as CASE tools) are important in their push for increases in productivity. Nothing truer, but the gains only happen when the processes are in place for the individuals to take advantage Page 3/13
  4. 4. Software Process Improvement with Agile Methods and Maturity Models of the tools. Untrained personnel cannot reduce the testing time even with the best of tools; neither can they reduce the number of escaped defects. It is a very common occurrence that organizations2 make poor use of their human resources and pay great sums for licenses that are scarcely used. Even if people and tools are important, it is process that clear the way to increases in productivity. We show this in Figure 1 with icons. The first “equation” competent personnel added to software tools and well defined processes yield (after the equals sign) in success and happiness. The second equation the lack of welldefined processes increases the risks and produces frequent problems in the resulting products. The discipline that processes induce makes it possible to take advantage of the tools and the skills. Without such discipline it is not possible to reproduce the possible successes that projects might have achieved, because the organizational memory is lost forever. The case study: Tahini-Tahini. We will follow in our book the development of an organization that is born out of an idea by university students. The organization they created started very small, and to joke about its size they have called it Tahini-Tahini. The name was born when Marcela, arriving late at the founding meeting, looked around at the 2 We will use the term ‘organization’ to make reference to any human endeavor that has as a goal producing software together, whether or not there is a formal organization and a business model. Page 4/13
  5. 5. Software Process Improvement with Agile Methods and Maturity Models small attendance, and quipped “we are a tiny-tiny organization”. The founding partners around the table found this funny and a name was chosen. We will often call it as they do, T2, which they pronounce t-squared. As any new company created by young entrepreneurs, it has not followed an ideal growth plan. It has been more a story of hiccups and jumps, but their predicaments made them stronger. The company’s problems are not unique; they are what we have found to be the most common ones for organizations of their size and with their profile, at each step in their growth. At each crossroads the partners have had to make decisions that affected results, and in each of them they have done it by changing processes that govern product development. At every chance they aimed at improving the quality and control of the processes to improve the quality and control over the product. The choices in techniques. Throughout the narration of the case study we will introduce Kanban for the initial steps in a process improvement project that aims at installing agile methods; Scrum for the most common project management techniques; XP (Extreme Programming) for what entails the engineering practices, covered in the MPS at Level D (Largely Defined) and in the CMMI in the five process areas of the Engineering category (Product Integration, Requirements Development, Technical Solution, Validation and Verification). When an organization grows, sometimes the above mentioned methods start showing cracks. The key to their breakdown is when the organizational projects grow beneath the normal parameters and it attempts ‘programming in the many’3, where the coordination across teams starts to require too much planning and the limitations it brings coerces teams to choose their tasks with almost no freedom. In those cases a shot to the past is not a bad thing. Using techniques borrowed from agile and chief programmer teams, Feature Driven Development (FDD) allows an organization to remain agile but grow in its scope of projects. These are our choices and we hope you will find them justified when we introduce them one by one as T2 grows. 3 Reference “The Mythical Man-Month”. Page 5/13
  6. 6. Software Process Improvement with Agile Methods and Maturity Models We have also made other choices. We have chosen to introduce SOA at one point, not because it is an agile method, but because it seemed to fit the requirements of the growth pattern of T2. In terms of our principal motor, the process improvement cycle, we have chosen Lean Six Sigma, in one of its many incarnations. We did this because it is our practice in practice: when we consult we chose to identify low hanging fruits that are ripe for process improvement and solve these first. Following a model for process improvement is only justified if it is improvement and not just changes. Without the guidance of Lean, one runs the risk of implementing changes that cannot be seen as improvements. Lean keeps the consultant honest. The contents of the book. In this chapter we also introduce all the remaining chapters. Each chapter that addresses a level in the MPS explains the required results that the model sets, or in CMMI terms, the expected practices. The book is thus divided into four parts, each covering a different need. In the first part, of which this is the initial chapter, (Chapters 1 – 4) we define the basic tenets of the book: the book’s contents, continuous process improvement, agile methods and maturity models. It is expected that this will help the reader understand our choices and, to some extent, the methods chosen. The second part (Chapters 5 and 6) deals with what is normally known as low maturity, levels G and F of the MPS and Maturity Level 2 of the CMMI. This is where the first growing pains of T2 are felt and the resolution of them that is based upon Kanban and Scrum. The third part (Chapters 7 to 9) develops the theme of middle maturity, levels E, D, and C of the MPS and the Defined Level of the CMMI. Here is where XP is introduced and later FDD. The fourth and final part of the book elaborates on how T2 grows on its defined level to achieve high maturity, by reaching a profound understanding (Deming calls it profound knowledge) of their processes, characterizing them quantitatively. The book closes with a view of the road that T2 navigated form its creation to its sale for billions. The chapters one by one. You are already more than halfway through chapter 1. In it we explain as much as we can about the book. Page 6/13
  7. 7. Software Process Improvement with Agile Methods and Maturity Models In chapter 2 we will explain our approach to process improvement. We will start by showing different alternatives and end up by choosing one of them, Lean. Lean is what we do when we consult. We will show you why it is better to do as little as possible to solve a problem rather than embark an organization in a process improvement project that shows little return in the short run. We have an eclectic approach, since no two organizations are identical, we adapt to their realities. However, we cannot adapt if we have only one tool. We will show you how different approaches can be used, by stressing their similarities and explaining their limitations. Also in chapter 2 we will discuss to some extent a systemic vision of organizations and multi-causality. A common cause of failure of organizational change is to consider problems linearly, created by a unique cause and hence easy to solve predictably. It is not the case and you should not be drawn to think this way. Linear thinking does not account for delays either, so that linear thinkers expect immediate results when patience is of the essence. A learning curve is not a step function. We quote freely from [POPPENDIECK, M. e POPPENDIECK, T.], Leading Lean Software Development, because we find it to be an inspiring and though provoking book. Sometimes we resort to the original primer in the subject of systems thinking, [MEADOWS, D.] Thinking in Systems, A Primer. Their joint influence on our managerial thoughts and our consulting experience is imposing. Chapter 3 is where we introduce agile methods in a little more depth. Even when Lean Software Development is an agile method on its own right, recognized as such by its creators and the agile community at large 4, its scope and depth are beyond the grasp of those starting this journey. It is even beyond the objectives we set for ourselves in this book. Instead, guided by the principles of Lean but using the simpler techniques advocated by Kanban, Scrum, XP and FDD we propose an easier adoption road that is equally rewarding. This chapter introduces them but in no way is designed to replace the original texts on the subject, which we strongly urge you to read. Chapter 4 deals with maturity models, in a sense the essence of the book. Maturity models are central in the strategy that we follow for process 4 Page 7/13
  8. 8. Software Process Improvement with Agile Methods and Maturity Models improvement, guiding the choices of solutions, not of problems. We follow closely the MPS but without disengaging with the CMMI-DEV. In any case, what is covered in this book is not enough for implementing any of the two models. We recommend that the reader finds the sources for these models elsewhere for a comprehensive coverage of the subject. Softex has published excellent guides that can be accessed at their website5. Similarly the CMMI Technical Report can be found on line. Both guides are incredibly useful and indispensable for those wanting to implement the suggestions in this book. In a little more detail we will cover the essential understanding required to choose a path in the labyrinth of practices that these models hold. In particular we will develop the cultural changes that an organization must engage in for it to fully realize the benefits of its transit through the different levels of maturity. We will compare in Chapter 4 both architectures, that of the CMMI and that of the MPS. We will also show how they match and where they differ. Details on some of the main differences will appear in the corresponding chapters where the differences are noticeable. Since MPS is the most encompassing of the two, we will follow it and underscore where applicable what the CMMI-DEV is lacking. To close chapter 4 we will discuss the appraisal methods and how an organization can make use of it to understand where it stands. Also we will briefly address the very useful concept of joint appraisals covering the two models at once. Part II starts in chapter 5, with the description of the first growing pains of T2. Using examples of our own activity as consultants and appraisers, we will describe the typical problems of an organization that works perfectly when everything is normal. However, small, even perhaps trivial problems of everyday life (a pregnancy, a visit to a doctor, an area without good cellular reception, a customer with new needs) can start a “perfect storm” that shatters the best of reputations. This is when our friends at T2 have chosen to introduce methods to control their development and, correctly advised; choose to include Kanban in their repertoire. With this small and painless change, they are capable of implementing several practices of the models. Such tempting beginnings induce the partners to invest in a full evaluation of their processes using the MPS model, 5 Page 8/13
  9. 9. Software Process Improvement with Agile Methods and Maturity Models mostly because the first level of the MPS (Level G) is so easy to achieve. This success encourages them to become adopters of the model. In Chapter 6 describe how the partners, inspired by the success they had had in process improvement, decide to increase their bets and go forward with the MPS. The controls that were put in place gave rise to configuration management, which was germinally implemented in level G. Measurement, which because of the professional background of the partners is considered essential to management in any organization, is formally defined and used to make decisions around daily activities. Quality assurance, as a means to keep track of compliance to processes and product standards, is implemented independently of the managerial structure of projects. The arrival of new customers, requesting services that sometimes generate projects around new developments, although often they are simple upgrades and adaptations of their existing product, makes the activities of project portfolio management germane to understanding how to assign their few and valuable resources in a way that best suits their short term and long term goals. This steady growth that T2 experiments soon puts them in a crossroads: grow internally, implying the extension of their facilities, with the attached investments; or subcontract part of their work, resigning some of the margins but lowering the investment, and hence the risk. It is a crucial decision for the tiny company. The partners once more resort to the MPS model, and go over the practices in the acquisition management process. They find that they are capable of managing subcontractors in a manner that they would like to be managed themselves and lower their risks, and they move forward with that decision. Even then, their small team reaches their limit and needs to split to handle the demands of the new customers, so a small number of new developers are hired. They decide to organize the teams minimally, and not wanting to lose their agility, add Scrum to Kanban. When implementing the method they make sure that they are compatible with MPS, and to prove it they undergo a new evaluation and reach Level F. Chapter 7 opens Part III, where the intermediate maturity processes of MPS are explained. As business expands and new opportunities flourish, our characters are forced to expand their office space, even when they had previously decided not Page 9/13
  10. 10. Software Process Improvement with Agile Methods and Maturity Models to. A crucial meeting takes place: Where do we want T2 to go? Remaining small is tempting, but denying growth is unthinkable. Young blood runs through their veins and the drive to succeed is large. Finally they decide to create an ambitious goal for themselves: Become one of the ten best software factories in the world in ten years. Happy with such a vision, they don’t even care to define what makes a software factory one of the best in the world, but they think they have a roadmap in the MPS. To prepare for growth, the partners understand that their goal is reachable only if a knowledge base exists that makes their skills shared, used and expanded by everyone in the company. They create the processes that will establish and maintain such a library of know-how. Their management processes evolve in unison to make better use of this new tool. As normal development of these activities the organizational processes are incrementally defined and updated, using the tool. They arbitrarily set a threshold on the number of employees that will trigger the formalization of the processes to improve and maintain the processes. They reach this number by calculating the overhead such activity will add and deciding on what is the overall income that would make it sustainable. When that number is reached they create the group and support it with rewards for those participating in it. The constant growth drives another need: T2 is now permanently identifying, hiring, training and retaining human resources. Again, they find inspiration in the MPS. The model gives them a coherence in their choices and a framework with which to train, de facto generating a growth culture that is supported by their success. The end result is a Learning Organization, with motivated employees that are constantly making suggestions that will improve the processes they and others follow. An excellent proposal suggests incorporating reuse practices to further improve the quality of the products and the productivity of the teams. The second chapter of this Part III, Chapter 8, deals with the practices and techniques of Extreme Programming (XP). They are incorporated in a manner that is consistent with the existing practices. It starts with a discussion that takes place Page 10/13
  11. 11. Software Process Improvement with Agile Methods and Maturity Models at one of the required retrospectives in each team. The significant difference in velocity across teams, now known because history is factored in when estimating commitments, is assigned to the very differing interpretations in different teams of what ‘development’ is. That misalignment is also blamed for a series of defects that are reaching the market, in a later process group meeting that is reviewing defects. In search for engineering practices that make sense in their environment, a proposal to adopt XP is made, with tepid reception. Nevertheless, the initiative is piloted, with great care not to break any of the processes that are already in place. Some of the practices require extra work, for example in Pair Programming initially a coach is added to capture statistics on time, defects and other variables. Later they adjust their production environment to help record these pieces of information without interruption of the programmers. This keeps them in track with the evidence requested by the maturity models MPS and CMMI, as they prepare for a new evaluation of their maturity. Challenged with the realities of their growth and the increasing risks size brings, our partners incorporate a strategic vision of their business and add practices that will allow them to have more control over their decisions. In Chapter 9 we describe the introduction of risk management as an integral management tool. T2 does not define itself as a risk avoidance company, rather as one that wants to know them and confront them. It is only through risk identification and analysis that a company can face them with chances to come out victorious. Having grown in maturity is used as a lever for absorbing risk. As an example, increasing productivity is not an easy task, but instead of making opportunistic reuse of existing products, the company organizes a component factory that brings them to the verge of product line framework production. It is easy to assemble parts from a library, following the model learnt by assembling processes to create a project’s own defined process, into a product that requires very few hours of work to become operational and fully documented. With this investment, risks of being late and having field defects, the most frequently cited risks, are all but gone and their teams’ productivity soars. This leads to a brief toying with Service Oriented Architectures for some of their product lines that is not generalized to the company. Page 11/13
  12. 12. Software Process Improvement with Agile Methods and Maturity Models In any organization decisions are made at all levels at all times. Each decision has its own cost and its own benefit. Yet T2 had not reached a point where lessons learned where captured for their systematical reuse. To avoid losing this rich experience, T2 incorporates methods of formal decision making that make use of past decisions to enrich the environment. Soon they are adopted and used by all projects, using them to make decisions about the composition of the process in order to attain the expected results, including decisions on reuse and subcontracting and subcontractors. By then T2 has grown to employ several hundred of developers, and their reputation for quality is growing worldwide. They are getting international calls for proposals. All of a sudden their customers are even more remote than before and their sprints have less chances of being closely followed by the product owner. T2 needs a method that lets them plan large projects with a reasonable chance of meeting the schedule and requirement commitments even when the client is less participating than what agile methods in general require. They decide on Feature Driven Development, based on their record and the need to keep their sprint structure that has worked so well for them. Piloted and accepted as a valid standard lifecycle, T2 feels it is ready for their first joint appraisal, simultaneously held with an MPS and a SCAMPI lead appraiser working together. They attempt a ML3 in the CMMI DEV and a Level C in the MPS, with great success. Part IV takes us to T2 at its peak. It has opened offices in several developed countries, has factories in all regions of Brazil and Latin America, and is enjoying its well-earned prestige. However, the partners are not laying back and resting on their laurels. In Chapter 10 we will show how they used their historical process database to further their understanding and improving their decision making. They identify the critical processes, those that tie up to their business goals, then they analyze their relative stability, build models around the data that allow predicting later results in early stages of a project, allowing managers to act timely and avoid serious problems. The techniques in use to support formal decision making are extended to include quantitative analysis and their causal analysis is also improved to make use of statistical models and previous data to calculate the financial outcome of each alternative. Project management becomes a scientific endeavor with parts of art instead of an art with pieces of engineering. Page 12/13
  13. 13. Software Process Improvement with Agile Methods and Maturity Models Chapter 11 closes the book with the partners discussing the sale of two product lines of the company to a mega business. So that their success story is not unique, we review in this chapter all contributing factors to their accomplishments. We briefly touch again on Lean, Kanban, Scrum, FDD and XP. Also, and with great emphasis, the role of maturity models in designing the roadmap to process improvements, thus eliminating costly trial and error and accelerating the growth with maximum returns. Page 13/13