IT BEST PRACTICES, WHITE PAPERS/INTERVIEWSFocus on Patrick O’Toole , Process Improvement GuruMatthew Kabik | September 20, 2012 |Could you tell us a little bit about yourself, yourbackground and what you are working on today.PATRICK O’TOOLE: I started out in IT back in the late 70’s. During that time period Iworked as a systems analyst, project manager, and ultimately program manager. I thenmade a career switch and assumed more of a systems engineering role working on high-speed imaging products that were being released to the financial services marketplace. Bythe early 90’s, we were capturing images of checks at 30 images or 30 checks per second,which was pretty high speed at that time. We were counting on Intel’s 486 chips to bereleased on time so we could actually do the processing – that’s how long ago this was.After that, I decided that I really wanted to consult in the area of project management, soI left the company I was with and started up my own consulting company. I found that Icould get jobs as a contractual project manager; nevertheless, I wasn’t really doingconsulting in the area of project management. Then one day I attended a conference at theSoftware Engineering Institute that focused on risk management. I was fortunate enoughto be seated next to the president of a risk management consulting company and westayed in touch over the Internet. About three years later, he gave me a call and asked if Iwas interested in working with him. That’s when I joined Teraquest, which is BillCurtis’s company. Bill is one of the authors of the CMM for software.I worked with Teraquest for about 4 and a half years, became a lead appraiser andworked with several organizations to help them achieve a higher level of maturity.However, when I noticed that I was spending 95 percent of my time on the road, Idecided that with 2 small kids at home I had better make an adjustment. So I gaveTeraquest about a year’s notice, made a separation plan, and started up my own companyin 2000. Since that time, I have essentially been focusing on training, assessing, andconsulting primarily in the area of CMMI, essentially same things that I was doing withTeraquest.Can you talk a little bit about your status as a visitingscientist at the SEI?PATRICK O’TOOLE: Essentially, the SEI uses part-time employees to help withcertain areas of specialization. I teach the intermediate concepts course which is a 5 dayupper level CMMI course that can only be offered by the SEI. It’s a simple pass fail class
that is a prerequisite for becoming an authorized lead appraiser or CMMI instructor. Ialso teach the intro course on behalf of the SEI. In fact, I just spent two weeks teachingthis class in Brazil. Another thing that I do at the SEI is observe first time instructors. Could you tell us a little bit about the historicalbackground of the CMM and the CMMI – how itstarted, how it evolved and any changes that arecurrently taking place today?PATRICK O’TOOLE: The story, as I’ve been told it, is that Watts Humphrey went toPhil Crosby’s quality college down in Florida, and as he was heading back home toPittsburgh he thought, “How can we apply something like the Crosby model, which is a5-scale model, to the wonderful world of software?” So on the proverbial back of anenvelope, he sketched out the 5 levels of the Crosby scale and what they would mean interms of software, and this ultimately evolved into the CMM.The CMM is simply a collection of good practices that organizations have found to besuccessful. When they were putting the model together, they invited several highqualitysoftware development organizations to congregate in Pittsburgh. They all sat around in aconference room and wrote the practices that had made them successful on yellowstickies. After they pasted the walls with these things, they went through an infinitydiagram exercise and clumped them into groups. They found a large group of projectmanagement practices, a large group of engineering practices and a large group ofsupport type activities. These groups ultimately became the key process series.That resulting set of practices ultimately turned into the CMM for software, which servedmany companies very well for several years. The CMM was primarily a software modelbecause companies were finding that they were having problems not so much withhardware, but more with software. They were more and more dependent on software andsoftware was increasingly letting them down. The need for software improvement waswhat ultimately led to the funding of the SEI and the development of the CMM forsoftware.After several years of use, the CMM started getting crusty around the edges. It was agood first attempt at a model, but the SEI decided that they needed to upgrade it andbegan revising it and developing the CMM for Software. The other thing that washappening was that more and more maturity models were being produced, including amaturity model for system engineering and another one for people. At one point, the SEIclaims they had 17 maturity models either published or on the drawing board to bedeveloped.In response to this, the Department of Defense said, “Wait a minute; every time the SEIreleases a new model we have to train people on it and we need another appraisal methodand another appraisal to go through. Let’s just get it down to one model.” That’s when
the SEI determined that it was time to do a consolidation. They brought together threemodels: the CMM for software, the System Engineering CMM, and the IPTD that wasjust about to be released (the integrated product team CMM). They consolidated thesethree into what is now known as the CMMI, which was published in early 2000.The CMMI went through a couple of trial periods, then ultimately fell into version 1.1,which became the official release for the model. Last August, it went through anotherfairly significant revision and now we’re up to version 1.2.What is your opinion of the progress that has beenmade to date in terms of CMMI penetration in largeorganizations?PATRICK O’TOOLE: I think one of the things we have found, especially in theDepartment of Defense area, is that the CMMI is almost a rite of passage. Organizationshave to adopt the model, because the Department of Defense gives preferred nation statusto any organization that has achieved maturity level 2 or maturity level 3, depending onwhat kind of software or engineering systems need to be developed. The use of theCMMI has now extended into hardware engineering, system engineering, electricalengineering and mechanical engineering. At this point, any project oriented engineeringactivity can use the CMMI to help guide its activities. The Department of Defense is stillon the leading edge of pushing it, but it’s also expanding into other areas. For example,the automotive industry is very adamant that software suppliers and other engineeringsuppliers achieve a certain level of maturity, or at least get on the path to doing so.General Motors has a requirement for their suppliers to be at maturity level 2 by a certaindate and maturity level 3 by another date. Chrysler has done the same thing, but it hastaken a slightly different approach. Rather than focusing on maturity levels, which is astaged representation of the model, Chrysler focused on the continuous representation ofthe model. They take the process areas they believe benefit them as a customer the most,and then they insist that their suppliers achieve certain capability levels in just thoseprocess areas. In some respects, I think this is a good trend because it’s pushingorganizations to do what they should be doing on their own. In other respects, however, Ithink it’s a bad trend because engineering organizations will wind up striving for thenumber, not necessarily for the improvement. Many organizations approach CMMI-based initiatives thinking, ”Let’s do anything we can to get to level 3, whether or not weachieve any improvement.”We hear a lot about Six Sigma, ITIL and of course theCMM. How do you differentiate between these differentimprovement programs and models and how doorganizations know where they should be putting their
energies? What, if any, is the interrelationship betweenall of these things?PATRICK O’TOOLE: There are a lot of models out there and it can be very confusingto people. If you’re doing data center or maintenance type support – a kind of flowactivity where it’s not really a project oriented activity – then ITIL is probably the properchoice. Six Sigma can be very beneficial, but it suffers from some of the problems thatwe see in the low maturity organizations. That is, if they are not doing things in areasonably consistent and repeatable way, Six Sigma can be applied, and can even bebeneficial, but the real benefit won’t materialize until a greater consistency is achieved.My typical recommendation is to get to CMMI maturity level 2, stabilize your projects,and then start thinking about how you might be able to apply a little more sophisticationto take those stable projects to the next level. At this point, I think Six Sigma and CMMIwork very nicely hand-in-hand, especially at higher maturity levels. And once you arelooking at maturity level 4 and 5, Six Sigma is a perfectly natural fit. The one thing Iwould suggest to organizations that have multiple quality initiatives going – Six Sigma,CMMI and ITIL – is that they make sure to pick one as the master. I don’t care which oneit is, but you’re going to get into bizarre turf wars over various quality initiatives if youdo not designate a primary initiative. For instance, you might say, “We’re doing SixSigma, and underneath that umbrella we’ve got an ITIL portion that we have applied tothe flow activities and a CMMI portion going on for the projectoriented activities, but it’sall being done under the umbrella of Six Sigma.” In other words, one intiative is masterand the others are subbordinate to it.For those organizations that have had a reasonablysuccessful CMM program in place, how would youcharacterize their return on investment?PATRICK O’TOOLE: That is always a tricky question, because it is kind of aselfselecting group. Who is going to report their data? Well, it is going to be theorganizations that had glorious success. Nevertheless, the numbers that they show arecompelling. They can show five-to-one payback – for every dollar invested you get a fivedollar return – and that is very good.I think the ones that have the data and are reporting it are being honest; nevertheless, theyare self-selected and that is something to keep in mind. What I would be more interestedin finding out is what occurred within those organizations that have not achieved success.What have been the failure points? Why did they abandon the effort or settle for less? Ithink part of the issue, and this is addressed a little bit in the CMMI, is that whenorganizations measure the benefits achieved or the progress that they’ve made, theymeasure this through the accomplishment of maturity levels. But maturity levels arenothing more than a recognition that you’ve achieved a certain level of process maturity.They do not reflect anything about the value that you have derived from that stabilization.
Organizations would be well served to say, “In order for us to be successful in themarketplace, we need to be a high quality producer, so the process we’re going to buildand the CMMI that we’re going to use as a guideline will have to be focused onachieving high quality.” Too often organizations say, “Let’s get to maturity level 3,” andthey don’t consider what they are really trying to achieve as a business.There are some statistics out there that suggest that80% of IT money is spent on maintaining existingsystems and infrastructure, whereas 20% of the totalinvestment goes toward development. To yourknowledge, is the SEI addressing maintenance at all? Isthere any possibility that we might see a CMM formaintenance in the future?PATRICK O’TOOLE: Well, the CMMI that was released in August was called CMMIfor development, so this presupposes that the SEI is recognizing the need for differentflavors, or what they call constellations. The CMMI for development is the first of threethat they have planned, the other two being the CMMI for services and the CMMI foracquisitions. To the best of my knowledge, though, they do not have maintenance on thedrawing board yet. A lot of organizations treat their maintenance projects. Consequently,you would leverage the CMMI for development, because teh CMMI for development isdesigned for project-oriented development involving things like minor enhancements.I think that the thinking around maintenance is that if we were to develop higher qualityproducts in our development activities and if we were to support these with betterdocumentation, better designs, and better architectures, then the maintenance effort overthe long term would naturally decrease. So rather than trying to fix the systems that wealready feel are eating up 80% of our resources, the best we can hope for is to reducethose same mistakes in the future and to release higher quality, more maintainable, moresupportable products in the beginning of the lifecycle.For organizations that are undertaking the CMMI andimplementing it, what are the critical success factors?PATRICK O’TOOLE: I think something that really helps a CMMI implementation besuccessful is the burning platform: the last release that went out was a major disaster, itblew up in the field, the customers were irate and it resulted in lawsuits. That failuremotivates an organization to say, “We can’t keep doing things the way we have beendoing them. It isn’t working.” Barring that, I have actually recommended to clients thatthey would be well served to have a major disaster, because that is what they need toconvince the workforce that things really need to change. They usually don’t listen to thatkind of advice, however.
Other than that, strong management support is crucial. And this goes beyond justsponsorship. To me sponsorship means we put a team together, we fund the team andafter a month we get together with the team and listen to what they have to say. Mypreference, however, is that have the executives actively get involved. I want them todocument some of their processes and to ensure that they are following their processesand make sure that they are improving them over time. I want the executive who canstand up and say, “Look, I’m becoming more process disciplined. It appears to beworking for me and my team, therefore I think it would be beneficial to the engineers andthe project managers as well.” We need executives who can stand up and lead byexample.For organizations that are attempting to work with theCMM and are having problems, could you outline someof the biggest pitfalls and some of the most commonmistakes made?PATRICK O’TOOLE: I think the biggest problem is the level. They may start off welland say, “We really need to improve and we’re using the CMM as a guidebook to help usdo that.” Then they go at it for six months, maybe twelve months, and because thechanges are gradual, the benefits that they’re achieving is not really evident unless theystep back and assess how far they have come. If they can take that longer termperspective they will see that things are getting better. But at some point the executivewill say, “This is going way too slow. I know we’re in it for improvement, but darn if weare going to make level 2 six months from now.” And so the pendulum swings fromdoing it for the right reasons to getting to the level. The level unfortunately becomes theend all and be all, and whatever process improvement group you have in place startsputting their energies into generating templates and forms to show that they haveachieved the level, rather than staying focused on gaining the day-to-day value. When Iwent to college, I didn’t get smarter the day I was handed my diploma. I got smarter overthe four years that I was at the University, went to the classes and did the homework.Achieving the diploma was just the recognition of the accomplishment; it was not theaccomplishment itself.What advice would you give to a Level 1 organization ifthey had 9-12 months to demonstrate value through aprocess improvement effort? What should they befocusing on and how should they undertake this?PATRICK O’TOOLE: I would take two different perspectives on the work beingperformed. The first question I would ask is, “Where’s the pain?” In other words, whatare the problems the organization is facing and how can we use the CMMI as a medical
reference book to find practices that we believe will address the pain? After that I wouldask, “Where do you need to be successful?”For example, if you’re trying to achieve high quality in order to be successful, why notlook at the CMMI for practices that will lead to higher quality? Peer reviews are oneexample of a practice that will lead to higher quality, but they happen to be staged atmaturity level 3. But who cares? If peer reviews are going to lead to higher quality, andhigher quality is what is going to make you successful, you should get implemented asearly as you can in order to get the value as quickly as possible.Is there anything else that you’d like to talk about, anyarea that we didn’t cover?PATRICK O’TOOLE: The one final thing I want to comment on is the appraisalprocess. The SEI has now come out with a family of appraisal methods. There are class Amethods, where you’re actually going for the gold, but there are also class B and Cmethods that are less rigourous. We typically use class B and C methods for gap analysisor progress analysis. These kinds of less rigourous appraisal methods can really be usedas a health check. We can not only look for compliance to the model, which I think is arather bizarre term anyway, but we can also determine how well the practices of themodel are being adopted by the organization. An organization is trying to become thehigh quality producer. Are they achieving that? Is the organization achieving that? Let’suse the appraisal method not just to look for conformance to the CMMI, but also toexplore personnel’s ideas on how they can achieve higher quality and win in themarketplace.I think appraisals are an underutilized tool. Most organizations look to them to determinewhether or not they have hit their number. But I think there is a whole other use for thesethings. They’re expensive, they’re time consuming and they’re somewhat disruptive, butwe’ve got to find a way to optimize their value.Biography of Patrick O’ToolePat O’Toole is the Principal Consultant at Process Assessment, Consulting & Training(PACT) where he provides a full range of services to his process improvement clients.Pat is one of the most active CMMI lead appraisers, and has led appraisals spanning allmaturity levels, including one of the largest and most complex CMM Level 5 assessmentconducted to date. He is an SEI authorized instructor for the “Intro to CMMI” course whohas taught this course more than 40 times in 6 countries. Pat is a Visiting Scientist at theSEI, and teaches the “Intermediate Concepts of CMMI” course on their behalf.