Case Study: Best-of-Breed ALM Technologies Help Blue Cross and Blue Shield of Florida Break Down Application Silos to Improve Application Performance
Case Study: Best-of-Breed ALM Technologies Help Blue Crossand Blue Shield of Florida Break Down Application SilosTranscript of a BrieﬁngsDirect podcast from HP Discover 2011 on how Blue Cross and BlueShield of Florida provided better visibility into apps and improved incident response.Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Sponsor:HPDana Gardner: Hello, and welcome to a special BrieﬁngsDirect podcast series coming to youfrom the HP Discover 2011 conference in Las Vegas. Were here on the Discover show ﬂoor thisweek, the week of June 6, to explore some major enterprise IT solution trends and innovations making news across HP’s ecosystem of customers, partners, and developers. Im Dana Gardner, Principal Analyst at Interarbor Solutions, and Ill be your host throughout this series of HP-sponsored Discover live discussions. [Disclosure: HP is a sponsor of BrieﬁngsDirect podcasts.] Were now going to focus on Blue Cross and Blue Shield of Florida and a casestudy about how they’ve been able to improve their applications performance and even changethe culture of how they test, provide, and operate their applications.Were here today with Victor Miller. He is Senior Manager of Systems Management at BlueCross and Blue Shield of Florida in Jacksonville, Florida. Welcome.Victor Miller: Thank you.Gardner: Tell me a little bit about this culture? When you shift from one way of doingapplications, you do employ technology, you do employ products. There are methodologies andprocess, but I am interested about how you changed your vision of how applications should bedone.Miller: The way we looked at applications was by their silos. It was a bunch of technology silosmonitoring and managing their individual ecosystems. There was no real way of pullinginformation together. We didn’t represent what the customer is actually feeling inside the applications. One of the things we started looking at was that we have to focus on the customers, seeing exactly what they were doing in the application to bring the information back. We were looking at the performance of the end-user transactions or what the end-users were doing inside the app, versus what Oracle database is doing, for example. When you start pulling that information together, it allows you to get full
traceability of the performance of the entire application from a development, test, staging,performance testing, and then also production side. You can actually compare that information tounderstand exactly where youre at. Also, youre breaking down those technology silos, whenyoure doing that. You move more towards a proactive transactional monitoring perspective.Gardner: It sounds as if you started looking at the experience of the application, rather than themetrics or the parts. Is that fair?Miller: That’s correct. Were looking at how the users are using it and what theyre doing insidethe applications, like you said, instead of the technology around it. The technology can change.You can add more resources or remove resources, but really its all up the end-user, what they aredoing in their performance of the apps.Overcome hurdlesGardner: In order to make this shift and to enjoy better performance and experience with yourapplications, you had to overcome some hurdles. Maybe you could explain what Blue Cross andBlue Shield of Florida is. I think I have a pretty good idea, but you can probably do a better job than I. After we learn a bit about your organization, what were some of the hurdles you had to overcome to get towards this culture? Miller: Blue Cross and Blue Shield is one of the 139 independent Blue Crosses throughout the United States. Were based out of Florida. Weve been around since about 1944. Were independent licensee of the Blue Cross Blue Shield Association. One of our main focuses is healthcare.We do sell insurance, but we also have our retail environment, where were bringing in morehealthcare services. It’s really about the well-being of our Florida population. We do things tohelp Florida as a whole, to make everyone more healthy where possible.Gardner: Let’s look at that problem set. In order to have a better experience for the health andwelfare of your clients and constituents, what was the problem? What did you need to change?Miller: Well, when we started looking at things we thought we were doing ﬁne until we actuallystarted bringing the data together to understand exactly what was really going on, and ourcustomers weren’t happy with IT performance of their application, the availability of theirapplications.We started looking at the technology silos and bringing them together in one holistic perspective.We started seeing that, from an availability perspective, we weren’t looking very good. So, wehad to ﬁgure out what we could do to resolve that. In doing that, we had to break down thetechnology silos, and really focus on the whole picture of the application, and not just theindividual components of the applications.
Gardner: So this sounds like you had to go deeper into the network, looking at the ecosystem ofthe applications. What did you have to do to start to get that full picture?Miller: Our previous directors reordered our environment and brought in a systems managementteam. It’s responsibility is to monitor and help manage the infrastructure from that perspective,centralize the tool suites, and understand exactly what were going to use for the capabilities. Wecreated a vision of what we wanted to do and weve been driving that vision for several years totry to make sure that it stays on target and focused to solve this problem.Gardner: And how did you go about choosing the products and the management capabilitiesyoure going to employ?Miller: We were such early adopters that we actually chose best in breed. We were agent-basedmonitoring environment, and we moved to agent-less. At the time, we adopted MercurySiteScope. Then, we also brought in Mercury’s BAC and a lot of Topaz technologies withdiagnostics and things like that. We had other capabilities like Bristol’s TransactionVision.Umbrella of productsHP purchased all the companies and brought them into one umbrella of product suites. Itallowed us to bind the best of breed. We bought technologies that didn’t overlap, could solve aproblem, and integrated well with each other. It allowed us to be able to get more traceabilityinside of these spaces, so we can get really good information about what the performanceavailability is of those applications that were focusing on?Gardner: In addition to adopting these products, I imagine you also had to change some of yourprocesses and methodologies like ITIL. Tell me about the combination of the products and theprocesses that led you to some pretty impressive results?Miller: One of the major things was that it was people, process, and technology that we werefocused on in making this happen. On the people side, we moved our command center from ourdowntown ofﬁce to our corporate headquarters where all the admins are, so they can be closer tothe command center. If there were a problem that command center can directly contact them andthey go down in there.We instituted what I guess I’d like to refer to as "butts in the seat." I cant come with a bettername for it, but its when the person is on call, they were in the command center working downthere. They were doing the regular operational work, but they were in the command center. So ifthere was an incident they would be there to resolve it.In the agent-based technologies we were monitoring thousands of measurement points. But, youhave to be very reactive, because you have to come after the fact trying to ﬁgure out which onetriggered. Moving to the agent-less technology is a different perspective on getting the data, but
you’re focusing on the key areas inside those systems that you want to pay attention to versus theeverything model.In doing that, our admins were challenged to be a little bit more speciﬁc as to what they wantedus to pay attention to from a monitoring perspective to give them visibility into the health of theirsystems and applications.Gardner: I imagine that this is translated back into your development earlier into therequirements. Is there a feedback loop of sorts now that you can look to that perhaps you didn’thave in the past?Miller: Yeah, there is a feedback loop and the big thing around that is actually movingmonitoring further back into the process.We’ve found out is if we ﬁx something in development, it may cost a dollar. If we ﬁx it intesting, it might cost $10. In production staging it may cost $1,000 It could be $10,000 or$100,000, when it’s in production, because that goes back to the entire lifecycle again, and morepeople are involved. So the idea is moving things further back in the lifecycle has been a verybig beneﬁt.Also, it involved working with the development and testing staffs to understand that you can’tthrow application over the wall and say, "Monitor my app, because it’s production." We have noidea which is your application, or we might say that it’s monitored, because were monitoringinfrastructure around your application, but we may not be monitoring a speciﬁc component ofthe application.Educating peopleThe challenge there is reeducating people and making sure that they understand that they haveto develop their app with monitoring in mind. Then, we can make sure that we can actually givethem visibility back into the application if there is a problem, so they can get to the root causefaster, if theres an incident.Gardner: This is all well and good and it sounds fabulous for a handful of apps, but I imagineyou’ve got to scale this. How do you take what you’ve been describing in terms of this journey,but make it for dozens or hundreds of applications? What is it that you rely on to automate this?Miller: We’ve created several different processes around this and we focused on monitoringevery single technology. We still monitor those from a siloed perspective, but then we also addeda few transactional monitors on top of that inside those silos, for example, transaction scripts thatrun at the same database query over-and-over again to get information out of there.At the same time, we had to make some changes, where we started leveraging the UniversalConﬁguration Management Database (UCMDB) or Run-time Service Model to bring it up andbuild business services out of this data to show how all these things relate to each other. The
UCMDB behind the scenes is one of the cornerstones of the technology. It brings all that silo-based information together to create a much better picture of the apps.Gardner: Some people call that a system of record.Miller: That’s correct. We don’t necessarily call it the system of record. We have multiplesystems of record. It’s more like the federation adapter for all these records to pull theinformation together. It guides us into those systems of record to pull that information out.Gardner: What does this get for you? Are there any metrics or examples you can point to thatvalidate that how effective this can be?Miller: About eight years ago when we ﬁrst started this, we had incident meetings where we hadbetween 15 and 20 people going over 20-30 incidents per week. We had those every day of theweek On Friday, we would review all the ones for the ﬁrst four days of the week. So, we werespending a lot of time doing that.Out of those meetings, we came up with what I call "the monitor of the day." If we foundsomething that was an incident that occurred in the infrastructure that was not caught by sometype of monitoring technology, we would then have it monitored. We’d bring that back, and closethat loop to make sure that it would never happen again.Another thing we did was improve our availability. We were taking something like ﬁve and sixhours to resolve some of these major incidents. We looked at the 80:20 rule. We solved 80percent of the problems in a very short amount of time. Now, we have six or seven peopleresolving incidents. Our command center staff is in the command center 24 hours a day to do thistype of work.Additional resourcesWhen they needed additional resources, they just pick up the phone and call the resourcesdown. So, it’s a level 1 or level 2 type person working with one admin to solve a problem, versushaving all hands on deck, where you have 50 admins in a room resolving incidents.Im not saying that we don’t have those now. We do, but when we do, it’s a major problem. It’snot something very small. It could be a ﬁrmware on a blade enclosure going down, which takesan entire group of applications down. Its not something you can plan for, because youre notmaking changes to your systems. Its just old hardware or stuff like that that can cause an outage.Another thing that is done for us is those 20 or 30 incidents we had per week are down to one ortwo. Knock on wood on that one, but it is really a testament to a lot of the things that our ITdepartment has done as a whole. Theyre putting a lot of effort into into reducing the number ofincidents that are occurring in the infrastructure. And, were partnering with them to get themonitoring in place to allow for them to get the visibility in the applications to actually throw
alerts on trends or symptoms, versus throwing the alert on the actual error that occurs in theinfrastructure.Gardner: Now, we started talking earlier about your philosophy and the experience of the user.Are there any metrics or anecdotes from the welfare and beneﬁt of your end-customers that havedeveloped from the way that you’ve been able to improve your applications?Miller: Customer satisfaction for IT is a lot higher now than it used to be. IT is being called in tosupport and partner with the business, versus business saying, "I want this," and then IT does itin a vacuum. It’s more of a partnership between the two entities to be able to bring stuff together.Operations is creating dashboards and visibility into business applications for the business, sothey can see exactly what theyre doing in the performance of their one department, versus justfrom an IT perspective. We can get the data down to speciﬁc people now.Gardner: Because these activities are a journey, you never perhaps get to an end destination.What are you looking forward next? What’s the roadmap for improving even beyond where youare now?Miller: Some of the big things I am looking at are close closed-loop processes, where I haveactually started to work with making some changes, working with our change management teamto make changes to the way that we do changes in our environment where everything isconﬁguration item (CI) based, and doing that allows for that complete traceability of an asset or aCI through its entire lifecycle.You understand every incident, request, problem request that ever occurred on that asset, but alsoyou can actually see ﬁnancial information. You can also see inventory information and locationinformation and start bringing the information together to make smart decisions based on thedata that you have in your environment.Gardner: That sounds like it could lead to some signiﬁcant cost savings in the long run?Miller: That’s my hope. The really big thing is really to help reduce the cost of IT in ourbusiness and be able to do whatever we can to help cut our cost and keep a lean ship going.Gardner: Well, great. We’ve been hearing about a user case study, Blue Cross and Blue Shieldof Florida, and how they’ve been improving their application performance and the userexperience, and then ultimately providing a better visibility for IT and the perception of IT alongwith overall reduction in total cost. We’ve been hearing this story from Victor Miller. He is theSenior Manager of Systems Management at Blue and Cross Blue Shield of Florida inJacksonville. Thank you.Miller: Thank you.Gardner: And thanks to our audience for joining this special BrieﬁngsDirect podcast coming toyou from the HP Discover 2011 Conference in Las Vegas. Im Dana Gardner, Principal Analyst at
Interarbor Solutions, your host for this series of user experience discussions. Thanks again forlistening, and come back next time.Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Sponsor:HPTranscript of a BrieﬁngsDirect podcast from HP Discover 2011 on how Blue Cross and BlueShield of Florida provided better visibility into apps and improved incident response. CopyrightInterarbor Solutions, LLC, 2005-2011. All rights reserved.You may also be interested in: • HP delivers applications appliance solutions that leverage converged infrastructure for virtualization, data management • HP takes plunge on dual cloud bursting: public and-or private apps support comes of age • HP rolls out EcoPOD modular data center, provides high-density converged infrastructure with extreme energy efﬁciency • HP at Discover releases converged infrastructure products and services aimed at helping IT migrate rapidly to the future • HPs IT Performance Suite empowers IT leaders with uniﬁed view into total operations, costs • HP Delivers NMC 9.1 as New Demands on Network Management Require Secure, Integrated, and Automated Response