Transcript of a discussion on how the reuse of proven performance scripts and replaying of synthetic transactions that mimic user experience have cut costs and gained insights into app behaviors.

1. Strategic DevOps—How Advanced Testing Brings Broad
Benefits to Operations and Systems Monitoring for
Independent Health
Transcript of a discussion on how the reuse of proven performance scripts and replaying of
synthetic transactions that mimic user experience have cut costs and gained insights into app
behaviors.
Listen to the podcast. Find it on iTunes. Get the mobile app. Download the
transcript. Sponsor: Hewlett Packard Enterprise.
Dana Gardner: Hello, and welcome to the next edition to the Hewlett Packard Enterprise
(HPE) Voice of the Customer podcast series. I’m Dana Gardner, Principal Analyst at Interarbor
Solutions, your host and moderator for this ongoing discussion on IT
Innovation -- and how it's making an impact on people's lives.
Our next applications performance innovation case study highlights how
Independent Health in Buffalo, New York has entered into a new phase of
DevOps. After a two-year drive to improve software development, speed to
value, and improved user experience of customer service applications,
Independent Health has now extended testing benefits to ongoing apps
production and performance monitoring.
We'll learn how reuse of proven performance scripts and replaying of synthetic transactions that
mimic user experience have cut costs and gained early warning and trending insights into app
behaviors and system status.
To describe how to attain a new strategic level of DevOps benefits, we're joined by Chris
Trimper, Manager of Quality Assurance Engineering at Independent Health in Buffalo, New
York. Welcome, Chris.
Chris Trimper: Thank you for having me, Dana.
Get the New Book
On Effective
Performance Engineering
Gardner: We are also here with Todd DeCapua, Chief Technology Evangelist at Hewlett
Packard Enterprise. Welcome back, Todd.
Todd DeCapua: Hello, everybody.
Gardner

2. Gardner: Let's start at the top. What were the major drivers that led you to increase the way in
which you used DevOps, particularly when you're looking at user-experience issues in the field
and in production?
Trimper: We were really hoping to get a better understanding of our users and
their experiences. The way I always describe it to folks is that we wanted to
have that opportunity to almost look over their shoulder and understand how
the system was performing for them.
Whether your user is internal or external, if they don't have that good user
experience, they're going to be very frustrated and they're going to have a poor
experience. Internally, time is money. So, if it takes longer for things to
happen, and you get frustrated potential turnover, it's an unfortunate barrier.
Gardner: What kind of applications are we talking about? Is this across the spectrum of
different type of apps, or did you focus on one particular type of app to start out?
End users important
Trimper: Well, when we started, we knew that the end user, our members, were the most
important thing to us, and we started off with the applications that our servicing center used,
specifically our customer relationship management (CRM) tool.
If the member information doesn’t pop fast when a member calls, it can lead to poor call quality,
queuing up calls, and it just slows down the whole business. We pride ourselves on our
commitment to our members. That goes even as far as, when you call up, making
sure that the person on the other end of the phone can service you well.
Unfortunately, they can only service you as well as the data that’s
provided to them to understand the member and their benefits.
Gardner: It’s one thing to look at user experience through
performance, but it's a whole new dimension or additional dimension when
you're looking at user experience in terms of how they utilize that application, how well it suits
their particular work progress, or the processes for their business, their line of business. Are you
able to take that additional step, or are you at the point where the feedback is about how users
behave and react in a business setting in addition to just how the application performs?
Trimper: We're starting to get to that point. Before, we only had as much information as we
were provided about how an application was used or what they were doing. Obviously, you can't
stand there and watch what they're doing 24x7.
Lately, we've been consuming an immense amount of log data from our systems and
understanding what they're doing, so that we can understand their problems and their woes, or
make sure that what we're testing, whether it's in production monitoring or pre-production
Trimper

3. testing, is an accurate representation of our user. Again, whether it’s internal or external, they're
both just as valuable to us.
Gardner: Before we go any further, Chris, tell us a little bit about Independent Health. What
kind of organization is it, how big is it, and what sort of services do you provide in your
communities?
Trimper: We're a healthcare company for the Western New York area. We're a smaller
organization. We define the red-shirt treatment that stands for the best quality care that we can
provide our members. We try to be very proactive in everything that we do for our members as
well. We drive members to the provider to do preventative things, that healthier lifestyle that
everybody is trying to go for.
Gardner: Todd, we're hearing this interesting progression towards a feedback loop of moving
beyond performance monitoring into behaviors and use patterns and improving that user
experience. How common is that, or is Independent Health on the bleeding edge?
Ahead of the curve
DeCapua: Independent Health is definitely moving with, or maybe a little bit ahead of, the
curve in the way that they're leveraging some of these capabilities.
If we were to step back and look at where we've been from an industry
perspective across many different markets, agile was hot, and now, as you
start to use agile and break all the right internal systems for all the right
reasons, you have to start adopting some of these DevOps practices.
Independent Health is moving a little bit ahead on some of those pieces, and
they're probably focusing on a lot of the right things, when you look across
other customers I work with. It's things like speed of time to value. That goes
across technology teams, business teams, and they're really focused on their
end customer, because they're talking about getting these new feature
functions to benefit their end customers for all the right reasons.
You heard Chris talking about that
improved end-user experience about
around their customer service
applications. This is when people are
calling in, and you're using tools to see
what’s going on and what your end users
are doing.
There's another organization that actually recorded what their customers were doing when they
were having issues. That was a production-monitoring type thing, but now you're recording a
I do think that Independent Health is
hitting the bleeding edge on that piece.
That’s what I've observed.
DeCapua

4. video of this. If you called within 10 minutes of having that online issue, as you are calling in
and speaking with that customer service representative, they're able to watch the video and see
exactly what you did to get that error online to cause that phone call. So having these different
types of users’ exceptions, being able to do the type of production monitoring that Independent
Health is doing is fantastic.
Another area that Chris was telling me about is some of the social media aspects and being able
to monitor that is another way of getting feedback. Now, I do think that Independent Health is
hitting the bleeding edge on that piece. That’s what I've observed.
Gardner: Let’s hear some more about that social media aspect, getting additional input,
additional data through all the available channels that you can?
Trimper: It would be foolish not to pay attention to all aspects of our members, and we're very
careful to make sure that they're getting that quality that we try to aim for. Whether it happens to
be Facebook, Twitter, or some other mechanism that they give us feedback on, we take all that
feedback very seriously.
I remember an instance or two where there might have been some negative feedback. That went
right to the product-management team to try to figure out how to make that person’s experience
better. It’s interesting, from a healthcare perspective, thinking about that. Normally, you think
about a member’s copay or their experience in the hospital. Now, it's their experience with this
application or this web app, but those are all just as important to us.
Broadened out?
Gardner: Now, you started this with those customer-care applications. Has this broadened out
into other application development? How do you plan to take the benefits that you've enjoyed
early and extend them into more and more aspects of your overall IT organization?
Trimper: We started off with the customer service applications and we've grown it into
observing our provider portals as well. A provider can come in and look at the benefits of a
member, the member portal that the members actually log in to. So, we're actually doing
production monitoring of pretty much all of our key areas,.
We also do pre-production monitoring of it. So, as we are doing a release, we don’t have to wait
until it gets to production to understand how it went. We're going a little bit beyond normal
performance testing We're running the same exact types of continuous monitoring in both our
pre-production region and our production regions to ensure that quality that we love to provide.
Gardner: And how are the operations people taking this? Has this been building bridges? Has
this been something that struck them as a foreign entity in their domain? How has that gone?

5. Trimper: At first, it was a little interesting. It felt like to them it was just another thing that they
had to check out and had to look at, but I took a unique approach with it. I sat down and talked to
them personally and said, "You hear about all these problems that people have, and it’s
impossible for you to be an expert on all these applications and understand how it works.
Luckily, coming from the quality organization, we test them all the time and we know the
business processes."
Get the New Book
On Effective
Performance Engineering
The way I sold it to them is, when you see an alert, when you look at the statistics, it’s for these
key business processes that you hear about, but you may not necessarily want to know all the
details about them or have the time to do that. So, we really gave them insight into the
applications.
As far as the alerting, there was a little bit of an adoption practice for that, but overall we've
noticed a decrease in the number of support tickets for applications, because we're allowing them
to be more proactive, whether it’s proactive of an unfortunately blown service-level agreement
(SLA), or it’s a degradation in quality of the performance. We can observe both of those, and
then they can react appropriately.
Gardner: Todd, he actually sat down and talked to the production people. Is this something
novel? Are we seeing more of that these days?
DeCapua: We're definitely seeing more of it, and I know it’s not unique for Chris. I know there
was some push back at the beginning from the operations teams.
There was another thing that was interesting. I was waiting for Chris to hit on it, and maybe he
can go into it a little bit more. It was the way that he rolled this out. When you're bringing a
monitoring solution in, it’s often the ops team that’s bringing in this solution.
Making it visible
What’s changing now is that you have these application-development testing teams that are
saying, "We also want to be able to get access to these types of monitoring, so that our teams can
see it and we can improve what we are doing and improve the quality of what we deliver to you,
the ops teams. We are going to do instrumenting and everything else that we want to get this type
of detail to make it visible."
Chris was sharing with me how he made this available first to the directors, and not just one
group of directors, but all the directors, making this very plain-sight visible, and helping to drive
some of the support for the change that needed to happen across the entire organization.

6. As we think about that as a proven practice, maybe Chris is one of the people blazing the trail
there. It was a big way of improving and helping to illuminate for all parties, this is what’s
happening, and again, we want to work to deliver better quality.
Gardner: Anything to add to that Chris?
Trimper: There were several folks in the development area that weren’t necessarily the happiest
when they learned that the perception of what they originally thought was there and what was
really there in terms of performance wasn’t that great.
One of the directors shared an experience
with me. He would go into our utilities
and look at the dashboards before he was
heading to a meeting in our customer
service center. He would understand what
kind of looks he was going to be given
when he walked in, because he was directly
responsible for the functionality and performance of all this stuff.
He was pleased that, as they went through different releases and were able to continually make
things better, he started seeing everything is green, everything is great today. So, when I walk in,
it’s going to be sunshine and happiness, and it was sunshine and happiness, as opposed to
potentially a little bit doomy and gloomy. It's been a really great experience for everyone to have.
There's a little bit of pain going through it, but eventually, it has been seen as a very positive
thing.
Gardner: What about the tools that you have in place? What allows you to provide these
organizational and cultural benefits? It seems to me that you need to have a little bit of data in
your hands. You need to have some ability to execute once you have got that data. What’s the
technology side of this; we've heard quite a bit about the people and the process?
Trimper: This whole thing came about because our CIO came to me and said. "We need to
know more about our production systems. I know that your team is doing all the performance
testing in pre-production. Some of the folks at HPE told me about this new tool called
Performance Anywhere. Here it is, check it out, and get back to me. "
We were doing all the pre-production testing and we learned that all the scripts that we did,
which had already been tried and true and been running and continuously get updates as we get
new releases, could just be turned into these production monitors. Then, we found through using
the tool, through our trial, and now all of our two plus years that we have been working with it
that it was a fairly easy process.
It was a big way of improving and
helping to illuminate for all parties, this
is what’s happening.

7. Difficult point
The most difficult point was understanding how to get production data that we could work with,
but you could literally take a test on your VUGen script and turn it into a production monitor in
5-10 minutes, and that was pretty invaluable to us.
That means that every time we get a release, we don’t have to modify two sets of scripts and we
don’t have two different teams working on everything. We have one team that is involved in the
full life cycle of these releases and that can very knowledgeably make the change to those
production monitors.
Gardner: HPE Performance Anywhere. Todd, are lot of people using it in the same fashion
where they're getting this dual benefit from preproduction and also in deployment and
operations?
DeCapua: Yes, it’s definitely something
that’s becoming more-and-more aware.
It’s a capability that's been around for a
little while. You'll also hear about things
like IT4IT, but I don’t want to open up
that whole can of worms unless we want
to dive into it. But as that starts to happen,
people like Chris, people like his CIO,
want to be able to get better visibility into all systems that are in production, and is there an easy
way to do that? Being able to provide that easy way for all of your stakeholders and all of your
customers are capabilities that we're definitely seeing people adopt. It was a big way of
improving and helping to illuminate for all parties, this is what’s happening
Gardner: Can you provide a bit more detail in terms of the actual products and services that
made this possible for you, Chris?
Trimper: We started with our HPE LoadRunner scripts, specifically the VUGen scripts, that we
were able to turn into the production monitors. Using the AppPulse Active tool from the
AppPulse suite of tools, we were able to build our scripts using their SaaS infrastructure and
have these monitors built for us and available to test our systems.
Gardner: So what do you see in our call center? Are you able to analyze in any way and say,
"We can point to these improvements, these benefits, from the ability for us to tie the loop back
on production and quality assurance across the production spectrum?"
Trimper: We can do a lot of trend analysis. To be perfectly honest, we didn’t think that the
report would run, but we did a year-to-date trend analysis and it actually was able to compile all
of our statistics. We saw really two neat things.
That means that every time we get a
release, we don’t have to modify two sets
of scripts and we don’t have two different
teams working on everything.

8. When you had open enrollment, we saw this little spike that shot up there, which we would
expect to see, but hopefully we can be more prepared for it as time goes. But we saw a gradual
decrease, and I think, due to the ability to monitor, due to the ability to react and plan better for a
better performing system, through the course of the year, for this one key piece of pulling
member data, we went from an average of about 12-14 seconds down to 4 seconds, and that
trend actually is continuing to go down.
I don’t know if it’s now 3 or less today, but if you think about that 12 or 14 down to about 4, that
was a really big improvement, and it spoke volumes to our capabilities of really understanding
that whole picture and being able to see all of that in one place was really helpful to us.
Where next?
Gardner: Looking to the future, now that you've made feedback loops demonstrate important
business benefits and even move into a performance benefit for the business at large, where can
you go next? Perhaps you're looking at security and privacy issues, given that you're dealing with
compliance and regulatory requirements like most other healthcare organizations. Can you start
to employ these methods and these tools
to improve other aspects of your SLAs?
Trimper: Definitely, in terms of the SLAs
and making sure that we're keeping
everything alive and well. As for some of
the security aspects, those are still things
where we haven’t necessarily gone down the channels yet. But we've started to realize that there
are an awful lot of places where we can either tie back or really start closing the gaps in our
understanding of just all that is our systems.
Gardner: Todd, last word, what should people be thinking about when they look at their tooling
for quality assurance and extending those benefits into full production and maybe doing some
cultural bonding at the same time?
DeCapua: The culture is a huge piece. No matter what we talk about nowadays, it starts with
that. When I look at somebody like Independent Health, the focus of that culture and the
organization is on their end user, on their customer.
When you look at what Chris and his team has been able to do, at a minimum, it’s reducing the
number of production incidents. And while you're reducing production incidents, you're doing a
number of things. There are actually hard costs there that you're saving. There are opportunity
costs now that you can have these resources working on other things to benefit that end customer.
We've talked a lot about DevOps, we've talked a lot about monitoring, we've mentioned now
culture, but where is that focus for your organization? How is it that you can start small and
The culture is a huge piece. No matter
what we talk about nowadays, it starts
with that.

9. incrementally show that value? Because now, what you're going to do is be able to illustrate that
in maybe two or three slides, two or three pages.
But some of the things that Chris has been doing, and other organizations are also doing, is
showing, "We did this, we made this investment, this is the return we got, and here's the value."
For Independent Health, their customers have a choice, and if you're able to move their
experience from 12-14 seconds to 4 seconds, that’s going to help. That’s going to be something
that Independent Health wants to be able to share with their potential new customers.
As far as acquiring new customers and retaining their existing customers, this is the real value.
That's probably my ending point. It's a culture, there are tools that are involved, but what is the
value to the organization around that culture and how is it that you can then take that and use that
to gain further support as you move forward?
Gardner: Clearly, the benefits can be pretty significant, easily measured, and then shared widely
for more buy-in. So, it's a very cool opportunity. I'm afraid we will have to leave it there.
We've been learning how Independent Health in Buffalo, New York has entered a new phase of
DevOps. And we've heard how the reuse of proven performance scripts and replaying of
synthetic transactions that mimic user experience have cut costs and gained early warning and
trending insights into app behaviors while they're in production and also how the systems and the
people are behaving.
So, please join me in thanking our guests. We've been here with Chris Trimper, Manager of
Quality Assurance Engineering at Independent Health in Buffalo, New York. Thanks, Chris.
Trimper: Thank you very much.
Get the New Book
On Effective
Performance Engineering
Gardner: And we've also been here with Todd DeCapua, Chief Technology Evangelist at
Hewlett Packard Enterprise. Thanks, Todd.
DeCapua: Thanks, everybody.
Gardner: And I'd like to thank our audience as well for joining us for this Hewlett Packard
Enterprise, Voice of the Customer podcast.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host for this ongoing series of
HPE-sponsored discussions. Thanks again for listening, and please come back next time.
Listen to the podcast. Find it on iTunes. Get the mobile app. Download the
transcript. Sponsor: Hewlett Packard Enterprise.

10. Transcript of a discussion on how the reuse of proven performance scripts and replaying of
synthetic transactions that mimic user experience have cut costs and gained insights into app
behaviors. Copyright Interarbor Solutions, LLC, 2005-2016. All rights reserved.
You may also be interested in:
• How Software-defined Storage Translates into Just-In-Time Data Center Scaling
• Big data enables top user experiences and extreme personalization for Intuit TurboTax
• Feedback loops: The confluence of DevOps and big data
• Spirent leverages big data to keep user experience quality a winning factor for telcos
• Powerful reporting from YP's data warehouse helps SMBs deliver the best ad campaigns
• IoT brings on development demands that DevOps manages best, say experts
• Big data generates new insights into what’s happening in the world's tropical ecosystems
• DevOps and security, a match made in heaven
• How Sprint employs orchestration and automation to bring IT into DevOps readiness
• How fast analytics changes the game and expands the market for big data value
• How HTC centralizes storage management to gain visibility and IT disaster avoidance
• Big data, risk, and predictive analysis drive use of cloud-based ITSM, says panel
• Rolta AdvizeX experts on hastening big data analytics in healthcare and retail
• The future of business intelligence as a service with GoodData and HP Vertica