JetBlue Airways has implemented a DevOps model to enhance performance feedback in their continuous integration process, utilizing virtual test environments for mobile applications to improve efficiency and reduce costs. The company's performance engineering efforts have led to significant savings and improvements in application reliability, focusing on proactive monitoring and early developer collaboration. This innovative approach has helped JetBlue achieve top rankings in industry benchmarks, emphasizing the importance of mobile apps in customer experience.

1. JetBlue Uses Virtual Test Environments for Mobile
Applications Quality Assurance to Enable Greater
Workforce Productivity
Transcript of a discussion on how JetBlue created a DevOps model by including more
performance feedback in the continuous integration process.
Listen to the podcast. Find it on iTunes. Get the mobile app. Sponsor: Hewlett
Packard Enterprise.
Dana Gardner: Hello, and welcome to the next edition to the Hewlett Packard Enterprise
(HPE) Voice of the Customer podcast series. I’m Dana Gardner, Principal Analyst at Interarbor
Solutions, your host and moderator for this ongoing discussion on IT
innovation and how it's making an impact on people's lives.
Our next performance engineering case study discussion examines how
JetBlue Airways in New York has used virtual environments to reduce
development costs, centralized performance testing, and created a climate
for continuous integration and real-time monitoring of mobile applications.
We'll now hear how JetBlue created a DevOps model by including
performance feedback in the continuous integration process to enable greater
workforce productivity.
To describe how efficient performance engineering has reduced testing, hardware and
maintenance costs by as much as 60 percent, we're joined by Mohammed Mahmud, the Senior
Software Performance Engineer at JetBlue Airways in New York. Welcome, Mahmud.
Mohammed Mahmud: Thank you.
Effective Performance Engineering
Download the Report
Gardner: Tell us why mobile is so important. I think it’s fairly obvious, but consumer
experience and user experience is very important, and so is workforce. So, it’s not just business
to consumer (B2C), it’s also business to employee (B2E). Tell me how important mobile apps are
to your organization.
Mahmud: It's really very important for us to give the customer an option to do check-in, book
flights, manage bookings, check flight status, and some other things. On flights, they have an
option to watch TV, listen to music, and buy stuff using mobile devices as well. But on board,
they have to use Fly-Fi, and that’s another thing. This is one of the most important business
drivers for JetBlue Airways.
Gardner

2. Gardner: What sort of climate or environment have you had to put together in order to make
sure that those mobile apps really work well and that your brand and reputation don’t suffer?
Mahmud: I believe a real-time monitoring solution is the key to success. We use HPE Business
Service Management (BSM), integrated with third-party applications for monitoring
purposes. We created some synthetic transactions and put them out
there on a real device to see, if it goes for a long time, how it
impacts performance. If there are any issues with that, we can fix it
before it happens in the production environment.
Also, we have a real-time monitoring solution in place. This solution uses real devices to get the
real user experience and to identify potential performance bottlenecks in live production
environment. If anything goes wrong there, we can get alert from the production environment,
and we can mitigate that issue right away.
DevOps benefits
Gardner: How have you been able to connect the development process in that operational
environment for DevOps benefits? It’s a great theory, and I know people seek to attain it, but it’s
more difficult to put into practice. What did you do in order to accomplish
that?
Mahmud: We have a DevOps model in our environment, but DevOps is not
my area. My area is strictly performance engineering, but we're in the
process of putting the performance effort into the DevOps model. We're
going to be part of the continuous integration process, so we can take part in
the development process and give performance feedback early in the
development phase.
In this model, an application module upgrade is kicking off the functional test cases and giving
feedback to the developers. Our plan is to take part of that CI process and include the
performance test cases to provide performance feedback in the very early stage of the
development process.
Gardner: How often are you updating these apps? Are you doing it monthly, quarterly, more
frequently?
Mahmud: Most of them on a two- or three-week basis.
Gardner: How are you managing the virtual environment to create as close to the operating
environment as you can. How do the virtualized services and network benefit you?
Mahmud

3. Mahmud: We're maintaining a complete virtualized environment for our performance testing
and performance engineering. Before our developers create any kind of a service or put it out
there, they do mockups using some third-party applications. The virtual environment they're
creating is similar to the production environment, so that when it’s being deployed out there in
the actual environment, it works efficiently and perfectly without any issue.
Effective Performance Engineering
Download the Report
Our developers recently started using the service virtualization technology. Also, we use network
virtualization technology to measure the latency for various geographical locations.
Gardner: How has performance engineering changed over the past few years? We've seen a lot
of changes in general, in development, mobile of course, DevOps, and the need for more rapid
development. But, how have you seen it shift in the past few years in performance engineering?
Mahmud: When I came to JetBlue Airways, LoadRunner was only one product they had. The
performance team was responsible for evaluating the application performance by running a
performance test and give the test results with identifying pass/fail based on the requirements
provided. It was strictly performance testing.
The statistics they used to provide were pretty straightforward, maybe some transaction response
times and some server statistics, but no other analysis or detailed information. But now, it’s more
than that. Now, we don’t just test the application and determine the pass/fail. We analyze the
logs, traffic flow, user behavior, and how they behave, etc. in order to create and design an
effective test. Now, this is more performance engineering than performance testing.
Early in the cycle
We're getting engaged early in the development cycle to provide performance feedback. We're
doing the performance testing, providing the response time in cases where multiple users are
using that application or that module, finding out how this is going to impact the performance,
and finding bottlenecks before it goes to the integration point.
So, it’s more of coming to the developers' table, sitting together, and figuring out any
performance issue.
Gardner: Understanding the trajectory forward, it seems that we're going to be doing more with
microservices, APIs, more points of contact, generating more data, trying to bring the analysis of
that data back into the application. Where do you see it going now that you've told us where it
has come from? What will be some of the next benefits that performance engineering can bring
to the overall development process?

4. Mahmud: Well, as I mentioned earlier, we're planning to be part of the continuous integration;
our goal is to become engaged earlier in the development process. That's sitting together with the
developers on a one-to-one basis to see what they need to make sure that we have performance-
efficient applications in our environment for our customers. Again, this is all about getting
involved in the earlier stages. That's number one.
Number two, we're trying to mitigate any kind of volume-related issue. Sometimes, we have
yearly sales. We don’t know when that's going to happen, but when it happens, it’s an enormous
pressure on the system. It's a big thing, and we need to make sure we're prepared for that kind of
traffic on our site.
Our applications are mostly JetBlue.com and JetBlue mobile applications. It’s really crucial for
us and for our business. We're trying to become engaged in the early stages and be part of the
development process as well.
Gardner: Of course it’s important to be able to demonstrate value. Do you have any metrics of
success or you can point to ways in which getting in early, getting in deep, has been of significant
value? How do you measure your effectiveness?
Mahmud: We did an assessment in our production environment to see, if JetBlue.com goes
down for an hour, how much it’s going to cost us? I'm not authorized to discuss any numbers, but
I can tell you that it was millions of dollars.
So, before it goes to production with any kind of performance-related issue, we make sure that
we're solving it before it happens. Right there, we're saving millions of dollars. That’s the value
we are adding.
Gardner: Of course more and more people identify the mobile app with the company. This is
how they interact; it becomes the brand. So, it's super important for that as well.
Adding value
Mahmud: Of course, and I can add another thing. When I joined JetBlue three years ago,
industry standard-wise our position was bottom on the benchmark list. Now, we're within the top
five in the benchmark list. So, we're adding value to our organization.
Gardner: It pays to get it done right the first time and get it early, almost in any activity these
days.
What comes next? Where would you like to extend continuous integration processes, to more
types of applications, developing more services? Where do you take the success and extend it?
Mahmud: Right now, we're more engaged with JetBlue.com and mobile applications. Other
teams are interested in doing performance testing for their systems as well. So, we're getting

5. engaged with the SAP, DB, HR, and Payroll team as well. We're getting engaged more day by
day. It’s getting bigger everyday.
Gardner: Well, great. I'm afraid we'll have to leave it there. We've been learning how JetBlue
Airways has created a climate for continuous integration and real-time monitoring of mobile
applications to enable greater workforce productivity. And we've heard how JetBlue created a
DevOps model by including more performance feedback in the continuous integration process.
So, join me in thanking our guest. We've been here with Mohammed Mahmud, the Senior
Software Performance Engineer at JetBlue Airways in New York. Thank you, Mohammed.
Mahmud: Thank you for having me.
Effective Performance Engineering
Download the Report
Gardner: And I would like to thank our audience as well for joining us for this Hewlett Packard
Enterprise Voice of the Customer Podcast.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host for this ongoing series of
HPE-sponsored discussions. Thanks again for listening, and do come back next time.
Listen to the podcast. Find it on iTunes. Get the mobile app. Sponsor: Hewlett
Packard Enterprise.
Transcript of a discussion on how JetBlue created a DevOps model by including more
performance feedback in the continuous integration process. Copyright Interarbor Solutions,
LLC, 2005-2016. All rights reserved.
You may also be interested in:
• How Software-defined Storage Translates into Just-In-Time Data Center Scaling
• Big data enables top user experiences and extreme personalization for Intuit TurboTax
• Feedback loops: The confluence of DevOps and big data
• Spirent leverages big data to keep user experience quality a winning factor for telcos
• Powerful reporting from YP's data warehouse helps SMBs deliver the best ad campaigns
• IoT brings on development demands that DevOps manages best, say experts
• Big data generates new insights into what’s happening in the world's tropical ecosystems
• DevOps and security, a match made in heaven
• How Sprint employs orchestration and automation to bring IT into DevOps readiness
• How fast analytics changes the game and expands the market for big data value
• How HTC centralizes storage management to gain visibility and IT disaster avoidance
• Big data, risk, and predictive analysis drive use of cloud-based ITSM, says panel
• Rolta AdvizeX experts on hastening big data analytics in healthcare and retail
• The future of business intelligence as a service with GoodData and HP Vertica