WestJet Customer Presentation
•
1 like•1,752 views
WestJet Airlines is a Canadian airline founded in 1996 that has grown to operate over 425 flights per day to over 90 destinations across North America and Central America. The Solutions Architect at WestJet discusses how they implemented Splunk to gain visibility into their various systems like websites and apps. Splunk has helped WestJet troubleshoot issues faster, identify performance problems, and answer ad-hoc questions by consolidating their logs in one place.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to WestJet Customer Presentation
Similar to WestJet Customer Presentation (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
WestJet Customer Presentation
- 1. Copyright © 2015 Splunk Inc. Splunk at WestJet Dan Ford Solutions Architect
- 2. 2 WestJet Airlines Headquarters Located In Calgary 1996 Founded by entrepreneur/pilot Clive and 3 other business partners 2003 WestJet expands service to 24 Canadian destinations coast to coast 2009 Continued growth with 11 new international destinations 2015 Operate an average of 425 flights and carry over 45,000 passengers per day with over 90 destinations across North America/ Central America/ Carribean and now Europe.
- 3. 3 My Role @ WestJet: Solutions Architect At Westjet since 2011 Responsible for: – Guest facing websites including westjet.com, guest portal, mobile apps, booking sites (desktop and mobile),and our self service checkin applications Splunk user for ½ a year Favorite Splunk T-shirt: Because Ninja’s are too busy Fun Fact – I’m pretty good at obscure sports/games (foosball/squash/curling/ping pong).
- 4. 4 Why We Needed Splunk! Voluminous logs dispersed across servers (20+ servers creating 5 Inter-dependent systems lacked integrated visibility Reactive approach to problems “False positives” from other tools Consolidate logs Reduce troubleshooting times Proactively address business critical issues Accurate real-time visibility
- 5. 5 Splunk at WestJet Today 5 50 GB license Self-managed installation 39 different data types forwarded to Splunk: Web logs, Application Server logs, F5 Load Balancers, Tivoli Access Management, Mqueue, API GW, Active Directory, etc. IT operations teams are currently the main users of Splunk
- 6. 6 Any Dashboard You Can Highlight? 6 Screenshot here
- 7. 7 What We Love About Splunk! Who talks to this server? Is a particular call taking too much time? Do we have bots hitting our sites? How many hits are we getting against a particular server? How many errors are we getting and what’s normal? What are the time-out rates? How much we learn from our outages / learn new patterns of what to look for / alert on / get better insight into our apps. Answers To Ad-hoc Questions On Demand Splunk helps us answer questions we didn’t know we had
- 8. 8 Splunk Helps With…. FILTERING OUT THE NOISE IDENTIFYING POOR TRANSACTION RESPONSES END-TO-END MONITORING MOBILE APPLICATIONS … and much more
- 9. 9 AHA! Moment(s) • Troubleshooting Checkin issues – Hours down to minutes • Found out our mobile booking timeouts were set too low • Figured out why my wife always gets selected for additional security checks
- 10. 10 What’s Next Integrate more systems and more data into Splunk Expand Splunk visibility to other application teams/business partners Extend Splunk deployment for security monitoring Look at putting Splunk in multiple data centers Upgrade to 6.2 and move from Search Head Pooling to Search Head Clustering
- 11. 11 Lessons Learned Make sure you architect the solution correctly – Don’t hesitate to get help from Splunk – Don’t skimp on investing in professional services Make logs more Splunk friendly – Build out an application on-boarding process and standardize on log formats Build example dashboards – and let the application teams go from there If you use the deployment server in production – set one up in your non production environments as well
Editor's Notes
- WestJet is Canada’s most preferred airline, the company was founded in 1996 by Pilot Clive Bedoe and 3 other business partners. At the time we were only serving 5 different cities in Western Canada and had 3 planes. In 1999 we expanded to serve a total of 12 different western locations (all in the West – hence the name WestJet). In 2003 WestJet expanded to 24 Canadian destinations – something they obviously weren’t expecting to do with the name WestJet and we’ve been growing since. Currently WestJet flies to over 90 destinations in North America, Central America the Caribbean and now Europe with over 100 aircraft and flies over 45000 guests a day.
- I’ve been at WestJet for about 3 and a half years now and in that time have been in charge of the implementation of our guest portal, booking engine implementations, mobile application and also work on a variety of other products like our self service checkin, mobile web booking and westjet.com applications. I demo’d Splunk many years ago but just starting using it in earnest in October of last year. My favorite shirt is the Because Ninja’s are too busy and when Splunk asked me a fun fact about myself this is all I could come up with is that I’m better than the average bear at semi obscure sports/hobbies like curling/squash/ping pong and foosball. My main focus at WestJet is our Internet Booking Engine and supporting that – which brings me to why I’m here today.
- Our booking system generates around 15Gb worth of logs per day across 25 servers for the booking application alone. Besides the application and web servers - our booking flow integrates with a lot of other systems like our Identity Management systems, Load balancers, Enterprise Service bus, API Gateways, and other front end web servers etc – not all of which our team supports. Before we had splunk – we had multiple scripts that went out to all the hosts and looked for the particular issue on each host – then you’d go to that server and look at the logs manually. Splunk gave us a single place to look for issues with all of these different systems besides the standard up / down monitoring we get from other tools. Our booking application is very verbose with it’s logging and throws a lot of “known good exceptions” – so looking for an issue in a sea of “known errors” unless you knew what you were looking for was very tedious and slow. With splunk we’re able to filter out these known good exceptions and get down to the root of issues much quicker. As well – with dashboards we get great visual representations in real time as to what’s going on with our systems. It’s now easy to tell at a glance from our dashboards what’s going on as opposed to waiting for other monitoring systems to tell us there’s issues and then go start looking at the logs. With our dashboards – which we constantly watch during heavy traffic times (seat sales) we can see when degredation / slowness in our services are happening ahead of time – and can react proactively before issues occur – allowing us to reduce downtime significantly. We’ve also taken a look at the data from the dashboards and have created alerts when we our performance starts to degrade because we can easily tell what’s “normal”. We use synthetic monitoring to continuously monitor our site (Gomez/Alertsite) – sometimes only once per 15 minutes which sometimes can go off due to issues with the location of the monitoring agent or the monitoring agent itself – instead of waiting 15 minutes for our next check – with Splunk we can at a glance tell if this is an issue with the monitoring software or if this is a widespread issue.
- Our installation is pretty straight forward – we have 3 search heads – which have search head pooling enabled using shared NFS storage, 3 Clustered Indexers, a Master Cluster server and our deployment server. We have the splunk forwarder installed on over 100 different clients and devices – and managing these manually would be painful, so I’m very glad we installed the splunk deployment server. We currently index around 35Gb worth of production data per day across a lot of different application. Currently the only users of our Splunk implementation are our IT Operations teams so we could focus on implementing it across multiple applications quickly. This has worked out well and now we’re at a point where we are looking at providing our business users access to Splunk as well.
- Here’s an example of one of our dashboards. We use this dashboard when we have a seat sale – where we can see large increases in traffic to our site (3 to 4 times a normal day’s traffic) and so we need to know how we’re doing because that amount of traffic can affect us quickly. On this one dashboard we can now quickly tell how fast we’re responding to our guests, how many errors we’re throwing, how many requests we’re getting per step of our booking flow and can watch the conversion rate. How our response times with our external vendors are and how each of the routes out to our external vendors are doing and more. This dashboard gives us a real time representation of how healthy our systems are – and what we’re actually responding to the guests with as opposed to what our synthetic monitoring is getting.
- Will just basically reiterate the questions on the slide.
- Filtering out the noise: During releases of our booking flow – we used to manually watch logs on all 20+ servers to see what was going on – and with the amount our application logs – just watching these logs go by while looking for errors was like looking for a needle in a haystack – and if you don’t see it after looking for a few seconds jumping into the haystack face first. With splunk we have a baseline from before the release – so we know what normal looks like and can easily filter out all the known errors so we can easily tell if anything new is happening. Poor transaction response times: The biggest thing for us is we didn't really kind of realize how bad some of our response times were for many different systems. We use multiple internet circuits to reach our source of record for bookings (Sabre) and splunk can easily show us which path is optimal which allows us to switch when we start seeing issues down one of the paths. Mobile: We use Splunk to monitor the API’s our mobile applications use, there was one application that was not responding properly. All the response times were really high, our internal monitoring didn’t show this app as down, our synthetic monitoring wasn’t triggering because our service is load balanced and response times were good over half of the time – looking at Splunk quickly showed us we had a problem with one server
- Thank you very much for taking the time to listen to how Splunk has made a big difference at WestJet. Any questions?