Learn how IHG runs Elastic on a hyper-converged infrastructure, processes more than 8 TB of data every day for operational log analytics, and maintains Kibana dashboards for more than 300 applications.

1. IHG© Private and confidential
IHG – ELK Hyperconverged
December 2018
August 2017

2. Systems Engineering OrganizationIHG© Private and confidential
Overview
• Introduction to IHG
• Global Hospitality Industry
– - 5000+ hotels
– Approximately 13,000 corporate employees
• East & West coast ELK deployments
• 1000+ application logs

3. Systems Engineering OrganizationIHG© Private and confidential
Wildly Successful!
• The ELK platform at IHG was a huge success
• AppDev teams usage of the platform drove the need to expand the deployment
• Expanding the Compute & Storage of Converged Infrastructure resources was not cost effective and firmware
upgrades were complex
• The demand for ELK drove us to explore Hyper-Converged to reduce Compute & Storage and the simplified
management of the Infrastructure (e.g. one-click upgrades)
• Best cost/performance ratio

4. Systems Engineering OrganizationIHG© Private and confidential
Where we started
• Log aggregation was a business need
• application logs written to nfs
• developers needed production access
• multiple terminals to grep across logs
• stale nfs mounts
• ELK easily became the new log aggregator
• lots of applications
• new customers
• Growth
• With Issues
• RAW unstructured application logs
• glorified grep from a single pane of glass

5. Systems Engineering OrganizationIHG© Private and confidential
Life before ELK

6. Systems Engineering OrganizationIHG© Private and confidential
Kibana – “Glorified Grep”

7. Systems Engineering OrganizationIHG© Private and confidential
Converged / Hyperconverged Architecture
• Converged
• Packages compute, network, and storage into a uniform platform with virtualization
• Typically same spec servers, switching fabric, and a SAN / NAS
• Hyperconverged
• Packages compute, network, and storage into a single physical unit with virtualization.
• Compact

8. Systems Engineering OrganizationIHG© Private and confidential
Converged Infrastructure
• Utilized existing converged infrastructure
• Dedicated Storage
• All flash
• Spindles
• Problems
• Growth
• Storage was very expensive
• Hot / Warm transfer was not transparent
• Complex firmware upgrades

9. Systems Engineering OrganizationIHG© Private and confidential
First Iteration
• Small team of 3 Engineers
• Did a POC with a limited budget
• Utilized existing corporate cloud
• Challenges of Virtualization
– Large VM’s in shared clusters
– Costly to run on a Private cloud managed service
– Daily fires with limited Storage
• The adoption of ELK allowed us to get funding to build out a dedicated platform, which started on a
Converged Platform
– Hot tier on all-flash
– Warm tier on NL-SAS drives
– Dedicated Compute Clusters

10. Systems Engineering OrganizationIHG© Private and confidential
Message delivery

11. Systems Engineering OrganizationIHG© Private and confidential
Hot / Warm

12. Systems Engineering OrganizationIHG© Private and confidential
The problem with queues
• Temporary storage
• Guarantee message delivery
• RabbitMQ
• tuning
• Elastic Support
• Kafka
• Easy
• Delivered out of the box

13. Systems Engineering OrganizationIHG© Private and confidential
New and Improved Pipeline

14. Systems Engineering OrganizationIHG© Private and confidential
Hyper-Converged!!
• Transparent hybrid storage
• Cost effective storage
• Easy to add more hardware
• Easy to manage
• Tunable

15. Systems Engineering OrganizationIHG© Private and confidential
Hyper-Converged!! – Lessons Learned
• Transparent hybrid storage
– Lesson learned: Be sure to size your flash tier appropriately
– Understand how many writes the hardware and software platform will do, e.g. - for us it was 6
per log line
• After troubleshooting with our partners (Elastic, Nutanix) we moved to having 1 Elastic data node per physical
Nutanix node
– Isolated the writes to prevent the flash tier from being overrun
• To get better throughput we striped multiple VMDK’s (200Gb max) across each data node
• Acropolis upgrade (Frodo) gave us even more throughput with better multi-threading for multiple writes per
VMDK
• 4Tb per day at each site (East & West Coast)
– Index rate max @ 120,000/sec, avg @ 40,000/sec

16. Systems Engineering OrganizationIHG© Private and confidential
Evolution
• AppDev teams started to write out new structured log files with enriched data in JSON format
• This evolved into more meaningful dashboards and the expectation that data was collected in real-time
• AppDev teams used the new data for troubleshooting transactions across application stacks
• Data collection rates continue to grow
• This drove us to find a more cost effective hardware platform, which led us to a Hyper-Converged solution
• AppDev teams work with
• 1 part-time Engineer can manage our ELK platform

17. Systems Engineering OrganizationIHG© Private and confidential

18. Systems Engineering OrganizationIHG© Private and confidential
Bios
• Brian Malia – Manager, Build Engineering, Global Infrastructure Operations (GIO)
• Mike Rich – Technical Advisor Systems Engineering (GIO)
• Kevin Patel – Technical Advisor Systems Engineering (GIO)
• Benjie Godfrey – Senior Systems Engineer (GIO)
• Santosh Vallurapalli -- Senior Systems Engineer (GIO)