NHS Choices: Managing complex infrastructure to deliver critical online services
•Download as PPTX, PDF•
1 like•1,060 views
Learn how NHS Choices analyses machine data to gain real-time insights into a complex hybrid infrastructure. With this operational intelligence NHS Choices can resolve issues faster, manage unpredictable traffic, easily report to management and ultimately keep the 'front-door to the NHS' open for more than 40 million visitors a month.
Recommended
More Related Content
Viewers also liked
Viewers also liked (11)
Similar to NHS Choices: Managing complex infrastructure to deliver critical online services
Similar to NHS Choices: Managing complex infrastructure to deliver critical online services (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
NHS Choices: Managing complex infrastructure to deliver critical online services
- 1. Copyright © 2015 Splunk Inc. NHS Choices: Managing complex infrastructure to deliver critical online services Neil Moran, NHS Choices
- 2. Copyright © 2015 Splunk Inc. Introduction Matt Davies, Splunk
- 3. © 2015 Splunk Inc. Proprietary and Confidential Information Not for Redistribution About Splunk Company (NASDAQ: SPLK) • Founded 2004, first software release in 2006 • HQ: San Francisco / Regional HQ: London, Hong Kong • Over 1,800 employees, based in 12 countries Business Model / Products • Free download to massive scale • Splunk Enterprise, Splunk Cloud, Splunk Light • Hunk: Splunk Analytics for Hadoop 10,000+ Customers • Customers in 100 countries • 80+ of the Fortune 100 • Largest license: Over 400 Terabytes per day
- 4. MAKE MACHINE DATA ACCESSIBLE, USABLE, AND VALUABLE TO EVERYONE
- 5. Platform for Machine Data Industrial Data and Internet of Things Security, Compliance and Fraud Application Delivery IT Operations Business Analytics OPERATIONAL INTELLIGENCE
- 6. European organisations transforming business with Splunk
- 7. Splunk for IT Operations and App Delivery Manage Operations Through a Single Pane of Glass Infrastructure Monitoring Service Intelligence Application Delivery Continuous Delivery IT Troubleshooting Platform for Machine Data
- 8. Splunk IT Service Intelligence Data-Driven Service Monitoring and Analytics At-a-Glance Problem Analysis Early Warning on Deviations Dynamic Service Models Seamless Workflow Integrations Time Series Index Schema-on-the-Fly Data Model Common Information Model Platform for Machine Data Splunk IT Service Intelligence
- 9. HOW VODAFONE GOT END-TO-END INSIGHT USING SPLUNK ITSI Glass table visualizations enable rapid and proactive issue resolution Custom KPIs empower teams across the business, operations & security Actionable service insights in two days, not months
- 10. Copyright © 2015 Splunk Inc. NHS Choices: Managing complex infrastructure to deliver critical online services Neil Moran, NHS Choices
- 11. About HSCIC The Health and Social Care Information Centre (HSCIC) was formed in April 2013 as an executive non-departmental public body and the national provider of information, data and IT systems for patients, service users, clinicians, commissioners, analysts and researchers in health and social care. We have responsibility for a number of national systems, including Spine, NHSMail, Electronic Referrals and Gps Systems of Choice and NHS Choices to name just a few. 11
- 12. We provide information, data and IT Systems for commissioners, analysts and clinicians in health and social care. Our work includes… • Setting standards that protect patient’s confidential information, reduce bureaucracy and improve data quality • Operating essential technology services that support the health and care system • Collecting, analysing and publishing national data and statistical information that helps inform decision making • Developing the next generation of national data and information systems HSCIC – What we do
- 13. About NHS Choices ● Online 'front door' to the NHS – the UK’s biggest health website ● More than 40 million visitors a month ● More than 20,000 regularly updated articles and hundreds of videos and interactive tools ● Compare health services in England 13
- 16. Our Original Challenge We want to continually improve services we provide for the public at the lowest possible cost. 16
- 17. Our Original Challenge Used external service to monitor site availability. When problems occurred we had to log on to each web server in turn to identify the root cause. Needed to identify availability issues faster. . 17
- 18. Our Original Challenge More than 40 million visits a month, generates a LOT of log data. 18
- 19. Our Splunk Journey 19 Nov 2010 • Tried out free version • Attended online demo • Minds blown! June 2011 •First purchase 5GB/day license •Reporting revolutionised Today •Multiple use cases •Real-time stats on display via big screens in the office Looking ahead • More custom dashboards/reports • Plans to expand into other areas of the business
- 20. Live Performance Stats on Display ● Main live operations dashboard: 20
- 21. Live Performance Stats on Display ● Public Health England campaigns dashboard: 21
- 22. Live Performance Stats on Display ● Other views (Google Maps, Particles, Sideview utils, XML, forms): 22
- 23. NHS Choices: Complex Infrastructure 23 ● 3 discrete technology stacks ● Sharepoint, SQL, Windows, .NET ● Ruby, Nginx, Ubuntu Linux ● Azure PaaS (Web Apps, SQL Azure) ● Multiple route to live environments per stack ● Automated creation of environments (introduction of DevOps principles) ● Continuous Integration – automated nightly builds/deployments ● External 3rd party integrations (search, tools, maps etc)
- 24. Indexers Single Search Head Searching Splunk Topology Reporting Alerting Dashboards NHS Choices Data segmented by user
- 25. Our Data Sources 25 Online Services Web Server Logs CDN (Akamai) Windows Event Logs RSS Logs Mail Server Logs Performance Counters from Windows and LINUX
- 26. Our Use Cases Troubleshooting Monitoring Impact of Changes to the Website Managing Unpredictable Traffic Reporting to Management 26
- 27. Troubleshooting: Alerts ● Correlation of events when something goes wrong to identify things that regularly cause issues ● Set up alerts to identify when these things happen in the future, e.g. – GP surgery comments - alert when message queue length exceeds a certain threshold which indicates that there is an issue in the backend causing the comments to get stuck. – Alert to ensure there’s plenty of cache on the servers. – Errors when AV scanning finds issues with files uploaded by users – Regular scheduled reports to developers for URLs generating unhandled exception errors – “closing the loop” 27
- 28. Troubleshooting: Root Cause Analysis DDOS attack blocked by CDN Reviewed data for previous year in Splunk to identify if there had ever been a similar attack that went undetected Splunk investigation revealed no previous attack We had the answer the same day 28 Content editors noticed user page rating averages were dropping Used Splunk to investigate Identified a small set of IP addresses in Germany that were bombarding pages with 1 and 2 star ratings Blocked those IPs and now have a dashboard to show average ratings over time to identify spam attacks Example: DDOS Attack Example: Spam page ratings
- 29. Troubleshooting: Root Cause Analysis ● Page rating dashboard – Count of ratings over time – Breakdown of different ratings (star rating between 1 and 5) – Detailed drilldown for source of ratings and correlation between URLs and source Ips 29
- 30. Monitoring Impact of Changes ● Impact on web server response time of recent change to database storage solution: 30
- 32. Managing Unpredictable Traffic ● Traffic levels – 10 Minute Shake Up campaign launch with TV ads! 32
- 33. Reporting to Management ● Uptime/availability reporting from external monitoring service for SLAs 33
- 34. Future Plans Make data available to the business Analytics team: Monitor real time searches – which browsers are they coming from, what search term took them to NHS Choices, journey through the website. Partner team: Track usage of content provided to over 600 syndication partners – usage rates etc. Product Owners: Build product specific dashboards showing detailed performance and transaction status for key areas of the site DevOps team: ingest data from other popular tools, such as Git, Puppet or TeamCity, for end to end view of builds/deployments 34
- 35. Value from Splunk More informed decisions Identifying business value in the data we already have Alerts and reports free up time to be more proactive and deliver more value Ability to correlate previously unconnected events Real-time and historical analysis 35
- 36. Thank You
Editor's Notes
- But I’m not going to talk about how you keep the lights on. If you’re in those seats, you already know that Splunk is used to troubleshoot, monitor, and generally improve IT Operations. As the backbone of the modern, data driven enterprise, Splunk gets the right data to the right person. Then you fix stuff. You are the ones who are making it happen. You know this. Instead I’m going to talk about something Splunk hasn’t done, until now. We’re re-framing all of that knowledge, all of that data, into the language real IT people speak. Services. Powered by Splunk.
- But I’m not going to talk about how you keep the lights on. If you’re in those seats, you already know that Splunk is used to troubleshoot, monitor, and generally improve IT Operations. As the backbone of the modern, data driven enterprise, Splunk gets the right data to the right person. Then you fix stuff. You are the ones who are making it happen. You know this. Instead I’m going to talk about something Splunk hasn’t done, until now. We’re re-framing all of that knowledge, all of that data, into the language real IT people speak. Services. Powered by Splunk.
- So are we making services exciting again? With this service layer in Splunk we’re going to keep bringing you new service oriented features. Adaptive thresholds? Check. Machine learning? Check. Content out of the box for common components? Check. But don’t believe me. We have over a dozen customer, running IT Service Intelligence live in production. Let’s hear from two of them.
- NHS Choices now part of HSCIC, Splunk already in use across Spine 2 and other systems
- Message Queues, Ruby on Rails/Unicorn, Akamai web application firewall, internal network infrastructure devices, etc, etc
- Managing unpredictable traffic – better understanding of traffic e.g. always spikes in January. Money saving expert Ehic card – nhschoices/ehic. Health campaigns e.g. Stoptober with TV ad campaign. One Show – huge spike in traffic. If a rare disease is mentioned in a TV drama people google it. Change4life. Webtrends identified the spike in traffic (users per hour) but Splunk has the granularity to identify the specific 60 second window. Splunk helps to identify any issues caused by these spikes. Can make more informed decisions e.g. not make site updates while it’s busy. Post-analysis with Splunk.
- Analytics team – search requests, page comments. Track server performance. Monitor real time searches – which browsers are they coming to, what search term took them to NHS Choices, journey through the website. Partner team – track usage of feeds/content syndicated to third parties. Over 600 syndication partners, more useful reporting on the rates of it being used. Monitor use and throttle to ensure no bug hit to the site. Identify repeat offenders and proactively reach out to them. Very dynamic – new people all the time, don’t always understand they need to stagger the content.