GovSummit London 2016 - NHS Choices
•Download as PPTX, PDF•
1 like•3,451 views
Splunk at NHS Choices
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (10)
Similar to GovSummit London 2016 - NHS Choices
Similar to GovSummit London 2016 - NHS Choices (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
GovSummit London 2016 - NHS Choices
- 1. NHS Choices: Managing complex infrastructure to deliver critical online services Presented by Andy Callow, Head of Technology Delivery
- 2. 2
- 3. About HSCIC 3 The Health and Social Care Information Centre (HSCIC) was formed in April 2013 as an executive non-departmental public body and the national provider of information, data and IT systems for patients, service users, clinicians, commissioners, analysts and researchers in health and social care.
- 4. About HSCIC 4 • We have responsibility for a number of national systems, including Spine, NHSMail, Electronic Referrals and GPs Systems of Choice and NHS Choices to name just a few. • April 2016 - Announced moving to new operating name NHS Digital
- 5. HSCIC - What We Do 5 We provide information, data and IT Systems for commissioners, analysts and clinicians in health and social care. Our work includes… • Setting standards that protect patient’s confidential information, reduce bureaucracy and improve data quality • Operating essential technology services that support the health and care system • Collecting, analysing and publishing national data and statistical information that helps inform decision making • Developing the next generation of national data and information systems
- 6. About NHS Choices 6 • UK’s biggest health website • More than 14,000 regularly updated articles and hundreds of videos and interactive tools • Compare health services in England 6
- 9. Our Build 9
- 11. Nov 2010 • Tried out free version June 2011 • 5GB/day license Today • Multiple use cases Future • Wider use Our Splunk Journey 11
- 12. Live Performance Stats on Display
- 13. Live Performance Stats on Display • Public Health England campaigns dashboard: 13
- 14. Live Performance Stats on Display 14
- 15. Live Performance Stats on Display 15
- 16. Live Performance Stats on Display 16
- 17. Internet CDN Web Application Firewall/Reverse Proxy Cloud Provider Search Provider (SaaS) Legacy Network (IaaS) Load Balancers Windows Web Servers Linux Web Servers Database Clusters Cloud Web Services (PaaS) Cloud Database Services (PaaS) Splunk Database Load Balancers
- 18. Indexers Single Search Head Searching Splunk Topology Reporting Alerting Dashboards NHS Choices Data segmented by user
- 19. Our Data Sources 19 Online Services Web Server Logs CDN (Akamai) Windows Event Logs Mail Server Logs Performance Counters from Windows & LINUX RSS Logs Searching Reporting Alerting Dashboards
- 20. Our Use Cases Troubleshooting Monitoring Impact of Changes to the Website Managing Unpredictable Traffic Reporting to Management 20
- 22. Troubleshooting: Root Cause Analysis DDOS attack blocked by CDN Reviewed data for previous year Investigation = no previous attack Answer the same day
- 23. Troubleshooting: Root Cause Analysis Page rating averages dropping Used Splunk to investigate Identified set of IP addresses in Germany Blocked IPs & introduced dashboard
- 24. Troubleshooting: Root Cause Analysis 24
- 25. Troubleshooting – using alerts 25
- 26. Monitoring Impact of Changes 26
- 29. Reporting to Management 29 Monitoring Data combined over period Grouped Results grouped into interest areas Availability Actual vs Target
- 30. Future Plans 30 Analytics team: Monitor real time searches Partner team: Track usage of content provided to syndication partners Product Owners: Build product specific dashboards DevOps team: ingest data from other popular tools • Make data available to the business
- 31. Value from Splunk More informed decisions Identifying business value in the data we already have Alerts and reports free up time Correlate previously unconnected events Real-time and historical analysis 31
- 32. Thank you 32
Editor's Notes
- Number of challenges: We want to continually improve services we provide for the public at the lowest possible cost – one way is to ensure our operation is running as efficiently as possible. However, if we’re going to capture sufficient information in order to make those improvements … 40M visits a month generates a LOT of log data – index around 40GB/day
- Nov 2010 Tried out free version Attended online demo Minds blown! June 2011 First purchase 5GB/day license Reporting revolutionised Today Multiple use cases Real-time stats on display via big screens in the office Looking ahead More custom dashboards/reports Plans to expand into other areas of the business
- Other views (Google Maps, Particles, Sideview utils, XML, forms):
- 3 discrete technology stacks Sharepoint, SQL, Windows, .NET Ruby, Nginx, Ubuntu Linux Azure PaaS (Web Apps, SQL Azure) Dual site for DR purposes (plus multiple operating bases) Multiple route to live environments per stack Automated creation of environments (introduction of DevOps principles) Continuous Integration – automated nightly builds/deployments External 3rd party integrations (search, tools, maps etc)
- Range of different sources – visible through single pane of glass
- Correlation of events when something goes wrong to identify things that regularly cause issues
- DDOS attack blocked by CDN Reviewed data for previous year in Splunk to identify if there had ever been a similar attack that went undetected Splunk investigation revealed no previous attack We had the answer the same day
- Example: Spam page ratings Content editors noticed user page rating averages were dropping Used Splunk to investigate Identified a small set of IP addresses in Germany that were bombarding pages with 1 and 2 star ratings Blocked those IPs and now have a dashboard to show average ratings over time to identify spam attacks
- Page rating dashboard Count of ratings over time Breakdown of different ratings (star rating between 1 and 5) Detailed drilldown for source of ratings and correlation between URLs and source IPs
- Set up alerts to identify when these things happen in the future, e.g. GP surgery comments - alert when message queue length exceeds a certain threshold which indicates that there is an issue in the backend causing the comments to get stuck. Alert to ensure there’s plenty of cache on the servers. Other examples: Errors when AV scanning finds issues with files uploaded by users Regular scheduled reports to developers for URLs generating unhandled exception errors – “closing the loop”
- Impact on web server response time of recent change to database storage solution:
- Managing unpredictable traffic – better understanding of traffic e.g. always spikes in January. Money saving expert Ehic card – nhschoices/ehic. Health campaigns e.g. Stoptober with TV ad campaign. One Show – huge spike in traffic. If a rare disease is mentioned in a TV drama people google it. Change4life. Webtrends identified the spike in traffic (users per hour) but Splunk has the granularity to identify the specific 60 second window. Splunk helps to identify any issues caused by these spikes. Can make more informed decisions e.g. not make site updates while it’s busy. Post-analysis with Splunk.
- Traffic levels – 10 Minute Shake Up campaign launch with TV ads!
- Uptime/availability reporting from external monitoring service for SLAs
- Analytics team: Monitor real time searches – which browsers are they coming from, what search term took them to NHS Choices, journey through the website. Partner team: Track usage of content provided to over 600 syndication partners – usage rates etc. Product Owners: Build product specific dashboards showing detailed performance and transaction status for key areas of the site DevOps team: ingest data from other popular tools, such as Git, Puppet or TeamCity, for end to end view of builds/deployments
- More informed decisions Identifying business value in the data we already have Alerts and reports free up time to be more proactive and deliver more value Ability to correlate previously unconnected events Real-time and historical analysis