Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CWP Meetup - Managing websites and emergencies

174 views

Published on

Gavin Hamilton from Central Agencies Shared Services shares their experience from the nation wide emergency alert exercise and the lessons learnt for the Civil Defence site when experiencing significant spikes in traffic.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

CWP Meetup - Managing websites and emergencies

  1. 1. Managing Websites and Emergencies - Lunch & Learn CWP Meetup – 20 Dec 2017
  2. 2. Here today from CASS IT & MCDEM • Gavin Hamilton
 Principal Advisor, Web and Publishing team, CASS IT • Leighton Corban
 Systems & Security Architect, Architecture team, CASS IT • Andrew Hammond-Tooke
 Senior Communications Advisor, Digital Channels, Communications team, MCDEM
  3. 3. Covering • Civil Defence website on CWP • Its use in emergencies • Emergency Mobile Alert (EMA) project and campaign • Preparation for EMA and its impact on the Civil Defence website • EMA test on 26 November • Lessons learned: website and EMA campaign • Application to your sites on CWP
  4. 4. CASS • Central Agencies Shared Services, unit of the Treasury • IT, Finance, HR functions for SSC, Treasury, DPMC • 26 websites • CMSs: Drupal, SilverStripe, Plone, static (bespoke), Squarespace • Hosting: CASS itself, SilverStripe CWP (7 instances), external, cloud
  5. 5. Ministry of Civil Defence and Emergency Management • A unit of DPMC since 2014 − DPMC oversees national security system − "4Rs" of risk reduction, readiness, response and recovery − MCDEM is a lead agency (geological, meteorological, infrastructure hazards) − Sarah Stuart-Black, Director • 6 websites − Civil Defence, Happens, GetThru, Shakeout, What's the Plan Stan, Takatu • Comms team of 7: Anthony Frith, Manager − Public education; social media & web content − Public information management during emergencies; manage 6 duty webmasters − BAU comms support for MCDEM & CDEM sector & Ministers
  6. 6. Civil Defence website hosting • Large instance on CWP • Active DR (disaster recovery, load balancing Akld/Wgtn) • Premium WAF/CDN (Incapsula) • Additional site for training
  7. 7. Civil Defence website support • DNA: Application support, software development • SilverStripe: 24/7 support, code warranty, stress testing • CASS IT: − Duty systems engineer/Duty IT helpdesk technician − Strategy/planning − Funding for 'keeping the lights on', planned work − Vendor relationships − Solution architects/security specialists − Content support • MCDEM Comms − Comms strategy − Primary content managers − Emergency dashboard − Duty Webmaster roster and training
  8. 8. Civil Defence website in emergencies • Declared emergencies notified on home page − Yellow for emergencies − Blue for awareness messages • All key information on home page • Reduced page size in KB (fewer images) • Sub-pages refer to home page • Dashboard for Duty Webmasters • RSS feeds from CDEM regions & manual curation
  9. 9. Civil Defence website in ‘peacetime’ > No declared emergencies Emergency checker on every page – updated every minute Banner 
 images
 displayed Banner 
 images
 displayed logo.png
  10. 10. Civil Defence website in emergencies > Declared emergencies notified on home page > Yellow for emergencies Yellow 
 emergency 
 content Banner
 images
 removed – 
 reduce page 
 load times logo.png
  11. 11. Civil Defence website in emergencies > Declared emergencies > Sub-pages > Emergency checker refers to home page https://mcdem2-uat.cwp.govt.nz Yellow emergency checker – refers to home page for emergency content
  12. 12. Civil Defence website in emergencies > Dashboard for managing regional and national emergencies and awareness messages One active emergency – Bay of Plenty region
  13. 13. Civil Defence website in emergencies > Dashboard > Review (manage) an emergency
  14. 14. Civil Defence website in emergencies > Dashboard > Review (manage) an awareness message
  15. 15. Emergency Mobile Alert (EMA) • Alerts to all EMA-capable phones reachable from cell towers on 2degrees, Vodafone, Sparks networks • Funding 2016/2020: $18.9m • System from Netherlands company: one2many.eu • “Serious hazards that involve threats to life, health or property, or in some cases for test purposes" • MCDEM (coordinator) + Police, Fire and Emergency New Zealand, MPI, MoH
  16. 16. EMA campaign • Current campaign: Mid-Oct (Get Ready Week) - Dec • TV, radio, motorway signs, social media • List of capable phones • FAQs/fact sheets (multiple languages) • Promotional resources: posters, videos, graphics/brand guide • Short URLs/redirects using htaccess on CWP instance − civildefence.govt.nz/ema − civildefence.govt.nz/emergency-mobile-alert
  17. 17. EMA campaign
  18. 18. EMA challenges for Civil Defence website • 2.5m smart phone handsets in NZ • Incomplete data from telcos eg cell tower time lag, 3G vs 4G, EMA-capable phones’ functionality & configuration • How many will get the alert? • Will alerts be staggered? • Can/should we include a URL in the message? Will it be clickable? • Will the platform handle the traffic generated? • Is our website design/code optimised?
  19. 19. Preparing website(s) for EMA • DNA/SilverStripe/CASS IT/MCDEM involved • Aug – Nov − SilverStripe/DNA analysing performance of site code − SilverStripe stress testing (test site on UAT, Incapsula/Revera involved) − DNA implemented priority performance improvements − CASS IT benchmarking • 4 Oct – Accidental test − Refined stress testing scenario (Android devices auto loading) − Needed ema.civildefence.govt.nz microsite • 26 Nov – Real nationwide EMA test − Monitoring, analyzing results − Survey Monkey feedback, Colmar Brunton survey
  20. 20. Analysing website performance • Range of tools used − www.webpagetest.org (free) − Chrome/Firefox developer tools (free) − Light stress testing also identifies issues • Focus on − Optimising caching (look at caching directives) − Errors − Requests for HTML, CSS, JS, images, fonts • How many • Size in bytes • www.civildefence.govt.nz − Emergency mode (lighter) − Peacetime (no declared emergencies) mode (heavier)
  21. 21. Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site summary https://mcdem2-uat.cwp.govt.nz
  22. 22. Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site summary https://mcdem2-uat.cwp.govt.nz
  23. 23. Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site waterfall view https://mcdem2-uat.cwp.govt.nz logo.png Google fonts, 
 analytics
 references 2.0 sec ‘Document complete’ page load 39 requests to fully load page
  24. 24. Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site performance review logo.png Ratings 39 requests to fully load page
  25. 25. Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site performance review Feedback/
 suggestions for optimisation of images and caching
  26. 26. Preparing website(s) for EMA > Analysing website performance > Chrome developer tools > Check cache directives in headers Headers for
 logo.png Age in cache
 1 year =
 31536000 secs Served from 
 Incapsula
  27. 27. Preparing website(s) for EMA > Analysing website performance > Chrome developer tools > Check site session cookies Incapsula session
 cookies OK
  28. 28. Priority Civil Defence website performance improvements • Goals − Maximise content delivery by Incapsula CDN (the cache) − Minimise ‘avoidable load’ ie requests to origin CWP server • Implemented before EMA nationwide test − Redeveloped the ‘emergency checker’ − No 404s (error messages can’t be cached) − Redundant session cookies removed − No mixed http and https references in site code − Effective and targeted caching directives • ‘Time to live in the cache’: home page & emergency content 30 secs, sub-pages 2 mins, unchanging images/assets 1 year • Addressing in 2018 − References to Google fonts − Size of SVG images/maps
  29. 29. Two websites strategy for 26 Nov EMA test www.civildefence.govt.nz • SilverStripe CMS • Hosted on CWP • Incapsula CDN • Stress tested by SilverStripe • URL not in test alert message • Traffic from social media or direct • Permanent ema.civildefence.govt.nz • Microsite, static HTML • Hosted by CASS • Cloudflare CDN • Stress tested by Integration QA • URL in test alert message • Traffic from message
 • Temporary only
  30. 30. Scenarios for website stress testing • 1. Real-world possibility – 2 major national emergencies − Permanent site: www.civildefence.govt.nz − Major earthquake in Hikurangi trench/subduction zone triggers major tsunami affecting East coast of NI and SI − 2M EMA-capable phones/handsets now, 3M in 2 years − Use psychology of Kaikoura earthquake behavior on site − Modelling of tests based on: requests to load page(s) * number of users * psychology of user behavior (including duration) • 2. Real-world campaign activity • 3. Impact of the EMA test − 350,000 users following a link to a Civil Defence website − Spikes with Android phones automatically loading link to the website
  31. 31. Civil Defence website stress testing Scenario Requests Hour User target count/hr First load Reload Total/hour Reqs/min Reqs/sec GB/sec Emergency events – notified by national EMA message with link 1 1,500,000 3,000,000 195,000,000 198,000,000 3,300,000 55,000 83.2 2 2,500,000 5,000,000 325,000,000 330,000,000 5,500,000 91,667 138.7 3 100,000 200,000 13,000,000 13,200,000 220,000 3,667 5.5 Campaign, no emergency eg ‘Happens’, ‘Stop What You’re Doing’ 1 200,000 5,200,000 4,800,000 10,000,000 166,667 2,778 2.9 2 100,000 2,400,000 2,400,000 4,800,000 80,000 1,333 1.4 3 50,000 1,200,000 1,200,000 2,400,000 40,000 667 0.7 Recent and ongoing stress testing of www.civildefence.govt.nz is based on these scenarios driving traffic to the site Also tested ema.civildefence.govt.nz to these levels
  32. 32. Civil Defence website stress testing • Conclusions from testing by SilverStripe and Integration in QA − High levels of confidence − Leighton’s view • Challenges during testing − Rate limiting by Revera − Test accounts and their DDoS protection limits − Testing tool constraints eg servers to generate load − Access to Incapsula and Cloudflare technical staff
  33. 33. The EMA Test – what happened? • Sun 26 Nov – staggered test during news hour 6pm-7pm − 98% of cell towers successfully broadcast the alert − Colmar Brunton poll: 49% of NZers received alert/were near someone who did
  34. 34. The EMA Test – what happened? • Both websites performed very well • ema.civildefence.govt.nz − Peak 25,000 concurrent visitors − 45,000 unique visitors 6pm-7pm • www.civildefence.govt.nz − Was in (light) emergency mode (message about the test) − 92,000 unique visitors – whole day − Referrals from news sites, direct, Survey Monkey • 25,000 Survey Monkey responses
  35. 35. EMA test – what happened? > ema.civildefence.govt.nz Cloudflare > 500K cached requests vs 0.5K uncached
  36. 36. EMA test – what happened? > www.civildefence.govt.nz > Peak 25,000 users shortly after 6pm; 92,000 users whole day
  37. 37. EMA test – what happened? > www.civildefence.govt.nz > 94% of requests/sec served from cache (Incapsula)
  38. 38. EMA test – what happened? > www.civildefence.govt.nz Incapsula > Test and day after Grey line: 
 Requests/sec passed to origin server Green line:
 Incapsula serving bulk from cache Dotted line: Total requests peak 340 requests/sec We tested for 
 92,000 requests/sec
  39. 39. EMA test – what happened? > CWP host (WLG) serving www.civildefence.govt.nz Spike in requests handled by origin server at 6.15pm – peaked at 
 1.5K requests/min Same data supplied by SilverStripe for AKL host No problems with origin server response times
  40. 40. Lessons learned - websites • Focus on performance/caching profile of pages • Vigilance: changes in technology stack, impact of software upgrades/development • Value of stress testing • Do regular spot performance tests (10 key pages) • Shared documentation/understanding between vendors/ IT/MCDEM
  41. 41. Lessons learned – EMA test & campaign • Handset EMA capability and configuration highly unpredictable • Largest ever mass broadcast alert in NZ − But telcos can’t provide data on who received it • Accidental test on 4 October − MCDEM took responsibility for the mistake swiftly − Spent the entire day apologising to every complaint − Ended up with an additional 1000 Facebook followers • Planned for public response to the actual test − MCDEM people responding on all social channels and phones − Survey Monkey feedback very useful, one-way was okay
  42. 42. What next? • More performance improvements • Switch from Incapsula to Cloudflare when available on CWP • Cache control module • More testing! • ‘Better Responses’ ministerial review findings • Digital strategy for MCDEM sites • Microsites for campaigns
  43. 43. Questions • Gavin Hamilton
 Principal Advisor, Web and Publishing team, CASS IT • Leighton Corban
 Systems & Security Architect, Architecture team, CASS IT • Andrew Hammond-Tooke
 Senior Communications Advisor, Digital Channels, Communications team, MCDEM
  44. 44. END - Some extra slides for reference follow
  45. 45. Accidental test EMA alert message - 
 4 Oct Source: https://www.stuff.co.nz/national/97522387/early- morning-civil-defence-mobile-alerts-anger-sleepinterrupted-kiwis

×