CHIME LEAD DC 2014 “Key Attributes for Success, Challenges and Critical Success Factors” with Paul Scheib, CISO and Senior Director IS Operations, Boston Children’s Hospital

1. A CHIME Leadership Education and Development Forum in collaboration with iHT2
Creating an Effective
Cyber Security Strategy
________
Key Attributes for Success, Challenges and Critical Success Factors
Paul Scheib Senior Director Information Services & CISO Boston Children’s Hospital
#LEAD14

2. Case Study: When Hacktivists Attack Your Hospital
A CHIME Leadership Education and Development Forum in collaboration with iHT2
The Cyber Threat
Under attack
Our response
Lessons Learned

3. Who is Boston Children’s Hospital
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•Regional medical center in Eastern Massachusetts with 13 satellite locations - 395 bed pediatric teaching hospital, affiliate of Harvard Medical School
•Approximately 25,000 inpatient admissions each year and 200+ specialized clinical programs schedule 557,000 visits annually
•One of the top rated pediatric institutions in the world (US News & World Report), World's largest research enterprise based at a pediatric hospital
•Over 8000 staff and ~14,000 users
•Diverse user community
•Full-time employees and Foundation physicians
•Residents, fellows, researchers and rotational staff

4. A Real Threat
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•March 20, 2014 – notified by external cyber intelligence group about Twitter/ Pastebin posting by Anonymous, threatening attack - result of highly publicized child custody case
•“d0x” of staff and presiding judge posted
•“Details” of BCH external web site posted

5. Who is Anonymous?
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•Anonymous is a loosely associated international network of activist and hacktivists
•Resume includes attacks on Bank of America, Sony, Boston Police, CIA and Sarah Palin.
•Weapons of choice are Distributed Denial of Service, web site defacing, & exposing confidential information.
•Seeks publicity to rally their followers
•Posted YouTube videos threatening Boston Children’s Hospital

6. Was This the Real “Anonymous”?
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•Convened Hospital’s general Incident Response Team
•Inventoried potentially impacted applications
•Began forming contingency plans - focused on potential of loosing or cutting ourselves off from Internet
•Message to entire organization emphasizing vigilance, email security best practices
•Contacted law enforcement
•Redoubled our security efforts and prepared for possible hacking attempts
Not hard to get details they posted
Not hard to post a video on YouTube
Should we take this seriously or is it a hoax?

7. The Cyber Attack
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•About 3 weeks later... low volume DDoS attack starts
•Mitigated by network changes
•Cat and mouse – we address attack, they change tactic/increase volume
•1 week later, Easter/Patriot’ Day weekend (Boston Marathon bombing 1 year anniversary)
•Massive uptick in DDoS volume
•Engaged 3rd party vendor’s Emergency Services and within 8 hours began blocking DDOS attack

8. Internet Traffic During DDoS Attack
A CHIME Leadership Education and Development Forum in collaboration with iHT2

9. The Cyber Attack Evolves
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•Direct attacks on exposed ports, web sites
•Proactively took down virtually all externally facing sites: research, philanthropy, patient and provider portals, etc…
•Massive influx of malware laden emails
•Proactively shut down entire email system for ~24 hrs
•Re-emphasized to staff to not open suspicious mails/attachments
•Ensured no malware made it through filters

10. What did we experience?
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•DDOS attack created short periods of web site outage.
•Attack reached 27 Gbps aimed at a 10Gbps connection. Congestion affected Harvard’s ISP.
•Additional attacks took down web sites of NStar, Wayside Youth, the Mass. Medical Society, and the Town of Framingham.
•Several attempts to deface BCH website.
•Massive influx of malware laden emails
•Proactively shut down entire email system for ~24 hrs. to ensure no malware made it through filters
•Re-emphasized to staff to not open suspicious mails/attachments
•Attempts to compromise systems to potentially expose patient and confidential data, through brute-force attacks, SQL injections, buffer overflows, and the recent HeartBleed vulnerability.

11. Cyber Attack Response
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•Initial attack mitigated by network architecture and changes
•Proactively shut down critical systems to reduce attack surface
•Projected likely attack escalations and formulated real time response plan
•Engaged outside security experts and law enforcement
•DDOS attack flitering
•Breach investigation services and penetration testing of our DMZ systems
•Web application firewall protection of DMZ ePHI systems
•Contingency plans developed to respond to extended Internet outage
•Internal systems (EMR, ERP, etc) remain available while external services (ePrescribe, some Pharmacy apps, etc) not available.
•External communication disruption – email, payers, portals, supply orders, …
•Impact across most functions – Finance, Supply Chain, HR, Clinical, Research.
•Staffed, and continue to staff, Intrusion Detection tools 24 by 7 to identify and block attacks

12. A CHIME Leadership Education and Development Forum in collaboration with iHT2
Cease Fire
•About 1 week after high volume DDoS started, it abruptly declined, to a low trickle
•Only gradually brought externally facing sites back online, after extensive 3rd party scanning and (re)penetration testing

13. What Did We Learn
A CHIME Leadership Education and Development Forum in collaboration with iHT2
•DDoS is a real threat and countermeasures are critical!
•Know what systems (or features within systems) depend on Internet access, and have contingency plans for those
•Recognize importance of email, and need for alternate forms of communication
•Challenging to defend an extended cyber attack with “peace time” staffing levels
•Difficult to separating signal from noise - need a baseline to help detect escalation of cyber activities

14. Q & A
Paul Scheib
paul.scheib@childrens.harvard.edu
A CHIME Leadership Education and Development Forum in collaboration with iHT2
Insert Twitter handle(s) here