The Transformation of Healthcare.gov: Two years of innovation in how our government serves its citizens
•Download as PPTX, PDF•
1 like•2,097 views
A presentation on how the Healthcare.gov debacle has transformed the technology landscape of our government.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Transformation of Healthcare.gov: Two years of innovation in how our government serves its citizens
Similar to The Transformation of Healthcare.gov: Two years of innovation in how our government serves its citizens (20)
More from New Relic
More from New Relic (20)
Recently uploaded
Recently uploaded (20)
The Transformation of Healthcare.gov: Two years of innovation in how our government serves its citizens
- 1. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 1 Transformation of Healthcare.gov 1 Sean O’Neil, Optum Technology
- 2. The Transformation of Healthcare.gov Two years of innovation in how our government serves its citizens Sean O’Neil Optum Technology
- 3. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 3 Optum joins the .war room • When the launch of Healthcare.gov ran into troubles, CMS looked for help from the tech sector and from Optum. • QSSI, an Optum owned company, was named Master System Integrator. • We held the MSI role for the next 24 months. This story is how those events have transformed our Public Sector technology in amazing ways.
- 4. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 4 October 2013 - What went wrong?
- 5. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 5 October 2013 - What went wrong? • Nothing special. It was just like most IT projects today. • There’s no need to cover all the details because you’ve all experienced them yourself. • The following list will be déjà vu for anyone that has worked in complex organizations:
- 6. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 6 Problems we’ve seen over and over… Processes missing: change, incident, problem, config, release, job control.. Automation and Monitoring missing. Load Testing inadequate. Unclear lines of responsibility. A high profile customer with high expectations. Technology change approval process complex and bureaucratic. Functional requirements changed at the last minute. Hundreds of serious defects known at launch. “Reference Architecture” overly and unnecessarily complex”. COTS software used inappropriately.
- 7. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 7 Partial list of key technology fixes: • Turn off transparent huge pages. • Turn down excessive logging (Layer 7, Log4j x 4, firewall). • Fix memory leaks. • Strategically replace small VMs with large VMs. • Fix data cache priming procedure. • Basic Apache worker tuning. • Change helpdesk interface to prevent locking on the whole portal. • Mark Logic xquery tuning (id generator). • Load balancing problems everywhere (introduce F5s and tuned distribution algorithms). • Physical tuning: – Placement of the Mark Logic forests within the NAS. – Spread hot spots around the compute pools. – Breaking out Mark Logic data and query nodes. – Moving VMs to consolidate subnets.
- 8. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 8 Partial list of process and culture changes: 1) Verbal communication is OK. 2) SOPs and pre-approved changes. 3) Perfection is an illusion. 4) Problem Management on a 12 hour cycle. 5) Diversity in thought is strength. 6) The passive voice (and passive thinking!) is too slow.
- 9. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 9 The Rules of the .war room
- 10. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 10 The Rules of the .war room 1) The .war room is for solving problems, not shifting blame. 2) The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. 3) We need to stay focused on the most urgent issues - things that will hurt us in the next 24 – 48 hours.
- 11. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 11 • The problems that plagued healthcare.gov weren’t anything special. • But the rescue of the site was the spark of a new revolution in IT that is transforming our public sector. • It’s not just “stopping another healthcare.gov”, it’s about embracing cutting edge technology and methods!
- 12. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 12 Our government joins the 21st Century The Past The Future is Now Proprietary private clouds Public and hybrid clouds Expensive proprietary hardware (Oracle Exadata) Amazon EC2 Expensive RDBMS (Oracle) MySQL – MariaDB Expensive on premise monitoring (Tivoli) New Relic APM Expensive decision support systems (CA Capacity Manager) New Relic Insights Java Application Servers Node.js and Nginx No automation Jenkins, Puppet “Cold stand-by” datacenter for Disaster Recovery Multiple EC2 availability zones, geo-dispersed datacenters, Akamai GTM Off-the-shelf, shrink-wrapped “bloatware” Roll your own. Use Open Source. Static content hosted in datacenters, backed by application servers. Dynamic, interactive JavaScript content hosted in Akamai. Limited visibility into customer behavior End user analytics, A/B Testing
- 13. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 13 New Relic Custom Plugins drive technology advances • Roll your own and community sponsored plugins allowed us freedom in our technology choices because we knew we could monitor it. • Quickly roll out new monitoring to respond to surprises in production. Community versions Developed by HC.gov team • Nginx • Terracotta • F5 LTM • CA Layer 7 XML Gateway • Apache HTTP Server • NFS • EC2 • Chartbeat • Ehcache • Gluster • MS SQL • RedHat Kernel • Node.js • LDAP • Oracle Identity Manager • MarkLogic
- 14. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 14 The Public sector using Public Clouds • The cloud is a wonderful place, right? –Leverage the cloud for cheap capacity • We’re all paying for it… you want the Government to use your money wisely, right? –Scale up for “Black Friday” scenarios • The last day of Healthcare.gov enrollment is 10x volume burst from day one. –Scale down after the surge to save your money.
- 15. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 15 Are public clouds secure? • Securing applications in of public clouds is top priority! –Remediated with secure network access using dedicated hardware –Hardened server images –Multiple layers of threat detection / prevention, data loss prevention, strong encryption everywhere. (…and lots of other stuff I won’t talk about!!)
- 16. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 16 Hybrid Clouds – The Best of Both Worlds • Because moving everything to the cloud is hard and expensive. –Some technology doesn’t work well in the cloud (yet) • Mark Logic still needs big iron. • Move assets to the cloud as they are ready.
- 17. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 17 A Case Study: The Scalable Login System • Oracle Identity Management Suite was the source of many problems. An expensive and complex COTS product. • Replaced with the Scalable Login System Node.js and Maria DB Amazon EC2, S3, ELB End of Year One End of Year Two SLS currently Logins per sec 53 154 346 Accounts per hour 60,000 120,000 132,000
- 18. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 18 A Case Study: App 2.0 • We found 70% of applications submitted the first year were “simple”. • Simple apps could be completed in less than 16 steps. • For year two, we created App 2.0 Year One “Classic” Year Two App 2.0 Application submittal time 40 minutes average for all apps. 10 minutes Steps per application 70+ 16 steps NGNIX, Node.js and Maria DB Calls into the restful layer of the “classic” Healthcare.gov system. No changes required to existing system.
- 19. Confidential property of Optum. Do not distribute or reproduce without express permission from Optum. 19 The United States Digital Service • A few of the original members of the Healthcare.gov rescue team have formed a new team within the government to transform it from the inside out. • Change Agents: USDS members are embedded within select agencies. • Change the culture. • Change the technology. • Oversight of agency technology budgets. • You can help! https://www.whitehouse.gov/digital/united-states-digital-service
- 20. Thank you. Sean O’Neil Optum Technology 612-508-4855 sean.oneil@optum.com
Editor's Notes
- Hi, I’m Sean O’Neil, from Optum Technology – the services business line within UnitedHealth Group. I’m here to tell you a little known story of how the Healthcare.gov debacle has transformed the technology landscape of our government.
- Back in October, 2013, during the problematic initial launch of Healthcare.gov, CMS (the Center for Medicare and Medicaid Services) looked for help from the tech sector and from Optum. QSSI, one of the subcontractors on the Federal Exchange project, had been more successful than some of the other subcontractors, and so was named Master System Integrator.
- This is the point in my speech where I usually disappoint people. They want to hear all the stupid things that everyone thinks must have happened. But that was not the case. (quick slide)
- The problems we had were just like any of the problems you’ve experienced yourself. (quick slide)
- Take a look at this list. These are things you’ve all experienced before. To go over this list, line by line would be to miss the point: that the problems weren’t anything special. Maybe the volume of problems was! It seemed that anything that could be broken, was broken. But I often disappoint people by telling them that there wasn’t any single mistake.
- And so it is true for the fixes. Also, very straightforward. Things we’ve all done before. Maybe special in the respect that we had so many things we had to fix in such a short period of time!
- However, I will pause to go over these changes as I see them as even more important than the technology fixes. Only by radically shaking up the culture of the .war room, were we able to make the progress necessary – and then sustain that progress over the next 24 months. This is also key to the transformation currently underway in how the Federal government approaches IT.
- Made famous by Mikey Dickerson in his article in Time. You may have heard these before. These were revolutionary ideas when we came to the .war room and needed to make rapid progress in a very chaotic situation.
- “ in the words of Mikey Dickerson: There are plenty of other venues where people devote their creative energies to shifting blame.” No badges. No titles.
- This is just a small list. Also using an enterprise implementation of Jira and Github. Pagerduty, Optimizely, Chartbeat, Geckoboard and Mixpanel
- We were able to tune the COTS product to 3x original capability. But that was as far as we could go. Using the new architecture approach, we created a system of nearly limitless throughput capabilities. I can’t show the architecture picture. I would love to – but it’s confidential. However, imagine any of the AWS, hybrid cloud apps you’re working on now and you’re on the right track!
- We were able to tune the COTS product to 3x original capability. But that was as far as we could go. Using the new architecture approach, we created a system of nearly limitless throughput capabilities. I can’t show the architecture picture. I would love to – but it’s confidential. However, imagine any of the AWS, hybrid cloud apps you’re working on now and you’re on the right track!