Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Becoming DevOps: athena’s Journey Through the Transformation


Published on

Chris Parsons, Sr. Manager, NOC Operations & Engineering, athenahealth

For athenahealth, a leading provider of cloud-based business services for 100,000 healthcare providers, delayed incident response times could put patients’ health and safety at risk. Learn how athenahealth has built an integrated ITOps collaboration system that combines five solutions for monitoring, service desk and chat, while automating problem management. See how this new toolchain increases visibility across the Enterprise and allows for the team to build self-service integrations at a rapid scale for the tools of today and the future. The end result is cost reduction, accelerated response times, and increases in customer satisfaction.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Becoming DevOps: athena’s Journey Through the Transformation

  1. 1. Becoming DevOps: athena’s Journey Through the Transformation athenahealth Christopher Parsons | Sr. Manager, NOC Operations & Engineering
  2. 2. Chris Parsons Sr Manager, NOC Operations & Engineering
  3. 3. athenahealth Products and Services For Medical Groups and Practices For Hospital and Health Systems Electronic Health Records Medical Billing Patient Engagement Order Transmission Secure Text Messaging Epocrates Electronic Health Records Revenue Cycle Management Care Coordination Population Health Epocrates athenahealth
  4. 4. We are a large organization... We are 5k+ employees strong, connecting more than 115,000 providers and health systems nationwide, on a mission to help deliver better care for all. billion posted client collections $30.8 medical providers 115K employees worldwide 5,282 patient records 91M web hits per day 1.1B healthcare interfaces 300K+
  5. 5. In a highly regulated and dynamic industry... HIPAA Compliant PCI-DSS Compliant SSAE 18/SOC 1 Sarbanes-Oxley Act (SOX) HITRUST Certified SOC 2 EHNAC Network and Penetration Tests EPCS DirectTrust TrustedSec Disaster Recovery Testing Meaningful Use ICD-10
  6. 6. The Business Problem
  7. 7. The Timeline... December 2016 ATHN Stock Price hits a 3-year low following Q3 2016 Investors Call August thru November 2017 Large scale disruptions cause us to miss 99.7% SLA for many clients July 2016 Agile/DevOps transformation and re-platform initiative begins November 2017 Corporate restructuring announced, Organization attempts to realign itself for success March 2017 Large scale disruption forces a change freeze 1997 thru 2015 Rainbows and butterflies, athenahealth is born, goes public and is regularly hitting 30% YoY growth goals Client NPS declines 24.3 points Q1 2018
  8. 8. DevOps was supposed to fix us, what happened? New platform was prioritized too heavily over the existing platform, existing product support was transitioned to teams who needed time to get familiar Existing development teams were broken up and reconfigured, replaced with many small scrum teams, support became even more difficult Tooling was lacking in many areas We had “DevOps Engineers!” The “us” vs. “them” mindset was in full effect Simply put… We tried to move too quickly and it felt like we were not prepared!
  9. 9. And our clients were feeling it, too... Net Promoter Score (NPS) was way down The term “death by a thousand paper cuts” used when discussing end user experience Complex problems were taking longer to resolve and were visible to the end user
  10. 10. Was this a Dev OOPS?
  11. 11. How are we fixing it?
  12. 12. Divide & Conquer Benefit: Teams remain small (2 pizza teams) and are able to move quickly. Each zone for any given product gets dedicated cross-functional resources. Decision making is pushed down to the lowest levels and teams are able to reduce dependencies and remove their own blockers. 1 Implemented a “zoning” strategy within each product; each zone is fully self-sufficient and has all of the cross-functional resources necessary to successfully execute on their charter 2 Each team in zone is responsible and accountable for their own code, tooling, automation, monitoring, on-call, etc. 3 Site Reliability Engineering (SRE) Teams replace legacy support teams (Linux, Windows, Database, etc.)
  13. 13. Build a Support Framework Benefit: Getting internal teams to follow a unified support model removes the guesswork from the process. Everybody follows the same support process and work is tracked using the same tools. Teams know exactly how to invoke support from another team if they need it and how to find out who the correct team is. Consistently associating a team identifier with everything owned by that team is an absolute must to avoid confusion! For us, 1 JIRA Project Key == 1 Support Team. 1 Adoption of the support framework is required for all teams that own code; required before anything goes into production 2 Team names & charters must be easy to find, logically categorized, and used for EVERYTHING in team’s support model (JIRA project, Slack channel, xMatters Group, Code Repos, Monitoring Alerts, etc.) 3 “You build it, you own it” 4 Work must be able to transfer seamlessly from system to system; it’s the only way to ensure things are not being dropped
  14. 14. What support looks like...
  15. 15. Stay Focused and Be Patient! Benefit: Creating a mantra and ensuring everyone is working toward a common goal is a must! Change is hard and people will be looking to leaders for guidance. Celebrate the wins often, fail quickly and learn through it all. 1 This is a disruptive change and people will be resistant , take time to refocus and ensure everyone knows what is important 2 Priorities should come from leadership; what are our organizational goals? Help people understand the “why” 3 Promote a culture of stability and excellent customer experience first, features second
  16. 16. Looking Good So Far... User NPS: +41.6% from Q4 2017 Client NPS: +51.6% from Q1 2018 Stock Price: +21.1% since December 2016 (5-year low) Uptime: 99.96% (+0.07% from 2017) Emergency Patches Deployed: -82.6% versus 2017 2018 Success Metrics
  17. 17. What’s Next? Benefit: Building “middleware” apps that front end our internal teams through Slack is a good way to enable all teams access to the vast amount of data stored in our DevOps toolset. As we continue to build additional integrations, users will be able to interact and pull information from any system that is currently integrated. Applying machine learning algorithms to historical data from all platforms in the DevOps ecosystem will allow us to virtually predict anomalous behavior! 1 Promote & Practice ChatOps 2 Continue to Integrate Tools, Remove Friction, and Enable Engineers 3 Begin the Push into leveraging AIOps
  18. 18. Key Takeaways
  19. 19. 1 2 3 4 5 Start small… the bigger you are and more processes you have, the harder this will be Change can be hard on your people… strong leadership and transparency makes all the difference Be patient, it’s called a journey for a reason! Be prepared for potentially years, not months Allow your Engineers to pick their tools, the cost is worth the benefit; enable autonomy Integrate with everything and record it all Considerations For Your Journey...
  20. 20. Thank You Becoming DevOps: athena’s Journey Through the Transformation athenahealth Christopher Parsons | Sr. Manager, NOC Operations & Engineering
  21. 21. Q&A