MedImpact’s Journey to Database Deployment Automation

Copyright © 2019 MedImpact Healthcare Systems, Inc. All rights reserved.
Confidential, Proprietary and Possible Trade Secret of MedImpact 1
The contents of this presentation are confidential and proprietary to MedImpact Healthcare Systems, Inc. and may contain material MedImpact considers
Trade Secrets. This presentation may not be reproduced, transmitted, published, or disclosed to others without MedImpact's prior written authorization.
Our Journey to Database Deployment Automation
MedImpact Healthcare Systems

Confidential, Proprietary and Possible Trade Secret of MedImpact 2Copyright © 2019 MedImpact Healthcare Systems, Inc. All rights reserved.
Mabuti Ng’andu
Director IT, Data Management, Database & Middleware
mabuti.ngandu@medimpact.com
As Director IT at MedImpact Healthcare Systems,
Mabuti Ng’andu is responsible for the installation,
configuration, availability and operational management
of key database and middleware technologies. He is also
responsible for the delivery of software into databases,
and related release automation capabilities.
Starting his career as an Oracle developer, Mabuti
transitioned into database administration and solutions
architecture before finally taking on leadership roles.
He has worked for a variety of private and public
organizations, small startups and fortune 100
companies including Abbott Laboratories where he
led database and infrastructure teams.
Mabuti holds a bachelors degree in Computer Information
Systems from La Sierra University.

AGENDA
1. Overview
2. Company Introduction
3. Deployment Challenges
4. Database Deployment Automation Initiative
5. The Implementation Process
6. Key Takeaways
7. Questions

OVERVIEW
Many – if not most – organizations have treated the deployment of code to databases
differently than they do other code stacks. This is in part because few organizations
have dedicated “Database Developers”, as they do java, python, swift or other
‘specialized’ software engineers.
It is also because databases have not traditionally been easily amenable to the standard
software deployment processes or toolsets. These differences can lead to occasional
deployment mishaps that can result in slow deployments or costly mistakes that are
difficult to roll back.
Ironically, making the case to justify an investment in database deployment automation
is not necessarily an easy ask because these mishaps, though costly, are viewed simply
as process failures due to resource or team “discipline” challenges. The truth is much
deeper than that.
This presentation seeks to describe MedImpact’s journey to improve database
deployment throughput and quality – a journey that required people, process and
tooling changes. It will also describe the justification process that finally enabled the
investment, and the improvements we have experienced since.

Who We Are

MedImpact is a Pharmacy Benefit Management (PBM) company founded in 1989 Headquartered in San
Diego, California
• PBMs reduce prescription drug costs and improve convenience and safety for consumers, employers,
unions, and government programs
Comprehensive PBM Services
• Claim adjudication
• Retail network management
• Mail-order pharmacy services
• Specialty pharmacy management
• Clinical pipeline monitoring/budget impact modeling
• Formulary management
• Utilization management
• Rebate management
• Fraud, waste and abuse management
• Clinical program management
Additional Services: 340B management, discount cash card

• MedImpact Offers PBM Solutions Focused on Low-Net Cost & Quality
Member Care
• As the nation’s largest independent, trend-focused pharmacy benefit
manager (PBM), MedImpact provides services for more than 50 million
lives including a robust cash card program. We work with health plans, self-
funded employers and government entities to help reduce prescription
drug costs, improve member health and increase medication adherence
and safety
• Our business model is unique. MedImpact offers all the services you need
from a PBM – with different goals for different outcomes than other PBMs.
Our independent business model, prioritization of member experience and
inherent flexibility allows us to align with plan sponsors in ways that other
PBMs and their misaligned incentives cannot. Our industry-differentiating
goal: Containing costs and driving value with quality member care while
reducing overall healthcare costs.

• MedImpact Our value proposition dictates unique and agile views of data
to support the reporting and analytics behind our service offerings
• Data objects need to be created and/or modified quickly but with reliable
quality
• We offer availability and quality-related performance guarantees to our
customers
• We are in the process of adopting agile software development
methodologies so as to be even more responsive to market needs
• All of the above mandate reliable data change management practices that
minimize or eliminate errors

Deployment Challenges

Database Deployment Process Challenges 1/2
Database deployments were not treated with the same rigor that
software deployments were
• Although source code was in GIT, the traceability of it from checkout to
deployment was limited due to manual processes
• Database deployment components were not part of the core build
package
• In fact, database changes were at times deployed days or even weeks
before the software deployments

Database Deployment Process Challenges 2/2
Deployment processes were mostly manual, save for some scripted
verifications/validations. This resulted in:
• Long, labor-intensive deployment processes (not agile)
• Inconsistent deployments across the enterprise
• Incomplete deployments within databases
• Limited ability to audit when changes occurred in a given environment
• Limited visibility into, and ability to review deployment failure points
• Labor intensive manual validation/verification of all of the above

Database Deployment Team Challenges
Three sets of teams had the ability to “deploy” DML or DDL. Each team
had their own set of deployment processes
• DBA’s deployed all DDL with exception of application-specific PL/SQL
• Software Engineering teams deployed application-specific DDL via
configuration management processes
• DBAs, Software Engineering teams or even some business teams could
deploy DML through manually controlled configuration management
processes
This variation of deployment processes limited visibility into and
traceability of deployments

Database Deployment Outcome Challenges
We employ rigorous manual verifications (unit, QA, test & validation, UAT)
Manual verifications are rigorous, but in the few instances when they fail,
additional challenges are introduced including:
• Long, arduous debugging processes when deployments fail for DBA,
Dev, QA and testing & validation (business) teams
• Unexpected post-deployment incidents/problems in QA, UAT and
production environments
• Longer software delivery timelines (not agile)
• More rework

The Database Deployment Automation Initiative

Database Deployment Automation
The primary objectives of the Database Deployment Automation initiative
were to:
1. Create a process that provided a consistent framework for all database
deployments including DDL, PL/SQL and DML to all non-COTS Oracle
databases using the same rigor for DB deployments that was expected in
software deployments
2. Identify and acquire a tool that would streamline database deployments
by:
• Enforcing automated deployments from source code repository (GIT) and
ability to audit this
• Deploying predictably, repeatably and consistently to individual or groups
targets (one QA database, or all QA databases, etc.)
• Logging all deployment script outcomes for immediate or future review
• Having the ability to integrate with standard DevOps tools including the
Atlassian stack

Executive Justification 1/3
First-round “naïve” justification described this potential value:
1. Improved deployment quality
2. More agile turnaround
Questions were asked as to why value couldn’t be achieved with more
efficient processes and better ‘people management’. Things to note:
1. You can only manage ‘fatigue’ to a point
2. Manual processes must by definition be expected to include a certain %
of error rate. “To err is human!”
3. Adding people to a manual process doesn’t necessarily make it faster

Second-round justification described this potential value:
1. Improved deployment quality
2. More agile turnaround
3. Reduction in deployment costs
4. Reduction in rework and associated costs and schedule impact
5. Seamless integration with planned CI/CD pipelines

Our second justification summary included the following components:
1. An opportunity cost calculation including:
1. Labor cost of “slow” deployments through all environments (DEV, QA, PRD)
2. Labor cost of manual verification
3. Cost of deployment-related rework
2. A roadmap to achieving automated deployment value, including:
a. Tool evaluation and selection process
b. Pilot (POV) of tool to prove it could deliver on said value
c. Alignment on success criteria
d. Reference calls with enterprises who have implemented selected tool

The Implementation Process

The Implementation Process
1. We evaluated four tools and aligned that Datical was the best fit for our
needs
2. We executed a Proof of Value, meeting all success criteria. All executive
leaders agreed to make the investment
3. We piloted implementation on 2 applications, so as to document the
implementation process
4. We modified our deployment processes and trained people
5. We are currently iterating through other applications
6. We are currently integrating Datical DB deployments with our broader CD
ecosystem
7. We are currently piloting the CI/CD automation with one team, after
which we will iterate through other applications

Confidential, Proprietary and Possible Trade Secret of MedImpact 21Copyright © 2019 MedImpact Healthcare Systems, Inc. All rights reserved.
Key Takeaways
Initial Investment
1. Be very clear on what your objectives are
2. Invest time in articulating the value that automated database deployments
provide. It may be “obvious” to the technical team, but it must still be well
articulated to executive leaders
Implementation
1. If you are not already actively agile, plan time to work through process
questions (e.g. Who owns deployments in which environments? What is the
process to trigger a deployment? Who needs to approve it? etc.)
2. Solicit input from and stay well aligned with all deployment stakeholders –
DevOps, software engineering, database administration/data management,
configuration management, release management, etc.
3. Never forget that this is a CHANGE MANAGEMENT activity. Identify the
advocates and utilize them to drive change. Communicate 7 times, 7 ways.
Prioritize those who understand the vision, and the rest will follow success

The contents of this presentation are confidential and proprietary to MedImpact Healthcare Systems, Inc. and may contain material MedImpact considers
Trade Secrets. This presentation may not be reproduced, transmitted, published, or disclosed to others without MedImpact's prior written authorization.
Thank You!

MedImpact’s Journey to Database Deployment Automation

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to MedImpact’s Journey to Database Deployment Automation

Similar to MedImpact’s Journey to Database Deployment Automation (20)

More from DevOps.com

More from DevOps.com (20)

Recently uploaded

Recently uploaded (20)

MedImpact’s Journey to Database Deployment Automation