Selecting An Intervention In A Complex Adaptive Health Care System & Evaluating Its Impact Malaz Boustani, MD, MPH Indianapolis Discovery Network for Dementia Indiana University center for Aging Research Regenstrief Institute, Inc www.indydiscoverynetwork.com

The US Health Care System Report Card Sub-optimal quality Dementia recognition:< 40% FDA approved Trx: < 10% Inappropriate Trx: > 25% Compromised safety 90,000 death / yr 1,000,000 injury / yr Waste 30% - 40% health expenditure (overuse, underuse, misuse, duplication) Discovery to Delivery: 17 yrs $800 millions Generalizability of Research Findings: < 1% of real world patients enrolled in AD studies Boustani, 2007; IOM, 2001, Westfall, 2007; Faison et al, 2003;

Sub-optimal quality

Dementia recognition:< 40%

FDA approved Trx: < 10%

Inappropriate Trx: > 25%

Compromised safety

90,000 death / yr

1,000,000 injury / yr

Waste

30% - 40% health expenditure (overuse, underuse, misuse, duplication)

Discovery to Delivery:

17 yrs

$800 millions

Generalizability of Research Findings:

< 1% of real world patients enrolled in AD studies

Independent Agent E Other Local CASs Stress Feedback Independent Agent D Independent Agent C Independent Agent B Independent Agent A Changing Environment Connection The health care CAS Emergent Performance

Complex Adaptive Health Care System An open, dynamic, flexible, adaptive, and complex network Complex due to Numerous interconnected, semi-autonomous, competing, and collaborating members Adaptive due to Its capability of learning from its prior experience Its flexibility to change its members connecting patterns to fit better with its surrounding environment Holden, 2005; Litaker et al. 2006 Feb.

An open, dynamic, flexible, adaptive, and complex network

Complex due to

Numerous interconnected, semi-autonomous, competing, and collaborating members

Adaptive due to

Its capability of learning from its prior experience

Its flexibility to change its members connecting patterns to fit better with its surrounding environment

Complex Adaptive Health Care System Emergent behaviors NOT predetermined ones Self-organized controls NOT central controls Holden, 2005; Litaker et al. 2006 Feb.

Emergent behaviors

NOT predetermined ones

Self-organized controls

NOT central controls

The Paradox of CAS Each CAS is unique in its Members diversity Members interactions Surrounding environment Previous history Evolving and learning process. Health care research centers generate scientific data from group-based studies Then regulators/guideline developers ask each unique CAS to implement the findings. Holden, 2005; Litaker et al. 2006 Feb; Stoebel et al, 2005.

Each CAS is unique in its

Members diversity

Members interactions

Surrounding environment

Previous history

Evolving and learning process.

Health care research centers generate scientific data from group-based studies

Then regulators/guideline developers ask each unique CAS to implement the findings.

The Paradox of CAS The performance of a CAS fluctuates over time The connecting patterns of its members are dynamic and change over time in respond to the stress or the fitness requirements of its surrounding environment. Holden, 2005; Litaker et al. 2006 Feb; Stoebel et al, 2005.

The performance of a CAS fluctuates over time

The connecting patterns of its members are dynamic and change over time in respond to the stress or the fitness requirements of its surrounding environment.

How Can I Assess CAUSALITY? Can I Predict? How can I identify or determine that a system is a CAS? How can a system with no bosses, no mandates and no plans produce repeatedly complicated structures and perform difficult tasks? Can I build a complex adaptive system model to study? What can I do with a simulator? What data do I have & how can I analyze data in CAS? How can I manage the problems of uncertainty, variability, and variable interdependency in a CAS? Is replication essential in science?

How can I identify or determine that a system is a CAS?

How can a system with no bosses, no mandates and no plans produce repeatedly complicated structures and perform difficult tasks?

Can I build a complex adaptive system model to study? What can I do with a simulator?

What data do I have & how can I analyze data in CAS?

How can I manage the problems of uncertainty, variability, and variable interdependency in a CAS?

Is replication essential in science?

How can I select a change in a CAS? How can I evaluate the success or the impact of my proposed intervention or change in a CAS? How are experiments performed in a complex adaptive system? Is a randomized controlled trial the best accurate study design to evaluate an intervention in a CAHS? How Can I Assess CAUSALITY? Can I Predict?

How can I select a change in a CAS?

How can I evaluate the success or the impact of my proposed intervention or change in a CAS?

How are experiments performed in a complex adaptive system?

Is a randomized controlled trial the best accurate study design to evaluate an intervention in a CAHS?

Selecting an Intervention

The 9 Principles of Selecting a Change in a Complex Health Care System View your system through the lens of complexity. Build a good enough vision with minimum specifications Lead from the edge by balancing data and intuition, planning and acting, safety and risk. Foster the &quot;right&quot; degree of information flow, diversity and difference, connection among agents, power differential, and anxiety. Uncover and work with paradox and tension. Paul E Plsek. 1997.

View your system through the lens of complexity.

Build a good enough vision with minimum specifications

Lead from the edge by balancing data and intuition, planning and acting, safety and risk.

Foster the &quot;right&quot; degree of information flow, diversity and difference, connection among agents, power differential, and anxiety.

Uncover and work with paradox and tension.

The 9 Principles of Selecting a Change in a Complex Health Care System Go for multiple actions at the fringes, let direction arise. Listen to the informal relationships, gossip, rumor, and hallway conversations that contribute significantly to the individuals’ perceptions about their surrounding environment and their subsequent actions. Allow complex systems to emerge out of the links among simple systems that work well and are capable of operating independently. Build a community of members who collaborate, create, learn and compete Paul E Plsek. 1997.

Go for multiple actions at the fringes, let direction arise.

Listen to the informal relationships, gossip, rumor, and hallway conversations that contribute significantly to the individuals’ perceptions about their surrounding environment and their subsequent actions.

Allow complex systems to emerge out of the links among simple systems that work well and are capable of operating independently.

Build a community of members who collaborate, create, learn and compete

Reflective Adaptive Process Build a local “Think-Tank: the RAP Team” responsible of introducing an acceptable, locally matched, flexible and effective change in its CAS. External or internal facilitators who encourage the RAP Team to select, adopt or create local processes to solve the CAS problem enhance the CAS performance guide the CAS respond to its surrounding environment RAP is the second generation of CQI Stroebel et al.2005.

Build a local “Think-Tank: the RAP Team” responsible of introducing an acceptable, locally matched, flexible and effective change in its CAS.

External or internal facilitators who encourage the RAP Team to select, adopt or create local processes to

solve the CAS problem

enhance the CAS performance

guide the CAS respond to its surrounding environment

RAP is the second generation of CQI

The RAP Principles Vision, mission, and shared values are fundamental in guiding ongoing change processes in a CAS Creating time and space for learning and reflection is necessary for a CAS to adapt to and plan change Tension and discomfort are essential and normal during CAS change Improvement teams should include a variety of system’s agents with different perspectives of the system and its environment System change requires supportive leadership that is actively involved in the change process, ensuring full participation from all members and protecting time for reflection McDaniel et al, 2003; Stroebel et al.2005; Boustani et al 2007.

Vision, mission, and shared values are fundamental in guiding ongoing change processes in a CAS

Creating time and space for learning and reflection is necessary for a CAS to adapt to and plan change

Tension and discomfort are essential and normal during CAS change

Improvement teams should include a variety of system’s agents with different perspectives of the system and its environment

System change requires supportive leadership that is actively involved in the change process, ensuring full participation from all members and protecting time for reflection

Evaluation of Interventions in CAS Selecting the content and the delivery of the intervention A- The content based on SER of past research or guidelines B- Develop a RAP team to Localize the content Localize / Invent the delivery process Monitor the delivery process Monitor the agents interactions Detect emergent behaviors

Selecting the content and the delivery of the intervention

A- The content based on SER of past research or guidelines

B- Develop a RAP team to

Localize the content

Localize / Invent the delivery process

Monitor the delivery process

Monitor the agents interactions

Detect emergent behaviors

Evaluating the Impact of the Selected Interventions

Methodological Quality “ Extent to which a study’s design, conduct, and analysis has minimized selection, measurement , and confounding biases” West et al, 2002

The “Safeguards” Evidence Random assignment against both unconscious and deliberate human influence on the assignment of subjects to different groups unknown confounders Blind outcome assessment ensures treatment and analysis of outcomes are not colored by prejudice Guyatt et al, 1993; West et al, 2002

Random assignment against

both unconscious and deliberate human influence on the assignment of subjects to different groups

unknown confounders

Blind outcome assessment ensures

treatment and analysis of outcomes are not colored by prejudice

The Hierarchical Strength of Evidence in Treatment Decisions N-of-1 RCT SER of RCT Single RCT SER of observational studies Single observational study Physiologic studies Unsystematic clinical observations Guyatt et al, 1993; West et al, 2002

N-of-1 RCT

SER of RCT

Single RCT

SER of observational studies

Single observational study

Physiologic studies

Unsystematic clinical observations

Design: N-OF-1 RCT System outcomes Overall Cost Overall Safety Overall Quality Overall Patient Satisfaction Overall Staff Satisfaction Intervention specific outcomes Proposed Evaluation Process

Design:

N-OF-1 RCT

System outcomes

Overall Cost

Overall Safety

Overall Quality

Overall Patient Satisfaction

Overall Staff Satisfaction

Intervention specific outcomes

Applying N-OF-1 RCT in CAS A CAS receives pairs of evaluation periods Each pair including one period of active intervention and another of usual care in random order. CAS members are kept masked to allocation Outcomes are monitored blindly and continuously

A CAS receives pairs of evaluation periods

Each pair including one period of active intervention and another of usual care in random order.

CAS members are kept masked to allocation

Outcomes are monitored blindly and continuously

N-OF-1 RCT in CAS 0----------1------------2-----------3-----------4----------5-----------6------------7-----------8-------------9-----------10------------11------------12 Time Outcome Data Outcome Data Outcome Data Outcome Data Outcome Data Outcome Data Outcome Data R to Int R to Int R to Int R to UC R to UC R to UC

Issues of –OF-1 RCT Selecting the Intervention: Is there enough data to suggest the efficacy of the intervention? Did the RAP team modify the intervention delivery and content to match the local resources? Determining the rapidity of onset of the intervention effect: How quickly does the intervention begin to act and cease acting? Optimizing dose: What is the best dose? Does it differ between systems? Guyatt et al, 1986; Price & Evans, 2002

Selecting the Intervention:

Is there enough data to suggest the efficacy of the intervention?

Did the RAP team modify the intervention delivery and content to match the local resources?

Determining the rapidity of onset of the intervention effect:

How quickly does the intervention begin to act and cease acting?

Optimizing dose:

What is the best dose?

Does it differ between systems?

Issues of –OF-1 RCT 4. Selecting the outcomes: the overall common system outcomes not related specifically to the intervention. the outcomes that the intervention effect directly. the process that mediates the effect of the interventions such as the relationship and the new connection between the CAS members. 5. Interpreting the Results of N-of-1 RCTs: What a priori criteria should be established for classifying a trial as definitely negative or indefinite? Guyatt et al, 1986; Price & Evans, 2002

4. Selecting the outcomes:

the overall common system outcomes not related specifically to the intervention.

the outcomes that the intervention effect directly.

the process that mediates the effect of the interventions such as the relationship and the new connection between the CAS members.

5. Interpreting the Results of N-of-1 RCTs:

What a priori criteria should be established for classifying a trial as definitely negative or indefinite?

The Requirement for N-of-1 RCT: The problem that requires intervention needs to be chronic. The problem that requires intervention needs to be stable. The effect of the new intervention begins to acts within days to weeks at most. The effect of the new change ceases acting after discontinuing within days to weeks at most. Guyatt et al, 1986; Price & Evans, 2002

The problem that requires intervention needs to be chronic.

The problem that requires intervention needs to be stable.

The effect of the new intervention begins to acts within days to weeks at most.

The effect of the new change ceases acting after discontinuing within days to weeks at most.

The Science of CAS “ A general feeling of unease when contemplating complex systems dynamics. Its devotees are practicing fact-free science. A fact for them is, at best, the outcome of a computer simulation; it is rarely a fact about the world.” (Smith 1995, p. 30).

“ A general feeling of unease when contemplating complex systems dynamics. Its devotees are practicing fact-free science. A fact for them is, at best, the outcome of a computer simulation; it is rarely a fact about the world.”

(Smith 1995, p. 30).

The Science of CAS “ Prediction is our best means of distinguishing science from pseudo-science” Karl Popper “ The history of 20th-century science should also give complexologists pause. Complexity is simply the latest in a long line of highly mathematical ‘theories of almost everything’ that have gripped the imaginations of scientists in this century” (Horgan 1995, p. 104).

“ Prediction is our best means of distinguishing science from pseudo-science”

Karl Popper

“ The history of 20th-century science should also give complexologists pause. Complexity is simply the latest in a long line of highly mathematical ‘theories of almost everything’ that have gripped the imaginations of scientists in this century”

(Horgan 1995, p. 104).