2. Overview
1. Overview of current data: Brief overview of
the current CSAT GAIN data
2. Using your own GAIN data: Reminder of
available GCC resources for evaluators
3. Collaborating with other sites: Review of
process for requesting cross-project data for
publications and available datasets
4. CSAT GAIN-I Data (through 9/09)
• This is a very LARGE dataset you are part of and have
access to!
• It is also important to have HIGH follow-up rates
• The goal is 100% with a minimum of 80% at each follow-
up interval
EAT FDC OJJDP SCY TCE YORP
6,405 193 179 2,312 2,034 2,250
AAFT ART ATDC ATM BIRT CYT DC
3,012 1,637 113 1,455 145 600 1,707
Total CSAT 2009
22,045
5. Follow-up rates* by program
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
AAFT ART ATDC ATM BIRT CYT DC EAT FDC OJJDP SCY TCE YORP
1+ Follow Up Rate 3-Month 6 Month 12 Month
Source: CSAT 2009 Summary Analytic Dataset
*Of those due for
that follow-up
10. Your local GAIN data
• GAIN-I, GAIN-M90, GAIN-Q, TxSI, and GRL (TTL
and FUL) - Your local raw data
• Your datasets are processed by the GCC Data
Team and “fully prepared” datasets are sent to
each site on a quarterly basis - Your analytical
data
• Each site may use its own local data for
analysis – Your Local Evaluation Plan
11. Fully prepared GAIN datasets
• GAIN-I (or GAIN-Q), GAIN-M90 (or GAIN-QM)
and TxSI
- “Analytic Datasets”
- All calculated variables, scales, and indices
- Includes all GAIN records in ABS
- All are vertical files
- Can request or create horizontal files
- Doesn’t include Follow-Up and Treatment Log
data
12. Analytic datasets: Horizontal vs. vertical file
• Vertical File (or stacked file):
- One row per observation (0, 3, 6, 9, or 12).
- When looking at comparisons or characteristics within a
point in time
- For example, comparing client sociodemographics
characteristics at baseline.
• Horizontal File (or spread file):
- One row per client containing all observations available (0,
3, 6, 9 and 12)
- When looking at comparisons or characteristics across
points in time.
- For example, assessing program outcomes such as change
in substance use over time.
- Grantees can create this dataset at anytime.
13. Analytic datasets: Types of measures
•Scale: a set of “symptoms” or items that are inter-
correlated (e.g., dependence, depression) where we
are interested in the pattern (i.e., common variance)
•Index: a set of items that may not be directly related
but add up to predict (e.g., sources of stress, barriers
to treatment, expenses)
•Ratio Estimators: one measure divided by another (e.g.,
percent of unprotected sex acts)
•Status measures: a categorical status based on a single
question or created across multiple (e.g., vocational
status, housing status)
•Survival: time to first event (e.g., time to first use)
14. Scales and Indices: Psychopathology Case Mix
General Individual Severity Scale (GISS)
Substance Problems Scale (SPS)
Substance Issues Index (SII)
Substance Abuse Index (SAI)
Substance Dependence Scale (SDS)
Internal Mental Distress Scale (IMDS)
Somatic Symptoms Index (SSI)
Depression Symptom Scale (DSS)
Homicidal/Suicidal Thoughts Scale (HSTS)
Anxiety/Fear Symptom Scale (AFSS)
Traumatic Stress Scale (TSS)
Behavior Complexity Scale (BCS)
Inattentiveness Disorder Scale (IDS)
Hyperactivity-Impulsivity Scale (HIS)
Conduct Disorder Scale (CDS)
Crime and Violence Scale (CVS)
General Conflict Tactic Scale (GCTS)
Property Crime Scale (PCS)
Interpersonal Crime Scale (ICS)
Drug Crime Scale (DCS)
* The GAIN-I has over 103 scales and indices
15. Scale and Index: Interpretative cut-points
• Definition of low, moderate and high clinical significance
bands to aid interpretation and decision making (scale
name + “g” for group)
• Useful for defining need at both the client and program
level
• Basis:
- DSM or other clinical standards where available (e.g.,
clinical is 3+/7 dependence)
- 50th & 90th percentile for common issues (e.g., days
of alcohol use)
- 0, +/- median of 1+ for zero saturated (more than half)
and right skewed variables
• Reverse-coded if “up” is low clinical significance (e.g.,
Treatment Motivation)
16. Using Characteristics and Outcomes Site Profiles
• Profiles include:
• Useful for
- Staff meetings, stakeholder meeting – fast!
- Understanding how your site compares to others.
- Determining analytical next steps.
- Program evaluation and program planning. Site Profiles
• Profiles are updated quarterly for all programs, posted
on APSS/GPSS site and e-mailed to each PI
• Located on the APSS/GPSS website
www.chestnut.org/li/APSS/ OR
www.chestnut.org/li/GPSS/ under your specific
- Demographics
- Substance use data
- Comorbidity data
- Risk data
- Treatment information
- Selected outcomes
- Individual site graphs
- Two site comparison graphs
17. FUL and TTL Reports
• Excel spreadsheet that is used to help track site
information, participants who are administered GAIN
assessments, and the treatment they've received
- Follow-up rates, tracking treatment, etc.
• Use beyond grant requirements
- Provide feedback loop to treatment program staff and
researchers
- Understand referral sources, discharges, transitions in
treatment. FUL/TxL
• Located at www.chestnut.org/li/APSS/ OR
• Located at www.chestnut.org/li/GPSS/
18. An Iceberg Analogy
• Analytic Dataset
• Site Profiles
• TxL and FU Reports
• GCC Resources to help
you with…
• Report Writing
• Local Program Evaluation
• Local Program Planning
• Detailed Analyses
• Multi-Site Collaboration
• Dissemination
20%
80%
19. FTP Common Site
* See the Evaluator Resource Sheet
ftp://data.chestnut.org/
Username: Common
Password: public
20. GAIN Evaluator Manual
• A learning tool for using GAIN data, GEM provides
information about our usual procedures and
examples from recent presentations or publications,
as well as detailed information with specific syntax.
This represents an option for analysis that is
pragmatic, robust and cost-effective, but is not
intended to be exhaustive or to represent the only
option.
• Evaluators can use the GEM to understand their GAIN
data, plan analyses, get answers to common
questions, see examples of syntax, and access key
documents that make up the numerous appendices.
• Located at ftp://data.chestnut.org
Evaluator FolderGAIN EvalManual
21. GAIN Scales and Variables File
• The “GAIN Code Book” Plus
• This Excel spreadsheet contains a listing of the major
scales and indices from the GAIN-I
- scale name, variable name, time period, GAIN-I
items used, number of items, pages, scale type
and cut points, purpose, a short description,
interpretation, references, syntax, and full text
items.
Scales and Variables File
- There is one for the GAIN-Q too
• Located at ftp://data.chestnut.org
in: Evaluator Folder/Data Summaries and Reports/
22. Adult and Adolescent Norms and Psychometrics
• GAIN Norms and Psychometric Tables
- Excel Spreadsheet
- Provides intake N, Mean, sd, alpha, severity group
percents, follow-up alpha and follow-up ICC for 3 groups
of clients (adult, young adult, adolescent).
- Includes intake N, Mean, sd, alpha for adolescents by
gender, age group and race.
- Psychometrics
• Located at ftp://data.chestnut.org
Evaluator FolderData Summaries and Reports
23. Syntax and Template Files
• SPSS syntax and information to help export
and prepare local data
- Create scales and indices (making a fully
prepared “analytic” dataset if not a grantee)
- Create horizontal file for analysis (cases to
vars)
- Other resources
• Located on the FTP site at: ftp://
data.chestnut.org/ under the evaluator
folder
24. Analytic Training Memos
• Series of documents created to help researchers
overcome specific problems associated with the
collection and analysis of data generated during
substance abuse treatment research.
• Evaluators can use these memos to conduct analyses
in the same manner as the GCC and/or to report on
data handling for an article or presentation
• They are available free for download
• Located at ftp://data.chestnut.org
Evaluator FolderTrainingAnalytic Training Series
25. GAIN Crosswalk
• A version-specific listing of each GAIN item or scale with
information about its status as a required or optional item
and whether optional items are treatment related or
recommended; and information regarding the purpose
for/source of the item and any comments regarding its use.
• Evaluators can use this crosswalk to determine item
comparability across study (optional items have lower
available Ns) and reasons for specific items.
• Helps in selecting which version of the GAIN to use. GAIN
Crosswalk 5.5
• Located at ftp://data.chestnut.org
in: Evaluator Folder/Data Summaries and Reports/
27. Collaborating with other sites
• Discuss your idea with the PI from the site(s)
you are interested in collaborating with or
using data from
• Sign Data Sharing Agreements
• Decide on the scope of the collaboration
- Informal Process: few sites, clear analysis plan,
clear roles, etc.
- Formal Process: many sites, sites want
analytical or methodological assistance form
Chestnut, need formal discussions/ permissions
among PIs, etc.
28. Collaborating with other sites
Acknowledge and Inform
Analyze and Disseminate
De-Identified Dataset Provided (GCC)
Sign Data Sharing Agreements
Seek Permissions
Review for Feasibility (GCC)
Develop the Abstract
Review GCC Resources OR “Do the Homework”
Develop an Idea to run with
29. Main components of an abstract
• Title
• Lead author
• Other (potential) authors
• Proposed forum(s) (journal or conference)
• Target Dates
• Data sources (what data set, data and/or time
periods)
• Objectives or questions to be addressed
• Methods/Design/Main analyses
• Variables to be created
• Relevance to field
30. Resources for Collaboration with other Sites
• Multisite Collaboration guidelines
- An overview of the annual datasets prepared by Chestnut Health
Systems including their purpose, inclusion rules and
expectations
- Located on the FTP site at: ftp://data.chestnut.org/ under the
Data Management folder/Data Management Required Reading
Guidelines
• Data request topic summary
- A list of other evaluators and researchers who have requested to
use GAIN data, including their name, contact information,
abstract title, and key topics.
- Located on the FTP site at: ftp://data.chestnut.org/ under the
evaluator folder/requesting and Using GAIN data
• Acknowledgement assistance in LI memos
• CSAT Publication Policy for your Initiative
31. Resources for Collaboration with other Sites
• Annual Summary Analytic Slides
- Four sets of Power Point slides using summary analytic data
for all records, adolescent sites, justice sites, or adolescents
(12-17). Includes characteristics, placement, treatment, and
outcomes (NOMS) by gender, age group, race, treatment type
and program.
- Located on the FTP site at: ftp://data.chestnut.org/ under
the evaluator folderData Summaries and Reports
• GAIN Publications
- A list of published articles and presentations that used GAIN
data in their analyses. The actual presentations and
publications are available as well.
- Located on the FTP site at: ftp://data.chestnut.org/ under
the evaluator folderGAIN Publications and Bibliographies
• Data Sharing Agreements
- Located at the GAIN Website: ww.chestnut.orgligain
32. Other ways to get help
• Use our e-mail support lines for questions with:
- general issues: gaininfo@chestnut.org
- evaluation/analysis/publication: gaineval@chestnut.org
- data submission/Site Profiles: datasubmit@chestnut.org
- Site Profiles Live Meeting: email your GAIN Project
Coordinator to set up a meeting
- administration/certification: gainsupport@chestnut.org
- software/web application: abssupport@chestnut.org
• This presentation is available on your flash drive!
Editor's Notes
Read briefly. This slide is used to provide an overview of the presentation.
How to Write GAIN A-QA Feedback Like It’s Your Job
The purpose of this slide is to provide grantees with an overview of the entire CSAT dataset and to understand how their data fits within a larger dataset.
More context for grantees. Importance of high FU rates. 3 or 6 month follow-up versus 3 and 6 month. We tend to think 3 or 6 is enough (meeting grant requirements), but 3 and 6 is what you have to deal with once you start analyzing longitudinal data. Also, some things to note (maybe mention) -OJJDP hasd only 20 clients with 3m data -9 month was not required for any studies funded after CYT - that's why is is not presented -12 month was not required for any CJ studies (YORP/DC) - so their rates are much lower
More context for grantees. Importance of high FU rates. Pay attention to the difference between orange and yellow lines. We tend to think the orange line is enough (meeting grant requirements), but the yellow line is what you have to deal with once you start analyzing longitudinal data. Barb’s notes just for kicks: Details for a Lost to follow-up discussion (from Kristman, Manno, and Cote. “Lost to follow-up in cohort studies: How much is too much?”; European Journal of Epidemiology,, 19:751-760, 2004) Types of Lost to follow-up: MCAR-Missing Completely At Random MAR-Missing At Random MNAR-Missing Not At Random MCAR: The probability that a subject remains in the study does not depend on the treatment exposure, confounders or the outcome. That is data from the remaining subjects are assumed to be a random sub-sample of the original study sample. Lost to follow-up MCAR should not bias the measures of association being studied and could be ignored. Only the power to detect the associations is compromised. For example a cohort study seeks to examine the effect of alcohol use on the incidence of mild traumatic brain injury (MTBI), controlling for smoking status. Lost to follow-up MCAR would not be related to smoking status, alcohol use or MTBI. MAR: The probability of a subject remaining in the study depends on the exposure or counfounders but not the outcomes. In this case subject loss is related to the variables observed at baseline or at follow-up. So in the MTBI example, loss to follow-up MAR would mean that dropouts are related to either alcohol use or smoking status at baseline or follow-up but not to MTBI. This is also considered ignorable because data that is collected (alcohol use, smoking status and other variables) can help explain the potential bias by controlling for the covariates that are associated with loss to follow-up in a multivariate analysis. MNAR: If the subject loss cannot be explained by existing data the lost data are considered MNAR or “non-ignorable”. Under this assumption the probability of being lost to follow-up depends on the outcome to be measured and cannot be completely explained by covariates. MNAR in the MTBI example, means drop out is dependent on MTBI – perhaps those with MTBI had difficulty participating in the study for any number of reasons. Although there are methods to distinguish between MCAR and MAR dropouts, there are no methods to differentiate MNAR data. The literature suggests that MNAR is the most likely reason for lost to follow-up in cohort studies because the subjects who drop out tend to have different outcomes than those who remain in the study. As the percentage of loss to follow-up increases under the MNAR mechanism the mean Odds Ratio (which is a measure of association that is more robust to non-randomly missing data than the relative risk or risk difference used in cohort studies) loses precision. With 20% of the data lost to follow-up the OR is under represented by half of its value. The precision of the estimate is directly related to the percentage of lost to follow-up as the SD narrow around the mean OR with increasing lost to follow-up. The 95% Confidence Interval coverage was impacted by the proportion of lost to follow-up. With 20% MNAR, the CI falls to 81.5% containing the true value OR in the author’s simulation studies. In addition, increasing subject loss MNAR leads to narrower variance estimates around the mean OR. This resulted from the imposed selection bias due to subject loss, which created a group of observations with similar characteristics. Therefore the narrower variance estimates can provide a false sense of precision around an already biased estimate. This combination of bias estimate and a tight interval leads to poor CI coverage and demonstrates the potential danger of relying on effect sizes obtained from studies with high loss to follow-up rates.
How to Write GAIN A-QA Feedback Like It’s Your Job
Ask audience the question to elicit responses and then reveal the graphic. OK, but where do I start??
Overview of the types of resources and tools available to evaluators. Read each. Will discuss in turn.
The first (and most obvious) resource is the raw data generated by conducting assessments.
So what does “fully prepared” mean? What files will I get from Chestnut and what are their purpose?
Basic differences in vertical and horizontal datasets. Ask audience for exampled from their own analysis or experience. Scale and index version differences are accounted for if possible, but where new items are added (e.g. medication question for MH treatment) the old version is calculated for all records so cross site comparison can still be done (MHTI3 v MHTI4). RFQ is another one where items were added (RFQ26 vs RFQ33)
Briefly describe the different types of measures. We are going to see more of this when the Scales and Variables file is described.
This diagram illustrates the organization of the main scales and indices in the GAIN. The overarching scale is the General Individual Severity Scale (GISS). Higher values indicate more severe problems across several life domains including substance use, mental health, emotional health, physical health, illegal activities. GISS contains 4 subscales. Substance Problems Scale (SPS): is a count of symptoms of substance abuse, dependence, and substance induced health and psychological disorders based on the DSM-IV. Subscales are the Substance Issues Index (SII), the Substance Abuse Index (SAI), and the Substance Use Disorders Scale (SUDS). Versions of SPS exist for lifetime, past year and past month. Internal Mental Distress Scale (IMDS): Count of past-year symptoms related to internalizing disorders including somatic, anxiety, depression, traumatic stress and suicide/homicide thoughts. Subscales are the Somatic Symptom Index (SSI), the Depressive Symptom Scale (DSS), the Homicidal Suicidal Thought Scale (HSTS), the Anxiety/Fear Symptom Scale (AFSS), and the Traumatic Distress Scale (TDS). Behavior Complexity Scale (BCS): Count of past-year symptoms related to externalizing disorders including attention deficit, hyperactivity/impulsivity, and conduct disorder based on the DSM-IV (APA, 1994; 2000). Subscales are the Conduct Disorder Scale (CDS) and the Attention Deficit Hyperactivity Disorder Scale (ADHDS) which consists of the Inattentive Disorder Scale (IDS) and Hyperactivity-Impulsivity Scale (HIS). Crime and Violence Scale (CVS): A count of strategies used in conflict and different types of illegal activities endorsed by the respondent. Subscales include: GCTS, PCS, ICS, DCS, and GCS.
Basic definition of the clinical cut points and the basis for them. More on this when the Scales and Variables file is described. Check in with the audience. This is the end of the discussion about their data. Now we move into the real resources and tools available to evaluators. Sub-bullet #3: If more than 50% are zero responses, the cut points are 0=lo, 1 to the median value between 1 and the highest value=1, and that median through the highest value=2. That is different than for non-zero-saturated where it is 0 to the median=0, median to 90th %ile=1, 90th %ile through hi=2.
Discuss what Site profiles contain and how they are useful. Then show an example.
Explain what these are. Grantees (evaluators) are very used to these reports because they are held to follow-up rates, and Chestnut reviews them frequently across all grants. However, grantees can use this information in their evaluation in several ways. Feedback loop to program staff (don’t forget to involve everyone in the data dissemination process). Discuss problems/successes with follow-up, referral sources, LOS, discharges. Are these optimal, expected? Use the data to improve programming.
The FTP can be found using your web browser (Internet Explorer). Use the address: ftp://data.chestnut.org/ Talk about the things that are listed here (use browser to go to the site if possible): Evaluation folder Data Management Folder Data Coordinator Folder
Take your time on this slide – the Excel spreadsheet is PACKED with information. Describe the scales and variables file then click on the link and demonstrate the spreadsheet. Explain the tabs at the bottom Go to the scales list tab and explain each column on the sheet Show different scale types, cut points, time period, syntax, items by showing examples such as Substance Frequency Scale – classical scale, change, past 90 days, etc Sex Protection Ratio – ratio, change, change, past 90 days, etc Recovery – status, change, past month, etc Treatment Motivation Index – summative index, change, current, etc
Read the slide to describe what the Norms and Psychometric tables are. Then show the Norms and/or Psychometrics table as a sample. The point is to familiarize grantees with this information and let them know where to get it NOT to describe every detail in the table. Recommend Norms as it is the most recent. Norms is the most recent (2008).
Read the slide. Really this is used if you are making your own datasets – especially cases to vars for horizontal files. Grantees should not need this very much otherwise. Cases to vars will append the variable name for follow-ups with _3, _6, etc.
Explain what these are and why they are helpful. Go to the pages if possible otherwise there is a sample on the next slide. Very helpful!!!! Even if you are not having difficulty, these are important to look over and read.
Take your time on this slide too – there is a lot of information on the Excel spreadsheet that is very useful. Column C-D, Q – Datasets Column J-N Grantee inclusion (Core, required, NA) Column P GPRA Column R-AA Use Column AB Comments
How to Write GAIN A-QA Feedback Like It’s Your Job
Steps to accessing Multi-site data Step 1 is a courtesy step that I (BEstrada) included.
List out the contents of the abstract. This is a non-trivial step and must be well thought out. Remember: If you are comparing to newer studies or across GAIN versions, be aware of version differences in scales and indices
List out the contents of the abstract. This is a non-trivial step and must be well thought out. Remember: If you are comparing to newer studies or across GAIN versions, be aware of version differences in scales and indices
List out the contents of the abstract. This is a non-trivial step and must be well thought out. Remember: If you are comparing to newer studies or across GAIN versions, be aware of version differences in scales and indices
