The NCQA Baseline Assessment Tool (BAT) aids in ensuring the reliability of health plan information systems for quality measure generation and compliance audits associated with HEDIS/QARR. The document details how information production process metadata establishes a common understanding among stakeholders, fostering process improvement, quality management, and accountability in information stewardship. It emphasizes the significance of measuring data quality in terms of timeliness, completeness, and accuracy while implementing continuous improvements in information production and reporting processes.

1. The NCQA Baseline
Assessment Tool
. . . and How Sharing Information
Production Process Metadata
Fosters Process Improvement

2. The BAT and Information
Production Process Metadata
• What is the Baseline Assessment Tool?
• What is this Metadata all about?
How it relates to:
– Quality Management and Process Improvement
– Information Production Processes
– HEDIS/QARR
• Process Improvement Details Useful for
the BAT

3. What is the BAT?

4. The NCQA Baseline
Assessment Tool
• The BAT collects information that helps
the NCQA assure that health plan
information systems can generate reliable
quality measures for HEDIS/QARR
• Supports the NCQA's HEDIS Compliance
Audit, an audit for compliance with HEDIS
specifications for measures.

5. Sections of the BAT
• Data Processing Systems:
– Encounters
– Members
– Providers
• Completeness of Data
• Integration of Data for Reporting
• Controlling Integrity of Reporting
• Medical Record Review
• Other / General Information

6. Quality Assurance Reporting
Requirements (QARR)
Childhood Immunization Status Annual Dental Visit
Lead Testing Frequency of Ongoing Prenatal Care
Use of Appropriate Medications for People with
Asthma Children's Access to Primary Care Practitioners
Antidepressant Medication Management Adults' Access to Preventive/Ambulatory Health
Breast Cancer Screening Services
Cervical Cancer Screening Ambulatory Care
Comprehensive Diabetes Care Inpatient Utilization
Controlling High Blood Pressure Births and Average Length of Stay, Newborns
Follow-up After Hospitalization for Mental Illness Discharges and Average Length of Stay-Maternity
Well Child and Preventive Care Visits in the 1st 15 Care
Months of Life Mental Health Utilization-Inpatient
Well Child and Preventive Care Visits - Children Chemical Dependency Utilization-Inpatient
Ages 3-6 Years Frequency of Selected Procedures
Adolescent Well-Care and Preventive Visits
Timeliness of Prenatal Care Practitioner Turnover
Postpartum Care Enrollment by County
Board Certification/Residency Completion

7. Generating QARR Measures
• Extract random samples of
member populations relevant to
each measure
• Calculate rates of those showing
compliance with requirements
for each measure
• Includes visiting primary care
sites, physically reviewing
medical records, recording
compliant instances

8. What is this Metadata all About?
• Metadata Categories
• Basic Metadata Purposes
• What Quality Management Does
• Overview of Relevant Processes
• Quality Approach to Information

9. Metadata Categories
• Production Process Factors
– People
– Process
– Tools
– Materials
• Quality Measures
– Timeliness
– Completeness
– Accuracy
• Barriers to Quality (Root Causes)
• Improvement Processes
– Quality Assessment
– Cleanup and Transfer
– Process Improvement

10. Basic Metadata Purposes:
• Create common understanding
• Foster process improvement
• Enable information stewardship

11. Metadata Creates
Common Understanding
Between:
• Information Producers:
– Business Process: data acquisition, entry and
updating
– Technical Process: developers and analysts
• Information Stakeholders:
– Anybody who uses the information for business

12. Metadata Fosters
Information Process Improvement
• Establishes common understanding, from an
enterprise standpoint, of all stakeholder
requirements for information
• Fosters collaboration, empowerment,
teamwork, professionalism, pride of
workmanship
• Makes environment of information
stewardship possible

13. Information Stewardship
Willingness to be accountable for a
set of business information for the
well-being of the larger organization,
by operating in service of (rather than
under control of) those around us.

14. What Quality Management Does:
• Brings producers and stakeholders together
• Makes sure stakeholder requirements are fully
represented
• Assures completeness and objectivity of measures so
they are reliable basis for understanding and managing
functionality and performance
• Establishes factual basis
• Makes process improvement possible
• Makes cross-functional teamwork possible
• Makes performance management possible
• Makes resource management possible

15. The Definition of Quality
Meeting and exceeding stakeholder
(customer) expectations and
requirements

16. Process Improvement:
PDCA (Shewhart Cycle)
• Plan an improvement based on factual
basis
• Do it for awhile
• Check it for improvement
• Act to standardize the improvement
(Repeat)

17. Process Improvement:
Root Cause Analysis
People Process
Defect
Tools Materials

18. Overview of Production Processes
Information Finance QM UM Etc.
Stakeholders
Information
Operational
Information Producers
Data Mart
Knowledge
Workers Data
Entry
Members
HEDIS/ Data
QARR Claims/ Acquisition
CRMS Encounters
Providers Systems
Performance Development
Management

19. Encounters Information Producers
Data Entry
Claims
Data Acquisition
CLAIMS/
ENCOUNTERS Care Providers
Vendors
Systems
Development
Developers/
Analysts

20. Members Information Producers
Data Entry
Enrollment
Data Acquisition
MEMBERS Marketing
Member Services
Recertification
Systems
Development
Developers/
Analysts

21. Providers Information Producers
Data Entry
Network Information
Data Acquisition
PROVIDERS Network Development
Provider Relations
Systems
Development
Developers/
Analysts

22. Encounters Stakeholders
Information
Stakeholders Finance QM UM Etc.
HEDIS/
QARR
Operational
Decision Support Information
System
Quality
Management
Utilization Members
Management
Community Claims/
Health CRMS Encounters
Institute
Data Mining Providers
P erformance
Management

23. HEDIS/QARR Quality Assessment and
Medical Record Review
Data Mart
• Build CRMS
• Create population samples
• Extract compliant instances CRMS
based on administrative data,
along with member and
provider site information, to a Special
QARR
special QARR database Review
Database
• Load laptops from this
database

24. Lack of Compliant Instances
by Administrative Data
• Compliance with measure conditions requires the
following data elements to be present and
accurate:
– Recipient information
– Service and diagnosis codes
– Service location information
– Provider information
• Missing or inaccurate information necessitates
the conduct of onsite medical record review

25. HEDIS/QARR Quality Assessment and
Medical Record Review
• Review claims
histories for all
members in samples
to identify most likely
primary care provider
• Sort samples and fax
lists to each site,
requesting charts for
onsite review

26. HEDIS/QARR Quality Assessment and
Medical Record Review
• Visit sites
and review
charts

27. HEDIS/QARR Quality Assessment and
Medical Record Review
Track down charts not
found at sites:
• Verify Member name, CIN, SSN
• Review claims history for other
providers and spans of care
• Check Healthy Beginnings /
Mammogram Incentives databases
• Check online Immunizations Registry
• Check Phone logs
• Check NY State Roster
• Check http://www.whitepages.com
• Call Providers
• Call Members

28. HEDIS/QARR Quality Assessment and
Medical Record Review
• Visit new
locations
and review
again

29. HEDIS/QARR Quality Assessment and
Medical Record Review
• Integrate and upload
laptop data to CRMS
• Use CRMS to
generate rates Data Mart
• Send results to NY
State Department of CRMS
Health

30. HEDIS/QARR Quality Assessment and
Medical Record Review
• Massive costs due to nonquality data
• Nurses to sites throughout network, three
times
• Hunting down members, what provider
they're getting care from, missing or
miscoded services and diagnoses
• 4300+ members in samples

31. Quality Approach
to Information
• Foster common understanding between producers
and stakeholders by use of shared metadata
• Accuracy can’t be automated; only people can
assess and correct for accuracy
• Information is an enterprise resource
• Measure information quality in voice of downstream
customers/stakeholders
• To improve a production process, measure its
product
• Improve information quality at the point of capture

32. Foster Common Understanding
Between Producers and
Stakeholders

33. Accuracy Can’t be Automated
• Only people can create, assess, or correct for accurate
information
• Automatic business rules can only prevent gross errors
• Checking against a
surrogate source:
comparison to external
Data Entry
datasets, forms
Information
• Checking against (Encounters,
authoritative source: Members,
Providers)
comparison to real Data Acquisition
world entity

34. Measure Information Quality
in Voice of Stakeholders
Information Finance QM UM Etc.
Stakeholders
Information
Operational
Information
Producers
Data Mart
Knowledge
Workers Data
Entry
Members
HEDIS/ Data
QARR Claims/ Acquisition
CRMS Encounters
Providers Systems
Performance Development
Management

35. Information Quality Measures
• Definitions/Specifications
• Timeliness
– Time from first capture to record of reference
• Completeness
– Represent each instance in the real world
– Completeness of values in existing fields
• Accuracy
– To surrogate source
– To authoritative source
• Usability
– Presence of fields to address functional requirements

36. Measure the Product
to Improve its Process
• Purpose is not to
improve a product by
measuring it, but to
improve its production
process
• Product is of concern only from a limited
perspective
• Quality measures show how well processes for
producing information are functioning; i.e., how
well they address all stakeholder requirements

37. Applications are not
the Product to Measure
• Assessing applications addresses
narrow requirements, not
downstream requirements for the
information
• Address information as the product, understood as
an enterprise resource
• Measure the information that applications work on to
assess development
• Applications are machines on the assembly line, not
the product

38. Applications are Machines
on the Assembly Line
Applications
Information

39. Improving Information (the Product)
is not the Chief Concern
• Right product to measure, but wrong purpose
• Improving the product doesn't prevent errors
or assure ongoing quality
• Improve production processes
• Information product improvement is all cost
basis/scrap and rework
• Correction of issues preventable by process

40. Three Ways to Correct
Information Quality
• As a one-time process
• As a regular conversion process
• At the point of capture

41. As a One-time Process for a
Special Table
• Not preventative; same problems recur
because information production process
has not changed
• Quality of data decays; not addressing
changes in the information
• Creates a redundant table for the entity
(encounters, services, providers,
members)

42. As a Regular Conversion for a
Data Mart or Data Warehouse
• Performed under constraint of a regular
load for up-to-the-moment, daily analysis
• Too late: requires automatic, estimated
corrections
• Corrections need to feed back to
operational tables

43. As a Regular Conversion for a
Data Mart or Data Warehouse
Regular Estimated,
Automated Corrections
Operational
Decision Support Systems
System
Cleanup &
Transfer Members
Claims/
CRMS Encounters
Providers

44. At the Point of Capture
• Actual process improvement
• Actually preventative
• Based on understanding of information as
enterprise resource
• Information producers assess and improve
business and technical/development
processes based on understanding of
stakeholder requirements

45. At the Point of Capture
Operational
Measure:
Information
Data - Definitions
- Timeliness
Entry
- Completeness
Members - Accuracy
- Usability
Data
Claims/ Acquisition Plan
Encounters
Do
Check
Providers Systems Act
Development
(Repeat)

46. Do Both:
• Improve at Point of Capture: Measure and
improve information production process quality
at the point of capture, from the standpoint of the
requirements of all downstream stakeholders
• Regular Cleanup and Transfer: Establish and
document business rules for integration (cleanup
and transfer), through stakeholder involvement
(i.e., for CRMS load)

47. Process Improvement for Information
• Plan:
– Identify stakeholders/customers
– Survey stakeholders for requirements
– Measure quality of product against requirements
– Determine root cause for identified issues
• Do:
– Implement improvement for awhile
• Check:
– Measure again to determine success of improvement
• Act:
– Act to standardize the improvement
(Repeat)

48. Root Cause Analysis for Information
Business Process Quality
Business Processes
Personnel
Methods / Procedures
Information Documented
Producers Procedures
Data Process
Intermediaries Steps
Specific
Defect
Application Source
Forms
Database Hardware External Data
Application /
Source Data
System

49. Root Cause Analysis for Information
Development Process Quality
Development
Personnel
Methods
Developers Application
Development
Analysts Data Analysis
Specific
Defect
Metadata Definitions/
Repository Specifications,
Business Rules
Forms, Reports,
Data Procedures, etc.
Dictionary
Development
Source Materials
Tools

50. Process Improvement Details
Useful for the NCQA Baseline
Assessment Tool

51. Process Improvement Steps
• Plan:
– Identify stakeholders/customers
– Identify requirements
– Measure quality of product against requirements
– Determine root cause for identified issues
• Do:
– Implement improvement for awhile
• Check:
– Measure again to determine success of improvement
• Act:
– Act to standardize the improvement
(Repeat)

52. Metadata Categories
• Production Process Factors
– People
– Process
– Tools
– Materials
• Quality Measures
– Timeliness
– Completeness
– Accuracy
• Barriers to Quality (Root Causes)
• Improvement Processes
– Quality Assessment
– Cleanup and Transfer
– Process Improvement

53. Production Process Factors
• People
– Stakeholders, Information Producers, Knowledge Workers
– Developers, Analysts
• Process
– Business Information Production Processes
– Application/Data Development Methods
• Tools
– Systems, Applications, Databases, Hardware
– Metadata Repositories, Data Dictionaries, CASE Tools
• Materials
– Requirements, Specifications, Business Rules, Documented
Procedures, Reports, Forms, External Data Sources

54. Simplified Picture
For each entity (Encounters, Members, Providers)
– Who are its stakeholders?
– What processes use it?
– What are their requirements?
– Then standards, measures, rules, definitions,
specifications, etc.

55. Slightly More Detailed Picture
• Definition Standards
• Goals / Motivations
• Product Specifications
• Roles – Owners, Stakeholders
• Business Requirements
• Production Processes
• Improvement Processes
• Performance Measures
• Performance Rates

56. Very Complex Structure

57. Purposes of Metadata
• Foster process improvement for information
• Enable information producers to understand
requirements from an enterprise standpoint, for both
business and technical/development processes
• See all stakeholders, their processes that depend on the
information, and their requirements
• Provide clear, accurate consensus definitions to enable
problem resolution
• Enable developers to standardize and reuse
specifications and business rules
• Manage information as a resource, including identifying
needless cost, how much value you're deriving, how and
where

58. Useful Steps to Take
• Work on information production processes by
surveying stakeholders for their requirements
• Measure business and development processes
according to requirements for the information as
voiced by stakeholders
• Improve information production processes at the
point of capture
• Devise rules for cleanup and transfer to decision
support systems (CRMS) in collaboration with
stakeholders
• Begin to store metadata: Document and share
stakeholders, processes, requirements,
performance measures, business rules, etc.

59. Foster Process Improvement through
Common Understanding of
Information Production Processes