Community IT Webinar - Data Quality
•Download as PPTX, PDF•
1 like•666 views
Join us for our free monthly webinar series as we review the importance of Data Quality for effective technology in nonprofit organizations.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Community IT Webinar - Data Quality
Similar to Community IT Webinar - Data Quality (20)
More from Community IT Innovators
More from Community IT Innovators (20)
Recently uploaded
Recently uploaded (20)
Community IT Webinar - Data Quality
- 1. How Data Quality Defines Your Organization BEST PRACTICES APPLIED TO REAL LIFE
- 2. Interact Ask questions via chat Connect on Twitter Focus Avoid multitasking. You may just miss the best part of the presentation Webinar PowerPoint & Recording PowerPoint and recording links will be shared after the webinar
- 3. Peter Mirus, Senior Strategist • 20 years of business strategy and information strategy experience with nonprofit organizations • Expert in helping organizations to learn and grow through the use of information technology. Peter’s Data Quality Story
- 4. Kyle Haines, Senior Strategist • 25 years working for, and with, nonprofit organizations seeking to improve their CRM maturity • Partner at Build Consulting, a frequent collaborator with Community IT on information strategy projects with CRM as an underpinning Kyle’s Data Quality Story
- 5. What We Will Talk About Today Why Data Quality? Dimensions of Data Quality 5 Dimensions 5 Accompanying Examples Solutions Q&A Agenda
- 8. Concept 2: The Real World Object
- 23. SOLUTIONS
- 24. Q&A
Editor's Notes
- My name is Peter Mirus – I am a nonprofit business and information strategy consultant working with Community IT Innovators. Over the past 20 years I have worked with a wide variety of nonprofit organizations ranging from small start-ups to large international organizations. What I have learned from working with these organizations is that data quality is a challenge for all organizations – whether they realize it or not, or whether or not they use the term “data quality” when discussing the related issues. During different parts of my career I have spent a considerable amount of time advising clients on strategic branding, marketing, and development initiatives. I have also spent time on program design and program management business process development. The organizations that tend to excel the most in these areas take care with their data – they seek to collect data beyond what is naturally created as a byproduct of their work -- and understand how to leverage that data to drive impact and to create clear and compelling stories. From my broad experience as an information strategy generalist, I can tell you that the potential for many organizations to create new standards of excellence – in part through a renewed emphasis on data quality as a broad concept – is huge. It is realizations such as this that inspire my work, and the work of all at Community IT Innovators, Build Consulting, and the firms with which we partner.
- You know, my data quality story really started from humble beginnings. 25 years ago, right out of college, I was asked to restart a CRM implementation that I would say was 'dead in the water'. With little experience, and probably mostly intuition, I struggled my way through getting the organization oriented around CRM. I didn't have the experience at the time, and candidly the organization didn't have the maturity to ask the question "What do we want to know about people?" CRM was viewed as a Rolodex and donation management system. You know, gifts in, acknowledgement letters out, and an hastily created annual report list. of donors. What I now know, through my work, is that so many organizations view systems as a data repository, without having a conversation about what data is important. Answering that question, among others, is a key underpinning of information strategy. It's sometimes hard to see the drain that these systems put on organizations, without meaningfully moving the needle on using data, even in its simplest form, to inform action.
- Through multiple information system design, implementation, integration, and migration projects, we have seen nonprofits encounter and then either deal with (or fail to deal with) data quality issues – and the kinds of decisions they make regarding data quality are interesting to us in three ways. First, data quality decisions have the ability to determine how successful an organization will be – how it sets itself up for success, mediocrity, or failure based on its ability to manage and leverage its data to make informed business decisions and drive impact. Second, data quality decisions are profoundly revealing of the strengths and limitations of an organization: does it have the strategic vision to make appropriate investments in data quality? Does it know what relationships the organization it has, and what data is profoundly important to nurturing and growing those relationships? Does it have the leadership and discipline to require staff to adhere to corporate policies, standards, and business processes that result in good data quality? Third, many organizations don’t fully comprehend the full picture of what data is. Data is all of the information that your organization collects: both qualitative and quantitative, both structured and unstructured, or in another way of putting it, both the narrative and the numbers. So what specifically are we going to talk about on this day? Today’s webinar is designed to be educational in nature – to answer the question “What is data quality?” We will do this by naming and defining some of the critical dimensions of data quality and relating them to real life scenarios. We hope that you will find something either in the more esoteric definitions or in the scenarios that resonates with you – that captures your intellect, your imagination, and perhaps your emotions.
- And now, to begin our formal presentation. You’ve doubtless heard the saying “Garbage in, Garbage Out” – meaning that if you put poor quality data into systems, you get poor quality reports from those systems. The above graphic (and many like it) helps to illustrate the critical nature of data quality, and I chose it because of its direct and ironic relevancy to that saying. The graphic – two maps of England, each divided into a 5km grid, shows in FIGURE A the geographic density of landfill sites for “special” and in FIGURE B “non-special or unknown” waste types (Figure B). This image is one part of a study titled “geographic density of landfill sites and risk of congenital anomalies in England. Similar work has been conducted in the United States, where nearly one of every six Americans live within three miles of a major hazardous waste site (Source: National Geographic via the EPA). It underscores the critical nature of carefully managing data to drive impact. In enabled a strong visual that can help to raise awareness of a public health and safety issue. Cleaning up hazardous waste—and reducing or eliminating its continued production-- in both England and the Unites States requires a coordinated effort between the public and private sectors. Reliable data that tells a story is critical to securing the funds and influence necessary to conduct these efforts. Reliable data tells the how, what, when, and where of what steps to take to prevent the air we breath and the water we drink from being contaminated. Just like we need to develop better habits in terms of putting “clean” waste into our landfills to produce better outcomes, we also need to put clean data into our systems in order to produce better outcomes. We put dangerous objects into our landfills and waste processing centers, and it runs the risk of causing harm. If we put poor quality data into our information systems, we run the risk of causing harm (or at the least, diminishing positive returns).
- How would you describe what you are seeing here? What does it look like to you? This is an aerial photo of the Washington Navy Yard, which is the “oldest shore establishment of the U.S. Navy.” The photo you see here was taken in 1985, and superficially it appears like a somewhat prosaic urban landscape. In 1980 Congress created the Superfund to pay for the cleanup of the country’s most hazardous waste sites. In 1998 the EPA put the Washington Navy Yard on the National Priorities List – one of the 1,700 prioritized cleanup projects of the 47,000 waste sites in the nation. Today, this site has the status of “active” – meaning that cleanup facilities have not yet been completed. Just looking at the superficial data provided in this photo you would likely not have been aware that something was amiss. The Anacostia River, on which the Navy Yard is situated, is part of a complex ecosystem about which much data has been collected but about which there is still much to be learned. In data quality we talk a lot about the “real world object.” Data exists to describe a real world object. The better the quality of your data, the better you are able to understand the real world object and hence your relationship to it – how your actions impact it. Just as the river flows continuously, sometimes the real world object is in a state of constant flux. Good data quality has been and continues to be part of the Anacostia Watershed cleanup process, which has seen participation from numerous organizations. Gathering and integrating high-quality from various sources is critical to the work of those organizations and to other organizations both large and small participating in conservation efforts worldwide.
- Whether you are trying to clean up landfills in England, the removing contaminates from the Washington Navy Yard and the Anacostia Watershed, running medical field trials in Africa, training small businesses in Afghanistan, empowering women and girls in Eastern Europe, or combatting domestic violence in the United States, the information you gather about the people you serve and their local environment has a good deal to do with the impact of your efforts. With so much data lacking that we wish we could obtain, the quality of the data we do have to describe those “real world objects” becomes of paramount importance. Data quality directly helps to create life quality. Data quality is not just about accuracy – as we will see in the upcoming parts of the presentation, it is multidimensional. We need to place the right information in the hands of people at all levels of the organization, from the C-suite, to the front-line staff, to the constituents. This will help to create greater efficiency and impact. By doing this, how can we help people do new and amazing things?
- And finally, the data that you have enables you to have a voice – to be heard. You want to get through to people. The stories that you tell with that data creates intellectual and emotional resonance with your key audiences. It communicates the value of your work and helps to secure the funds and influence necessary to do the work well. That’s why data quality.
- Today, for the sake of simplicity, we are going to talk about five dimensions of data quality. Those who are familiar with data quality terminology as used by such major funders as USAID will recognize some points of familiarity. These dimensions are: Completeness Validity and Accuracy (which are distinct, but well paired together) Consistency Timeliness Integrity We are going to take each of these in turn.
- When data is complete it has all of the necessary or appropriate parts. We use the terms “breadth” and “depth” to describe these qualities of completeness: When we say “Breadth” we are asking: Does the data have all of the expected attributes? And when we say “Depth” we are asking: Is the data set as fully populated as expected? To provide a very simple illustration: when gathering constituent information, you do not have complete addresses if you are not capturing the state in which each constituent lives. The fact that you have a field (or column of data) to capture the state speaks to the completeness of breadth within the data set. The fact that you have a value in that column for each of the constituents is indicative of the depth of the data.
- Let’s say that your organization is running an economic development program in Afghanistan, focused on training small business owners and entrepreneurs to engage in the global economy. As an aspect of data collection, the program managers and chiefs of party are tasked with collecting information on women served through these programs – an imperative coming from both the funder and your own organization. The program is executed through multiple technical assistance projects run out of various locations within Afghanistan, and the data collection is being done on paper and then entered into spreadsheets. However, the data collection method was not standardized and some projects had a field to capture beneficiary gender and some did not. Some of the projects had a gender field but did not consistently manage the paper documents that captured that information, resulting in loss of “backup information” and contributing to data gaps. The problems within this scenario impact both the breadth and depth of the information. The data was not complete and in order to successfully provide reports based on the data, the program managers had to make “best guesses” to fill in the information. This impacted the data quality and the trustworthiness of that data. This is the kind of completeness problem that you want to avoid. You want to make sure that the gender field is available as an attribute of the object you want to describe, and you want to make sure that the field has information in it. This is a simple example of the problems that can be created by incomplete data. When a major funder like USAID loses confidence in the data quality and trustworthiness of the data because they see problems, they lose confidence in your ability to deliver on the program. On the flip side, having good quality data and backup for that data causes the funder to gain confidence in the program and increases the chances that the program will be expanded or renewed.
- Let’s talk a little bit about validity and accuracy. In the world of data management, including performance data management, “validity” is the “conformity to a domain of values.” That domain of values might be: An actual set of valid values A range of values A rule that would generate a value In other words, validity is judged by comparing data to criterion or constraints. An example might be validating zip code based on five digits, or five numeric values ins a string. So in order to test and measure validity, you need to know the values or rules to which the data should be compared. These are the rules that prevent invalid data from being entered, which goes a long way to helping avoid inaccuracy. You can extend the definition of validity to create rules for preventing or assessing duplicate data, which in data sets over a very small size is difficult to assess visually. For example, you can create a rule for a beneficiary database indicating that it should be impossible to have two records for with the same last name and street address. You can be using a piece of valid data that is nonetheless inaccurate. In order to examine accuracy you must examine the “real world object” and compare it to the data. So for example, if I was filling in a state field and entered “MD” for Maryland, which is valid response, and I lived in Virginia, it would be valid but not accurate.
- Here’s another example of how data validity and accuracy is important. Last week, this man was a $10,000 per year giver to your organization. This week, he is no longer your donor. Why? This man’s wife was on your mailing list. Some donations were made in her name, and some in his. Then his wife died. The man let you know that his wife was diseased, and a development person at your organization accessed the woman’s record and selected the wife’s status as “Inactive – Diseased” from ta drop-down list. The drop-down list ensured that the person could only select from a range of values – therefore it was impossible for the entry to be invalid, and in the donor database that information also accurately reflected the real world situation. However, because of poor data management between the donor database and the mailing list, the wife was never taken off the organization’s mailing list – in that separate data set, she continued to have a status of “Active”. Therefore, even though the husband notified the organization several times that his wife was deceased, the organization continued to send mailings to her. The person doing that data entry into the donor database reported the problem to a manager, but it was not red flagged as a potential persistent data quality issue. As it turned out, the problem took only $1,000 to resolve, but by the time the organization recognized and reacted to the data quality problem resulting in inaccuracy in one part of the data set, the man was both emotionally upset and had lost confidence in the competency of the organization. He therefore cancelled his pledge and began giving the same amount to another organization. For lack of a $1,000 data quality decision being made proactively, the organization lost $200,000 in potential revenue – or what it expected to get from this donor during his lifetime.
- Data consistency is ensured by the methodology applied for data collection remaining consistent over time, and is measured by the degree of similarity (or the absence of variety) within the data. We can measure consistency of similar data over time by looking for trends and differences from the trend. Consistency is what allows for the authentic analysis (interpretation of data) over time and across space. In order to ensure this is possible, data collection process must be well documented and standardized to the fullest extent possible. For example, considering every program within a vacuum can be deadly from the standpoint of organizational “indicators” that aggregated will provide performance data to both the organization and its key funders, as well as to industry analysts and influencers.
- So in this case story we are going to talk about consistency and youth services. Consistency will frequently be a problem when critical business definitions change over time, but that fact is not recorded as a separate data point that can be applied to help interpret data trends. For example, an organization’s definition of “at risk” youth might change over the course of a prolonged period of time and data might be collected relative to that definition. This means that there is a lack of consistency within the data collection and hence within the data itself. The fact of the definition changing is a critical piece of data necessary to interpret the lack of consistency within the data, and make adjustments accordingly when doing analysis. Otherwise, it might seem as if the real world object (in this case, the number of “at risk” youth) had changed, but in actual fact what changed is how the organization defines “at risk.”
- Timeliness refers to both the regularity of the data collection AND the availability of that data to decision-makers, which are two distinct things. The overall goal is to decrease the time between when the “real world object” changes and when the change is known by all relevant parties (program staff, organizational strategists, funders, and other information consumers). Important points to consider regarding timeliness: If the data has a high degree of volatility, there is an increased risk that the data will not meet standards for timeliness. Volatility is the degree to which data is likely to change over time. An example of low volatility data is gender. An example of high volatility data is salary. Data that has high volatility will need to be checked more frequently to ensure it remains timely. Data fails to be timely if there is a lag between when a fact becomes known and when it becomes available for use. Example: if a program staffer in the field becomes aware of a change to a “real world object” that is important to program outcomes but fails to communicate it to program decision-makers in a timely manner, perhaps because the change occurred between regular reporting intervals. This highlights the need for all staff to follow both the letter and the spirit of the law when collecting and transmitting data. Data fails to be timely when there is a lag between when the data is updated at the source and when it becomes available to decision-maker at the other end of the information chain. For example, if data is contained in spreadsheet at the program office but does not get transmitted to HQ, then you have problems with timeliness.
- This a picture of the devastation that occurred on the New Jersey coastline during Hurricane Sandy. Timeliness of critical importance particularly when it comes to disaster response. The timeliness of accurate data, such as weather forecasting and resource availability and positioning, helps governments and relief organizations to successfully deliver services to assist those in urgent need because of a natural disaster. Or in another example, some sort of mass human rights violation or civil strife. In the events leading up to, during, and following a large scale event like Hurricane Sandy, the timeliness of data can make a huge difference to providing timely relief. You will find in these particular kinds of situations that the efficiency with which the organizations involved exchange information, and the timeliness of that data being able to be combined and interpreted for different actions, has a lot to do with how organizations and agencies fare in the effectiveness of their activities and hence in the eyes of public perception in the assessment that takes place following the aftermath of an incident such as this.
- Data integrity is how all of the parts fit together; how the pieces make one whole thing. If you have a large database, data warehouse, data-mart, or even a file share with multiple Excel files -- containing records from a range of systems -- people who are using that data have an expectation that they will be able to connect it – they expect the data to be integrated. The integrity of the data is one of the first things people will question when they cannot use the data in the ways they expect. So it is important to have a plan for how you are joining data from different data sets or tables, and how well the joined data fits together. When you have multiple information systems, and the data within them relates to each other, you need to create a way to make sure the integrity of that data is present when it is related for the purpose of gaining insight and making decisions. At a very high level, “Measures of integrity” are measures between different tables in the database; how you join data from different data sets or tables, and how well it fits together. It is basically your ability to relate the data cohesively.
- This is a brief story about integrity and healthcare. Integrity matters in the healthcare industry when you need to be able to bring together data related to diseases from a wide variety of sources. Viewing all of the medical records from a particular hospital is relatively easy by comparison to collecting and analyzing disease-related data from a variety of different hospital systems and other similar institutions. Without being able to easily integrate the various data sets it is very time consuming to be able to analyze the data, perhaps even to perform predictive analysis that will help to control the spread of a particular disease. So in this case, as in many cases – and including the last one around disaster relief where there is a high degree of criticality associated with making a quick response – integrity is extremely important – the ease with which the different parts of the data fit together. And you’ll notice that there has been a lot of work done to standardize the collection of healthcare data within the United States and globally, particularly over the past decade. It is a continuing challenge and a continuing battle. One of the reasons why that work is being done is to ensure there is integrity across multiple different systems, across multiple different means of collecting data. And that eases the ways in which people can use that data to increase public health and mitigate risk
- So to do a recap, here are the aspects of data quality that we covered today: Completeness – the breadth and depth of the data, as considered through the lens of gender equality Validity and Accuracy – as considered through the lens of donor retention Consistency – as considered through the lens of youth services Timeliness – as considered through the lens of disaster response Integrity – as considered through the lens of healthcare
- For each of the data dimensions covered in today’s presentation, we cited an real-life example that included some sort of problem or challenge. One might reasonably ask how these challenges can be addressed. The approach that I prefer is rooted in two principles: There is no such thing as perfect data quality. For data to be perfect it would have to always be precisely representative of the real world object. We know that this is extremely difficult, if not impossible, for the objects we describe with data in the course of performing our work. Data is meaningless absent the context of relationships. Your relationship to a real world object and the relative importance of that relationship compared to other relationships dictates your data quality priorities. This is what drives you to ask the critical questions: how complete does my data need to be? how accurate does it need to be? Etc. The specific approach to data quality for each organization, therefore, relies on clarity and strength in organizational strategy both at a high level and as it is extended through programs and projects. This strategy should tell you the relationships most critical to success and the value placed on each relationship. Your can then prioritize your resources according to that value. The resources should be used to do three things: The first is to describe the content and flow of data within those relationships from the perspective of the dimensions of data quality that we described today. This is effectively represented as a business process mapping or design exercise, and is done absent the context of managing that data within a particular database or a particular information system. The second is to select or design – and then implement -- information systems that come as close to reflecting those business processes as the budget will allow. These might be extremely low-tech, low-integration information systems (such as pen and paper) or high-tech, high-integration information systems (such as Salesforce). The third step is to train staff extremely well on the business processes necessary to ensure proper data quality, including actions that take place both inside and outside of the information systems. Organizations of any size and complexity that do all three of these things well typically have good data quality relative to their need for such, while organizations that do not will generally have poor data quality.