How Data Quality Defines Your Nonprofit
•Download as PPTX, PDF•
1 like•493 views
Peter Mirus from Build Consulting uses examples from the nonprofit and public sectors to show how good data quality can have big fundraising and program impacts. He explores the role of good data at your organization, provides tips on how to get everyone at your organization on the same page, and gives ideas on how to prioritize data for quality improvement. This webinar was presented by Community IT Innovators in November 2019.
Recommended
Recommended
More Related Content
What's hot
What's hot (16)
Similar to How Data Quality Defines Your Nonprofit
Similar to How Data Quality Defines Your Nonprofit (20)
More from Community IT Innovators
More from Community IT Innovators (20)
Recently uploaded
Recently uploaded (20)
How Data Quality Defines Your Nonprofit
- 1. HowDataQualityDefinesYourOrganization Best practices applied to real life.
- 2. Interact Ask questions using the available Questions feature in GTW. Focus Avoid multitasking. You may just miss the best part of the presentation! Webinar Recording and Slides Links to the recordings (video and MP3) will be shared via email after the webinar.
- 3. NextCommunity ITwebinar: “5SecurityTipstoProtectYourLogin Credentials andMore” Wednesday, November20 from3:00-3:40PMET
- 4. Invested Work exclusively with nonprofit organizations; have served over 1,000. Strategic Help our clients make IT and IS decisions that support mission. Collaborative Empower you to make informed choices.
- 5. HowBuildleadsinthesocialgoodsector: Interim or Part-Time CIOs Outsourced CRM Management Assessments and Roadmaps Software Selections Implementation Support
- 6. Peter Mirus Partner, Build Consulting
- 8. Tomorrow’s best nonprofits will use technology to transform themselves... and the world.
- 9. More than 50% of nonprofit technology projects fail.
- 10. The technology moves forward... but the organization does not.
- 11. Transformation is critical to your success.
- 12. Build’s Information Strategy Framework Leadership and Governance DataOperations Process Technology TM
- 13. Our Observations Making the Case for Data Quality The Dimensions of Data Quality 5 Dimensions 5 Accompanying Examples Recommendations Q&A Agenda
- 15. Can data quality determine success?
- 19. Concept 2: The Real World Object
- 22. “After 10 years of flunking, [in 2018] Anacostia River passes annual health-check with D rating”
- 29. Validity, Accuracy & Donor Retention
- 37. Recommendations
- 38. There’s no such thing as perfect data quality
- 40. Transformation is critical to your success.
- 41. Data Quality Success Be Like…
- 43. Q&A
Editor's Notes
- Welcome to the Community IT and Build Consulting webinar for November 2019 titled “How Data Quality Defines Organization.” Today I’ll be introducing a simple framework for understanding data quality that you can use to help drive data quality conversations at your organization, leading to improvements in the quality of your data, in fundraising outcomes, and in mission impact.
- Before we get started on the webinar, here are a few housekeeping notes….
- November 20 from 3:00-3:40 pm Matt Eshleman will present “5 Security Tips to Protect Your Login Credentials and More”  There will be a special offer for attendees during the webinar!
- Now, a little bit about Community IT and Build Consulting. We both work exclusively with nonprofit organizations to help them make information technology and information system decisions that support their mission. We have a collaborative approach, empowering our clients to make informed choices for their organizations.
- All our services are designed to help clients transform themselves to better serve constituents of all types, including funders, donors, program beneficiaries, staff, volunteers, board and committee members, and the general public.
- 20 years serving all manner of nonprofit organizations, ranging in size from small/local to enterprise/global, and across a wide variety of industry categories and mission orientations. Over the past 7 year, I’ve worked exclusively with nonprofit organizations. I’ve worked with over 100 clients in both the nonprofit, government, and for-profit spaces. I have three primary areas of expertise: information strategy, constituent relationship management, and communications.
- We had over 90 registrants from this webinar from a wide range of nonprofit organizations, from throughout the United States and Canada. I see that [X number] of those registrants are currently attending! Thanks so much for being with us today. I hope we’ll have some time remaining at the end of the presentation for questions and answers. But if you asked a question during the registration process or during today’s presentation that remains unaddressed by the end of today’s session, my personal contact information will be provided immediately prior to Q&A – please feel free to reach out to me directly with a follow-up.
- Through multiple information system design, implementation, integration, and migration projects, we have seen social good organizations encounter and then either deal with (or fail to deal with) data quality issues – and the kinds of decisions they make regarding data quality are interesting to us in three ways.
- First, data quality decisions have the ability to determine how successful an organization will be – how it sets itself up for success, mediocrity, or failure based on its ability to manage and leverage its data to make informed business decisions and drive impact.
- Second, data quality decisions are profoundly revealing of the strengths and limitations of an organization: Does it have the strategic vision to make appropriate investments in data quality? Does it know what relationships the organization has, and what data is profoundly important to nurturing and growing those relationships? Does it have the leadership and discipline to require staff to adhere to corporate policies, standards, and business processes that result in good data quality?
- Third, many organizations don’t fully comprehend the full picture of what data is. Data is all of the information that your organization collects: both qualitative and quantitative, both structured and unstructured, or in another way of putting it, both the narrative and the numbers.
- This is an aerial photo of the Washington Navy Yard, which is the “oldest shore establishment of the U.S. Navy.” The photo you see here was taken in 1985, and superficially it appears like a somewhat prosaic urban landscape.
- In 1980 Congress created the Superfund to pay for the cleanup of the country’s most hazardous waste sites. In 1998 the EPA put the Washington Navy Yard on the National Priorities List – one of the 1,700 prioritized cleanup projects of the 47,000 waste sites in the nation. This site has the status of “active” – meaning that cleanup facilities had not yet been completed.
- The Anacostia River, on which the Navy Yard is situated, is part of a complex ecosystem about which much data has been collected but about which there is still much to be learned. In data quality we talk a lot about the “real world object.” Data exists to describe a real world object. The better the quality of your data, the better you are able to understand the real world object and hence your relationship to it – how your actions impact it. Just as the river flows continuously, sometimes the real world object is in a state of constant flux.
- Good data quality has been and continues to be part of the Anacostia Watershed cleanup process, which has seen participation from numerous organizations. Gathering and integrating high-quality from various sources is critical to the work of those organizations and to other organizations both large and small participating in conservation efforts worldwide.
- Whether removing contaminates from the Washington Navy Yard and the Anacostia Watershed, running medical field trials in Africa, training small businesses in Afghanistan, empowering women and girls in Eastern Europe, or combatting domestic violence in the United States, the information you gather about the people you serve and their local environment has a good deal to do with the impact of your efforts. With so much data lacking that we wish we could obtain, the quality of the data we do have to describe those “real world objects” becomes of paramount importance. Data quality directly helps to create life quality. Data quality is not just about accuracy – as we will see in the upcoming parts of the presentation, it is multidimensional. We need to place the right information in the hands of people at all levels of the organization, from the C-suite, to the front-line staff, to the constituents. This will help to create greater efficiency and impact. By doing this, how can we help people do new and amazing things?
- And finally, the data that you have enables you to have a voice – to be heard. You want to get through to people. The stories that you tell with that data creates intellectual and emotional resonance with your key audiences. It communicates the value of your work and helps to secure the funds and influence necessary to do the work well.
- Today, for the sake of simplicity, we are going to talk about five dimensions of data quality. Those who are familiar with data quality terminology as used by such major funders as USAID will recognize some points of familiarity. These dimensions are: Completeness Validity and Accuracy (which are distinct, but well paired together) Consistency Timeliness Integrity We are going to take each of these in turn.
- When data is complete it has all of the necessary or appropriate parts. We use the terms “breadth” and “depth” to describe these qualities of completeness: When we say “Breadth” we are asking: Does the data have all of the expected attributes? And when we say “Depth” we are asking: Is the data set as fully populated as expected? To provide a very simple illustration: when gathering constituent information, you do not have complete addresses if you are not capturing the state in which each constituent lives. The fact that you have a field (or column of data) to capture the state speaks to the completeness of breadth within the data set. The fact that you have a value in that column for each of the constituents is indicative of the depth of the data.
- Let’s say that your organization is running an economic development program in Afghanistan, focused on training small business owners and entrepreneurs to engage in the global economy. As an aspect of data collection, the program managers and chiefs of party are tasked with collecting information on women served through these programs – an imperative coming from both the funder and your own organization. The program is executed through multiple technical assistance projects run out of various locations within Afghanistan, and the data collection is being done on paper and then entered into spreadsheets. However, the data collection method was not standardized and some projects had a field to capture beneficiary gender and some did not. Some of the projects had a gender field but did not consistently manage the paper documents that captured that information, resulting in loss of “backup information” and contributing to data gaps. The problems within this scenario impact both the breadth and depth of the information. The data was not complete, and in order to successfully provide reports based on the data, the program managers had to make “best guesses” to fill in the information. This impacted the data quality and the trustworthiness of that data. This is the kind of completeness problem that you want to avoid. You want to make sure that the gender field is available as an attribute of the object you want to describe, and you want to make sure that the field has information in it. This is a simple example of the problems that can be created by incomplete data. When a major funder like USAID loses confidence in the data quality and trustworthiness of the data because they see problems, they lose confidence in your ability to deliver on the program. On the flip side, having good quality data and backup for that data causes the funder to gain confidence in the program and increases the chances that the program will be expanded or renewed.
- Let’s talk a little bit about validity and accuracy. In the world of data management, including performance data management, “validity” is the “conformity to a domain of values.” That domain of values might be: An actual set of valid values (such as a list of states) A range of values (such as a number between 1 and 100) A rule that would generate a value (such GPS coordinate fields being automatically populated from an address) In other words, validity is judged by comparing data to criterion or constraints. So in order to test and measure validity, you need to know the values or rules to which the data should be compared. These are the rules that prevent invalid data from being entered, which goes a long way to helping avoid inaccuracy. You can extend the definition of validity to create rules for preventing or assessing duplicate data, which in data sets over a very small size is difficult to assess visually. For example, you can create a rule for a beneficiary database indicating that it should be impossible to have two records for with the same last name and street address. You can be using a piece of valid data that is nonetheless inaccurate. In order to examine accuracy you must examine the “real world object” and compare it to the data. So for example, if I was filling in a state field and entered “MD” for Maryland, which is valid response, and I lived in Virginia, it would be valid but not accurate.
- Here’s another example of how data validity and accuracy is important. Last week, this man was a $10,000 per year giver to your organization. This week, he is no longer your donor. Why? This man’s wife was on your mailing list. Some donations were made in her name, and some in his. Then his wife died. The man let you know that his wife was diseased, and a development person at your organization accessed the woman’s record and selected the wife’s status as “Inactive – Deceased” from a drop-down list. The drop-down list ensured that the person could only select from a range of values – therefore it was impossible for the entry to be invalid, and in the donor database that information also accurately reflected the real world situation. However, because of poor data integration between the donor database and the mailing list, the wife was never taken off the organization’s mailing list – in that separate data set, she continued to have a status of “Active”. Therefore, even though the husband notified the organization several times that his wife was deceased, the organization continued to send mailings to her. The person doing that data entry into the donor database reported the problem to a manager, but it was not red flagged as a potential persistent data quality issue. As it turned out, the problem took only $1,000 to resolve, but by the time the organization recognized and reacted to the data quality problem resulting in inaccuracy in one part of the data set, the man was both emotionally upset and had lost confidence in the competency of the organization. He therefore cancelled his pledge and began giving the same amount to another organization. For lack of a $1,000 data quality decision being made proactively, the organization lost $200,000 in potential revenue – or what it expected to get from this donor during his lifetime.
- Data consistency is ensured by the methodology applied for data collection remaining consistent over time, and is measured by the degree of similarity (or the absence of variety) within the data. We can measure consistency of similar data over time by looking for trends and differences from the trend. Consistency is what allows for the authentic analysis (interpretation of data) over time and across space. In order to ensure this is possible, data collection process must be well documented and standardized to the fullest extent possible. For example, considering every program within a vacuum can be deadly from the standpoint of organizational “indicators” that aggregated will provide performance data to both the organization and its key funders, as well as to industry analysts and influencers. If the consistent collection rules for organizational indicators are not applied across all programs, then the organization’s performance may be off in ways that are very difficult to spot just by looking at the data, absent the ability to evaluate the record of differing data collection methodologies and make appropriate corrections.
- So in this case story we are going to talk about consistency and youth services. Consistency will frequently be a problem when critical business definitions change over time, but that fact is not recorded as a separate data point that can be applied to help interpret data trends. For example, an organization’s definition of “at risk” youth might change over the course of a prolonged period of time and data might be collected relative to that definition. This means that there is a lack of consistency within the data collection and hence within the data itself. The fact of the definition changing is a critical piece of data necessary to interpret the lack of consistency within the data, and make adjustments accordingly when doing analysis. Otherwise, it might seem as if the real world object (in this case, the number of “at risk” youth) had changed, but in actual fact what changed is how the organization defines “at risk.”
- Timeliness refers to both the regularity of the data collection AND the availability of that data to decision-makers, which are two distinct things. The overall goal is to decrease the time between when the “real world object” changes and when the change is known by all relevant parties (program staff, organizational strategists, funders, and other information consumers). Important points to consider regarding timeliness: If the data has a high degree of volatility, there is an increased risk that the data will not meet standards for timeliness. Volatility is the degree to which data is likely to change over time. An example of low volatility data is gender. An example of high volatility data is salary. Data that has high volatility will need to be checked more frequently to ensure it remains timely. Data fails to be timely if there is a lag between when a fact becomes known and when it becomes available for use. Example: if a program staffer in the field becomes aware of a change to a “real world object” that is important to program outcomes but fails to communicate it to program decision-makers in a timely manner, perhaps because the change occurred between regular reporting intervals. This highlights the need for all staff to follow both the letter and the spirit of the law when collecting and transmitting data. Data fails to be timely when there is a lag between when the data is updated at the source and when it becomes available to decision-maker at the other end of the information chain. For example, if data is contained in spreadsheet at the program office but does not get transmitted to HQ, then you have problems with timeliness.
- This a picture of the devastation that occurred on the New Jersey coastline during Hurricane Sandy. Timeliness of critical importance particularly when it comes to disaster response. The timeliness of accurate data, such as weather forecasting and resource availability and positioning, helps governments and relief organizations to successfully deliver services to assist those in urgent need because of a natural disaster. Or in another example, some sort of mass human rights violation or civil strife. In the events leading up to, during, and following a large scale event like Hurricane Sandy, the timeliness of data can make a huge difference to providing timely relief. You will find in these particular kinds of situations that the efficiency with which the organizations involved exchange information, and the timeliness of that data being able to be combined and interpreted for different actions, has a lot to do with how organizations and agencies fare in the effectiveness of their activities and hence in the eyes of public perception in the assessment that takes place following the aftermath of an incident such as this.
- Data integrity is how all of the parts fit together; how the pieces make one whole thing. If you have a large database, data warehouse, data-mart, or even a file share with multiple Excel files -- containing records from a range of systems -- people who are using that data have an expectation that they will be able to connect it – they expect the data to be integrated. The integrity of the data is one of the first things people will question when they cannot use the data in the ways they expect. So it is important to have a plan for how you are joining data from different data sets or tables, and how well the joined data fits together. When you have multiple information systems, and the data within them relates to each other, you need to create a way to make sure the integrity of that data is present when it is related for the purpose of gaining insight and making decisions. At a very high level, “Measures of integrity” are measures between different tables in the database; how you join data from different data sets or tables, and how well it fits together. It is basically your ability to relate the data cohesively.
- This is a brief story about integrity and healthcare. Integrity matters in the healthcare industry when you need to be able to bring together data related to diseases from a wide variety of sources. Viewing all of the medical records from a particular hospital is relatively easy by comparison to collecting and analyzing disease-related data from a variety of different hospital systems and other similar institutions. Without being able to easily integrate the various data sets it is very time consuming to be able to analyze the data, perhaps even to perform predictive analysis that will help to control the spread of a particular disease. So in this case, as in many cases – and including the last one around disaster relief where there is a high degree of criticality associated with making a quick response – integrity is extremely important – the ease with which the different parts of the data fit together. And you’ll notice that there has been a lot of work done to standardize the collection of healthcare data within the United States and globally, particularly over the past decade. It is a continuing challenge and a continuing battle. One of the reasons why that work is being done is to ensure there is integrity across multiple different systems, across multiple different means of collecting data. And that eases the ways in which people can use that data to increase public health and mitigate risk
- So to do a recap, here are the aspects of data quality that we covered today: Completeness – the breadth and depth of the data, as considered through the lens of gender equality Validity and Accuracy – as considered through the lens of donor retention Consistency – as considered through the lens of youth services Timeliness – as considered through the lens of disaster response Integrity – as considered through the lens of healthcare
- For each of the data dimensions covered in today’s presentation, we cited an real-life example that included some sort of problem or challenge. One might reasonably ask how these challenges can be addressed. The approach that I prefer is rooted in two principles:
- There is no such thing as perfect data quality. For data to be perfect it would have to always be precisely representative of the real world object. We know that this is extremely difficult, if not impossible, for the objects we describe with data in the course of performing our work.
- 2. Data is meaningless absent the context of relationships. Your relationship to a real world object and the relative importance of that relationship compared to other relationships dictates your data quality priorities. This is what drives you to ask the critical questions: how complete does my data need to be? how accurate does it need to be? Etc.
- The specific approach to data quality for each organization, therefore, relies on clarity and strength in organizational strategy both at a high level and as it is extended through programs and projects. This strategy should tell you the relationships most critical to success and the value placed on each relationship. Your can then prioritize your resources according to that value. This often requires some degree of transformation at an organization level before improvements to technology may be effectively leveraged.
- Organizations of any size and complexity that make data quality a cultural endeavor typically have good data quality relative to their need for such, while organizations that do not will generally have poor data quality. So do it well, and celebrate your successes! For all the talk about data quality in the social sector, having truly high-quality data is relatively rare, and the ability to use that data to inform meaningful action is even rarer.