Adventures in Data Profiling

•

8 likes•2,634 views

Vendor-neutral presentation about the common functionality provided by data profiling tools, which can help automate some of the work needed to begin your preliminary data analysis.

Jim Harris
Blogger‐in‐Chief
www.ocdqblog.com
Jim Harris
Digitally signed by Jim Harris
DN: cn=Jim Harris, o=Obsessive-Compulsive Data Quality (OCDQ), ou, email=jim.harris@ocdqblog.
com, c=US
Date: 2010.03.04 10:55:20 -06'00'

Jim Harris
Blogger‐in‐Chief
www.ocdqblog.com

E‐mail
jim.harris@ocdqblog.com

Twitter
twitter.com/ocdqblog

LinkedIn
linkedin.com/in/jimharris

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 2

Let the Adventures Begin . . .
This will be a vendor‐neutral presentation:

Focusing on general methodology of data profiling
and common functionality of data profiling tools

Discussing how a data profiling tool helps automate
some of the work needed for preliminary data analysis

Reviewing fictional data and results produced by a
fictional data profiling tool to illustrate basic concepts

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 3

Understanding Your Data
Understanding your data is essential to using it
effectively and improving its quality

You need a reality check for the perceptions and
assumptions you have about the quality of your data

You need to prepare meaningful questions to ask your
business analysts and subject matter experts

There is simply no substitute for data analysis

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 4

Profiling Your Data
Data profiling includes many types of analysis such as:

Verify data matches the metadata that describes it

Identify representations for the absence of data

Identify potential default and invalid values

Check data formats for inconsistencies

Assess domain, structural, and relational integrity

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 5

Getting Your Data Freq On
Data profiling tools can help you by automating some
of the grunt work needed to begin your data analysis

One of their basic features is the ability to generate
statistical summaries and frequency distributions for
the unique values and formats found within your fields

Therefore, I like to refer to using a data profiling tool as:
“Getting Your Data Freq On”

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 6

Let Me Count The Ways

NULL – record count of NULL values Cardinality – count of the number of
distinct actual values
Missing – record count of Missing values
(i.e., non‐NULL absence of data) Uniqueness – percentage calculated as
Cardinality divided by total record count
Actual – record count of Actual values
(i.e., non‐NULL and non‐Missing) Distinctness – percentage calculated as
Cardinality divided by Actual
Completeness – percentage calculated as
Actual divided by total record count
Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 7

You Uniquely Complete Me
Completeness and
Uniqueness are useful in
evaluating potential key
fields and especially a
single primary key,
which should be both:
100% Complete
100% Unique

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 8

It’s a Distinct Possibility

Distinctness can be useful
in evaluating the potential
for duplicate records
< 100% Distinct means some
distinct actual values occur on
more than one record
Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 9

Gimme the lo down, Drill‐down

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 10

Freq’ing Distribution of Values

Frequency distribution of
values is useful for fields
with a low cardinality
Extremely low cardinality
might be an indication of
default values

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 11

Reviewing the Top N List

Reviewing the Top N most
frequently occurring values
Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 12

Freq’ing Distribution of Formats

Frequency distribution of formats is useful for fields having
both a high cardinality and free‐form values
Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 13

Unlocking the Combination

Combination of values
and formats can help with
unlocking the mystery of
more complex fields

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 14

. . . the Adventures Conclude
What can just your analysis of data tell you about it?

Understand your data better by first looking at it from a
starting point of blissful ignorance and curiosity

A tool can help automate some of the grunt work, but
the actual data analysis can not be automated

Your analytical goal is not to try to find answers, but to
discover the right questions

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 15

A brief introduction to Data Quality rule development and implementation covering: - What are Data Quality Rules. - Examples of Data Quality Rules. - What are the benefits of rules. - How can I create my own rules? - What alternate approaches are there to building my own rules? The presentation also includes a very brief overview of our Data Quality Rule services. For more information on this please contact us.

Data Quality Definitions

Michael Küsters

Successful Big Data initiatives rely on accurate, complete data, but the information they draw on is often not validated when it enters an organization. In this session we will look at the challenges big data brings to an organization, and how data quality principles are adapting to ensure business goals and return on investments in big data are realised. We will cover: - Challenges of big data - Turning data lakes into reservoirs - How data quality tools are adapting - Why data governance disciplines remain crucial

Lecture 21

Shani729

Differential Privacy Case Studies (CMU-MSR Mindswap on Privacy 2007)

Denny Lee

Lecture 23

Shani729

Tamr overview

Meg Vorland

Data analytics

Canopus InfoSystems Pvt.Ltd

Foundation of data quality

Khaled Mosharraf

Enterprise Analytics: Serving Big Data Projects for Healthcare

DATA360US

Data analytics

Dr.Bhuvaneswari Velumani

Machine Learning and Multi Drug Resistant(MDR) Infections case study

AlgoAnalytics Financial Consultancy Pvt. Ltd.

Data quality management Basic

Khaled Mosharraf

Data quality and data profiling

Shailja Khurana

2007 Tidc India Profilingdanrinkes

What's hot

Building a Data Quality Program from Scratchdmurph4

Data quality - The True Big Data Challenge

Stefan Kühn

Data quality architecture

anicewick

Sound Data Quality for CRMDivya Malik

Data profiling-best-practices

Blaise Cheuteu

Data Quality Technical ArchitectureHarshendu Desai

Lecture 22

Shani729

Data Quality: The Data Science struggle nobody mentions - Data Science MeetUp...

University of Twente

Introduction to Data Analytics

Dr. C.V. Suresh Babu

Big Data Expo 2015 - Trillium software Big Data and the Data Quality

BigDataExpo

Lecture 21

Shani729

Differential Privacy Case Studies (CMU-MSR Mindswap on Privacy 2007)

Denny Lee

Lecture 23

Shani729

Tamr overview

Meg Vorland

Data analytics

Canopus InfoSystems Pvt.Ltd

Foundation of data quality

Khaled Mosharraf

Enterprise Analytics: Serving Big Data Projects for Healthcare

DATA360US

Data analytics

Dr.Bhuvaneswari Velumani

Machine Learning and Multi Drug Resistant(MDR) Infections case study

AlgoAnalytics Financial Consultancy Pvt. Ltd.

Data quality management Basic

Khaled Mosharraf

What's hot (20)

Building a Data Quality Program from Scratch

Data quality - The True Big Data Challenge

Data quality architecture

Sound Data Quality for CRM

Data profiling-best-practices

Data Quality Technical Architecture

Lecture 22

Data Quality: The Data Science struggle nobody mentions - Data Science MeetUp...

Introduction to Data Analytics

Big Data Expo 2015 - Trillium software Big Data and the Data Quality

Lecture 21

Differential Privacy Case Studies (CMU-MSR Mindswap on Privacy 2007)

Lecture 23

Tamr overview

Data analytics

Foundation of data quality

Enterprise Analytics: Serving Big Data Projects for Healthcare

Data analytics

Machine Learning and Multi Drug Resistant(MDR) Infections case study

Data quality management Basic

Viewers also liked

Data quality and data profiling

Shailja Khurana

2007 Tidc India Profilingdanrinkes

Marketing Data Utilization

CRMIT

Data is an integral and invaluable asset of any organization to pursue the vision of value-creation for customers. Given its significance, efficient management of organizational data can be a potential differentiator between thriving and failing in today’s competitive world. Leverage CRMIT’s Marketing Utilization Services for CRM, including 1. Data Utilization - Data sampling, data de-duplication, data scrubbing & data cleansing. 2. Data enrichment - Data profiling and data segmentation 3. Campaign management – E-campaigns, lead nurturing and lead scoring 4. Analytics - Analysis to Insights

Uncover Untold Stories in Your Data: A Deep Dive on Data Profiling

Josiah Renaudin

How well do you know your data? Organizations are discovering the value in their data—as evidence of what they have done and a clue to how they can improve the bottom line. With the increase in analytics, it is no secret that there are more eyes on the data. And analyzing data can give valuable insight into patterns that drive efficiencies or errors. It is important to use this information and make sure it is being used correctly. However, excavating the data is not always as simple as it seems. Catherine Cruz Agosto and Shauna Ayers are your guides as they define data profiling and its importance, delve into different strategies you can use, and discuss how to get the most out of your data. Come and learn useful tools and strategies you can take back to get to know and better use your data.

NCDM Datamining Case Study 2010

Jim Stafford

Bitcoin, Transaction Fees and The Cost of Poor Quality

RSky215

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

huguk

The task of “data profiling”—assessing the overall content and quality of a data set—is a core aspect of the analytic experience. Traditionally, profiling was a fairly cut-and-dried task: load the raw numbers into a stat package, run some basic descriptive statistics, and report the output in a summary file or perhaps a simple data visualization. However, data volumes can be so large today that traditional tools and methods for computing descriptive statistics become intractable; even with scalable infrastructure like Hadoop, aggressive optimization and statistical approximation techniques must be used. In this talk Sean will cover technical challenges in keeping data profiling agile in the Big Data era. He will discuss both research results and real-world best practices used by analysts in the field, including methods for sampling, summarizing and sketching data, and the pros and cons of using these various approaches. Sean is Trifacta’s Chief Technical Officer. He completed his Ph.D. at Stanford University, where his research focused on user interfaces for database systems. At Stanford, Sean led development of new tools for data transformation and discovery, such as Data Wrangler. He previously worked as a data analyst at Citadel Investment Group.

Telling Visual Stories About Data

Congressional Budget Office

Exposing Your Hidden Costs of Performance

Juran Global

An ongoing challenge today for most organizations is to do more with less. Many companies are spending more time with their customers, while customers are demanding competitive pricing on products and services. As the planned profit margin erodes, the question management is posing to their staff is, "How do we maintain our margins and still meet the customers’ demands?" Basically, how do we leave less dollars on the table? The answer is Cost of Poor Quality (COPQ) analysis, which can be used to identify and reduce operational wastes while maintaining margins. Dr. Joseph DeFeo, Chairman and CEO of Juran Global, shares: * Typical misconceptions about quality. * How COPQ affects the bottom line. * How to identify the "tip of the iceberg." * The costs hidden in the bottom of the iceberg. * How to estimate costs using total resources and unit costs.

Cost of poor quality

Abdullah Sasy

Cost of poor quality presentation5Imran Jamil

Cost of-poor-quality - juran institute

Manish Chaurasia

Fifth Elephant 2014 talk - Crafting Visual Stories with Data

Amit Kapoor

Quality is a costbatch18

QUALITY & COSTRajeev Sharan

Crafting Visual Stories with Data

Amit Kapoor

Data visualization has enabled us to compress data and express them visually in many interesting new ways. It is often cited that we are trying to tell stories through them. Is that really the case? How can we ensure that the audience is able to retain, recall and retell our data-driven stories. Using examples and videos learning from different storytelling mediums, I walk through why stories are important and what we can learn about how stories work from these mediums. I then detail out a framework built on the See the data | Show the Visual | Tell the Story | Engage the Audience paradigm to convert the data in to a data-visual-story. This slide deck was used in Bangalore Meetup - Crafting Visual Stories with Data - in March 2014 @ InMobi's Bangalore Office.

Big Data Profiling

eXascale Infolab

Data profiling comprises a broad range of methods to efficiently analyze a given data set. In a typical scenario, which mirrors the capabilities of commercial data profiling tools, tables of a relational database are scanned to derive metadata, such as data types and value patterns, completeness and uniqueness of columns, keys and foreign keys, and occasionally functional dependencies and association rules. Individual research projects have proposed several additional profiling tasks, such as the discovery of inclusion dependencies or conditional functional dependencies. Data profiling deserves a fresh look for two reasons: First, the area itself is neither established nor defined in any principled way, despite significant research activity on individual parts in the past. Second, current data profiling techniques hardly scale beyond what can only be called small data. Finally, more and more data beyond the traditional relational databases are being created and beg to be profiled. The talk proposes new research directions and challenges, including interactive and incremental profiling and profiling heterogeneous and non-relational data. Speaker: Felix Naumann studied mathematics, economy, and computer sciences at the University of Technology in Berlin. After receiving his diploma (MA) in 1997 he joined the graduate school "Distributed Information Systems" at Humboldt University of Berlin. He completed his PhD thesis on "Quality-driven Query Answering" in 2000. In 2001 and 2002 he worked at the IBM Almaden Research Center on topics around data integration. From 2003 - 2006 he was assistant professor for information integration at the Humboldt-University of Berlin. Since then he holds the chair for information systems at the Hasso Plattner Institute at the University of Potsdam in Germany.

Cost of Poor quality

Raghvendra Rangaswamy Gopal

Cost of qualityMohit Singla

Cost of qualityHassan Asif

Viewers also liked (20)

Data quality and data profiling

2007 Tidc India Profiling

Marketing Data Utilization

Uncover Untold Stories in Your Data: A Deep Dive on Data Profiling

NCDM Datamining Case Study 2010

Bitcoin, Transaction Fees and The Cost of Poor Quality

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

Telling Visual Stories About Data

Exposing Your Hidden Costs of Performance

Cost of poor quality

Cost of poor quality presentation5

Cost of-poor-quality - juran institute

Fifth Elephant 2014 talk - Crafting Visual Stories with Data

Quality is a cost

QUALITY & COST

Crafting Visual Stories with Data

Big Data Profiling

Cost of Poor quality

Cost of quality

Similar to Adventures in Data Profiling

Privacy in AI/ML Systems: Practical Challenges and Lessons Learned

Krishnaram Kenthapadi

How do we protect the privacy of users when building large-scale AI based systems? How do we develop machine learning models and systems taking fairness, accuracy, explainability, and transparency into account? Model fairness and explainability and protection of user privacy are considered prerequisites for building trust and adoption of AI systems in high stakes domains. We will first motivate the need for adopting a “fairness, explainability, and privacy by design” approach when developing AI/ML models and systems for different consumer and enterprise applications from the societal, regulatory, customer, end-user, and model developer perspectives. We will then focus on the application of privacy-preserving AI techniques in practice through industry case studies. We will discuss the sociotechnical dimensions and practical challenges, and conclude with the key takeaways and open challenges.

IDOL presentation

Andrey Karpov

Analytics At Work T. DavenportSaurabh Shah

Thomas Davenport: Analytics at Work: How to Make Better Decisions and Get Bet...

SAS Institute India Pvt. Ltd

Thomas H. Davenport, best-selling co-author of Competing on Analytics and Analytics at Work, and President's Distinguished Professor at Babson College presented at the Premier Business Leadership Series 2010 http://www.sas.com/theserieshk Davenport will present straightforward, practical advice from his new book Analytics at Work: Smarter Decisions, Better Results, including laying out a plan of action for deploying and succeeding with business analytics inside your company. Learn how to: - Use an analytics approach to run your business. Put the right assets in place and deploy them most effectively. - Launch an analytics initiative. - Sustain an analytics focus over time. - Evaluate your organisation's current analytical capabilities. - Use analytics to make better decisions.

Information Management and Analytics AKAGroup

Big Data, Analytics, and Social Good - The Challenges, The OpportunityJaime Fitzgerald

Week2day2 communicating data for impact

Nishant Kumar

Why does telling a story with your data matters Explain the impo.docx

franknwest27899

Why does telling a story with your data matters? Explain the importance of accurate data in today's business environment. Data Storytelling: What It Is, Why It Matters Telling a compelling story with your data helps you get your point across effectively. Here are four tips to keep your data from getting lost in translation. 8 Non-Tech Skills IT Pros Need To Succeed (Click image for larger view and slideshow.) Organizations can do a lot more with their data if they understand it better than they do. While businesses continue to invest dollars in business intelligence (BI) and analytics tools, they aren't necessarily getting the information they need to improve business decision-making. Data visualizations help by transforming complex information into something easier to understand. However, two people can interpret the same data visualization differently. Notably, data visualizations tend to answer "what" questions, but they don't tend to explain the "why," or provide other contextual information. Data storytelling does exactly that. "Data storytelling weaves data and visualizations into a narrative tailored to a specific audience in order to convey credibility in the analytical approach, confidence in the results, and a compelling set of insights that is actionable to the audience." said Ryan Fuller, general manager at Microsoft and former CEO and cofounder of enterprise analytics company VoloMetrix, in an interview. "The narrative is the key vehicle to convey insights, and the visualizations are important proof points to back up the narrative." Executives, managers, and employees have always told stories as part of their everyday work experience, but they are increasingly being required to use data to support their points of view, claims, and recommendations. The danger, of course, is data can be tortured into saying almost anything. "One of the biggest mistakes is trying to fit the data to the story, which often results in a jumbled narrative that doesn't arrive at a compelling conclusion," said Francois Ajenstat, VP of product development at BI and analytics solution provider Tableau , in an interview. "Always start with the data, then build your story around it, rather than vice versa." After speaking with experts in data science and analytics, we've developed the following four tips to help guide your data storytelling. 1. General Storytelling Rules Apply Effective data storytelling is a lot like storytelling generally. The data story should have a beginning, a middle, and an end. It should also include a thesis (or a hypothesis), supporting facts (data), a logical structure, and a compelling presentation. Yet, all too often, those responsible for analyzing data are unable to present it in a way that's meaningful to the audience. "A common mistake is spending too much time on the technical aspect or methodology and not providing much creativity in pointing out how the data can help the business," said David Liebskind, VP of anal.

Big Data v2.pptx

Pecific University

Complete Guide to Data QualityJaime Muirhead SVP Sales, RingLead

Data quality metrics infographic

Intellspot

How Data as a Service can make your campaigns more successful - Dun and Brads...

B2B Marketing

Many organisations have invested in technology to help with their lead generation programmes. However, this often has not delivered the anticipated benefits and as a result, a high proportion of marketing generated leads are not worked by sales, wasting valuable budget. Sales and marketing alignment seems as far away as ever. New Data as a Service (DaaS) based solutions allow you to integrate real time external data, directly into your marketing and sales automation systems to transform your campaigns, access insights not previously available and generate the right leads for your business. In this session you will learn: What DaaS is and how it works How to generate the right leads to drive sales and marketing alignment Know all about your inbound leads, in an instant Tailor your conversations to accelerate the sales cycle How you can link social media data into your CRM

Great Learning Amit Sharma March 2018

Amit Sharma

2017 06-14-getting started with data science

Thinkful

For the Love of Big Data

Robert Sutor

4 Critical Requirements for Building Truly Intelligent AI Models

Innodata, Inc

HPE IDOL Technical Overview - july 2016

Andrey Karpov

Search and Analytics Platform for Text and Rich Media Open Innovation is transforming everything Connected people, apps and things generating massive data in many forms How do you bridge the gap between data and outcomes? Augmented Intelligence power apps for competitive advantage Machine Learning at the Service of Business Augmented Intelligence HPE Big Data Advanced Analytics Software Solutions Strong information and weak information HPE IDOL: Natural Language Processing (NLP) engine

Healthcare Best Practices in Data Warehousing & Analytics

Dale Sanders

Dark data

Amir Sedighi

Of Unicorns, Yetis, and Error-Free Datasets (or what is data quality?)

Gianluca Tarasconi

Similar to Adventures in Data Profiling (20)

Privacy in AI/ML Systems: Practical Challenges and Lessons Learned

IDOL presentation

Analytics At Work T. Davenport

Thomas Davenport: Analytics at Work: How to Make Better Decisions and Get Bet...

Information Management and Analytics

Big Data, Analytics, and Social Good - The Challenges, The Opportunity

Week2day2 communicating data for impact

Why does telling a story with your data matters Explain the impo.docx

Big Data v2.pptx

Complete Guide to Data Quality

Data quality metrics infographic

How Data as a Service can make your campaigns more successful - Dun and Brads...

Great Learning Amit Sharma March 2018

2017 06-14-getting started with data science

For the Love of Big Data

4 Critical Requirements for Building Truly Intelligent AI Models

HPE IDOL Technical Overview - july 2016

Healthcare Best Practices in Data Warehousing & Analytics

Dark data

Of Unicorns, Yetis, and Error-Free Datasets (or what is data quality?)

Recently uploaded

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

The Future of Platform Engineering

Jemma Hussein Allen

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

ODC, Data Fabric and Architecture User Group

CatarinaPereira64715

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Product School

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

Recently uploaded (20)

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

The Future of Platform Engineering

Connector Corner: Automate dynamic content and events by pushing a button

ODC, Data Fabric and Architecture User Group

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

Leading Change strategies and insights for effective change management pdf 1.pdf

Search and Society: Reimagining Information Access for Radical Futures

DevOps and Testing slides at DASA Connect

Key Trends Shaping the Future of Infrastructure.pdf

Neuro-symbolic is not enough, we need neuro-*semantic*

FIDO Alliance Osaka Seminar: Overview.pdf

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

"Impact of front-end architecture on development cost", Viktor Turskyi

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

How world-class product teams are winning in the AI era by CEO and Founder, P...

Epistemic Interaction - tuning interfaces to provide information for AI support

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Adventures in Data Profiling

1. Jim Harris Blogger‐in‐Chief www.ocdqblog.com Jim Harris Digitally signed by Jim Harris DN: cn=Jim Harris, o=Obsessive-Compulsive Data Quality (OCDQ), ou, email=jim.harris@ocdqblog. com, c=US Date: 2010.03.04 10:55:20 -06'00'

2. Jim Harris Blogger‐in‐Chief www.ocdqblog.com E‐mail jim.harris@ocdqblog.com Twitter twitter.com/ocdqblog LinkedIn linkedin.com/in/jimharris Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 2

3. Let the Adventures Begin . . . This will be a vendor‐neutral presentation: Focusing on general methodology of data profiling and common functionality of data profiling tools Discussing how a data profiling tool helps automate some of the work needed for preliminary data analysis Reviewing fictional data and results produced by a fictional data profiling tool to illustrate basic concepts Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 3

4. Understanding Your Data Understanding your data is essential to using it effectively and improving its quality You need a reality check for the perceptions and assumptions you have about the quality of your data You need to prepare meaningful questions to ask your business analysts and subject matter experts There is simply no substitute for data analysis Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 4

5. Profiling Your Data Data profiling includes many types of analysis such as: Verify data matches the metadata that describes it Identify representations for the absence of data Identify potential default and invalid values Check data formats for inconsistencies Assess domain, structural, and relational integrity Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 5

6. Getting Your Data Freq On Data profiling tools can help you by automating some of the grunt work needed to begin your data analysis One of their basic features is the ability to generate statistical summaries and frequency distributions for the unique values and formats found within your fields Therefore, I like to refer to using a data profiling tool as: “Getting Your Data Freq On” Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 6

7. Let Me Count The Ways NULL – record count of NULL values Cardinality – count of the number of distinct actual values Missing – record count of Missing values (i.e., non‐NULL absence of data) Uniqueness – percentage calculated as Cardinality divided by total record count Actual – record count of Actual values (i.e., non‐NULL and non‐Missing) Distinctness – percentage calculated as Cardinality divided by Actual Completeness – percentage calculated as Actual divided by total record count Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 7

8. You Uniquely Complete Me Completeness and Uniqueness are useful in evaluating potential key fields and especially a single primary key, which should be both: 100% Complete 100% Unique Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 8

9. It’s a Distinct Possibility Distinctness can be useful in evaluating the potential for duplicate records < 100% Distinct means some distinct actual values occur on more than one record Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 9

11. Freq’ing Distribution of Values Frequency distribution of values is useful for fields with a low cardinality Extremely low cardinality might be an indication of default values Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 11

13. Freq’ing Distribution of Formats Frequency distribution of formats is useful for fields having both a high cardinality and free‐form values Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 13

14. Unlocking the Combination Combination of values and formats can help with unlocking the mystery of more complex fields Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 14

15. . . . the Adventures Conclude What can just your analysis of data tell you about it? Understand your data better by first looking at it from a starting point of blissful ignorance and curiosity A tool can help automate some of the grunt work, but the actual data analysis can not be automated Your analytical goal is not to try to find answers, but to discover the right questions Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 15

Adventures in Data Profiling

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Adventures in Data Profiling

Similar to Adventures in Data Profiling (20)

Recently uploaded

Recently uploaded (20)

Adventures in Data Profiling