Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov

•Download as PPTX, PDF•

1 like•696 views

The quality of business decisions, machine learning insights, and executive reports depend on the quality and integrity of the underlying data. There are many ways that data can get corrupted in an analytical data platform from de-synchronization with the system-of-record to defects in data pipelines. We will show how to detect and prevent data corruption with automation, open source tools, and machine learning.

2
Data Quality
Automatic enforcement in real-time with machine learning.
Max Martynov, VP of Technology

Introducing Grid Dynamics technology services
Digital transformation Big data, real time analytics, ML & AI
Microservices replatforming DevOps & cloud enablement
Open Source Cloud-ready Scalable Automated

12 years of
experience in digital
transformation.

9.8
9.3 9.6 9.4
10.1
17.5
16.9
8.9 9.6 9.2
10.3
9.8
5.1
4.1 3.9 4 3.9
4.5
7.5 7.1
3.8 4.1 4 4.5
4.2
2.3
0
5
10
15
20
7/1/19 7/2/19 7/3/19 7/4/19 7/5/19 7/6/19 7/7/19 7/8/19 7/9/19 7/10/19 7/11/19 7/12/19 7/13/19 7/14/19
Retailer X, Daily Sales – Executive Summary
Revenue, $M Gross Profit, $M
weekend weekend

Cloud
Data Lake
EDW
DBDBDB FileFileFile MQ CloudCloud
AppAppAPI

Cloud
EDW
DBDBDB FileFileFile MQ CloudCloud
AppAppAPI
AppAppApp
Data Lake

1 0 1 1
0 1 1 0
1 0 1 0
1 0 0 1
1. Trust is hard to build and easy to lose.
2. Distrust in data slows down decisions.
3. Slow decisions prevent agility.

Data corruption reasons
1. Code 2. Data Sources 3. Infrastructure

Test environment
Input
Actual
Expected
ETL
code
compareTest suite
run test
Traditional approach to testing

Data quality goals
Detect
data corruption
Prevent
it from spreading
Alert
support team

Production data lake
Real-time data quality enforcement
DBDBDB
FileFileFile
MQ
AppAppAPI
data processing job

Production data lake
Real-time data quality enforcement
DBDBDB
FileFileFile
MQ
AppAppAPI
data processing job data quality job

Production data lake
Real-time data quality enforcement
DBDBDB
FileFileFile
MQ
AppAppAPI
data processing job data quality job
alert &
stop pipeline
alert &
continue
x

Data Lake
Data
source
data
1. Compare with
SoR
2. Validate
business rules
3. Data profiling and
anomaly detection
Main data processing pipeline
confidence
data
confidence

1. Control divergence from SoR
Data Lake
Data
source
Imported
dataset
Compare data in
SoR and data lake
1. Validate correctness of
import.
2. Prevent stale data.
3. Prevent corruption
accumulation in stream
processing use cases.
4. Check data before it gets in
the lake.

2. Validate business rules
Data Lake
Dataset
Check for nulls and
data ranges
1. Enforce schema.
2. Check for nulls.
3. Validate data ranges.
4. Specify and enforce
data invariants.

Anomaly detection (counts): anomaly & return

The document defines and discusses the key characteristics of data quality: accuracy, precision, relevance, completeness, consistency, transparency, and timeliness. It provides examples to illustrate each characteristic, defining them as the degree to which data matches reality (accuracy), the specificity of data values (precision), how closely data meets the needs of its consumers (relevance), how fully the needs of consumers are met (completeness), how synchronized data is across systems (consistency), the ability to trace data back to its origin (transparency), and the availability of data when it is needed (timeliness).

Big Data Expo 2015 - Trillium software Big Data and the Data Quality

BigDataExpo

Successful Big Data initiatives rely on accurate, complete data, but the information they draw on is often not validated when it enters an organization. In this session we will look at the challenges big data brings to an organization, and how data quality principles are adapting to ensure business goals and return on investments in big data are realised. We will cover: - Challenges of big data - Turning data lakes into reservoirs - How data quality tools are adapting - Why data governance disciplines remain crucial

Data quality and data profiling

Shailja Khurana

This document discusses data quality and data profiling. It begins by describing problems with data like duplication, inconsistency, and incompleteness. Good data is a valuable asset while bad data can harm a business. Data quality is assessed based on dimensions like accuracy, consistency, completeness, and timeliness. Data profiling statistically examines data to understand issues before development begins. It helps assess data quality and catch problems early. Common analyses include analyzing null values, keys, formats, and more. Data profiling is conducted using SQL or profiling tools during requirements, modeling, and ETL design.

Data Quality Strategy: A Step-by-Step Approach

FindWhitePapers

Data quality is critical for organizations to realize full benefits from their enterprise systems. A data quality strategy involves making decisions across six factors: context, storage, data flow, workflow, stewardship, and continuous monitoring. These factors determine the processes, solutions, and resources needed to improve data quality. The document provides guidance on developing a comprehensive data quality strategy.

Data Quality

Michael Collins

The document discusses the importance of data quality and having a data strategy. It notes that poor quality data can lead to skewed analysis, improper campaign targeting, and wasted resources. It also outlines steps for improving data quality such as data audits, profiling data sources, data cleansing, and establishing business rules for data management. Maintaining high quality data requires both internal processes and leveraging external data services and is a key part of building data as a strategic asset for the business.

COVID Data Challenges - Updated 2021

303Computing

Data Quality Presentation

Stephen McCarthy

This document discusses data quality and its importance for business decision making. It defines data quality as ensuring information is fit for its intended purpose and helps data consumers make the right decisions. Poor data quality can significantly impact business performance, with 75% of companies reporting financial losses due to low quality data. The document outlines different data quality needs and metrics for various use cases and decision makers. It also presents examples of companies that have benefited financially from implementing thorough data quality management programs.

Sound Data Quality for CRM

Divya Malik

Poor data quality costs businesses significantly through wasted labor, lost productivity and direct financial losses. Rapid changes in circumstances like addresses, phone numbers, jobs and relationships mean that master data changes at a rate of around 2% per month. Siebel's data management solution and new data quality products from Oracle help address these issues through comprehensive data profiling, cleansing, matching and enrichment capabilities. Regular data stewardship is important for ongoing monitoring and governance to maintain high quality master data.

This document discusses data quality testing. It begins by defining data quality and listing its key dimensions such as accuracy, consistency, completeness and timeliness. It then notes common business problems caused by poor data quality and the benefits of improving data quality. Key aspects of data quality testing covered include planning, design, execution, monitoring and challenges. Best practices emphasized include understanding the business, planning for data quality early, being proactive about data growth and thoroughly understanding the data.

Data Quality Management - Data Issue Management & Resolutionn / Practical App...

Burak S. Arikan

One of the key stepping stones to turn the theoretical Data Governance concept to reality is the implementation of data issue management and resolution (IMR) process which includes tools, processes, governance and most importantly persistence to get to the bottom of the each data quality issue. This presentation lays down the basic components of IMR process and tries to guide practitioners. This process was applied along with an in-house configured SharePoint management tool with workflows.

The data quality challenge

Lenia Miltiadous

This document discusses data quality and its importance for businesses. It provides a case study of how British Airways improved data quality which increased efficiency and decision making. An insurance case study shows how improving data quality led to better customer understanding and risk assessment. Finally, the document outlines key drivers of data quality including regulatory compliance, business intelligence, and customer-centric models.

Data quality overview

Alex Meadows

This document discusses data quality and provides facts about the high costs of poor data quality to businesses and the US economy. It defines data quality as ensuring data is "fit for purpose" by measuring it against its intended uses and dimensions of quality. The document outlines best practices for measuring data quality including profiling data to understand metadata and trends, using statistical process control, master data management to create standardized "gold records", and implementing a data governance program to centrally manage data quality.

Data quality and bi

jeffd00

Data Quality Rules introduction

datatovalue

A brief introduction to Data Quality rule development and implementation covering: - What are Data Quality Rules. - Examples of Data Quality Rules. - What are the benefits of rules. - How can I create my own rules? - What alternate approaches are there to building my own rules? The presentation also includes a very brief overview of our Data Quality Rule services. For more information on this please contact us.

Enterprise Analytics: Serving Big Data Projects for Healthcare

DATA360US

Why You Need to Govern Big Data

IBM Analytics

Unlocking Business Value Using Data

Splunk

This document discusses how businesses can unlock value from data using analytics. It notes that the volume and variety of available data is growing rapidly due to sources like IoT. Analytics can help businesses make faster, better decisions by delivering insights across departments like marketing, sales, and customer experience. The goal is for businesses to be able to manage operations in real-time using analytics to track things like customer journeys, behavior, and sales. This requires flexible technology and expertise to collect data from anywhere, correlate all information, and deliver agile analytics.

( Big ) Data Management - Data Quality - Global concepts in 5 slides

Nicolas Sarramagna

This document discusses data quality and management. It covers: - The importance of data quality metrics like accuracy, completeness, conformity, and timeliness for verifying data. - Why data quality is needed, as poor quality data can cause issues in applications and business processes. - Approaches to improving data quality, including identifying quality issues, acting on the data directly through normalization and rules, or acting on underlying processes.

Data Quality Integration (ETL) Open Source

Stratebi

Data quality is the process of ensuring data values conform to business requirements. It is important for business intelligence projects which involve data integration from multiple sources. Pentaho Data Integration and DataCleaner are open source tools that can be used together for data integration and quality tasks like extraction, transformation, loading, cleansing and profiling. Performing data quality as part of the ETL process through tools like these helps standardize processes and improve scalability.

COVID-19 - How to Improve Outcomes By Improving Data

303Computing

Data Quality Testing Generic (http://www.geektester.blogspot.com/)

raj.kamal13

The document discusses the importance of data quality and information management for business success. It states that virtually all business today relies on managing information effectively, as poor data quality can cost companies billions per year. It also outlines various categories and types of data quality checks that should be tested, such as row counts, completeness, consistency, validity, redundancy, and integrity.

The Merger is Happening, Now What Do We Do?

DATUM LLC

This was presented on October 24, 2018 at the ASUG EIM Conference. One of the many challenges presented by an acquisition and divestiture event is unifying disparate data and integrating systems together. If you are leading an integration, you may have more questions than answers on how to approach this event. Learn how to best leverage the momentum and budgets that accompany these activities to jump start good governance practices up front, as well as how to measure the return on investment, ensuring data and EIM professionals' ongoing success.

Webinar: Decoding the Mystery - How to Know if You Need a Data Catalog, a Dat...

DATAVERSITY

This document discusses the importance of metadata and data governance. It describes how a data catalog can consolidate metadata from various sources like a business glossary, data dictionary, and data profiling. Automating data lineage is key to harvesting metadata at scale and establishing relationships between different metadata objects. When integrated in a data catalog, metadata provides a single source of truth about an organization's data that improves data literacy and trust.

Тестирование данных с помощью Data Quality Services (MS SQL 12)

SQALab

This document summarizes a presentation about using Microsoft's Data Quality Services (DQS) for data quality assurance. It introduces DQS and its components for cleansing, matching, profiling, and monitoring data. DQS uses a knowledge-based approach where domain experts build knowledge bases of valid values and rules. It then uses these knowledge bases to identify data quality issues like invalid, inaccurate, incomplete or inconsistent data. The document provides examples of how DQS could help quality assurance engineers by introducing structure, measurement, reusability and manageability to the data quality process.

Five Things to Consider About Data Mesh and Data Governance

DATAVERSITY

Data mesh was among the most discussed and controversial enterprise data management topics of 2021. One of the reasons people struggle with data mesh concepts is we still have a lot of open questions that we are not thinking about: Are you thinking beyond analytics? Are you thinking about all possible stakeholders? Are you thinking about how to be agile? Are you thinking about standardization and policies? Are you thinking about organizational structures and roles? Join data.world VP of Product Tim Gasper and Principal Scientist Juan Sequeda for an honest, no-bs discussion about data mesh and its role in data governance.

Introduction to Data Governance

John Bao Vuu

This introduction to data governance presentation covers the inter-related DM foundational disciplines (Data Integration / DWH, Business Intelligence and Data Governance). Some of the pitfalls and success factors for data governance. • IM Foundational Disciplines • Cross-functional Workflow Exchange • Key Objectives of the Data Governance Framework • Components of a Data Governance Framework • Key Roles in Data Governance • Data Governance Committee (DGC) • 4 Data Governance Policy Areas • 3 Challenges to Implementing Data Governance • Data Governance Success Factors

Predictive analytics in decision management systems

Decision Management Solutions

To ensure that Decision Management Systems are analytic and adaptive you must embed the results of data mining and predictive analytics in them. In this webinar you will learn what can be discovered using data mining and predictive analytic techniques and how this can be applied to the decision-making embedded in Decision Management Systems. The role of analytics in predicting risk, fraud and opportunity and the importance of continuous improvement and learning is also be covered.

Next generation Data Governance

Vladimiro Borsi

This whitepaper outlines a reference architecture for real-time data governance using DataOps principles. The architecture defines logical units including lines of business, spaces, datasets, actors and identities to enable federated governance over heterogeneous systems. It allows each line of business to be aligned with the appropriate data, applications, rules and processes. The architecture specifies roles for access management and enforcement of governance policies across business units and technical environments.

"Implementing data quality automation with open source stack" - Max Martynov,...

Grid Dynamics

CWIN17 India / Bigdata architecture yashowardhan sowale

Capgemini

The document provides an overview of big data architecture and concepts. It discusses big data dimensions like volume, velocity, variety, veracity and value. It also outlines applications of big data analytics in various domains like homeland security, finance, healthcare, telecom etc. The document presents a general reference architecture for big data analytics and describes layers for data ingestion, storage, processing, governance and access. It provides conceptual and logical views of a business data lake reference architecture.

What's hot

Data Quality

Vijaya K

Data Quality Management - Data Issue Management & Resolutionn / Practical App...

Burak S. Arikan

The data quality challenge

Lenia Miltiadous

Data quality overview

Alex Meadows

Data quality and bi

jeffd00

Data Quality Rules introduction

datatovalue

Enterprise Analytics: Serving Big Data Projects for Healthcare

DATA360US

Why You Need to Govern Big Data

IBM Analytics

Unlocking Business Value Using Data

Splunk

( Big ) Data Management - Data Quality - Global concepts in 5 slides

Nicolas Sarramagna

Data Quality Integration (ETL) Open Source

Stratebi

COVID-19 - How to Improve Outcomes By Improving Data

303Computing

Data Quality Testing Generic (http://www.geektester.blogspot.com/)

raj.kamal13

The Merger is Happening, Now What Do We Do?

DATUM LLC

Webinar: Decoding the Mystery - How to Know if You Need a Data Catalog, a Dat...

DATAVERSITY

Тестирование данных с помощью Data Quality Services (MS SQL 12)

SQALab

Five Things to Consider About Data Mesh and Data Governance

DATAVERSITY

Introduction to Data Governance

John Bao Vuu

Predictive analytics in decision management systems

Decision Management Solutions

Next generation Data Governance

Vladimiro Borsi

What's hot (20)

Data Quality

Data Quality Management - Data Issue Management & Resolutionn / Practical App...

The data quality challenge

Data quality overview

Data quality and bi

Data Quality Rules introduction

Enterprise Analytics: Serving Big Data Projects for Healthcare

Why You Need to Govern Big Data

Unlocking Business Value Using Data

( Big ) Data Management - Data Quality - Global concepts in 5 slides

Data Quality Integration (ETL) Open Source

COVID-19 - How to Improve Outcomes By Improving Data

Data Quality Testing Generic (http://www.geektester.blogspot.com/)

The Merger is Happening, Now What Do We Do?

Webinar: Decoding the Mystery - How to Know if You Need a Data Catalog, a Dat...

Тестирование данных с помощью Data Quality Services (MS SQL 12)

Five Things to Consider About Data Mesh and Data Governance

Introduction to Data Governance

Predictive analytics in decision management systems

Next generation Data Governance

Similar to Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov

"Implementing data quality automation with open source stack" - Max Martynov,...

Grid Dynamics

CWIN17 India / Bigdata architecture yashowardhan sowale

Capgemini

Real-Time Analytics for Industries

Avadhoot Patwardhan

Cloud & Big Data - Digital Transformation in Banking

Sutedjo Tjahjadi

Keeping the Pulse of Your Data: Why You Need Data Observability

Precisely

With the explosive growth of DataOps to drive faster and better-informed business decisions, proactively understanding the health of your data is more important than ever. Data observability is one of the foundational capabilities of DataOps and an emerging discipline used to expose anomalies in data by continuously monitoring and testing data using artificial intelligence and machine learning to trigger alerts when issues are discovered. Join Paul Rasmussen and Shalaish Koul from Precisely, to learn how data observability can be used as part of a DataOps strategy to prevent data issues from wreaking havoc on your analytics and ensure that your organization can confidently rely on the data used for advanced analytics and business intelligence. Topics you will hear addressed in this webinar: Data observability – what is it and how it is different from other monitoring solutions Why now is the time to incorporate data observability into your DataOps strategy How data observability helps prevent data issues from impacting downstream analytics Examples of how data observability can be used to prevent real-world issues

Big data and the data quality imperative

Trillium Software

The document discusses the challenges of big data and data quality. It defines big data as large volumes of data that are difficult to process using traditional database systems. Big data comes from various sources like social media, machines, and open data. It highlights that poor data quality will undermine the value of big data investments, and that data quality foundations are needed to build successful big data and analytics programs. Effective data integration, profiling, standardization, and governance are critical to addressing the data quality imperative of big data.

What is Big Data - Edvicon

edviconin

Fundamentals of Big Data

The Wisdom Daily

Big data refers to the massive amounts of data that are being created every day from sources like mobile devices, the internet, and sensors. As data volumes and variety have increased exponentially, traditional data processing tools are no longer adequate. This has led to the development of new techniques for data storage, processing, and analysis that can handle "big data". Some key aspects of big data include volume, velocity, and variety of data. Common big data uses cases include customer analytics, fraud detection, and scientific research. Terms related to big data include data pipelines, distributed processing, machine learning, and data visualization.

Machine Data Analytics

Nicolas Morales

Gain New Insights by Analyzing Machine Logs using Machine Data Analytics and BigInsights. Half of Fortune 500 companies experience more than 80 hours of system down time annually. Spread evenly over a year, that amounts to approximately 13 minutes every day. As a consumer, the thought of online bank operations being inaccessible so frequently is disturbing. As a business owner, when systems go down, all processes come to a stop. Work in progress is destroyed and failure to meet SLA’s and contractual obligations can result in expensive fees, adverse publicity, and loss of current and potential future customers. Ultimately the inability to provide a reliable and stable system results in loss of $$$’s. While the failure of these systems is inevitable, the ability to timely predict failures and intercept them before they occur is now a requirement. A possible solution to the problem can be found is in the huge volumes of diagnostic big data generated at hardware, firmware, middleware, application, storage and management layers indicating failures or errors. Machine analysis and understanding of this data is becoming an important part of debugging, performance analysis, root cause analysis and business analysis. In addition to preventing outages, machine data analysis can also provide insights for fraud detection, customer retention and other important use cases.

Anomaly Detection in Telco Networks

Esbert Frédéric

This document discusses anomaly detection in telecommunications networks using big data analytics and machine learning. It notes that communication service providers lose over $38 billion annually to fraud and network abuse. Big data approaches are ideally suited to manage the high volume, velocity, and variety of data available to CSPs. The document promotes a company called Argyle Data that provides a platform and application suite leveraging machine learning to help CSPs detect fraudulent activity in real-time from their network data and gain other business insights. Their approach includes building a comprehensive enterprise data lake and using a native Hadoop architecture to operate at large scale.

Transforming GE Healthcare with Data Platform Strategy

Databricks

Data and Analytics is foundational to the success of GE Healthcare’s digital transformation and market competitiveness. This use case focuses on a heavy platform transformation that GE Healthcare drove in the last year to move from an On prem legacy data platforming strategy to a cloud native and completely services oriented strategy. This was a huge effort for an 18Bn company and executed in the middle of the pandemic. It enables GE Healthcare to leap frog in the enterprise data analytics strategy.

Delivering Analytics at Scale with a Governed Data Lake

Jean-Michel Franco

Building a Data Strategy – Practical Steps for Aligning with Business Goals

DATAVERSITY

Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.

The Bigger They Are The Harder They Fall

Trillium Software

This document discusses big data and the importance of data quality for big data initiatives. It defines big data as large, diverse digital data sets that require new techniques to enable capture, storage, analysis and visualization. The key challenges of big data include integrating diverse structured and unstructured data sources and ensuring high quality data. The document emphasizes that poor data quality can undermine big data analytics efforts and lead to wrong insights. It promotes establishing a data quality framework including profiling, standardization, matching and enrichment to enable valid big data analytics.

Big Data Matching - How to Find Two Similar Needles in a Really Big Haystack

Precisely

When consolidating multiple sources of information from across your organization, how do you find the records that relate to the same customer, the same company or the same product? This is the challenge faced by many businesses today when putting a data lake to work. The problem is made far worse when different systems may not have the same contact entered the same way. Is Bob Smith the same as Robert Smith? How about Dr. Robert L. Smith - is he the same person? What about Syncsort, Inc and Sinksort Corp.? Are those the same company? One must compare each individual record to every other record in the dataset with some very sophisticated matching algorithms to determine who is who, and you may have to compare the data multiple times in multiple ways to resolve each entity. Just to add to the difficulty, let’s say your organization has very large volumes of records in your data lake - you don’t have to compare a thousand records to a thousand other records multiple times - you must compare a million to a million, or 100 million to 100 million. This kind of compute intensive comparison can bring even a powerful cluster to its knees. This is a problem Syncsort customers must solve, and we have developed some very powerful and intelligent software to tackle it. View this presentation as we discuss the challenges of entity resolution at scale, how Syncsort’s Trillium data quality software line has tackled them successfully in production clusters and see a demonstration of this software in action.

Li charles biometrics analytics & big data 122013a for release

Charles Li

This document discusses leveraging biometrics and identity data through big data analytics. It describes challenges in managing identity at large scales, such as with billions of individuals. It proposes addressing this through entity resolution across multiple data sources, providing biometrics as cloud services, and monitoring systems in real-time. Big data technologies can help establish identity from diverse information and gain insights through analytics on high-volume, variety and velocity of identity and biometrics data.

Don't think DevOps think Compliant Database DevOps

Red Gate Software

DevOps and data privacy do not need to oppose each other. Rather, they can complement one another. The automation and audit trails that DevOps processes introduce to database development can ease compliance with data protection regulations and enable organizations to balance the need to deliver software faster with the requirement to protect and preserve personal data. So how can the promise of releasing changes to the database faster and easier be balanced with the need to keep data safe and remain compliant with legislation? Redgate’s Data Privacy and Protection Specialist Chris Unwin, shows how the answer lies in in going one step further than database DevOps and thinking about Compliant Database DevOps: • Introduce standardized team-based development • Automate deployments • Monitor performance and availability • Protect and preserve data

SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"

MDS ap

The document discusses digital transformation and the journey to data-driven insights. It provides an overview of data types and how data has grown exponentially over time. Both structured and unstructured data are discussed, with examples of semi-structured data like emails and reports. The value of understanding all data sources is emphasized for gaining competitive advantages through analytics. New technologies like complex event processing are enabling lightning-fast action based on diverse data. Finally, the presentation introduces SAP HANA Vora for bridging the divide between enterprise and big data systems to facilitate precision decision making.

¿En qué se parece el Gobierno del Dato a un parque de atracciones?

Denodo

Watch full webinar here: https://bit.ly/3Ab9gYq Imagina llegar a un parque de atracciones con tu familia y comenzar tu día sin el típico plano que te permitirá planificarte para saber qué espectáculos ver, a qué atracciones ir, donde pueden o no pueden montar los niños… Posiblemente, no podrás sacar el máximo partido a tu día y te habrás perdido muchas cosas. Hay personas que les gusta ir a la aventura e ir descubriendo poco a poco, pero cuando hablamos de negocios, ir a la aventura puede ser fatídico... En la era de la explosión de la información repartida en distintas fuentes, el gobierno de datos es clave para garantizar la disponibilidad, usabilidad, integridad y seguridad de esa información. Asimismo, el conjunto de procesos, roles y políticas que define permite que las organizaciones alcancen sus objetivos asegurando el uso eficiente de sus datos. La virtualización de datos, herramienta estratégica para implementar y optimizar el gobierno del dato, permite a las empresas crear una visión 360º de sus datos y establecer controles de seguridad y políticas de acceso sobre toda la infraestructura, independientemente del formato o de su ubicación. De ese modo, reúne múltiples fuentes de datos, las hace accesibles desde una sola capa y proporciona capacidades de trazabilidad para supervisar los cambios en los datos. En este webinar aprenderás a: - Acelerar la integración de datos provenientes de fuentes de datos fragmentados en los sistemas internos y externos y obtener una vista integral de la información. - Activar en toda la empresa una sola capa de acceso a los datos con medidas de protección. - Cómo la virtualización de datos proporciona los pilares para cumplir con las normativas actuales de protección de datos mediante auditoría, catálogo y seguridad de datos.

Advanced Analytics and Machine Learning with Data Virtualization

Denodo

Watch full webinar here: https://bit.ly/32c6TnG Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Attend this webinar and learn: - How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo - How you can use the Denodo Platform with large data volumes in an efficient way - About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization

Similar to Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov (20)

"Implementing data quality automation with open source stack" - Max Martynov,...

CWIN17 India / Bigdata architecture yashowardhan sowale

Real-Time Analytics for Industries

Cloud & Big Data - Digital Transformation in Banking

Keeping the Pulse of Your Data: Why You Need Data Observability

Big data and the data quality imperative

What is Big Data - Edvicon

Fundamentals of Big Data

Machine Data Analytics

Anomaly Detection in Telco Networks

Transforming GE Healthcare with Data Platform Strategy

Delivering Analytics at Scale with a Governed Data Lake

Building a Data Strategy – Practical Steps for Aligning with Business Goals

The Bigger They Are The Harder They Fall

Big Data Matching - How to Find Two Similar Needles in a Really Big Haystack

Li charles biometrics analytics & big data 122013a for release

Don't think DevOps think Compliant Database DevOps

SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"

¿En qué se parece el Gobierno del Dato a un parque de atracciones?

Advanced Analytics and Machine Learning with Data Virtualization

More from Grid Dynamics

Are you keeping up with your customer

Grid Dynamics

Industry-leading brands have adapted to the ever-changing business environment and consumer expectations significantly faster than the competition. The key driving factor behind this speed-to-market is the adoption of intelligent cloud-native digital commerce architectures. In our talk, we will describe the best practices and case studies of achieving business agility that is imperative to excel in the post-pandemic era.

"How to build cool & useful voice commerce applications (such as devices like...

Grid Dynamics

In this talk, we describe how to build conversational e-commerce applications for the growing market of voice-powered AI devices using Dialog Flow. This talk demonstrates the capabilities of "Flower Genie," a teaching-oriented chatbot that can recommend a bouquet for any occasion, then take an order and deliver the flower arrangement via Alexa. We present the overall architecture of a voice application developed with Dialog Flow, including dialog management and NLU, and then discuss the finer points of testing and publishing a voice application for Alexa.

"Challenges for AI in Healthcare" - Peter Graven Ph.D

Grid Dynamics

Dynamic Talks Portland: The use of AI in many industries has revolutionized operations and efficiency. In healthcare, the progress is just beginning. Despite the promise of AI, why has the development lagged other industries? What issues are unique to healthcare that create challenges for common approaches? How can data scientists overcome these challenges and deliver on the promise of using data to reach multiple goals of improved quality, decreased cost, and greater patient satisfaction?

Dynamic Talks: "Applications of Big Data, Machine Learning and Artificial Int...

Grid Dynamics

The global HIV pandemic continues, particularly in Sub-Saharan Africa. By 2025, 40 million people will be living with HIV. The global cost of the pandemic is in the hundreds of billions of dollars. Better treatment means more people are living longer and costs will increase. The use of existing and emerging technologies is rare. Research institutions don’t share data. Data that drive US HIV policy in 2019 are from 2017 because of the time it takes for the CDC and NIH to combine data. The are many opportunities for big data, ML and AI to have a broader and continued impact on the HIV crisis. The use of these technologies can identify new avenues of research and help prioritize and focus efforts. We are starting to see these technologies used more and more. Several case studies will be presented. For example, advances in HIV vaccine research by Dr. David Heckerman; research at UCLA and Georgetown University looking at how social media can be used for tracking and predicting the spread of the epidemic; and work by researchers to improve care utilization in South Carolina. The opportunities for commercial and non-profit ventures to apply existing and emerging technologies like big data, ML and AI are countless. Tech4HIV is an organization working to drive these efforts into the tech sector and provides opportunities and resources for engagement.

Dynamic Talks: "Digital Transformation in Banking & Financial Services… a per...

Grid Dynamics

Retail industry has gone through a significant transformation in the last decade; caused by the change in consumer behaviour and expectations which have been driven by the convenience and low cost offered by Amazon. There are clear signs of a similar disruption in the financial services industry now, driven by new entrants who are changing consumer expectations. In this talk, Rahul looks at the evolution of retailers and draws conclusions about the path players in the financial services have to take to compete and provide customer experiences that will be necessary to compete.

Dynamic Talks: "Data Strategy as a Conduit for Data Maturity and Monetization...

Grid Dynamics

Dynamics Talks: "Writing Spark Pipelines with Less Boilerplate Code" - Egor P...

Grid Dynamics

Apache Spark is a general-purpose big data execution engine. You can work with different data sources with the same set of API in both batch and streaming mode. Such flexibility is great if you are experienced Spark developer solving a complicated data engineering problem, which might include ML or streaming. In Airbnb, 95% of all data pipelines are daily batch jobs, which read from Hive tables and write to Hive tables. For such jobs, you would like to trade some flexibility for more extensive functionality around writing to Hive or multiple days processing orchestration. Another advantage of reducing flexibility is creating "best practices", which can be followed by less experienced data engineers. In Airbnb, we've created a framework called "Sputnik," which tries to address these issues. In this talk, I'll show the typical boilerplate code, which Sputnik tries to reduce and concepts it introduces to simplify pipeline development.

"Trends in Building Advanced Analytics Platform for Large Enterprises" - Atul...

Grid Dynamics

The New Era of Public Safety Records Management: Dynamic talks Chicago 9/24/2019

Grid Dynamics

For businesses large and small, cloud-based software platforms are breaking down data siloes between teams, adding new capabilities, and boosting efficiency. While not known as early adopters, there's a compelling case for public safety to do the same. In this session, we'll explore how a comprehensive records and evidence platform can simplify operations and streamline data management and access, securely and responsibly. We'll also highlight the role of artificial intelligence, and how it can help agencies respond more quickly and improve outcomes.

"Implementing AI for New Business Models and Efficiencies" - Parag Shrivastav...

Grid Dynamics

Dynamic talks Seattle: Artificial Intelligence (AI) and data are foundational to ideation of new business models that bring growth and efficiencies. This is in action in pharmaceutical supply chain for reducing costs, increasing volumes, and optimizing the contracts with suppliers. The implementation involves process and tools that make data usable, and overcome the challenges of culture, ethics, data scalability, compute and engineering, and. Learn about data collection, data management, and metadata management tools implementation and modern data architecture to support them. Discuss machine learning algorithms for growth and efficiency scenarios.

Reducing No-shows and Late Cancelations in Healthcare Enterprise" - Shervin M...

Grid Dynamics

Dynamic Talks Seattle: Patient No-shows and Late Cancelations is estimated to cost around $150B across the nation. Providence St. Joseph Health, Healthcare Intelligence Group has been working on this pain point for the past couple of years and has developed a software solution leveraging a predictive engine to identify high risk patients at risk of No-show and Late Cancellation on a daily basis. More than 25 clinics have been using this solution for about 2 years now to optimize their reminder call strategy and to have a meaningful reduction of No-shows and Late Cancellations. In addition, a call center pilot was designed recently to standardize this process end-end. Pilot results will be evaluated to prove the impact at scale.

Customer intelligence: a Machine Learning Approach: Dynamic talks Atlanta 8/2...

Grid Dynamics

In this talk, we will discuss automatic decision-making and AI techniques for customer relationship management. First, we will present a methodology that helps to develop highly automated promotion and loyalty management systems. Next, we will walk through practical examples of how advanced customer and content signals can be generated using predictive models, and how optimization and reinforcement learning techniques can be used for targeting, budgeting, and pricing decisions. This talk is for Data Scientists, Product Owners, and Software Engineers involved in marketing operations or development of marketing automation software and interested in ML-based decision automation techniques.

"ML Services - How do you begin and when do you start scaling?" - Madhura Dud...

Grid Dynamics

Dynamic talks SF: So you have heard all the hype around how Machine Learning is going to change the world. But within your business context, where do you start? How do you get leadership buy-in for investment? And how and when you start scaling your ML Services? In this session, you will walk away with an actionable framework to bootstrap and scale a machine learning services team. You will see the framework in action through an actual 0 to 1 product journey involving deep learning where we delivered value in record speed in-spite of not having a dataset when we started. You will get practical tips on how to make decisions about when and how to scale your capability to scale ML Services and platform. Some of the tips are pretty counterintuitive and revealed themselves with our experience of productizing ML services over the last 5+ years. (Using a diverse range of technologies - Vision, Language, Graph, Anomaly Detection, Search Relevance, Personalization)

Realtime Contextual Product Recommendations…that scale and generate revenue -...

Grid Dynamics

Dynamic Talks SF: Recommendation systems are all around us. E-commerce companies like Amazon recommend products we are likely to buy based on our browsing behavior. Netflix suggests what shows we should watch based on our binging habits. Spotify builds a personalized playlist we would enjoy listening to, based on their understanding of what musical genre we are into. In this talk, we will explore recent advances in the area of product recommendations in both research and practice. We will see how machine learning, design thinking and solid data engineering principles are combined to create an engaging customer experience that positively impacts the bottom line. We will look at how we use various deep learning architectures to obtain image and text embeddings that supplement user and product based features to generate product recommendations that align closely with a consumer’s aesthetic preferences. The talk would be of interest to data scientists, data engineers, product managers, UX designers and anyone interested in machine learning.

Decision Automation in Marketing Systems using Reinforcement Learning: Dynami...

Grid Dynamics

- ML-based decision automation systems can make billions of micro-decisions in real-time to target customers, time promotions, and determine messaging and budgets. - These systems use techniques like propensity scoring, recommendation algorithms, and multi-armed bandits to optimize for business objectives within complex environments. - An example case study describes how a promotion targeting system for retailers and manufacturers can drive traffic, improve loyalty, and increase market share by automating decisions around targeting, timing, outreach, and promotion properties.

Best practices for enterprise-grade microservices implementations with Google...

Grid Dynamics

When migrating to a cloud and microservices architecture, companies need to invest in foundational capabilities, such as a microservices platform, continuous delivery, and an immutable infrastructure. In this talk, we will discuss our experience implementing these capabilities on the enterprise-scale with Google Cloud, Kubernetes, Istio, Envoy, Spinnaker, and Hashicorp stack. We will also discuss best practices of onboarding the cloud to facilitate DevOps, SRE without sacrificing quality or control.

Attribution Modelling 101: Credit Where Credit is Due!: Dynamic talks Seattle...

Grid Dynamics

Building an algorithmic price management system using ML: Dynamic talks Seatt...

Grid Dynamics

In this talk, we will discuss how predictive modeling and reinforcement learning can be used to build advanced price management systems that unlock the potential of dynamic and personalized pricing. We will present price optimization methods for a number of use cases including introductory pricing, promotion calendars, replenishable and seasonal products, targeted offers, and flash sales. We will also review case studies that demonstrate how these methods were applied in practice and how algorithmic price management components were fitted into pricing strategies.

Customer intelligence: a machine learning approach- Dynamic talks Dallas Q2

Grid Dynamics

Improving Customer Experience via Experimentation Dynamic Talks: San Francisc...

Grid Dynamics

She'll be sharing use cases, lessons learned and business impacts made using customer experience data and experimentation. She'll also share how combining quantitative data with unstructured quantitative data could provide powerful insights into user behavior that can be leveraged for improving business KPIs. Her team is using AI to support thousands of employees at PayPal and fueling innovation and data driven culture.

More from Grid Dynamics (20)