Hadoop in Validated Environment - Data Governance Initiative

•

2 likes•1,174 views

DataWorks Summit

Hadoop Summit 2015

Hadoop in Validated Environment
Data Governance Initiative

Martin Ryzl
Director, Analytics Platform
Ivo Lasek
Architect, Analytics Platform

Research
Manufacturing
Marketing

Search
Data
Integration
Data
Analytics
Open
Data

90 Days

Laboratory
Information
Management
SAP
Enterprise
Asset
Management
Manufacturing
Execution
Systems
Data
Analytics
Data
Integration

Who is the
dataset owner?
How can I
get access?
What does the
data mean?
How can
I reproduce
the results?
Where is the
data I need?

6 Months

Where Is the Data I Need?

Data Lake

Data Lake
Merge
Clean

Data Lake
Merge
Clean
Security and Data Governance

Data Catalog
Source: http://www.data.gov/

Who Is the Dataset Owner?

Entitlements
Dataset
Owner
Dataset
User

Entitlements
Dataset
Owner
Dataset
User

Entitlements
Dataset
Owner
Dataset
User

How Can I Get Access?

Entitlements
Dataset
Owner
Entitlement
Steward

Entitlements
Dataset
Owner
Entitlement
Steward
Dataset
User

Entitlements
Dataset
Owner
Entitlement
Steward
Dataset
User

Entitlements
Dataset
Owner
Entitlement
Steward
Dataset
User

What Does the Data Mean?

Semantic meaning – Metastore
id name ssn birth_n
o
phone id
personal_number
employee_number
first_name
division

Metastore
Dataset
Owner

Metastore
Dataset
Owner
Data
Steward

Metastore
Dataset
Owner
Data
Steward

Metastore
Dataset
Owner
Data
Steward
Entitlement
Steward
Dataset
User

Metastore
Dataset
Owner
Data
Steward
Entitlement
Steward
Dataset
User

Metastore
Dataset
Owner
Data
Steward
Entitlement
Steward
Dataset
User

How Can I Reproduce the Data?

Reporting Data
Reproducibility
delta1
delta2
delta3
Raw Data
Aggregate
v1.0
delta1..delta3
Reporting Data

Reproducibility
delta1
delta2
delta3
Raw Data
delta1..delta3
Reporting Data
Aggregate
v1.0
delta4
delta5
delta6 delta1..delta7
Reporting Data
Aggregate
v1.0
delta7

Traceability
delta1
delta2
delta3
Raw Data
delta1..delta3
Reporting Data
Aggregate
v1.0
delta1..delta3
Cleaned Data
Clean
v1.0
delta4
delta5
delta6 delta1..delta7
Reporting Data
Aggregate
v1.0
delta1..delta7
Cleaned Data
Clean
v1.1
delta7

Data Lineage

Access Logs

Process Logs
2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1702)) - Get metadata for subqueries
17865102 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Get metadata for destination tables
2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1726)) - Get metadata for destination
tables
17865345 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Completed getting MetaData in Semantic Analysis
2015-06-04 12:53:31,844 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(10004)) - Completed getting MetaData
in Semantic Analysis
17865347 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Not invoking CBO because the statement has too few joins
2015-06-04 12:53:31,846 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:canHandleAstForCbo(10258)) - Not invoking CBO
because the statement has too few joins
Heart beat
17866695 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException [Error 10044]: Line 2:18 Cannot insert into target
table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:18 Cannot insert into target table because column number/types are different
''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns.
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:6535)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6336)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8977)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8868)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9713)

Who is the
dataset owner?
How can I
get access?
What does the
data mean?
How can
I reproduce
the results?
Where is the
data I need?
HDFS/Hive Metastore Ranger
Metastore Falcon

Contacts
• Martin Ryzl (martin.ryzl@merck.com)
• Ivo Lasek (ivo.lasek@merck.com)
• http://www.merck.com/
• http://www.msdit.cz/

More Related Content

What's hot

Deploying a Governed Data Lake

Deploying a Governed Data Lake

Deploying a Governed Data Lake

Finance Data Lake objective is to create a centralized enterprise data repository for all Finance and Supply Chain data. It serves as the single source of truth. It enables a self-service discovery Analytics platform for business users to answer adhoc business questions and derive critical insights. The data lake is based on open source Hadoop big data platform and a very cost effective solution in breaking the ERP data silos and simplifying the data architecture in the enterprise. POCs were conducted on in-house Hortonworks Hadoop data platform to validate the cluster performance for Production volumes. Based on business priorities, an initial roadmap was defined using 3 data sources including 2 SAP ERPs and Peoplesoft (OLTP systems). Development environment was established in AWS Cloud for agile delivery. The near real time data ingestion architecture for the data lake was defined using replication tools and custom SQOOP based micro-batching framework and data persisted in Apache Hive DB in ORC format. Data and user security is implemented using Apache Ranger and sensitive data stored at rest in encryption zones. Business data sets were developed in Hive scripts and scheduled using Oozie. Multiple reporting tools connectivity including SQL tools, Excel and Tableau were enabled for Self-service Analytics. Upon successful implementation of the initial phase, a full roadmap is established to extend the Finance data lake to over 25 data sources and enhance data ingestion to scale as well as enable OLAP tools on Hadoop.

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

DataWorks Summit

Jethro + Symphony Health at Qlik Qonnections

Jethro + Symphony Health at Qlik Qonnections

Jethro + Symphony Health at Qlik Qonnections

Designing Data Pipelines for Automous and Trusted Analytics

Designing Data Pipelines for Automous and Trusted Analytics

Designing Data Pipelines for Automous and Trusted Analytics

DataWorks Summit

Hadoop,Big Data Analytics and More

Hadoop,Big Data Analytics and More

Hadoop,Big Data Analytics and More

Trendwise Analytics

Apache Hadoop is quickly becoming the technology of choice for organizations investing in big data, powering their next generation data architecture. With Hadoop serving as both a scalable data platform and computational engine, data science is re-emerging as a center-piece of enterprise innovation, with applied data solutions such as online product recommendation, automated fraud detection and customer sentiment analysis. In this talk Ofer will provide an overview of data science and how to take advantage of Hadoop for large scale data science projects: * What is data science? * How can techniques like classification, regression, clustering and outlier detection help your organization? * What questions do you ask and which problems do you go after? * How do you instrument and prepare your organization for applied data science with Hadoop? * Who do you hire to solve these problems? You will learn how to plan, design and implement a data science project with Hadoop

Data Science with Hadoop: A Primer

Data Science with Hadoop: A Primer

Data Science with Hadoop: A Primer

DataWorks Summit

The key to unlocking the Value in the IoT? Managing the Data!

The key to unlocking the Value in the IoT? Managing the Data!

The key to unlocking the Value in the IoT? Managing the Data!

DataWorks Summit/Hadoop Summit

Aster getting started

Aster getting started

Aster getting started

JESUS WORLD MISSION

Pentaho Analytics on MongoDB

Pentaho Analytics on MongoDB

Pentaho Analytics on MongoDB

Since the early days of Hive, SQL on Hadoop has evolved from being a SQL wrapper on top of MapReduce to a viable replacement for the traditional EDW. In the meantime, while SQL-on-Hadoop vendors were busy adding enterprise capabilities and comparing their TPC-DS prowess against Hive, a niche industry emerged on the side for OLAP (a.k.a. “Interactive BI”) on Hadoop data. Unlike general-purpose SQL-on-Hadoop engines, which deal with the multiple aspects of warehousing, including reporting, OLAP-on-Hadoop engines focus almost exclusively on answering OLAP queries fast by using implementation techniques that had not been part of the SQL-on-Hadoop toolbox so far. But SQL-on-Hadoop engines are not standing still. After having made huge progress in catching up to traditional EDWs for reporting workloads, SQL-on-Hadoop engines are now setting their sights on interactive BI. This is great news for enterprises. As the line between reporting and OLAP gets blurred, enterprises can now start considering using a single engine for both reporting and Interactive BI on their Hadoop data, as opposed to having to host, manage, and license two separate products. Can a single engine satisfy both your reporting and Interactive BI needs? This may be a hard question to answer. Vendors use inconsistent terminology to describe their products and make ambitious and sometimes conflicting claims. This makes it very hard for enterprises to compare products, let alone decide which is the product that best matches their needs. In this presentation, we’ll provide an overview of the different approaches to OLAP on Hadoop, and explain the key technologies behind each of them. We’ll use consistent terminology to describe what you get from multiple proprietary and open source products and outline advantages and disadvantages. You’ll come out equipped with the knowledge you need to read past marketing and sales pitches. You’ll be able to compare products and make an informed decision on whether a single engine for both reporting and Interactive BI on Hadoop is right for you. Speaker Gustavo Arocena, Big Data Architect, IBM

The convergence of reporting and interactive BI on Hadoop

The convergence of reporting and interactive BI on Hadoop

The convergence of reporting and interactive BI on Hadoop

DataWorks Summit

Benefits of Hadoop as Platform as a Service

Benefits of Hadoop as Platform as a Service

Benefits of Hadoop as Platform as a Service

DataWorks Summit/Hadoop Summit

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we'll look at how SDC's "intent-driven" approach keeps the data flowing, whether you're processing data 'off-cluster', in Spark, or in MapReduce. StreamSets software delivers performance management for data flows that feed the next generation of big data applications. Its mission is to bring operational excellence to the management of data in motion, so that data arrives on time and with quality, accelerating analysis and decision making. StreamSets Data Collector is in use at hundreds of companies where it brings unprecedented visibility into and control over data as it moves between an expanding variety of sources and destinations. Speakers: Pat Patterson has been working with Internet technologies since 1997, building software and working with communities at Sun Microsystems, Huawei, Salesforce and StreamSets. At Sun, Pat was the community lead for the OpenSSO open source project, while at Huawei he developed cloud storage infrastructure software. Part of the developer evangelism team at Salesforce, Pat focused on identity, integration and the Internet of Things. Now community champion at StreamSets, Pat is responsible for the care and feeding of the StreamSets open source community.

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

Yahoo Developer Network

Big Data Analytics Projects - Real World with Pentaho

Big Data Analytics Projects - Real World with Pentaho

Big Data Analytics Projects - Real World with Pentaho

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

DataWorks Summit

Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes. In this presentation we discuss Microsoft HDInsight offering of Spark. Azure HDInsight, Microsoft’s managed Hadoop and Spark cloud service that runs the Hortonworks Data Platform. Spark for Azure HDInsight offers customers an enterprise-ready Spark solution that’s fully managed, secured, and highly available and made simpler for users with compelling and interactive experiences.

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Hadoop based data Lakes have become increasingly popular within today’s modern data architectures for their ability to scale, handle data variety and low cost. Many organizations start slow with the data lake initiatives but as they grow bigger, they suffer with challenges on data consistency, quality and security, resulting in losing confidence in their data lake initiatives. This talk will discuss the need for good data governance mechanisms for Hadoop data lakes and it relationship with productivity and how it helps organizations meet regulatory and compliance requirements. The talk advocates carrying a different mindset for designing and implementing flexible governance mechanisms on Hadoop data lakes.

Data Governance for Data Lakes

Data Governance for Data Lakes

Data Governance for Data Lakes

Brian Brownlow is an experienced senior analyst programmer for Mayo Clinic. He is made a workshop presentation at the 2014 BDPA Technology Conference on the topic, 'Big Data Implementation - Mayo Clinic Case Study'. This presentation will show part of the Mayo Clinic story on the embarking of an exploration of the application of `Big Data' technologies. `Big Data' is seen as one set of tools that can be used to enhance medical research, medical education and practice management. Mayo Clinic is always searching for better, faster and cheaper ways to use its data to improve patient care and sustain financial outcomes in a challenging reimbursement environment. Our approach uses several components that are open source and combines them with data from various sources to provide information to decision makers in near real time. We have created a center of `Big Data' excellence using in-house staff and vendor engagements. `Big Data' is one element of our Enterprise Data Trust framework.

A Mayo Clinic Big Data Implementation

A Mayo Clinic Big Data Implementation

A Mayo Clinic Big Data Implementation

BDPA Education and Technology Foundation

Come to this deep dive on how Pivotal's Data Lake Vision is evolving by embracing next generation in-memory data exchange and compute technologies around Spark and Tachyon. Did we say Hadoop, SQL, and what's the shortest path to get from past to future state? The next generation of data lake technology will leverage the availability of in-memory processing, with an architecture that supports multiple data analytics workloads within a single environment: SQL, R, Spark, batch and transactional.

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Speaker: Geetha Balasundaram, Developer at ThoughtWorks From tools and technology to people and requirements, what's different in the data engineering space? App development is traditional now. All enterprises want to become data-guided. Data lake is good start yet the know-hows and do-hows are so many. Experiences from building a data lake in the retail domain, the talk will be covering. - What is this vast new space of data engineering, - Why it is critical to think in terms of data rather than features - How important it is to understand these technologies and create a data lake that is usable and insightful to business

Building a Data Lake - An App Dev's Perspective

Building a Data Lake - An App Dev's Perspective

Building a Data Lake - An App Dev's Perspective

GeekNightHyderabad

10 Amazing Things To Do With a Hadoop-Based Data Lake

10 Amazing Things To Do With a Hadoop-Based Data Lake

10 Amazing Things To Do With a Hadoop-Based Data Lake

What's hot (20)

Deploying a Governed Data Lake

Deploying a Governed Data Lake

Deploying a Governed Data Lake

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Jethro + Symphony Health at Qlik Qonnections

Jethro + Symphony Health at Qlik Qonnections

Jethro + Symphony Health at Qlik Qonnections

Designing Data Pipelines for Automous and Trusted Analytics

Designing Data Pipelines for Automous and Trusted Analytics

Designing Data Pipelines for Automous and Trusted Analytics

Hadoop,Big Data Analytics and More

Hadoop,Big Data Analytics and More

Hadoop,Big Data Analytics and More

Data Science with Hadoop: A Primer

Data Science with Hadoop: A Primer

Data Science with Hadoop: A Primer

The key to unlocking the Value in the IoT? Managing the Data!

The key to unlocking the Value in the IoT? Managing the Data!

The key to unlocking the Value in the IoT? Managing the Data!

Aster getting started

Aster getting started

Aster getting started

Pentaho Analytics on MongoDB

Pentaho Analytics on MongoDB

Pentaho Analytics on MongoDB

The convergence of reporting and interactive BI on Hadoop

The convergence of reporting and interactive BI on Hadoop

The convergence of reporting and interactive BI on Hadoop

Benefits of Hadoop as Platform as a Service

Benefits of Hadoop as Platform as a Service

Benefits of Hadoop as Platform as a Service

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

Big Data Analytics Projects - Real World with Pentaho

Big Data Analytics Projects - Real World with Pentaho

Big Data Analytics Projects - Real World with Pentaho

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Spark with Azure HDInsight - Tampa Bay Data Science - Adnan Masood, PhD

Data Governance for Data Lakes

Data Governance for Data Lakes

Data Governance for Data Lakes

A Mayo Clinic Big Data Implementation

A Mayo Clinic Big Data Implementation

A Mayo Clinic Big Data Implementation

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Building a Data Lake - An App Dev's Perspective

Building a Data Lake - An App Dev's Perspective

Building a Data Lake - An App Dev's Perspective

10 Amazing Things To Do With a Hadoop-Based Data Lake

10 Amazing Things To Do With a Hadoop-Based Data Lake

10 Amazing Things To Do With a Hadoop-Based Data Lake

Viewers also liked

HBase and Drill: How loosley typed SQL is ideal for NoSQL

HBase and Drill: How loosley typed SQL is ideal for NoSQL

HBase and Drill: How loosley typed SQL is ideal for NoSQL

DataWorks Summit

Karta an ETL Framework to process high volume datasets

Karta an ETL Framework to process high volume datasets

Karta an ETL Framework to process high volume datasets

DataWorks Summit

50 Shades of SQL

50 Shades of SQL

50 Shades of SQL

DataWorks Summit

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

DataWorks Summit

Realistic Synthetic Generation Allows Secure Development

Realistic Synthetic Generation Allows Secure Development

Realistic Synthetic Generation Allows Secure Development

DataWorks Summit

Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...

Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...

Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...

DataWorks Summit

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

DataWorks Summit

Practical Distributed Machine Learning Pipelines on Hadoop

Practical Distributed Machine Learning Pipelines on Hadoop

Practical Distributed Machine Learning Pipelines on Hadoop

DataWorks Summit

Running Spark and MapReduce together in Production

Running Spark and MapReduce together in Production

Running Spark and MapReduce together in Production

DataWorks Summit

Carpe Datum: Building Big Data Analytical Applications with HP Haven

Carpe Datum: Building Big Data Analytical Applications with HP Haven

Carpe Datum: Building Big Data Analytical Applications with HP Haven

DataWorks Summit

The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...

The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...

The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...

DataWorks Summit

Hadoop for Genomics__HadoopSummit2010

Hadoop for Genomics__HadoopSummit2010

Hadoop for Genomics__HadoopSummit2010

Yahoo Developer Network

One Click Hadoop Clusters - Anywhere (Using Docker)

One Click Hadoop Clusters - Anywhere (Using Docker)

One Click Hadoop Clusters - Anywhere (Using Docker)

DataWorks Summit

Inspiring Travel at Airbnb [WIP]

Inspiring Travel at Airbnb [WIP]

Inspiring Travel at Airbnb [WIP]

DataWorks Summit

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN

DataWorks Summit

Spark Application Development Made Easy

Spark Application Development Made Easy

Spark Application Development Made Easy

DataWorks Summit

Open Source SQL for Hadoop: Where are we and Where are we Going?

Open Source SQL for Hadoop: Where are we and Where are we Going?

Open Source SQL for Hadoop: Where are we and Where are we Going?

DataWorks Summit

NoSQL Needs SomeSQL

NoSQL Needs SomeSQL

NoSQL Needs SomeSQL

DataWorks Summit

Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...

Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...

Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...

DataWorks Summit

Big Data Challenges in the Energy Sector

Big Data Challenges in the Energy Sector

Big Data Challenges in the Energy Sector

DataWorks Summit

Viewers also liked (20)

HBase and Drill: How loosley typed SQL is ideal for NoSQL

HBase and Drill: How loosley typed SQL is ideal for NoSQL

HBase and Drill: How loosley typed SQL is ideal for NoSQL

Karta an ETL Framework to process high volume datasets

Karta an ETL Framework to process high volume datasets

Karta an ETL Framework to process high volume datasets

50 Shades of SQL

50 Shades of SQL

50 Shades of SQL

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

Realistic Synthetic Generation Allows Secure Development

Realistic Synthetic Generation Allows Secure Development

Realistic Synthetic Generation Allows Secure Development

Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...

Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...

Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...

Practical Distributed Machine Learning Pipelines on Hadoop

Practical Distributed Machine Learning Pipelines on Hadoop

Practical Distributed Machine Learning Pipelines on Hadoop

Running Spark and MapReduce together in Production

Running Spark and MapReduce together in Production

Running Spark and MapReduce together in Production

Carpe Datum: Building Big Data Analytical Applications with HP Haven

Carpe Datum: Building Big Data Analytical Applications with HP Haven

Carpe Datum: Building Big Data Analytical Applications with HP Haven

The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...

The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...

The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...

Hadoop for Genomics__HadoopSummit2010

Hadoop for Genomics__HadoopSummit2010

Hadoop for Genomics__HadoopSummit2010

One Click Hadoop Clusters - Anywhere (Using Docker)

One Click Hadoop Clusters - Anywhere (Using Docker)

One Click Hadoop Clusters - Anywhere (Using Docker)

Inspiring Travel at Airbnb [WIP]

Inspiring Travel at Airbnb [WIP]

Inspiring Travel at Airbnb [WIP]

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN

DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN

Spark Application Development Made Easy

Spark Application Development Made Easy

Spark Application Development Made Easy

Open Source SQL for Hadoop: Where are we and Where are we Going?

Open Source SQL for Hadoop: Where are we and Where are we Going?

Open Source SQL for Hadoop: Where are we and Where are we Going?

NoSQL Needs SomeSQL

NoSQL Needs SomeSQL

NoSQL Needs SomeSQL

Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...

Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...

Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...

Big Data Challenges in the Energy Sector

Big Data Challenges in the Energy Sector

Big Data Challenges in the Energy Sector

Similar to Hadoop in Validated Environment - Data Governance Initiative

Workshop with Joe Caserta, President of Caserta Concepts, at Data Summit 2015 in NYC. Data science, the ability to sift through massive amounts of data to discover hidden patterns and predict future trends and actions, may be considered the "sexiest" job of the 21st century, but it requires an understanding of many elements of data analytics. This workshop introduced basic concepts, such as SQL and NoSQL, MapReduce, Hadoop, data mining, machine learning, and data visualization. For notes and exercises from this workshop, click here: https://github.com/Caserta-Concepts/ds-workshop. For more information, visit our website at www.casertaconcepts.com

Introduction to Data Science

Introduction to Data Science

Introduction to Data Science

If information is the new oil, then governance is its “safety data sheet.” As demand for data as the raw material for competitive differentiation continues to rise in enterprises, enterprises are having bigger challenges identifying and valuing data and ensuring its appropriate use to extract the right information. In order for organizations to make effective business decisions, organizations need to have trust in their data so that they can impute the right value and use it for the right purposes while satisfying any organizational or regulatory mandates. A number of analytics and data science initiatives fail to reach their potential due to lack of an information governance framework in place. Robust information governance capabilities can help organizations develop trust in their data and empower them to make decisions confidently. In this session Sanjeev Mohan, Research Analyst at Gartner, and Srikanth Venkat, Sr. Director of Product Management at Hortonworks, will walk you through an end-to-end architectural blueprint for information governance and best practices for helping organizations understand, secure, and govern diverse types of data in enterprise data lakes. Speaker Sanjeev Mohan, Gartner, Research Analyst Srikanth Venkat, Hortonworks, Senior Director, Product Management In this session Sanjeev Mohan, Research Analyst at Gartner, and Srikanth Venkat, Sr. Director of Product Management at Hortonworks, will walk you through an end-to-end architectural blueprint for information governance and best practices for helping organizations understand, secure, and govern diverse types of data in enterprise data lakes.

Balancing data democratization with comprehensive information governance: bui...

Balancing data democratization with comprehensive information governance: bui...

Balancing data democratization with comprehensive information governance: bui...

DataWorks Summit

Big datarevealed hadoop catalog

Big datarevealed hadoop catalog

Big datarevealed hadoop catalog

In our recent Big Data Warehousing Meetup, we discussed Data Governance, Compliance and Security in Hadoop. As the Big Data paradigm becomes more commonplace, we must apply enterprise-grade governance capabilities for critical data that is highly regulated and adhere to stringent compliance requirements. Caserta and Cloudera shared techniques and tools that enables data governance, compliance and security on Big Data. For more information, visit www.casertaconcepts.com

Data Governance, Compliance and Security in Hadoop with Cloudera

Data Governance, Compliance and Security in Hadoop with Cloudera

Data Governance, Compliance and Security in Hadoop with Cloudera

Intro to Data Science on Hadoop

Intro to Data Science on Hadoop

Intro to Data Science on Hadoop

This presentation starts off by discussing powerful examples of The Power of Data and the benefits of Data Driven architectures. A Data Governance program is important for the success of Data Driven architectures. We then discuss the challenges of implementing a Data Governance framework on a Big Data Data Lake with open source software including DataPlane, Apache Atlas and Apache Ranger. And finally, we discuss the importance of the democratization of data and the switching to a speed of thought framework with Hive LLAP.

The Power of Data

The Power of Data

The Power of Data

DataWorks Summit

Defining and Applying Data Governance in Today’s Business Environment

Defining and Applying Data Governance in Today’s Business Environment

Defining and Applying Data Governance in Today’s Business Environment

Oracle Big Data Governance Webcast Charts

Oracle Big Data Governance Webcast Charts

Oracle Big Data Governance Webcast Charts

Jeffrey T. Pollock

big data

BigData Analytics

BigData Analytics

BigData Analytics

Mayank Kumar Sharma

Big Data and BI Tools - BI Reporting for Bay Area Startups User Group

Big Data and BI Tools - BI Reporting for Bay Area Startups User Group

Big Data and BI Tools - BI Reporting for Bay Area Startups User Group

Achieving Flexible Scalability of Hadoop to Meet Enterprise Workload Requirem...

Achieving Flexible Scalability of Hadoop to Meet Enterprise Workload Requirem...

Achieving Flexible Scalability of Hadoop to Meet Enterprise Workload Requirem...

Anzo Smart Data Lake 4.0 - a Data Lake Platform for the Enterprise Informatio...

Anzo Smart Data Lake 4.0 - a Data Lake Platform for the Enterprise Informatio...

Anzo Smart Data Lake 4.0 - a Data Lake Platform for the Enterprise Informatio...

Cambridge Semantics

Dataware housing

Dataware housing

Dataware housing

Big data analytics - Introduction to Big Data and Hadoop

Big data analytics - Introduction to Big Data and Hadoop

Big data analytics - Introduction to Big Data and Hadoop

BAR360 open data platform presentation at DAMA, Sydney

BAR360 open data platform presentation at DAMA, Sydney

BAR360 open data platform presentation at DAMA, Sydney

TSE_Pres12.pptx

TSE_Pres12.pptx

TSE_Pres12.pptx

The book of elephant tattoo

The book of elephant tattoo

The book of elephant tattoo

Getting down to business on Big Data analytics

Getting down to business on Big Data analytics

Getting down to business on Big Data analytics

The Marketing Distillery

Today, practically every firm uses big data to gain a competitive advantage in the market. With this in mind, freely available big data tools for analysis and processing are a cost-effective and beneficial choice for enterprises. Hadoop is the sector’s leading open-source initiative and big data tidal roller. Moreover, this is not the final chapter! Numerous other businesses pursue Hadoop’s free and open-source path.

Big Data Tools: A Deep Dive into Essential Tools

Big Data Tools: A Deep Dive into Essential Tools

Big Data Tools: A Deep Dive into Essential Tools

Similar to Hadoop in Validated Environment - Data Governance Initiative (20)

Introduction to Data Science

Introduction to Data Science

Introduction to Data Science

Balancing data democratization with comprehensive information governance: bui...

Balancing data democratization with comprehensive information governance: bui...

Balancing data democratization with comprehensive information governance: bui...

Big datarevealed hadoop catalog

Big datarevealed hadoop catalog

Big datarevealed hadoop catalog

Data Governance, Compliance and Security in Hadoop with Cloudera

Data Governance, Compliance and Security in Hadoop with Cloudera

Data Governance, Compliance and Security in Hadoop with Cloudera

Intro to Data Science on Hadoop

Intro to Data Science on Hadoop

Intro to Data Science on Hadoop

The Power of Data

The Power of Data

The Power of Data

Defining and Applying Data Governance in Today’s Business Environment

Defining and Applying Data Governance in Today’s Business Environment

Defining and Applying Data Governance in Today’s Business Environment

Oracle Big Data Governance Webcast Charts

Oracle Big Data Governance Webcast Charts

Oracle Big Data Governance Webcast Charts

big data

BigData Analytics

BigData Analytics

BigData Analytics

Big Data and BI Tools - BI Reporting for Bay Area Startups User Group

Big Data and BI Tools - BI Reporting for Bay Area Startups User Group

Big Data and BI Tools - BI Reporting for Bay Area Startups User Group

Achieving Flexible Scalability of Hadoop to Meet Enterprise Workload Requirem...

Achieving Flexible Scalability of Hadoop to Meet Enterprise Workload Requirem...

Achieving Flexible Scalability of Hadoop to Meet Enterprise Workload Requirem...

Anzo Smart Data Lake 4.0 - a Data Lake Platform for the Enterprise Informatio...

Anzo Smart Data Lake 4.0 - a Data Lake Platform for the Enterprise Informatio...

Anzo Smart Data Lake 4.0 - a Data Lake Platform for the Enterprise Informatio...

Dataware housing

Dataware housing

Dataware housing

Big data analytics - Introduction to Big Data and Hadoop

Big data analytics - Introduction to Big Data and Hadoop

Big data analytics - Introduction to Big Data and Hadoop

BAR360 open data platform presentation at DAMA, Sydney

BAR360 open data platform presentation at DAMA, Sydney

BAR360 open data platform presentation at DAMA, Sydney

TSE_Pres12.pptx

TSE_Pres12.pptx

TSE_Pres12.pptx

The book of elephant tattoo

The book of elephant tattoo

The book of elephant tattoo

Getting down to business on Big Data analytics

Getting down to business on Big Data analytics

Getting down to business on Big Data analytics

Big Data Tools: A Deep Dive into Essential Tools

Big Data Tools: A Deep Dive into Essential Tools

Big Data Tools: A Deep Dive into Essential Tools

More from DataWorks Summit

Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL). Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW). Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models. Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.

Data Science Crash Course

Data Science Crash Course

Data Science Crash Course

DataWorks Summit

In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort. This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.

Floating on a RAFT: HBase Durability with Apache Ratis

Floating on a RAFT: HBase Durability with Apache Ratis

Floating on a RAFT: HBase Durability with Apache Ratis

DataWorks Summit

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.

HBase Tales From the Trenches - Short stories about most common HBase operati...

HBase Tales From the Trenches - Short stories about most common HBase operati...

HBase Tales From the Trenches - Short stories about most common HBase operati...

DataWorks Summit

LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

DataWorks Summit

Managing the Dewey Decimal System

Managing the Dewey Decimal System

Managing the Dewey Decimal System

DataWorks Summit

Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL. Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist). In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.

Practical NoSQL: Accumulo's dirlist Example

Practical NoSQL: Accumulo's dirlist Example

Practical NoSQL: Accumulo's dirlist Example

DataWorks Summit

Data serves as the platform for decision-making at Uber. To facilitate data driven decisions, many datasets at Uber are ingested in a Hadoop Data Lake and exposed to querying via Hive. Analytical queries joining various datasets are run to better understand business data at Uber. Data ingestion, at its most basic form, is about organizing data to balance efficient reading and writing of newer data. Data organization for efficient reading involves factoring in query patterns to partition data to ensure read amplification is low. Data organization for efficient writing involves factoring the nature of input data - whether it is append only or updatable. At Uber we ingest terabytes of many critical tables such as trips that are updatable. These tables are fundamental part of Uber's data-driven solutions, and act as the source-of-truth for all the analytical use-cases across the entire company. Datasets such as trips constantly receive updates to the data apart from inserts. To ingest such datasets we need a critical component that is responsible for bookkeeping information of the data layout, and annotates each incoming change with the location in HDFS where this data should be written. This component is called as Global Indexing. Without this component, all records get treated as inserts and get re-written to HDFS instead of being updated. This leads to duplication of data, breaking data correctness and user queries. This component is key to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. This component will need to have strong consistency and provide large throughputs for index writes and reads. At Uber, we have chosen HBase to be the backing store for the Global Indexing component and is a critical component in allowing us to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. In this talk, we will discuss data@Uber and expound more on why we built the global index using Apache Hbase and how this helps to scale out our cluster usage. We’ll give details on why we chose HBase over other storage systems, how and why we came up with a creative solution to automatically load Hfiles directly to the backend circumventing the normal write path when bootstrapping our ingestion tables to avoid QPS constraints, as well as other learnings we had bringing this system up in production at the scale of data that Uber encounters daily.

HBase Global Indexing to support large-scale data ingestion at Uber

HBase Global Indexing to support large-scale data ingestion at Uber

HBase Global Indexing to support large-scale data ingestion at Uber

DataWorks Summit

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Cybersecurity requires an organization to collect data, analyze it, and alert on cyber anomalies in near real-time. This is a challenging endeavor when considering the variety of data sources which need to be collected and analyzed. Everything from application logs, network events, authentications systems, IOT devices, business events, cloud service logs, and more need to be taken into consideration. In addition, multiple data formats need to be transformed and conformed to be understood by both humans and ML/AI algorithms. To solve this problem, the Aetna Global Security team developed the Unified Data Platform based on Apache NiFi, which allows them to remain agile and adapt to new security threats and the onboarding of new technologies in the Aetna environment. The platform currently has over 60 different data flows with 95% doing real-time ETL and handles over 20 billion events per day. In this session learn from Aetna’s experience building an edge to AI high-speed data pipeline with Apache NiFi.

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

DataWorks Summit

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”

Security Framework for Multitenant Architecture

Security Framework for Multitenant Architecture

Security Framework for Multitenant Architecture

DataWorks Summit

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Presto: Optimizing Performance of SQL-on-Anything Engine

Presto: Optimizing Performance of SQL-on-Anything Engine

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Extending Twitter's Data Platform to Google Cloud

Extending Twitter's Data Platform to Google Cloud

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

DataWorks Summit

Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

DataWorks Summit

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

DataWorks Summit

Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

Computer Vision: Coming to a Store Near You

Computer Vision: Coming to a Store Near You

Computer Vision: Coming to a Store Near You

DataWorks Summit

Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

DataWorks Summit

More from DataWorks Summit (20)

Data Science Crash Course

Data Science Crash Course

Data Science Crash Course

Floating on a RAFT: HBase Durability with Apache Ratis

Floating on a RAFT: HBase Durability with Apache Ratis

Floating on a RAFT: HBase Durability with Apache Ratis

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

HBase Tales From the Trenches - Short stories about most common HBase operati...

HBase Tales From the Trenches - Short stories about most common HBase operati...

HBase Tales From the Trenches - Short stories about most common HBase operati...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Managing the Dewey Decimal System

Managing the Dewey Decimal System

Managing the Dewey Decimal System

Practical NoSQL: Accumulo's dirlist Example

Practical NoSQL: Accumulo's dirlist Example

Practical NoSQL: Accumulo's dirlist Example

HBase Global Indexing to support large-scale data ingestion at Uber

HBase Global Indexing to support large-scale data ingestion at Uber

HBase Global Indexing to support large-scale data ingestion at Uber

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Security Framework for Multitenant Architecture

Security Framework for Multitenant Architecture

Security Framework for Multitenant Architecture

Presto: Optimizing Performance of SQL-on-Anything Engine

Presto: Optimizing Performance of SQL-on-Anything Engine

Presto: Optimizing Performance of SQL-on-Anything Engine

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Extending Twitter's Data Platform to Google Cloud

Extending Twitter's Data Platform to Google Cloud

Extending Twitter's Data Platform to Google Cloud

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Computer Vision: Coming to a Store Near You

Computer Vision: Coming to a Store Near You

Computer Vision: Coming to a Store Near You

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Recently uploaded

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

[BuildWithAI] Introduction to Gemini.pdf

[BuildWithAI] Introduction to Gemini.pdf

[BuildWithAI] Introduction to Gemini.pdf

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Cyberprint. Dark Pink Apt Group [EN].pdf

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

When you’re building (micro)services, you have lots of framework options. Spring Boot is no doubt a popular choice. But there’s more! Take Quarkus, a framework that’s considered the rising star for Kubernetes-native Java. It always depends on what's best for your situation, but how to choose the best solution if you're comparing 2 frameworks? Both Spring Boot and Quarkus have their positives and negatives. Let us compare the two by live coding a couple of common use cases in Spring Boot and Quarkus. After this talk, you’ll be ready to get started with Quarkus yourself, and know when to select Quarkus or Spring Boot.

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Recently uploaded (20)

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

[BuildWithAI] Introduction to Gemini.pdf

[BuildWithAI] Introduction to Gemini.pdf

[BuildWithAI] Introduction to Gemini.pdf

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Cyberprint. Dark Pink Apt Group [EN].pdf

Cyberprint. Dark Pink Apt Group [EN].pdf

Cyberprint. Dark Pink Apt Group [EN].pdf

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Hadoop in Validated Environment - Data Governance Initiative

1. Hadoop in Validated Environment Data Governance Initiative

2. Martin Ryzl Director, Analytics Platform Ivo Lasek Architect, Analytics Platform

3. Research Manufacturing Marketing

4. Search Data Integration Data Analytics Open Data

6. Laboratory Information Management SAP Enterprise Asset Management Manufacturing Execution Systems Data Analytics Data Integration

7. Who is the dataset owner? How can I get access? What does the data mean? How can I reproduce the results? Where is the data I need?

9. Where Is the Data I Need?

11. Data Lake Merge Clean

12. Data Lake Merge Clean Security and Data Governance

13. Data Catalog Source: http://www.data.gov/

14. Who Is the Dataset Owner?

15. Entitlements Dataset Owner Dataset User

16. Entitlements Dataset Owner Dataset User

17. Entitlements Dataset Owner Dataset User

18. How Can I Get Access?

19. Entitlements Dataset Owner Entitlement Steward

20. Entitlements Dataset Owner Entitlement Steward Dataset User

21. Entitlements Dataset Owner Entitlement Steward Dataset User

22. Entitlements Dataset Owner Entitlement Steward Dataset User

23. What Does the Data Mean?

24. Semantic meaning – Metastore id name ssn birth_n o phone id personal_number employee_number first_name division

25. Metastore Dataset Owner

26. Metastore Dataset Owner Data Steward

27. Metastore Dataset Owner Data Steward

28. Metastore Dataset Owner Data Steward Entitlement Steward Dataset User

29. Metastore Dataset Owner Data Steward Entitlement Steward Dataset User

30. Metastore Dataset Owner Data Steward Entitlement Steward Dataset User

31. How Can I Reproduce the Data?

32. Reporting Data Reproducibility delta1 delta2 delta3 Raw Data Aggregate v1.0 delta1..delta3 Reporting Data

33. Reproducibility delta1 delta2 delta3 Raw Data delta1..delta3 Reporting Data Aggregate v1.0 delta4 delta5 delta6 delta1..delta7 Reporting Data Aggregate v1.0 delta7

34. Traceability delta1 delta2 delta3 Raw Data delta1..delta3 Reporting Data Aggregate v1.0 delta1..delta3 Cleaned Data Clean v1.0 delta4 delta5 delta6 delta1..delta7 Reporting Data Aggregate v1.0 delta1..delta7 Cleaned Data Clean v1.1 delta7

35. Data Lineage

36. Access Logs

37. Process Logs 2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1702)) - Get metadata for subqueries 17865102 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Get metadata for destination tables 2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1726)) - Get metadata for destination tables 17865345 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Completed getting MetaData in Semantic Analysis 2015-06-04 12:53:31,844 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(10004)) - Completed getting MetaData in Semantic Analysis 17865347 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Not invoking CBO because the statement has too few joins 2015-06-04 12:53:31,846 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:canHandleAstForCbo(10258)) - Not invoking CBO because the statement has too few joins Heart beat 17866695 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException [Error 10044]: Line 2:18 Cannot insert into target table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns. org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:18 Cannot insert into target table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns. at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:6535) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6336) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8977) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8868) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9713)

38. Who is the dataset owner? How can I get access? What does the data mean? How can I reproduce the results? Where is the data I need? HDFS/Hive Metastore Ranger Metastore Falcon

39. Contacts • Martin Ryzl (martin.ryzl@merck.com) • Ivo Lasek (ivo.lasek@merck.com) • http://www.merck.com/ • http://www.msdit.cz/