Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

•

1 like•1,303 views

As Hadoop became mainstream, the need to simplify and speed up analytics processes grew rapidly. Data wrangling emerged as a necessary step in any analytical pipeline, and is often considered to be its crux, taking as much as 80% of an analyst's time. In this presentation we will discuss how data wrangling solutions can be leveraged to streamline, strengthen and improve data analytics initiatives on Hadoop, including use cases from Trifacta customers. Bio: Olivier is EMEA Solutions Lead at Trifacta. He has 7 years experience in analytics with prior roles as technical lead for business analytics at Splunk and quantitative analyst at Accenture and Aon.

Technology

Hadoop User Group London: Data Wrangling on Hadoop
September 8 2016
Olivier de Garrigues, EMEA Solutions Lead

Creating radical productivity
for people who analyze data.
JEFFREY HEER
Co-Founder & CXO
VISUALIZATION
JOE HELLERSTEIN
Co-Founder & CSO
BIG DATA
SEAN KANDEL
Co-Founder & CTO
HUMAN-COMPUTER INTERACTION

What is Data Wrangling?
4
QUESTION ANALYZE INSIGHTDISCOVER STRUCTURE CLEANSE ENRICH VALIDATE PUBLISH

The Bridge Between Raw Data & Analysis
5
v
Ingestion Storage Processing
ANALYSIS & VISUALIZATION
LOBCLEANING ENRICHMENT DISTILLATIONSTRUCTURINGDISCOVERY
End-User Capabilities
IT
GOVERNANCE INTEGRATION AVAILABILTIYSCALABILITYSECURITY
Technical Capabilities

Conventional Approaches Inhibit User Empowerment
Hand-Coding Technical Workﬂow Mapping

Trifacta Approach: It’s All About The Experience
Interact Predict
Preview

TRIFACTA
DATA WRANGLING WORKFLOW
Trifacta. Confidential & Proprietary.
Sample Scale Up
Reﬁne
Sample
Results
Identify/Register Data
1.
Predictive Interaction
2
.
Consume
Schedulers
Monitor and Adjust
3
.
Schedule
Visualization & Analysis
Secure Access

Ingestion Processing Storage
ANALYSIS & CONSUMPTION
v
Discover Structure Clean Enrich Distill
LOB
IT
News
Topics
Time
Trades
Tickers
Date
$
eMails
Recipients
Topics
Phone Logs
Call Details
Recipients
Corporations
Company Relations
Individuals
Financial Services use case: Trader Fraud

Data Wrangling Beneﬁts
➔  Empower the people who know the data best
➔  Accelerate time to value
➔  Lower business risk with more accurate data
➔  Unlock innovation using a wider variety of data

What's hot

Apouc 2014-business-analytics-and-big-data

OUGTH Oracle User Group in Thailand

As a knowledge graph database, Grakn is ideal for storing metadata and data lineage information. Many applications, such as data discovery, data governance, and data marketplaces, depend upon metadata for management. User experiences can be enhanced by leveraging a hyper-scalable graph database like Grakn, rather than traditional graph databases. Additionally, inference-driven use cases predominantly depended on RDF Triple Stores, requiring additional plug-ins to derive the inferences. With Grakn, this can now be achieved natively.

Unified Information Governance, Powered by Knowledge Graph

Vaticle

Learn about the industry's new open metadata standard Egeria, introduced in September by ODPi, The Linux Foundation’s Open Data Platform initiative. Egeria supports the free flow of standardized metadata between different technologies and vendor platforms, enabling organizations to locate, manage and use their data resources more effectively. Explore how Egeria's set of open APIs, types and interchange protocols to allow all metadata repositories to share and exchange metadata. From this common base, it adds governance, discovery and access frameworks for automating the collection, management and use of metadata across an enterprise. The result is an enterprise catalog of data resources that are transparently assessed, governed and used in order to deliver maximum value to the enterprise. This presentation by ODPi Director John Mertic provides an introduction to Egeria, and explores how the standard provides a vendor-neutral approach to data governance. Learn how a group of companies led by ING, IBM and Hortonworks came together through the open source community to re-imagining data governance and delivered Egeria -- to automate the collection, management and use of metadata across organizations of any size and complexity. Learn how Egeria was built on open standards and delivered via Apache 2.0 open source license.

Streamline Data Governance with Egeria: The Industry's First Open Metadata St...

DataWorks Summit

Data Science Application in Business Portfolio & Risk Management

Data Science Thailand

Scalability and Graph Analytics with Neo4j - Stefan Kolmar, Neo4j

Neo4j

In this webinar, AnzoGraph’s graph database guru Barry Zane (former co-founder of Netezza) and data governance author Steve Sarsfield talk about how graph databases fit into the data warehouse modernization trend. They also explore how certain workloads can be better served with an analytical graph database and how today’s technology stacks offer new paradigms for deployment like the cloud, containers and graph analytics.

Should a Graph Database Be in Your Next Data Warehouse Stack?

Cambridge Semantics

In their webinar "Big Data Fabric 2.0 Drives Data Democratization" Ben Szekley, Cambridge Semantics’ SVP of Field Operations, and guest speaker, Forrester’s Noel Yuhanna, author of the Forrester report: “Big Data Fabric 2.0 Drives Data Democratization”, explored why data-driven businesses are making a big data fabric part of their data strategy to minimize data complexity, integrate siloed data, deliver real-time trusted insights, and to create new business opportunities. These are the slides from that webinar.

Big Data Fabric 2.0 Drives Data Democratization

Cambridge Semantics

Using Cloud Automation Technologies to Deliver an Enterprise Data Fabric

Cambridge Semantics

Infochimps + CloudCon: Infinite Monkey Theorem

Infochimps, a CSC Big Data Business

BI architecture drivers have to change to satisfy new requirements in format, volume, latency, hosting, analysis, reporting, and visualization. In this presentation delivered at the 2014 SATURN conference, SoftServe`s Serhiy and Olha showcased a number of reference architectures that address these challenges and speed up the design and implementation process, making it more predictable and economical: - Traditional architecture based on an RDMBS data warehouse but modernized with column-based storage to handle a high load and capacity - NoSQL-based architectures that address Big Data batch and stream-based processing and use popular NoSQL and complex event-processing solutions - Hybrid architecture that combines traditional and NoSQL approaches to achieve completeness that would not be possible with either alone The architectures are accompanied by real-life projects and case studies that the presenters have performed for multiple companies, including Fortune 100 and start-ups.

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

SoftServe

[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics

Infochimps, a CSC Big Data Business

Importance of Big Data Analytics

Impetus Technologies

# Automating Data Science over a Human Genomics Knowledge Base Radouane Oudrhiri, the CTO of Eagle Genomics, will talk about how Eagle Genomics is building a platform for automating data science over a human genomics knowledge base. Rad will dive into the architecture Eagle Genomics and also discuss how Grakn serves as the knowledge base foundation of the system. Rad also give a brief history of databases, semantic expressiveness and how Grakn fits in the big picture. # Radouane Oudrhiri, CTO, Eagle Genomics Radouane has an extensive experience in leading world-class software and data-intensive system developments in different industries from Telecom to Healthcare, Nuclear, Automotive, Financials. Radouane is Lean/Six Sigma Master Black Belt with speciality in high-tech, IT and Software engineering and he is recognised as the leader and early adaptor of Lean/Six Sigma and DFSS to IT and Software. He is a fellow of the Royal Statistical Society (RSS) and member of the ISO Technical Committee (TC69: Applications of Statistical methods) where he is co-author of the Lean & Six Sigma Standard (ISO 18404) as well as the new standard under development (Design for Six Sigma). He is also part of the newly formed international Group on Big Data (nominated by BSI as the UK representative/expert). Radouane has also been Chair of the working group on Measurement Systems for Automated Processes/Systems within the ISPE (International Society for Pharmaceutical Engineering).

Automating Data Science over a Human Genomics Knowledge Base

Vaticle

Data is both our most valuable asset and our biggest ongoing challenge. As data grows in volume, variety and complexity, across applications, clouds and siloed systems, traditional ways of working with data no longer work. Unlike traditional databases, which arrange data in rows, columns and tables, Neo4j has a flexible structure defined by stored relationships between data records. We'll discuss the primary use cases for graph databases Explore the properties of Neo4j that make those use cases possible Look into the visualisation of graphs Introduce how to write queries. Webinar, 23 July 2020

Introduction to Neo4j

Neo4j

Big Data & Analytics continues to redefine business. Data has transitioned from an underused asset to the lifeblood of the organisation, and a critical component of business intelligence, insight and strategy. Big Data Scotland is the largest annual data analytics conference held in Scotland: it is supported by ScotlandIS and The Data Lab and free for delegates to attend. The conference is geared towards senior technologists and business leaders and aims to provide a unique forum for knowledge exchange, discussion and cross-pollination. The programme will explore the evolution of data analytics; looking at key tools and techniques and how these can be applied to deliver practical insight and value. Presentations will span a wide array of topics from Data Wrangling and Visualisation to AI, Chatbots and Industry 4.0. Key Topics • Tools and techniques • Corporate data culture, business processes, digital transformation • Business intelligence, trends, decision making • AI, Real-time Analytics, IoT, Industry 4.0, Robotics • Security, regulation, privacy, consent, anonymization • Data visualisation, interpretation and communication • CRM and Personalisation

Big Data Scotland 2017

Ray Bugg

Big Data Ecosystem

Ivo Vachkov

Combining a Knowledge Graph and Graph Algorithms to Find Hidden Skills at NASA

Neo4j

Big data ecosystem

magda3695

Banking organizations are having to deal with highly complex fraud rings today. Fraudsters are spreading their activities across a large number of transactions and geographical regions in a more coordinated fashion, making traditional anomaly identification methods nearly obsolete. Join us for this on-demand webinar with Jie Wu, Director of Product Marketing, DataStax and guest speaker Scott Heath, Chief Revenue Officer, Expero for a discussion on how to use real-time graph database to effectively detect fraud and reduce risk in real time. View recording: https://youtu.be/XzpTICtJ2Fk Explore all DataStax webinars: https://www.datastax.com/resources/webinars

Webinar - Fighting Bank Fraud with Real-time Graph Database

DataStax

Risk Analytics Using Knowledge Graphs / FIBO with Deep Learning

Cambridge Semantics

What's hot (20)

Apouc 2014-business-analytics-and-big-data

Unified Information Governance, Powered by Knowledge Graph

Streamline Data Governance with Egeria: The Industry's First Open Metadata St...

Data Science Application in Business Portfolio & Risk Management

Scalability and Graph Analytics with Neo4j - Stefan Kolmar, Neo4j

Should a Graph Database Be in Your Next Data Warehouse Stack?

Big Data Fabric 2.0 Drives Data Democratization

Using Cloud Automation Technologies to Deliver an Enterprise Data Fabric

Infochimps + CloudCon: Infinite Monkey Theorem

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics

Importance of Big Data Analytics

Automating Data Science over a Human Genomics Knowledge Base

Introduction to Neo4j

Big Data Scotland 2017

Big Data Ecosystem

Combining a Knowledge Graph and Graph Algorithms to Find Hidden Skills at NASA

Big data ecosystem

Webinar - Fighting Bank Fraud with Real-time Graph Database

Risk Analytics Using Knowledge Graphs / FIBO with Deep Learning

Viewers also liked

At Google Cloud Platform, we're combining the Apache Spark and Hadoop ecosystem with our software and hardware innovations. We want to make these awesome tools easier, faster, and more cost-effective, from 3 to 30,000 cores. This presentation will showcase how Google Cloud Platform is innovating with the goal of bringing the Hadoop ecosystem to everyone. Bio: "I love data because it surrounds us - everything is data. I also love open source software, because it shows what is possible when people come together to solve common problems with technology. While they are awesome on their own, I am passionate about combining the power of open source software with the potential unlimited uses of data. That's why I joined Google. I am a product manager for Google Cloud Platform and manage Cloud Dataproc and Apache Beam (incubating). I've previously spent time hanging out at Disney and Amazon. Beyond Google, love data, amateur radio, Disneyland, photography, running and Legos."

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

huguk

See the new release: https://www.talend.com/products/talend-6/ Talend Summer ’17 delivers the latest cloud and big data innovations so you can get a 360-degree view of your customer across multiple cloud platforms. Accelerate AWS, Microsoft Azure and Google Cloud Platform adoption, with the flexibility and portability to easily reuse development work across the cloud. In this presentation, we break down new features for data quality, ESB, cloud, big data and more.

Talend Summer '17 Release: New Features and Tech Overview

Talend

Role of Analytics in Consumer Packaged Goods Industry

Perceptive Analytics

Extend Governance in Hadoop with Atlas Ecosystem: Waterline, Attivo & Trifacta

DataWorks Summit/Hadoop Summit

Préparation de Données Hadoop avec Trifacta

Victor Coustenoble

Like all consumer packaged goods (CPG) companies, PepsiCo relies on huge volumes of data to accurately replenish its retailers with the appropriate amount and type of product. Across the CPG industry, most analysts exclusively rely on Excel and Access for data wrangling, but as PepsiCo’s data surpassed the capabilities of those tools, they knew they needed a better way. https://www.brighttalk.com/webcast/9573/227935

How PepsiCo's Big Data Strategy is Disrupting CPG Retail Analytics

Hortonworks

Apache Atlas: Tracking dataset lineage across Hadoop components

DataWorks Summit/Hadoop Summit

Comparison of Data Preparation vs. Data Wrangling Programming Languages, Frameworks and Tools in Machine Learning / Deep Learning Projects. A key task to create appropriate analytic models in machine learning or deep learning is the integration and preparation of data sets from various sources like files, databases, big data storages, sensors or social networks. This step can take up to 80% of the whole project. This session compares different alternative techniques to prepare data, including extract-transform-load (ETL) batch processing (like Talend, Pentaho), streaming analytics ingestion (like Apache Storm, Flink, Apex, TIBCO StreamBase, IBM Streams, Software AG Apama), and data wrangling (DataWrangler, Trifacta) within visual analytics. Various options and their trade-offs are shown in live demos using different advanced analytics technologies and open source frameworks such as R, Python, Apache Hadoop, Spark, KNIME or RapidMiner. The session also discusses how this is related to visual analytics tools (like TIBCO Spotfire), and best practices for how the data scientist and business user should work together to build good analytic models. Key takeaways for the audience: - Learn various options for preparing data sets to build analytic models - Understand the pros and cons and the targeted persona for each option - See different technologies and open source frameworks for data preparation - Understand the relation to visual analytics and streaming analytics, and how these concepts are actually leveraged to build the analytic model after data preparation Video Recording / Screencast of this Slide Deck: https://youtu.be/2MR5UynQocs

Data Preparation vs. Inline Data Wrangling in Data Science and Machine Learning

Kai Wähner

Viewers also liked (8)

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

Talend Summer '17 Release: New Features and Tech Overview

Role of Analytics in Consumer Packaged Goods Industry

Extend Governance in Hadoop with Atlas Ecosystem: Waterline, Attivo & Trifacta

Préparation de Données Hadoop avec Trifacta

How PepsiCo's Big Data Strategy is Disrupting CPG Retail Analytics

Apache Atlas: Tracking dataset lineage across Hadoop components

Data Preparation vs. Inline Data Wrangling in Data Science and Machine Learning

Similar to Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

Democratizing Advanced Analytics Propels Instant Analysis Results to the Ubiq...

Dana Gardner

Benefiting from Big Data - A New Approach for the Telecom Industry

Persontyle

In today’s context, the big data market is rapidly undergoing contortions that define market maturity, such as consolidation. Big data refers to large volumes of data. This can be both structured and unstructured data. Big data is data that is huge in size and grows exponentially with time. As the data is too large and complex, traditional data management tools are not sufficient for storing or processing it efficiently. But analyzing big data is crucial to know the patterns and trends to be adopted to improve your business.

Top 10 renowned big data companies

Robert Smith

Insights Success is The Best Business Magazine in the world for enterprises. Being a platform, it focuses distinctively on emerging as well as leading fastest growing companies, their confrontational style of doing businesses and the way of delivering effective and collaborative solutions to strengthen market share. Here, we talk about the leader’s viewpoints & ideas, latest products/services, etc. Insights Success magazine reaches out to all the ‘C’ Level Professionals, VPs, Consultants, VCs, Managers, and HRs of various industries.

Insights success the 10 best hadoop solution provider companies nov 2017

Insights success media and technology pvt ltd

Alitora Innovation Networks

alitora

Seminario OpenStack - Universita' Tor Vergata - Cattedra Cloud Computing

Fausto Pasqualetti

World’s 10 Best Data Integration Solution Providers 2022.pdf

InsightsSuccess4

The Long Road of IT Systems Management Enters the Domain of AIOps-Fueled Auto...

Dana Gardner

AI is moving from its academic roots to the forefront of business and industry

Digital Transformation EXPO Event Series

GraphTalks Frankfurt - Graph Database Überblick

Neo4j

Watch the full session: Denodo DataFest 2016 sessions: https://goo.gl/yVJnti Data virtualization starts with democratizing data access for business users, but goes well beyond to enable entire analytics life cycle. This session will discuss the critical role of data virtualization in the four key phases of big data analytics: Discovery of raw and enriched data, Analytic Exploration, Real-time Operationalization, and Predictive Intervention. In this session, you will learn: • Design of advanced analytics with view towards business goal realization • The role of data virtualization in enabling analytics through four key phases • How to exploit product capabilities relevant to each stage • Creating a system of governed self-service and collaborative analytics This session is part of the Denodo DataFest 2016 event. You can also watch more Denodo DataFest sessions on demand here: https://goo.gl/VXb6M6

Denodo DataFest 2016: Data Science: Operationalizing Analytical Models in Rea...

Denodo

As more companies grow their business in global markets, they discover the need to capture new opportunities in a matter of days rather than months to have competitive advantage and to capture new market share. Their machines are producing terabytes of various data types — video, audio, Microsoft® SharePoint®, sensor data, Microsoft Excel® files — and leaders are searching for the right technologies to capture this data and help provide a better understanding of their business. The HDS big data product roadmap will help customers build a big data enterprise plan that ingests data faster and correlate meaningful data sets to create intelligence that’s easy to consume and helps leaders make the right business decisions. View this webcast to learn about Hitachi’s product roadmap to big data. For more information on HDS Big Data Solutions please visit: http://www.hds.com/solutions/it-strategies/big-data/?WT.ac=us_mg_sol_bigdat

Hitachi Data Systems Big Data Roadmap

Hitachi Vantara

Record manager 8.0 presentation

Andrey Karpov

Modernizing Your IT Infrastructure with Hadoop - Cloudera Summer Webinar Seri...

Cloudera, Inc.

HP Software Performance Tour 2014 - Vincere i Big Data con HP HAVEn

HP Enterprise Italia

How HudsonAlpha Transforms Hybrid Cloud Deployment Complexity Into a Manageme...

Dana Gardner

Even from the “man in the street” perspective, there is a sense that we are living in an increasingly algorithmic world. Self-driving cars, pizza delivery by drone, and smart houses are commonplace. The technologies enabling this revolution are both simultaneously mature and evolving rapidly. In this session, we’ll took a look at a real world problem, the recent global outbreak of the ZIka virus, and used data analytics technologies to gain valuable insights that can assist authorities and the general public to understand and potentially prevent the spread of this disease. Bardess Group, a sponsor of the event and business analytics consulting firm, will demonstrate how huge, extremely jagged data from a variety of sources can be collected and prepared and rapidly made available for analysis. Advanced machine learning and predictive analysis further enhance the value of those insights. Finally, Bardess will make the case that using a systematic approach to conceptually visualize the strategic journey to insightful business analytics, the analytics value chain, can assist any organization prepare for this revolution in analytics. Also see http://cloudera.qlik.com for the demos.

Revolution in Business Analytics-Zika Virus Example

Bardess Group

SIMPosium presentation_Bardess Qlik

Bardess Group

It is clear that all employees must have access to data wherever they are to make decisions. However, tools that allow to share data just as easily as the best collaborative tools such as a google doc or an office 365 should be used. Open source driven by the big data ecosystem and a number of large companies have provided solutions to allow organizations to federate data systems and secure their access. After a quick overview of existing open source solutions, and how such projects can be organized, it will be necessary to detail Dremio implementation, a unique and centralized interface on all your data. Some real feedbacks will conclude the presentation.

All data accessible to all my organization - Presentation at OW2con'19, June...

OW2

Big Data Analytics

Sreedhar Chowdam

Similar to Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta (20)

Democratizing Advanced Analytics Propels Instant Analysis Results to the Ubiq...

Benefiting from Big Data - A New Approach for the Telecom Industry

Top 10 renowned big data companies

Insights success the 10 best hadoop solution provider companies nov 2017

Alitora Innovation Networks

Seminario OpenStack - Universita' Tor Vergata - Cattedra Cloud Computing

World’s 10 Best Data Integration Solution Providers 2022.pdf

The Long Road of IT Systems Management Enters the Domain of AIOps-Fueled Auto...

AI is moving from its academic roots to the forefront of business and industry

GraphTalks Frankfurt - Graph Database Überblick

Denodo DataFest 2016: Data Science: Operationalizing Analytical Models in Rea...

Hitachi Data Systems Big Data Roadmap

Record manager 8.0 presentation

Modernizing Your IT Infrastructure with Hadoop - Cloudera Summer Webinar Seri...

HP Software Performance Tour 2014 - Vincere i Big Data con HP HAVEn

How HudsonAlpha Transforms Hybrid Cloud Deployment Complexity Into a Manageme...

Revolution in Business Analytics-Zika Virus Example

SIMPosium presentation_Bardess Qlik

All data accessible to all my organization - Presentation at OW2con'19, June...

Big Data Analytics

ether.camp - Hackathon & ether.camp intro

huguk

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

huguk

Big organisations have a wealth of rich customer data which opens up huge new opportunities. However, they have the challenge of how to extract value from this data while protecting the privacy of their individual customers. He will talk about the risks organisations face, and what they should do about it. He will survey the techniques which can be used to make data safe for analysis, and talk briefly about how they are solving this problem at Privitar.

Extracting maximum value from data while protecting consumer privacy. Jason ...

huguk

IBM is developing the Watson Ecosystem to leverage its Developer Cloud, APIs, Content Store and Talent Hub. This is part of IBM's recent announcement of the $1B investment in Watson as a new business unit including Silicon Alley NYC headquarters. For the first time, IBM will open up Watson as a development platform in the Cloud to spur innovation and fuel a new ecosystem of entrepreneurial software app providers who will bring forward a new generation of applications infused with Watson's cognitive computing intelligence.

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

huguk

In this talk about Apache Flink we will touch on three main things, an introductory look at Flink, a look under the hood and a demo. * In the introduction we will briefly look at the history of Flink and then go on to the API and different use cases. Here we will also see how it can be deployed in practice and what some of the pitfalls in a cluster setting can be. * In the second section we will look at the streaming execution engine that lies at the heart of Flink. Here we will see what makes it tick and also what distinguishes it from other approaches, such as the mini-batch execution model. Ufuk Celebi - PMC member at Apache Flink and co-founder and software engineer at data Artisans * In the final section we will see a live demo of a fault-tolerant streaming job that performs analysis of the wikipedia edit-stream.

Streaming Dataflow with Apache Flink

huguk

Sean Owen – Director of Data Science @Cloudera Building machine learning models is all well and good, but how do they get productionized into a service? It's a long way from a Python script on a laptop, to a fault-tolerant system that learns continuously, serves thousands of queries per second, and scales to terabytes. The confederation of open source technologies we know as Hadoop now offers data scientists the raw materials from which to assemble an answer: the means to build models but also ingest data and serve queries, at scale. This short talk will introduce Oryx 2, a blueprint for building this type of service on Hadoop technologies. It will survey the problem and the standard technologies and ideas that Oryx 2 combines: Apache Spark, Kafka, HDFS, the lambda architecture, PMML, REST APIs. The talk will touch on a key use case for this architecture -- recommendation engines.

Lambda architecture on Spark, Kafka for real-time large scale ML

huguk

Martin Oberhuber and Eliano Marques, Senior Data Scientists @Think Big International In this talk Think Big International Lead Data Scientists will discuss the options that exist today for engineering and data science teams aiming to use big data patterns to solve new business problems. With the enterprise adoption of the Hadoop ecosystem and the emerging momentum of open source projects like Spark it is becoming mandatory to have an approach that solves for business results but remains flexible to adapt and change with the open source market.

Today’s reality Hadoop with Spark- How to select the best Data Science approa...

huguk

Jonathon Southam: Venture Capital, Funding & Pitching

huguk

Signal Media: Real-Time Media & News Monitoring

huguk

Dean Bryen: Scaling The Platform For Your Startup

huguk

Peter Karney: Intro to the Digital catapult

huguk

Cytora: Real-Time Political Risk Analysis

huguk

Cubitic: Predictive Analytics

huguk

Bird.i: Earth Observation Data Made Social

huguk

Aiseedo: Real Time Machine Intelligence

huguk

This talk will cover the design and implementation decisions that have been key to the success of Apache Spark over other competing cluster computing frameworks. It will be delving into the whitepaper behind Spark and cover the design of Spark RDDs, the abstraction enables the Spark execution engine to be extended to support a wide variety of use cases: Spark SQL, Spark Streaming, MLib and GraphX. RDDs allow Spark to outperform existing models by up to 100x in multi-pass analytics.

Secrets of Spark's success - Deenar Toraskar, Think Reactive

huguk

Technical developments in the area of data warehousing have allowed companies to push their analysis a step further and, therefore, allowed data scientists to deliver more value to business areas. In that session, we will focus on the case of performance marketing at King and demonstrate how we use Hadoop capabilities to exploit user-level data efficiently. That approach results in obtaining a more holistic view in a return-on-investment analysis of TV advertisement.

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

huguk

Hadoop - Looking to the Future By Arun Murthy (Founder of Hortonworks, Creator of YARN) The Apache Hadoop ecosystem began as just HDFS & MapReduce nearly 10 years ago in 2006. Very much like the Ship of Theseus (http://en.wikipedia.org/wiki/Ship_of_Theseus), Hadoop has undergone incredible amount of transformation from multi-purpose YARN to interactive SQL with Hive/Tez to machine learning with Spark. Much more lies ahead: whether you want sub-second SQL with Hive or use SSDs/Memory effectively in HDFS or manage Metadata-driven security policies in Ranger, the Hadoop ecosystem in the Apache Software Foundation continues to evolve to meet new challenges and use-cases. Arun C Murthy has been involved with Apache Hadoop since the beginning of the project - nearly 10 years now. In the beginning he led MapReduce, went on to create YARN and then drove Tez & the Stinger effort to get to interactive & sub-second Hive. Recently he has been very involved in the Metadata and Governance efforts. In between he founded Hortonworks, the first public Hadoop distribution company.

Hadoop - Looking to the Future By Arun Murthy

huguk

By Kevin Schmidt (Head of Data Science at Mind Candy) Luis Vicente (Senior Data Engineer at Mind Candy) For mobile games, constant tweaks are the difference between success and failure. Data and analytics have to be available in real-time, but calculating, for example, uniqueness or newness of a data point requires a list of seen data points - both memory intensive and tricky when using real-time stream processing like Spark Streaming. Probabilistic data structures allow approximation of these properties with a fixed memory representation, and are very well suited for this kind of stream processing. Getting from the theory of approximation to a useful metric at a low error rate even for many millions of users is another story. In our talk we will look at practical ways of achieving this: which approximation we used for selection of useful metrics, why we picked a specific probabilistic data structure, how we stored it in Cassandra as a time series and how we implemented it in Spark Streaming.

Fast real-time approximations using Spark streaming

huguk

The task of “data profiling”—assessing the overall content and quality of a data set—is a core aspect of the analytic experience. Traditionally, profiling was a fairly cut-and-dried task: load the raw numbers into a stat package, run some basic descriptive statistics, and report the output in a summary file or perhaps a simple data visualization. However, data volumes can be so large today that traditional tools and methods for computing descriptive statistics become intractable; even with scalable infrastructure like Hadoop, aggressive optimization and statistical approximation techniques must be used. In this talk Sean will cover technical challenges in keeping data profiling agile in the Big Data era. He will discuss both research results and real-world best practices used by analysts in the field, including methods for sampling, summarizing and sketching data, and the pros and cons of using these various approaches. Sean is Trifacta’s Chief Technical Officer. He completed his Ph.D. at Stanford University, where his research focused on user interfaces for database systems. At Stanford, Sean led development of new tools for data transformation and discovery, such as Data Wrangler. He previously worked as a data analyst at Citadel Investment Group.

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

huguk

More from huguk (20)

ether.camp - Hackathon & ether.camp intro

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

Extracting maximum value from data while protecting consumer privacy. Jason ...

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

Streaming Dataflow with Apache Flink

Lambda architecture on Spark, Kafka for real-time large scale ML

Today’s reality Hadoop with Spark- How to select the best Data Science approa...

Jonathon Southam: Venture Capital, Funding & Pitching

Signal Media: Real-Time Media & News Monitoring

Dean Bryen: Scaling The Platform For Your Startup

Peter Karney: Intro to the Digital catapult

Cytora: Real-Time Political Risk Analysis

Cubitic: Predictive Analytics

Bird.i: Earth Observation Data Made Social

Aiseedo: Real Time Machine Intelligence

Secrets of Spark's success - Deenar Toraskar, Think Reactive

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

Hadoop - Looking to the Future By Arun Murthy

Fast real-time approximations using Spark streaming

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

Recently uploaded

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Scalable LLM APIs for AI and Generative AI Application Development Ettikan Karuppiah, Director/Technologist - NVIDIA Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

apidays

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Recently uploaded (20)

Corporate and higher education May webinar.pptx

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

MS Copilot expands with MS Graph connectors

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Ransomware_Q4_2023. The report. [EN].pdf

Exploring the Future Potential of AI-Enabled Smartphone Processors

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

presentation ICT roal in 21st century education

Automating Google Workspace (GWS) & more with Apps Script

Manulife - Insurer Transformation Award 2024

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Axa Assurance Maroc - Insurer Innovation Award 2024

Data Cloud, More than a CDP by Matt Robison

Apidays New York 2024 - The value of a flexible API Management solution for O...

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

1. Hadoop User Group London: Data Wrangling on Hadoop September 8 2016 Olivier de Garrigues, EMEA Solutions Lead

2. Creating radical productivity for people who analyze data. JEFFREY HEER Co-Founder & CXO VISUALIZATION JOE HELLERSTEIN Co-Founder & CSO BIG DATA SEAN KANDEL Co-Founder & CTO HUMAN-COMPUTER INTERACTION

3. 3 3,000+ Companies 10,000+ Users

4. What is Data Wrangling? 4 QUESTION ANALYZE INSIGHTDISCOVER STRUCTURE CLEANSE ENRICH VALIDATE PUBLISH

5. The Bridge Between Raw Data & Analysis 5 v Ingestion Storage Processing ANALYSIS & VISUALIZATION LOBCLEANING ENRICHMENT DISTILLATIONSTRUCTURINGDISCOVERY End-User Capabilities IT GOVERNANCE INTEGRATION AVAILABILTIYSCALABILITYSECURITY Technical Capabilities

6. Conventional Approaches Inhibit User Empowerment Hand-Coding Technical Workﬂow Mapping

7. Trifacta Approach: It’s All About The Experience Interact Predict Preview

8. Data Wrangling for Financial Fraud

9. TRIFACTA DATA WRANGLING WORKFLOW Trifacta. Confidential & Proprietary. Sample Scale Up Reﬁne Sample Results Identify/Register Data 1. Predictive Interaction 2 . Consume Schedulers Monitor and Adjust 3 . Schedule Visualization & Analysis Secure Access

10. Ingestion Processing Storage ANALYSIS & CONSUMPTION v Discover Structure Clean Enrich Distill LOB IT News Topics Time Trades Tickers Date $ eMails Recipients Topics Phone Logs Call Details Recipients Corporations Company Relations Individuals Financial Services use case: Trader Fraud

11. Data Wrangling Beneﬁts ➔  Empower the people who know the data best ➔  Accelerate time to value ➔  Lower business risk with more accurate data ➔  Unlock innovation using a wider variety of data

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

Similar to Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta (20)

More from huguk

More from huguk (20)

Recently uploaded

Recently uploaded (20)

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta