The DAP - Where YARN, HBase, Kafka and Spark go to Production

•Download as PPTX, PDF•

2 likes•1,131 views

DataWorks Summit/Hadoop Summit

The DAP - Where YARN, HBase, Kafka and Spark go to Production

cask.co
About Me
2

cask.co
The Many Faces of Hadoop
3
Developer Data Scientist IT Pro / Ops
LOB Manager
Advanced Programming
Focuses on App Logic
Basic Programming
Focuses on Data
Configuration & Monitoring
Focuses on Operations
Analysis & Decision Making
Focuses on Insights

cask.co
Big Data Challenges
4

cask.co
Building a Big Data App
5

cask.co
Deploying and Operating a Big Data App
6

cask.co
Today’s Integration Solutions are Silo’ed
7
Data Integration App Integration Cloud Integration Governance

cask.co
Introducing the DAP
8

cask.co9
Enter Cask
Key Customers and Partners
Named a Gartner Cool Vendor 2016
Founded in 2011 by early Hadoop engineers from Facebook and Yahoo!

cask.co
Introducing the Cask Data App Platform
10

cask.co
CDAP Overview
11
Open Source, Integrated Framework for Building and Running
Data Applications on Hadoop and Spark

cask.co12
● Provides a platform with framework level correctness
● Dataset abstractions & self-service data
● One framework: Prototype to Production
● Unified approach across all paradigms
○ Metrics & Log collection
○ Lineage, Audit, Access Control
CDAP Consolidates Big Data App Lifecycle

cask.co
CDAP Extensions
13

cask.co
CDAP Architecture
14
● Application Container Architecture
● Reusable Programming Abstractions
● Global User and Machine Metadata

cask.co
CDAP Application Structure
15

cask.co
CDAP Deployment Architecture
16

cask.co
Hadoop in the Enterprise – Simplified with CDAP
17

cask.co
Common Use Cases
18

cask.co
Summary
19

cask.co
Thank You !
20

More Related Content

What's hot

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

DataWorks Summit

Is your Enterprise Data lake Metadata Driven AND Secure?

Is your Enterprise Data lake Metadata Driven AND Secure?

Is your Enterprise Data lake Metadata Driven AND Secure?

DataWorks Summit/Hadoop Summit

Predictive Analytics and Machine Learning…with SAS and Apache Hadoop

Predictive Analytics and Machine Learning…with SAS and Apache Hadoop

Predictive Analytics and Machine Learning…with SAS and Apache Hadoop

No matter if you are new to Hadoop or have a mature cluster in production, scale will be a critical factor of your success with Hadoop. Are you ready to take the next big step as you scale out your data architecture? Talend and Hortonworks discuss where we will help you learn how to implement an effective big data and Hadoop strategy across your IT infrastructure. You will learn: How to grow a pilot into production How to scale-out architecture & systems affordably How to leverage the flexibility of Hadoop to optimize your data integration processes Recording: http://www.talend.com/resources/webinars/starting-small-and-scaling-big-with-hadoop

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Big Data at your Desk with KNIME

Big Data at your Desk with KNIME

Big Data at your Desk with KNIME

DataWorks Summit/Hadoop Summit

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

DataWorks Summit/Hadoop Summit

Over the last eighteen months, we have seen significant adoption of Hadoop eco-system centric big data processing in Microsoft Azure and Amazon AWS. In this talk we present some of the lessons learned and architectural considerations for cloud-based deployments including security, fault tolerance and auto-scaling. We look at how Hortonworks Data Cloud and Cloudbreak can automate that scaling of Hadoop clusters, showing how it can react dynamically to workloads, and what that can deliver in cost-effective Hadoop-in-cloud deployments.

Cloudy with a chance of Hadoop - real world considerations

Cloudy with a chance of Hadoop - real world considerations

Cloudy with a chance of Hadoop - real world considerations

DataWorks Summit

Modernise your EDW - Data Lake

Modernise your EDW - Data Lake

Modernise your EDW - Data Lake

DataWorks Summit/Hadoop Summit

Splunk-hortonworks-risk-management-oct-2014

Splunk-hortonworks-risk-management-oct-2014

Splunk-hortonworks-risk-management-oct-2014

Back in 2014, our team set out to change the way the world exchanges and collaborates with data. Our vision was to build a single tenant environment for multiple organisations to securely share and consume data. And we did just that, leveraging multiple Hadoop technologies to help our infrastructure scale quickly and securely. Today Data Republic’s technology delivers a trusted platform for hundreds of enterprise level companies to securely exchange, commercialise and collaborate with large datasets. Join Head of Engineering, Juan Delard de Rigoulières and Senior Solutions Architect, Amin Abbaspour as they share key lessons from their team’s journey with Hadoop: * How a startup leveraged a clever combination of Hadoop technologies to build a secure data exchange platform * How Hadoop technologies helped us deliver key solutions around governance, security and controls of data and metadata * An evaluation on the maturity and usefulness of some Hadoop technologies in our environment: Hive, HDFS, Spark, Ranger, Atlas, Knox, Kylin: we've use them all extensively. * Our bold approach to expose APIs directly to end users; as well as the challenges, learning and code we created in the process * Learnings from the front-line: How our team coped with code changes, performance tuning, issues and solutions while building our data exchange Whether you’re an enterprise level business or a start-up looking to scale - this case study discussion offers behind-the-scenes lessons and key tips when using Hadoop technologies to manage data governance and collaboration in the cloud. Speakers: Juan Delard De Rigoulieres, Head of Engineering, Data Republic Pty Ltd Amin Abbaspour, Senior Solutions Architect, Data Republic

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

DataWorks Summit

Discover.hdp2.2.h base.final[2]

Discover.hdp2.2.h base.final[2]

Discover.hdp2.2.h base.final[2]

The world’s largest enterprises run their infrastructure on Oracle, DB2 and SQL and their critical business operations on SAP applications. Organisations need this data to be available in real-time to conduct necessary analytics. However, delivering this heterogeneous data at the speed it’s required can be a huge challenge because of the complex underlying data models and structures and legacy manual processes which are prone to errors and delays. Unlock these silos of data and enable the new advanced analytics platforms by attending this session. Find out how to: • To overcome common challenges faced by enterprises trying to access their SAP data • You can integrate SAP data in real-time with change data capture (CDC) technology • Organisations are using Attunity Replicate for SAP to stream SAP data in to Kafka Speakers: John Hol, Regional Director, Attunity Mike Hollobon, Director Business Development, IBT

Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

DataWorks Summit

Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production ? In this session learn the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of DSX with HDP with the focus on integration, security and model deployment and management. Speakers: Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM Vikram Murali, Program Director, Data Science and Machine Learning, IBM

Scaling Data Science on Big Data

Scaling Data Science on Big Data

Scaling Data Science on Big Data

DataWorks Summit

Swimming Across the Data Lake, Lessons learned and keys to success

Swimming Across the Data Lake, Lessons learned and keys to success

Swimming Across the Data Lake, Lessons learned and keys to success

DataWorks Summit/Hadoop Summit

Format Wars: from VHS and Beta to Avro and Parquet

Format Wars: from VHS and Beta to Avro and Parquet

Format Wars: from VHS and Beta to Avro and Parquet

DataWorks Summit

HDFS: Optimization, Stabilization and Supportability

HDFS: Optimization, Stabilization and Supportability

HDFS: Optimization, Stabilization and Supportability

DataWorks Summit/Hadoop Summit

End-to-End Security and Auditing in a Big Data as a Service Deployment

End-to-End Security and Auditing in a Big Data as a Service Deployment

End-to-End Security and Auditing in a Big Data as a Service Deployment

DataWorks Summit/Hadoop Summit

The world’s largest enterprises run their infrastructure on Oracle, DB2 and SQL and their critical business operations on SAP applications. Organisations need this data to be available in real-time to conduct necessary analytics. However, delivering this heterogeneous data at the speed it’s required can be a huge challenge because of the complex underlying data models and structures and legacy manual processes which are prone to errors and delays. Unlock these silos of data and enable the new advanced analytics platforms by attending this session. Find out how to: • To overcome common challenges faced by enterprises trying to access their SAP data • You can integrate SAP data in real-time with change data capture (CDC) technology • Organisations are using Attunity Replicate for SAP to stream SAP data in to Kafka

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

DataWorks Summit/Hadoop Summit

High Performance Spatial-Temporal Trajectory Analysis with Spark

High Performance Spatial-Temporal Trajectory Analysis with Spark

High Performance Spatial-Temporal Trajectory Analysis with Spark

DataWorks Summit/Hadoop Summit

Hybrid Data Platform

Hybrid Data Platform

Hybrid Data Platform

DataWorks Summit/Hadoop Summit

What's hot (20)

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

Big Data Simplified - Is all about Ab'strakSHeN

Is your Enterprise Data lake Metadata Driven AND Secure?

Is your Enterprise Data lake Metadata Driven AND Secure?

Is your Enterprise Data lake Metadata Driven AND Secure?

Predictive Analytics and Machine Learning…with SAS and Apache Hadoop

Predictive Analytics and Machine Learning…with SAS and Apache Hadoop

Predictive Analytics and Machine Learning…with SAS and Apache Hadoop

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Big Data at your Desk with KNIME

Big Data at your Desk with KNIME

Big Data at your Desk with KNIME

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

Cloudy with a chance of Hadoop - real world considerations

Cloudy with a chance of Hadoop - real world considerations

Cloudy with a chance of Hadoop - real world considerations

Modernise your EDW - Data Lake

Modernise your EDW - Data Lake

Modernise your EDW - Data Lake

Splunk-hortonworks-risk-management-oct-2014

Splunk-hortonworks-risk-management-oct-2014

Splunk-hortonworks-risk-management-oct-2014

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

Discover.hdp2.2.h base.final[2]

Discover.hdp2.2.h base.final[2]

Discover.hdp2.2.h base.final[2]

Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud

Scaling Data Science on Big Data

Scaling Data Science on Big Data

Scaling Data Science on Big Data

Swimming Across the Data Lake, Lessons learned and keys to success

Swimming Across the Data Lake, Lessons learned and keys to success

Swimming Across the Data Lake, Lessons learned and keys to success

Format Wars: from VHS and Beta to Avro and Parquet

Format Wars: from VHS and Beta to Avro and Parquet

Format Wars: from VHS and Beta to Avro and Parquet

HDFS: Optimization, Stabilization and Supportability

HDFS: Optimization, Stabilization and Supportability

HDFS: Optimization, Stabilization and Supportability

End-to-End Security and Auditing in a Big Data as a Service Deployment

End-to-End Security and Auditing in a Big Data as a Service Deployment

End-to-End Security and Auditing in a Big Data as a Service Deployment

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

High Performance Spatial-Temporal Trajectory Analysis with Spark

High Performance Spatial-Temporal Trajectory Analysis with Spark

High Performance Spatial-Temporal Trajectory Analysis with Spark

Hybrid Data Platform

Hybrid Data Platform

Hybrid Data Platform

Viewers also liked

Using Apache Cassandra, Apache Spark and Apache Kafka is a powerful combination to realize streaming analytics, from online recommendations to metrics applications. We will have a look at this setup in the context of popular architectures such as Lambda and Kappa and discuss requirements such as latency and processing guarantees in a practical setup. This talk includes a demo of a streaming analytics application using the Mesosphere DCOS as a deployment environment.

Cassandra summit 2015 - Simplifying Streaming Analytics

Cassandra summit 2015 - Simplifying Streaming Analytics

Cassandra summit 2015 - Simplifying Streaming Analytics

Brenden Matthews

Manage your compactions before they manage you!

Manage your compactions before they manage you!

Manage your compactions before they manage you!

Carlos Juzarte Rolo

Every society has structures which inﬂuence collective behavior. With all of the possible conﬁgurations of people in a population, the question emerges for designers of how to implement a method to use the collective information and create a successful design solution. We show how the fractal shape can emerge from a generative process that takes information on the scale of individuals or groups, and uses it to design a permanent infrastructure on the scale of a city. In this sense, we grow cities consisting of individuals and roads, starting from just individuals.

A Generative Method for Infrastructure Emergence

A Generative Method for Infrastructure Emergence

A Generative Method for Infrastructure Emergence

Ambari talk at Hadoop Summit, Tokyo 2016 Abstract Apache Ambari has become an indispensable tool for operating Hadoop clusters from as small as 10s of nodes to 1000s of nodes. Ambari’s deep knowledge of the Hadoop stack allows it to deploy a cluster within minutes and manage the entire lifecycle: scaling, security, upgrades, and more. This talk will cover the central features important to cluster operators and the latest innovations from the community. We will discuss automatically deploying clusters with Blueprints, adding custom services, scaling the number of hosts as the data needs grow, adding High Availability for critical services, securing with MIT kerberos, and upgrading the Hadoop stack with features like Rolling & Express Upgrade. More advanced users will also be interested in using Ambari’s powerful REST API to automate workflows. For users and data scientists, Ambari provides LDAP sync, Role-Based Access Control to handle user permissions, and a framework to host Ambari Views such as as the newly added Views for Hive, Oozie, Capacity Scheduler, Tez, Storm, and Zeppelin. Lastly, we will cover how to monitor the health of the cluster via Alerts and troubleshoot problems by using new features like LogSearch and Ambari Metrics Systems integrated with Grafana UI.

Streamline Hadoop DevOps with Apache Ambari

Streamline Hadoop DevOps with Apache Ambari

Streamline Hadoop DevOps with Apache Ambari

Cassandra ne permet ni jointure, ni agrégats et limite drastiquement vos capacités à requêter vos données pour permettre une scalabilité linéaire dans une architecture masterless. L'outil de choix pour effectuer des traitements analytiques sur vos tables Cassandra est Spark mais ce dernier complexifie des opérations pourtant simples en SQL. SparkSQL permet de retrouver une syntaxe SQL dans Spark et nous allons voir comment l'utiliser en Scala, Java et en Python pour travailler sur des tables Cassandra, et retrouver jointures et agrégats (entre autres).

SparkSQL et Cassandra - Tool In Action Devoxx 2015

SparkSQL et Cassandra - Tool In Action Devoxx 2015

SparkSQL et Cassandra - Tool In Action Devoxx 2015

Alexander DEJANOVSKI

A complete hadoop stack

A complete hadoop stack

A complete hadoop stack

The SparkSQL things you maybe confuse

The SparkSQL things you maybe confuse

The SparkSQL things you maybe confuse

Speaker: Russ Savage, from Cask Big Data Applications Meetup, 09/14/2016 Palo Alto, CA More info here: http://www.meetup.com/BigDataApps/ Link to talk: https://youtu.be/4j78g3WvC4Y About the talk: As data lake sizes grow, and more users begin exploring and including that data in their everyday analysis, keeping track of the sources for data becomes critical. Understanding how a dataset was generated and who is using it allows users and companies to ensure their analysis is leveraging the most accurate and up to date information. In this talk, we will explore the different techniques available to keep track of your data in your data lake and demonstrate how we at Cask approached and attempted to mitigate this issue.

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

Getting started with SparkSQL - Desert Code Camp 2016

Getting started with SparkSQL - Desert Code Camp 2016

Getting started with SparkSQL - Desert Code Camp 2016

Speaker: Yuanchi Ning from Uber Big Data Applications Meetup, 08/19/2015 Palo Alto, CA More info here: http://www.meetup.com/BigDataApps/ Link to talk: https://www.youtube.com/watch?v=SY1YSU8cFLI About the talk: Athena is a stream processing platform for Uber's near real time analytics applications, built using Samza. We will be discussing some of the existing and upcoming use cases and how they impact the Uber partners / riders. The talk will go through the tooling built around Samza for easier user on-boarding - such as deployment manager, integration with typesafe config system, unit test framework, Graphite integration, metric whitelisting and so on. We'll also go over some of the issues observed during this process.

Introducing Athena: 08/19 Big Data Application Meetup, Talk #3

Introducing Athena: 08/19 Big Data Application Meetup, Talk #3

Introducing Athena: 08/19 Big Data Application Meetup, Talk #3

TITLE: ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating) SPEAKER: Poorna Chandra, Cask Data DATE: May 25, 2016 LOCATION: PhoenixCon, San Francisco CA http://www.meetup.com/SF-Bay-Area-Apache-Phoenix-Meetup/events/230545182/ TALK ABSTRACT: This talk is about how Apache Phoenix added support for ACID transactions using Apache Tephra™ (incubating), an open source transaction engine on top of Apache HBase. To start off with, the talk will examine the need for Phoenix data operations to be transactional. The talk will then discuss how Tephra implements transactional semantics using Optimistic Concurrency Control by giving an overview of the transactional model of Tephra along with the high level architecture. The talk will then describe the details of Phoenix integration with Tephra, and present performance some benchmark results for Phoenix operations with transactions. The talk will conclude with discussion on some challenges with scaling Tephra and potential solutions.

ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...

ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...

ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...

Cask Webinar Date: 08/10/2016 Link to video recording: https://www.youtube.com/watch?v=XUkANr9iag0 In this webinar, Nitin Motgi, CTO of Cask, walks through the new capabilities of CDAP 3.5 and explains how your organization can benefit. Some of the highlights include: - Enterprise-grade security - Authentication, authorization, secure keystore for storing configurations. Plus integration with Apache Sentry and Apache Ranger. - Preview mode - Ability to preview and debug data pipelines before deploying them. - Joins in Cask Hydrator - Capabilities to join multiple data sources in data pipelines - Real-time pipelines with Spark Streaming - Drag & drop real-time pipelines using Spark Streaming. - Data usage analytics - Ability to report application usage of data sets. - And much more!

Webinar: What's new in CDAP 3.5?

Webinar: What's new in CDAP 3.5?

Webinar: What's new in CDAP 3.5?

Galvanise NYC - Scaling R with Hadoop & Spark. V1.0

Galvanise NYC - Scaling R with Hadoop & Spark. V1.0

Galvanise NYC - Scaling R with Hadoop & Spark. V1.0

HBaseConEast2016: HBase and Spark, State of the Art

HBaseConEast2016: HBase and Spark, State of the Art

HBaseConEast2016: HBase and Spark, State of the Art

Many people refer to Apache Hadoop as their system of choice for big data management but few actually use just Apache Hadoop. Hadoop has become a proxy for a much larger system which has HDFS storage at its core. The Apache Hadoop based "big data stack" has changed dramatically over the past 24 months and will chance even more over the next 24 months. This talk talks about trends in the evolution of the Hadoop stack, change in architecture and changes in the kinds of use cases that are supported. It will also talk about the role of interoperability and cohesion in the Apache Hadoop stack and the role of Apache Bigtop in this regard.

Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...

Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...

Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...

Transactions for Apache HBase™: Apache Tephra provides globally consistent transactions on top of Apache HBase. While HBase provides strong consistency with row- or region-level ACID operations, it sacrifices cross-region and cross-table consistency in favor of scalability. This trade-off requires application developers to handle the complexity of ensuring consistency when their modifications span region boundaries. By providing support for global transactions that span regions, tables, or multiple RPCs, Tephra simplifies application development on top of HBase, without a significant impact on performance or scalability for many workloads.

Transactions Over Apache HBase

Transactions Over Apache HBase

Transactions Over Apache HBase

Apache Spark is the next big data processing tool for Data Scientist. As seen on the recent StackOverflow analysis, it's the hottest big data technology on their site! In this talk, I'll use the PySpark interface to leverage the speed and performance of Apache Spark. I'll focus on the end to end workflow for getting data into a distributed platform, and leverage Spark to process the data for advanced analytics. I'll discuss the popular Spark APIs used for data preparation, SQL analysis, and ML algorithms. I'll explain the performance differences between Scala and Python, and how Spark has bridged the gap in performance. I'll focus on PySpark as the interface to the platform, and walk through a demo to showcase the APIs. Talk Overview: Spark's Architecture. What's out now and what's in Spark 2.0Spark APIs: Most common APIs used by Spark Common misconceptions and proper techniques for using Spark. Demo: Walk through ETL of the Reddit dataset. SparkSQL Analytics + Visualizations of the Dataset using MatplotLibSentiment Analysis on Reddit Comments

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

Miklos Christine

Given at Data Day Seattle 2015. Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Data Science at Scale: Using Apache Spark for Data Science at Bitly

In this talk, we’ll discuss technical designs of support of HBase as a “native” data source to Spark SQL to achieve both query and load performance and scalability: near-precise execution locality of query and loading, fine-tuned partition pruning, predicate pushdown, plan execution through coprocessor, and optimized and fully parallelized bulk loader. Point and range queries on dimensional attributes will benefit particularly well from the techniques. Preliminary test results vs. established SQL-on-HBase technologies will be provided. The speaker will also share the future plan and real-world use cases, particularly in the telecom industry.

Spark meetup v2.0.5

Spark meetup v2.0.5

Spark meetup v2.0.5

Parquet Hadoop Summit 2013

Parquet Hadoop Summit 2013

Parquet Hadoop Summit 2013

Viewers also liked (20)

Cassandra summit 2015 - Simplifying Streaming Analytics

Cassandra summit 2015 - Simplifying Streaming Analytics

Cassandra summit 2015 - Simplifying Streaming Analytics

Manage your compactions before they manage you!

Manage your compactions before they manage you!

Manage your compactions before they manage you!

A Generative Method for Infrastructure Emergence

A Generative Method for Infrastructure Emergence

A Generative Method for Infrastructure Emergence

Streamline Hadoop DevOps with Apache Ambari

Streamline Hadoop DevOps with Apache Ambari

Streamline Hadoop DevOps with Apache Ambari

SparkSQL et Cassandra - Tool In Action Devoxx 2015

SparkSQL et Cassandra - Tool In Action Devoxx 2015

SparkSQL et Cassandra - Tool In Action Devoxx 2015

A complete hadoop stack

A complete hadoop stack

A complete hadoop stack

The SparkSQL things you maybe confuse

The SparkSQL things you maybe confuse

The SparkSQL things you maybe confuse

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

Getting started with SparkSQL - Desert Code Camp 2016

Getting started with SparkSQL - Desert Code Camp 2016

Getting started with SparkSQL - Desert Code Camp 2016

Introducing Athena: 08/19 Big Data Application Meetup, Talk #3

Introducing Athena: 08/19 Big Data Application Meetup, Talk #3

Introducing Athena: 08/19 Big Data Application Meetup, Talk #3

ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...

ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...

ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...

Webinar: What's new in CDAP 3.5?

Webinar: What's new in CDAP 3.5?

Webinar: What's new in CDAP 3.5?

Galvanise NYC - Scaling R with Hadoop & Spark. V1.0

Galvanise NYC - Scaling R with Hadoop & Spark. V1.0

Galvanise NYC - Scaling R with Hadoop & Spark. V1.0

HBaseConEast2016: HBase and Spark, State of the Art

HBaseConEast2016: HBase and Spark, State of the Art

HBaseConEast2016: HBase and Spark, State of the Art

Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...

Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...

Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...

Transactions Over Apache HBase

Transactions Over Apache HBase

Transactions Over Apache HBase

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Spark meetup v2.0.5

Spark meetup v2.0.5

Spark meetup v2.0.5

Parquet Hadoop Summit 2013

Parquet Hadoop Summit 2013

Parquet Hadoop Summit 2013

Similar to The DAP - Where YARN, HBase, Kafka and Spark go to Production

Hadoop or Spark: is it an either-or proposition? An exodus away from Hadoop to Spark is picking up steam in the news headlines and talks! Away from marketing fluff and politics, this talk analyzes such news and claims from a technical perspective. In practical ways, while referring to components and tools from both Hadoop and Spark ecosystems, this talk will show that the relationship between Hadoop and Spark is not of an either-or type but can take different forms such as: evolution, transition, integration, alternation and complementarity.

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Transitioning Compute Models: Hadoop MapReduce to Spark

Transitioning Compute Models: Hadoop MapReduce to Spark

Transitioning Compute Models: Hadoop MapReduce to Spark

spark_v1_2

Frank Schroeter

Analysts predict that the Hadoop market will reach $50.2 billion USD by 2020.1 Applications driving these large expenditures are some of the most important workloads for businesses today including: • Analyzing clickstream data, including site-side clicks and web media tags. • Measuring sentiment by scanning product feedback, blog feeds, social media comments, and Twitter streams. • Analysis of behavior and risk by capturing vehicle telematics. • Optimizing product performance and utilization by gathering data from built-in sensors. • Tracking and analyzing people and material movement with location-aware systems. • Identifying system performance and intrusion attempts by analyzing server and network log. • Enabling automatic document and speech categorization. • Extracting learning from digitized images, voice, video, and other media types. Predictive analytics on large data sets provides organizations with a key opportunity to improve a broad variety of business outcomes, and many have embraced Apache Hadoop as the platform of choice. In the last few years, large businesses have adopted Apache Hadoop as a next-generation data platform, one capable of managing large data assets in a way that is flexible, scalable, and relatively low cost. However, to realize predictive benefits of big data, organizations must be able to develop or hire individuals with the requisite statistics skills, then provide them with a platform for analyzing massive data assets collected in Hadoop “data lakes.” As users adopted Hadoop, many discovered performance and complexity limited Hadoop’s use for broad predictive analytics use. In response, the Hadoop community has focused on the Apache Spark platform to provide Hadoop with significant performance improvements. With Spark atop Hadoop, users can leverage Hadoop’s big-data management capabilities while achieving new performance levels by running analytics in Apache Spark. What remains is a challenge—conquering the complexity of Hadoop when developing predictive analytics applications. In this white paper, we’ll describe how Microsoft R Server helps data scientists, actuaries, risk analysts, quantitative analysts, product planners, and other R users to capture the benefits of Apache Spark on Hadoop by providing a straightforward platform that eliminates much of the complexity of using Spark and Hadoop to conduct analyses on large data assets.

Accelerating R analytics with Spark and Microsoft R Server for Hadoop

Accelerating R analytics with Spark and Microsoft R Server for Hadoop

Accelerating R analytics with Spark and Microsoft R Server for Hadoop

Willy Marroquin (WillyDevNET)

The Big Picture on Hadoop

The Big Picture on Hadoop

The Big Picture on Hadoop

Spark vs Hadoop: Which Big Data Framework to Choose?

Spark vs Hadoop: Which Big Data Framework to Choose?

Spark vs Hadoop: Which Big Data Framework to Choose?

Hadoop Vs Spark — Choosing the Right Big Data Framework

Hadoop Vs Spark — Choosing the Right Big Data Framework

Hadoop Vs Spark — Choosing the Right Big Data Framework

View on big data technologies

View on big data technologies

View on big data technologies

Krisshhna Daasaarii

In spite of investments in big data lakes, there is wide use of expensive proprietary products for data ingestion, integration, and transformation (ETL) while bringing and processing data on the lake. Enterprises have successfully tested Apache Spark for its versatility and strengths as a distributed computing framework that can completely handle all needs for data processing, analytics, and machine learning workloads. Since the Hadoop distributions and the public cloud already include Apache Spark, there is nothing new to be procured. However, the skills required to put Spark to good use are typically unavailable today. In this webinar, we will discuss how Apache Spark can be an inexpensive enterprise backbone for all types of data processing workloads. We will also demo how a visual framework on top of Apache Spark makes it much more viable. The following scenarios will be covered: On-Prem Data quality and ETL with Apache Spark using pre-built operators Advanced monitoring of Spark pipelines On Cloud Visual interactive development of Apache Spark Structured Streaming pipelines IoT use-case with event-time, late-arrival and watermarks Python based predictive analytics running on Spark

Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...

Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...

Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...

Impetus Technologies

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Athens BigData Meetup - Sept 17

Athens BigData Meetup - Sept 17

Athens BigData Meetup - Sept 17

Learn more about the SnapLogic Fall 2014 Release in this impact brief from Enterprise Management Associates (EMA): In September 2014 SnapLogic announced the Fall 2014 release of its integration solution with new data wrangling capabilities and major SnapReduce enhancements for Hadoop 2.0 deployments. This ENTERPRISE MANAGEMENT ASSOCIATES® (EMA™) impact brief covers the announcement, identifying significant new capabilities and direction that the new release provides.

SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments

SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments

SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments

Custom Development - SAP HANA

Custom Development - SAP HANA

Custom Development - SAP HANA

Sandish3Certs

Sandish Kumar H N

Introduction to Big Data Analytics using Apache Spark on HDInsights on Azure (SaaS) and/or HDP on Azure(PaaS) This workshop will provide an introduction to Big Data Analytics using Apache Spark using the HDInsights on Azure (SaaS) and/or HDP deployment on Azure(PaaS) . There will be a short lecture that includes an introduction to Spark, the Spark components. Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes. The lecture will be followed by demo . There will be a short lecture on Hadoop and how Spark and Hadoop interact and compliment each other. You will learn how to move data into HDFS using Spark APIs, create Hive table, explore the data with Spark and SQL, transform the data and then issue some SQL queries. We will be using Scala and/or PySpark for labs.

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Analytics and machine learning continue to be the top use cases for deploying big data platforms such as Hadoop. SAS recognised the potential and power of Hadoop platform early on and has been integrating analytical solutions with Hadoop to leverage the power and flexibility of Hadoop for analytical workloads. The combination of SAS and Hadoop offers developers and organisations an approach that can accelerate the development and deployment of big data analytics applications that are mature, proven and scalable. Furthermore, by giving developers and analysts analytical applications that are rich, proven and collaborative, SAS allows more users across different skill levels to unleash the value of data stored in big data platform more easily and quickly. In this session, we will cover common big data analytics use cases, the depth and breadth of SAS analytical capabilities on Hadoop, and how SAS solutions are integrated into the Hadoop ecosystem via technologies such as Hive, YARN and Spark. Speaker Felix Liao, SAS Institute Australia & New Zealand

Accelerate Your Big Data Analytics Efforts with SAS and Hadoop

Accelerate Your Big Data Analytics Efforts with SAS and Hadoop

Accelerate Your Big Data Analytics Efforts with SAS and Hadoop

DataWorks Summit

Emerging trends in data analytics

Emerging trends in data analytics

Emerging trends in data analytics

Wei-Chiu Chuang

Storm Demo Talk - Colorado Springs May 2015

Storm Demo Talk - Colorado Springs May 2015

Storm Demo Talk - Colorado Springs May 2015

Learn Apache Spark: A Comprehensive Guide

Learn Apache Spark: A Comprehensive Guide

Learn Apache Spark: A Comprehensive Guide

Leveraging SAP HANA with Apache Hadoop and SAP Analytics

Leveraging SAP HANA with Apache Hadoop and SAP Analytics

Leveraging SAP HANA with Apache Hadoop and SAP Analytics

Similar to The DAP - Where YARN, HBase, Kafka and Spark go to Production (20)

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Transitioning Compute Models: Hadoop MapReduce to Spark

Transitioning Compute Models: Hadoop MapReduce to Spark

Transitioning Compute Models: Hadoop MapReduce to Spark

spark_v1_2

Accelerating R analytics with Spark and Microsoft R Server for Hadoop

Accelerating R analytics with Spark and Microsoft R Server for Hadoop

Accelerating R analytics with Spark and Microsoft R Server for Hadoop

The Big Picture on Hadoop

The Big Picture on Hadoop

The Big Picture on Hadoop

Spark vs Hadoop: Which Big Data Framework to Choose?

Spark vs Hadoop: Which Big Data Framework to Choose?

Spark vs Hadoop: Which Big Data Framework to Choose?

Hadoop Vs Spark — Choosing the Right Big Data Framework

Hadoop Vs Spark — Choosing the Right Big Data Framework

Hadoop Vs Spark — Choosing the Right Big Data Framework

View on big data technologies

View on big data technologies

View on big data technologies

Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...

Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...

Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala

Athens BigData Meetup - Sept 17

Athens BigData Meetup - Sept 17

Athens BigData Meetup - Sept 17

SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments

SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments

SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments

Custom Development - SAP HANA

Custom Development - SAP HANA

Custom Development - SAP HANA

Sandish3Certs

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Accelerate Your Big Data Analytics Efforts with SAS and Hadoop

Accelerate Your Big Data Analytics Efforts with SAS and Hadoop

Accelerate Your Big Data Analytics Efforts with SAS and Hadoop

Emerging trends in data analytics

Emerging trends in data analytics

Emerging trends in data analytics

Storm Demo Talk - Colorado Springs May 2015

Storm Demo Talk - Colorado Springs May 2015

Storm Demo Talk - Colorado Springs May 2015

Learn Apache Spark: A Comprehensive Guide

Learn Apache Spark: A Comprehensive Guide

Learn Apache Spark: A Comprehensive Guide

Leveraging SAP HANA with Apache Hadoop and SAP Analytics

Leveraging SAP HANA with Apache Hadoop and SAP Analytics

Leveraging SAP HANA with Apache Hadoop and SAP Analytics

More from DataWorks Summit/Hadoop Summit

Running Apache Spark & Apache Zeppelin in Production

Running Apache Spark & Apache Zeppelin in Production

Running Apache Spark & Apache Zeppelin in Production

DataWorks Summit/Hadoop Summit

State of Security: Apache Spark & Apache Zeppelin

State of Security: Apache Spark & Apache Zeppelin

State of Security: Apache Spark & Apache Zeppelin

DataWorks Summit/Hadoop Summit

Unleashing the Power of Apache Atlas with Apache Ranger

Unleashing the Power of Apache Atlas with Apache Ranger

Unleashing the Power of Apache Atlas with Apache Ranger

DataWorks Summit/Hadoop Summit

Enabling Digital Diagnostics with a Data Science Platform

Enabling Digital Diagnostics with a Data Science Platform

Enabling Digital Diagnostics with a Data Science Platform

DataWorks Summit/Hadoop Summit

Revolutionize Text Mining with Spark and Zeppelin

Revolutionize Text Mining with Spark and Zeppelin

Revolutionize Text Mining with Spark and Zeppelin

DataWorks Summit/Hadoop Summit

Double Your Hadoop Performance with Hortonworks SmartSense

Double Your Hadoop Performance with Hortonworks SmartSense

Double Your Hadoop Performance with Hortonworks SmartSense

DataWorks Summit/Hadoop Summit

Hadoop Crash Course

Hadoop Crash Course

Hadoop Crash Course

DataWorks Summit/Hadoop Summit

Data Science Crash Course

Data Science Crash Course

Data Science Crash Course

DataWorks Summit/Hadoop Summit

Apache Spark Crash Course

Apache Spark Crash Course

Apache Spark Crash Course

DataWorks Summit/Hadoop Summit

Dataflow with Apache NiFi

Dataflow with Apache NiFi

Dataflow with Apache NiFi

DataWorks Summit/Hadoop Summit

Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats. SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc. In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.

Schema Registry - Set you Data Free

Schema Registry - Set you Data Free

Schema Registry - Set you Data Free

DataWorks Summit/Hadoop Summit

There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time. The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

DataWorks Summit/Hadoop Summit

DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

DataWorks Summit/Hadoop Summit

QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful. At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.

Mool - Automated Log Analysis using Data Science and ML

Mool - Automated Log Analysis using Data Science and ML

Mool - Automated Log Analysis using Data Science and ML

DataWorks Summit/Hadoop Summit

Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together. This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear: • How and why the business and IT requirements originated • How we leverage the platform to fulfill security and production requirements • How we organize a community to: o Guard all the players, no one gets left on the ground! o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead) • What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match! DETAILS This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.

How Hadoop Makes the Natixis Pack More Efficient

How Hadoop Makes the Natixis Pack More Efficient

How Hadoop Makes the Natixis Pack More Efficient

DataWorks Summit/Hadoop Summit

HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.

HBase in Practice

HBase in Practice

HBase in Practice

DataWorks Summit/Hadoop Summit

There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases. In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.

The Challenge of Driving Business Value from the Analytics of Things (AOT)

The Challenge of Driving Business Value from the Analytics of Things (AOT)

The Challenge of Driving Business Value from the Analytics of Things (AOT)

DataWorks Summit/Hadoop Summit

In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

DataWorks Summit/Hadoop Summit

In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs. Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

DataWorks Summit/Hadoop Summit

While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.

Backup and Disaster Recovery in Hadoop

Backup and Disaster Recovery in Hadoop

Backup and Disaster Recovery in Hadoop

DataWorks Summit/Hadoop Summit

More from DataWorks Summit/Hadoop Summit (20)

Running Apache Spark & Apache Zeppelin in Production

Running Apache Spark & Apache Zeppelin in Production

Running Apache Spark & Apache Zeppelin in Production

State of Security: Apache Spark & Apache Zeppelin

State of Security: Apache Spark & Apache Zeppelin

State of Security: Apache Spark & Apache Zeppelin

Unleashing the Power of Apache Atlas with Apache Ranger

Unleashing the Power of Apache Atlas with Apache Ranger

Unleashing the Power of Apache Atlas with Apache Ranger

Enabling Digital Diagnostics with a Data Science Platform

Enabling Digital Diagnostics with a Data Science Platform

Enabling Digital Diagnostics with a Data Science Platform

Revolutionize Text Mining with Spark and Zeppelin

Revolutionize Text Mining with Spark and Zeppelin

Revolutionize Text Mining with Spark and Zeppelin

Double Your Hadoop Performance with Hortonworks SmartSense

Double Your Hadoop Performance with Hortonworks SmartSense

Double Your Hadoop Performance with Hortonworks SmartSense

Hadoop Crash Course

Hadoop Crash Course

Hadoop Crash Course

Data Science Crash Course

Data Science Crash Course

Data Science Crash Course

Apache Spark Crash Course

Apache Spark Crash Course

Apache Spark Crash Course

Dataflow with Apache NiFi

Dataflow with Apache NiFi

Dataflow with Apache NiFi

Schema Registry - Set you Data Free

Schema Registry - Set you Data Free

Schema Registry - Set you Data Free

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

Mool - Automated Log Analysis using Data Science and ML

Mool - Automated Log Analysis using Data Science and ML

Mool - Automated Log Analysis using Data Science and ML

How Hadoop Makes the Natixis Pack More Efficient

How Hadoop Makes the Natixis Pack More Efficient

How Hadoop Makes the Natixis Pack More Efficient

HBase in Practice

HBase in Practice

HBase in Practice

The Challenge of Driving Business Value from the Analytics of Things (AOT)

The Challenge of Driving Business Value from the Analytics of Things (AOT)

The Challenge of Driving Business Value from the Analytics of Things (AOT)

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

Backup and Disaster Recovery in Hadoop

Backup and Disaster Recovery in Hadoop

Backup and Disaster Recovery in Hadoop

Recently uploaded

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

What Are The Drone Anti-jamming Systems Technology?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Recently uploaded (20)

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Handwritten Text Recognition for manuscripts and early printed texts

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

What Are The Drone Anti-jamming Systems Technology?

What Are The Drone Anti-jamming Systems Technology?

What Are The Drone Anti-jamming Systems Technology?

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

The DAP - Where YARN, HBase, Kafka and Spark go to Production

1. The DAP – Where YARN, HBase, Kafka and Spark Go to Production Hadoop Summit - June 30th, 2016 cask.co Cask, CDAP, Cask Hydrator and Cask Tracker are trademarks or registered trademarks of Cask Data. Apache Spark, Spark, the Spark logo, Apache Hadoop, Hadoop and the Hadoop logo are trademarks or registered trademarks of the Apache Software Foundation. All other trademarks and registered trademarks are the property of their respective owners.

2. cask.co About Me 2

3. cask.co The Many Faces of Hadoop 3 Developer Data Scientist IT Pro / Ops LOB Manager Advanced Programming Focuses on App Logic Basic Programming Focuses on Data Configuration & Monitoring Focuses on Operations Analysis & Decision Making Focuses on Insights

4. cask.co Big Data Challenges 4

5. cask.co Building a Big Data App 5

6. cask.co Deploying and Operating a Big Data App 6

7. cask.co Today’s Integration Solutions are Silo’ed 7 Data Integration App Integration Cloud Integration Governance

8. cask.co Introducing the DAP 8

9. cask.co9 Enter Cask Key Customers and Partners Named a Gartner Cool Vendor 2016 Founded in 2011 by early Hadoop engineers from Facebook and Yahoo!

10. cask.co Introducing the Cask Data App Platform 10

11. cask.co CDAP Overview 11 Open Source, Integrated Framework for Building and Running Data Applications on Hadoop and Spark

12. cask.co12 ● Provides a platform with framework level correctness ● Dataset abstractions & self-service data ● One framework: Prototype to Production ● Unified approach across all paradigms ○ Metrics & Log collection ○ Lineage, Audit, Access Control CDAP Consolidates Big Data App Lifecycle

13. cask.co CDAP Extensions 13

14. cask.co CDAP Architecture 14 ● Application Container Architecture ● Reusable Programming Abstractions ● Global User and Machine Metadata

15. cask.co CDAP Application Structure 15

16. cask.co CDAP Deployment Architecture 16

17. cask.co Hadoop in the Enterprise – Simplified with CDAP 17

18. cask.co Common Use Cases 18

19. cask.co Summary 19

20. cask.co Thank You ! 20