Big data: Loading your data with flume and sqoop

•

15 likes•8,413 views

This document discusses loading data into Hadoop 2 using SQOOP and Flume. It covers batch loading versus stream loading with SQOOP, describing SQOOP imports and exports. Flume is described as a distributed, reliable, and available system for collecting and aggregating data from various sources and writing to sinks like HDFS. Examples are provided for using both SQOOP and Flume to import data into HDFS and Hive.

Technology

Loading data in
Hadoop 2
with SQOOP and Flume

Christophe Marchal | Software Architect

Flume

Source
Source
Source
Source
Web
Web
Server
Web
Server
Server

Agent
Agent
Agent
Agent

Sink
Sink
Sink
Sink

Channel
Channel
Channel
Channel
HDFS

Flume

vs

SQOOP

● distributed

● Data imports

● reliable (transaction)

● Parallelizes data

● available (backup
routes)
● collecting data
● aggregating data

transfer
● Copies data quickly

Thanks

Christophe Marchal | Software Architect

@toff63

The driving question behind redesigns of countless data collection architectures has often been, ?how can we make the data available to our analytical systems faster?? Increasingly, the go-to solution for this data collection problem is Apache Flume. In this talk, architectures and techniques for designing a low-latency Flume-based data collection and delivery system to enable Hadoop-based analytics are explored. Techniques for getting the data into Flume, getting the data onto HDFS and HBase, and making the data available as quickly as possible are discussed. Best practices for scaling up collection, addressing de-duplication, and utilizing a combination streaming/batch model are described in the context of Flume and Hadoop ecosystem components.

Sqoop2 refactoring for generic data transfer - NYC Sqoop Meetup

gethue

New Data Transfer Tools for Hadoop: Sqoop 2DataWorks Summit

Large-Scale Stream Processing in the Hadoop Ecosystem

DataWorks Summit/Hadoop Summit

Apache Sqoop: A Data Transfer Tool for Hadoop

Cloudera, Inc.

The Future of Apache Storm

DataWorks Summit/Hadoop Summit

ApacheCon-Flume-Kafka-2016Jayesh Thakrar

Apache Flume’s extensible architecture allows Cisco to stream system and application logs from worldwide production data centers to a central Hadoop cluster and Solr. This architecture enables a new level of scalable indexing so that a larger volume of logs is searchable within seconds. Using Solr 4.0′s near real time features together with Hadoop, we can execute mission critical tasks much quicker, improving our ability to meet tight SLAs. At the same time, using the same infrastructure, we can perform large-scale historical analysis and pattern extraction to help further improve our services. This talk will explore our infrastructure and decisions we?ve made to meet key requirements, i.e. high indexing load, high availability and disaster recovery. We will further explore other uses of Flume and SolrCloud within Cisco including dynamic event routing, parsing and multi-tenancy.

Introduction to Sqoop Aaron Kimball Cloudera Hadoop User Group UK

Skills Matter

In this talk of Hadoop User Group UK meeting, Aaron Kimball from Cloudera introduces Sqoop, the open source SQL-to-Hadoop tool. Sqoop helps users perform efficient imports of data from RDBMS sources to Hadoop's distributed file system, where it can be processed in concert with other data sources. Sqoop also allows users to export Hadoop-generated results back to an RDBMS for use with other data pipelines. After this session, users will understand how databases and Hadoop fit together, and how to use Sqoop to move data between these systems. The talk will provide suggestions for best practices when integrating Sqoop and Hadoop in your data processing pipelines. We'll also cover some deeper technical details of Sqoop's architecture, and take a look at some upcoming aspects of Sqoop's development roadmap.

Flexible and Real-Time Stream Processing with Apache Flink

DataWorks Summit

Stream Processing made simple with Kafka

DataWorks Summit/Hadoop Summit

Apache Flume - Streaming data easily to Hadoop from any source for Telco oper...

DataWorks Summit

HBaseConEast2016: Coprocessors – Uses, Abuses and Solutions

Michael Stack

Highlights Of Sqoop2

Alexander Alten

Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident

DataWorks Summit/Hadoop Summit

HBaseConEast2016: HBase and Spark, State of the Art

Michael Stack

October 2016 HUG: Architecture of an Open Source RDBMS powered by HBase and ...

Yahoo Developer Network

Splice Machine is an open-source database that combines the benefits of modern lambda architectures with the full expressiveness of ANSI-SQL. Like lambda architectures, it employs separate compute engines for different workloads - some call this an HTAP database (Hybrid Transactional and Analytical Platform). This talk describes the architecture and implementation of Splice Machine V2.0. The system is powered by a sharded key-value store for fast short reads and writes, and short range scans (Apache HBase) and an in-memory, cluster data flow engine for analytics (Apache Spark). It differs from most other clustered SQL systems such as Impala, SparkSQL, and Hive because it combines analytical processing with a distributed Multi-Value Concurrency Method that provides fine-grained concurrency which is required to power real-time applications. This talk will highlight the Splice Machine storage representation, transaction engine, cost-based optimizer, and present the detailed execution of operational queries on HBase, and the detailed execution of analytical queries on Spark. We will compare and contrast how Splice Machine executes queries with other HTAP systems such as Apache Phoenix and Apache Trafodian. We will end with some roadmap items under development involving new row-based and column-based storage encodings. Speakers: Monte Zweben, is a technology industry veteran. Monte’s early career was spent with the NASA Ames Research Center as the Deputy Chief of the Artificial Intelligence Branch, where he won the prestigious Space Act Award for his work on the Space Shuttle program. He then founded and was the Chairman and CEO of Red Pepper Software, a leading supply chain optimization company, which merged in 1996 with PeopleSoft, where he was VP and General Manager, Manufacturing Business Unit. In 1998, he was the founder and CEO of Blue Martini Software – the leader in e-commerce and multi-channel systems for retailers. Blue Martini went public on NASDAQ in one of the most successful IPOs of 2000, and is now part of JDA. Following Blue Martini, he was the chairman of SeeSaw Networks, a digital, place-based media company. Monte is also the co-author of Intelligent Scheduling and has published articles in the Harvard Business Review and various computer science journals and conference proceedings. He currently serves on the Board of Directors of Rocket Fuel Inc. as well as the Dean’s Advisory Board for Carnegie-Mellon’s School of Computer Science.

Apache kafka

Shravan (Sean) Pabba

Introduction to Spark Streaming

Knoldus Inc.

Spark+flume seattle

Hari Shreedharan

Realtime Detection of DDOS attacks using Apache Spark and MLLib

Ryan Bosshart

In this talk we will show how Hadoop Ecosystem tools like Apache Kafka, Spark, and MLLib can be used in various real-time architectures and how they can be used to perform real-time detection of a DDOS attack. We will explain some of the challenges in building real-time architectures, followed by walking through the DDOS detection example and a live demo. This talk is appropriate for anyone interested in Security, IoT, Apache Kafka, Spark, or Hadoop. Presenter Ryan Bosshart is a Systems Engineer at Cloudera and is the first 3 time presenter at BigDataMadison!

Near-realtime analytics with Kafka and HBase

dave_revell

From oracle to hadoop with Sqoop and other tools

Guy Harrison

HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket

Cloudera, Inc.

Solbase is an exciting new open-source, real-time search engine being developed at Photobucket to service the over 30 million daily search requests Photobucket handles. Solbase replaces Lucene’s file system-based index with HBase. This allows the system to update in real-time and linearly scale to serve millions of daily search requests on a large dataset. This session will explore the architecture of Solbase as well as some of Lucene/Solr’s inherent issues we overcame. Finally, we’ll go over performance metrics of Solbase against production traffic.

Ingest and Stream Processing - What will you choose?

DataWorks Summit/Hadoop Summit

Apache sqoop with an use case

Davin Abraham

ITSS Overview

IMC Institute

What's hot

Introduction to streaming and messaging flume,kafka,SQS,kinesis

Omid Vahdaty

Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...

DataWorks Summit/Hadoop Summit

Large scale near real-time log indexing with Flume and SolrCloud

DataWorks Summit

Introduction to Sqoop Aaron Kimball Cloudera Hadoop User Group UK

Skills Matter

Flexible and Real-Time Stream Processing with Apache Flink

DataWorks Summit

Stream Processing made simple with Kafka

DataWorks Summit/Hadoop Summit

Apache Flume - Streaming data easily to Hadoop from any source for Telco oper...

DataWorks Summit

HBaseConEast2016: Coprocessors – Uses, Abuses and Solutions

Michael Stack

Highlights Of Sqoop2

Alexander Alten

Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident

DataWorks Summit/Hadoop Summit

HBaseConEast2016: HBase and Spark, State of the Art

Michael Stack

October 2016 HUG: Architecture of an Open Source RDBMS powered by HBase and ...

Yahoo Developer Network

Apache kafka

Shravan (Sean) Pabba

Introduction to Spark Streaming

Knoldus Inc.

Spark+flume seattle

Hari Shreedharan

Realtime Detection of DDOS attacks using Apache Spark and MLLib

Ryan Bosshart

Near-realtime analytics with Kafka and HBase

dave_revell

From oracle to hadoop with Sqoop and other tools

Guy Harrison

HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket

Cloudera, Inc.

Ingest and Stream Processing - What will you choose?

DataWorks Summit/Hadoop Summit

What's hot (20)

Introduction to streaming and messaging flume,kafka,SQS,kinesis

Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...

Large scale near real-time log indexing with Flume and SolrCloud

Introduction to Sqoop Aaron Kimball Cloudera Hadoop User Group UK

Flexible and Real-Time Stream Processing with Apache Flink

Stream Processing made simple with Kafka

Apache Flume - Streaming data easily to Hadoop from any source for Telco oper...

HBaseConEast2016: Coprocessors – Uses, Abuses and Solutions

Highlights Of Sqoop2

Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident

HBaseConEast2016: HBase and Spark, State of the Art

October 2016 HUG: Architecture of an Open Source RDBMS powered by HBase and ...

Apache kafka

Introduction to Spark Streaming

Spark+flume seattle

Realtime Detection of DDOS attacks using Apache Spark and MLLib

Near-realtime analytics with Kafka and HBase

From oracle to hadoop with Sqoop and other tools

HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket

Ingest and Stream Processing - What will you choose?

Viewers also liked

Apache sqoop with an use case

Davin Abraham

ITSS Overview

IMC Institute

Big Data Analytics using Mahout

IMC Institute

สมุดกิจกรรม Code for Kids

IMC Institute

Thai Software & Software Market Survey 2015

IMC Institute

Introduction to Apache Sqoop

Avkash Chauhan

Big data processing using Hadoop with Cloudera Quickstart

IMC Institute

Advanced Sqoop

Yogesh Kulkarni

Mobile User and App Analytics in China

IMC Institute

Install Apache Hadoop for Development/Production

IMC Institute

Machine Learning using Apache Spark MLlib

IMC Institute

Kanban boards step by step

Giulio Roggero

Updated version at https://www.slideshare.net/GiulioRoggero/kanban-board-82363781 Do you have a team that works on both project and maintenance? Do you need to organize your team activities? Do you have a lot of activities in parallel and the time to market it's a problem? With a Kanban board and an Agile approach you can solve your problems! Take a look of the animation of the slides to discover how it works.

Flume vs. kafka

Omid Vahdaty

Viewers also liked (13)

Apache sqoop with an use case

ITSS Overview

Big Data Analytics using Mahout

สมุดกิจกรรม Code for Kids

Thai Software & Software Market Survey 2015

Introduction to Apache Sqoop

Big data processing using Hadoop with Cloudera Quickstart

Advanced Sqoop

Mobile User and App Analytics in China

Install Apache Hadoop for Development/Production

Machine Learning using Apache Spark MLlib

Kanban boards step by step

Flume vs. kafka

Similar to Big data: Loading your data with flume and sqoop

Data Integration

Datio Big Data

Bigdata

sweetysweety8

A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)

Robert Metzger

Kafka connect-london-meetup-2016

Gwen (Chen) Shapira

Storm – Streaming Data Analytics at Scale - StampedeCon 2014

StampedeCon

At StampedeCon 2014, Scott Shaw (Hortonworks) and Kit Menke (Enteprise Holdings) presented "Storm – Streaming Data Analytics at Scale" Storm’s primary purpose is to provide real-time analytics against fast moving data before its stored. The use cases range from fraud detection, machine learning, to ETL. Storm has been clocked at over 1 million tuples processed per second per node. It’s fast, scalable, and language agnostic. This session provides an architecture overview as well as a real-world discussion of its use and implementation at Enterprise Holdings.

QCon London - Stream Processing with Apache Flink

Robert Metzger

Building Scalable Data Pipelines - 2016 DataPalooza Seattle

Evan Chan

GOTO Night Amsterdam - Stream processing with Apache Flink

Robert Metzger

Flink history, roadmap and vision

Stephan Ewen

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)

Apache Flink Taiwan User Group

Data ingestion

nitheeshe2

SQOOP PPT

Dushhyant Kumar

K. Tzoumas & S. Ewen – Flink Forward Keynote

Flink Forward

Introduction to sqoop

Uday Vakalapudi

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Visual Mapping of Clickstream DataDataWorks Summit

Hadoop pycon2011ukAditya Sakhuja

Discover HDP2.1: Apache Storm for Stream Data Processing in Hadoop

Hortonworks

For the first time, Hortonworks Data Platform ships with Apache Storm for processing stream data in Hadoop. In this presentation, Himanshu Bari, Hortonworks senior product manager, and Taylor Goetz, Hortonworks engineer and committer to Apache Storm, cover Storm and stream processing in HDP 2.1: + Key requirements of a streaming solution and common use cases + An overview of Apache Storm + Q & A

Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop

Hortonworks

Beginning with HDP 2.1, Hortonworks Data Platform ships with Apache Falcon for Hadoop data governance. Himanshu Bari, Hortonworks senior product manager, and Venkatesh Seetharam, Hortonworks co-founder and committer to Apache Falcon, lead this 30-minute webinar, including: + Why you need Apache Falcon + Key new Falcon features + Demo: Defining data pipelines with replication; policies for retention and late data arrival; managing Falcon server with Ambari

Robust stream processing with Apache Flink

Aljoscha Krettek

Aljoscha Krettek offers a very short introduction to stream processing before diving into writing code and demonstrating the features in Apache Flink that make truly robust stream processing possible, with a focus on correctness and robustness in stream processing. All of this will be done in the context of a real-time analytics application that we’ll be modifying on the fly based on the topics we’re working though, as Aljoscha exercises Flink’s unique features, demonstrates fault recovery, clearly explains why event time is such an important concept in robust, stateful stream processing, and covers the features you need in a stream processor to do robust, stateful stream processing in production. We’ll also use a real-time analytics dashboard to visualize the results we’re computing in real time, allowing us to easily see the effects of the code we’re developing as we go along. Topics include: * Apache Flink * Stateful stream processing * Event time versus processing time * Fault tolerance * State management in the face of faults * Savepoints * Data reprocessing

Similar to Big data: Loading your data with flume and sqoop (20)

Data Integration

Bigdata

A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)

Kafka connect-london-meetup-2016

Storm – Streaming Data Analytics at Scale - StampedeCon 2014

QCon London - Stream Processing with Apache Flink

Building Scalable Data Pipelines - 2016 DataPalooza Seattle

GOTO Night Amsterdam - Stream processing with Apache Flink

Flink history, roadmap and vision

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)

Data ingestion

SQOOP PPT

K. Tzoumas & S. Ewen – Flink Forward Keynote

Introduction to sqoop

Cloud lunch and learn real-time streaming in azure

Visual Mapping of Clickstream Data

Hadoop pycon2011uk

Discover HDP2.1: Apache Storm for Stream Data Processing in Hadoop

Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop

Robust stream processing with Apache Flink

More from Christophe Marchal

Elasticsearch avoiding hotspots

Big data: Loading your data with flume and sqoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (13)

Similar to Big data: Loading your data with flume and sqoop

Similar to Big data: Loading your data with flume and sqoop (20)

More from Christophe Marchal

More from Christophe Marchal (20)

Recently uploaded

Recently uploaded (20)

Big data: Loading your data with flume and sqoop