While the last few years have seen great advancements in computing paradigms for big data stores, there remains one critical bottleneck in this architecture - the ingestion process. Instead of immediate insights into the data, a poor ingestion process can cause headaches and problems to no end. On the other hand, a well-designed ingestion infrastructure should give you real-time visibility into how your systems are functioning at any given time. This can significantly increase the overall effectiveness of your ad-campaigns, fraud-detection systems, preventive-maintenance systems, or other critical applications underpinning your business. In this session we will explore various modes of ingest including pipelining, pub-sub, and micro-batching, and identify the use-cases where these can be applied. We will present this in the context of open source frameworks such as Apache Flume, Kafka, among others that can be used to build related solutions. We will also present when and how to use multiple modes and frameworks together to form hybrid solutions that can address non-trivial ingest requirements with little or no extra overhead. Through this discussion we will drill-down into details of configuration and sizing for these frameworks to ensure optimal operations and utilization for long-running deployments.

1. Always-on Ingestion
Con$nuous
Inges$on
for
Data
at
Scale
©
2015
StreamSets
Inc.,
All
rights
reserved
Arvind
Prabhakar
Big
Data
Day
LA,
June
2015

2. © 2015 StreamSets, Inc.
About me
❏ Founder/CTO
Apache
So?ware
FoundaBon
❏ Flume
-­‐
PMC
Chair
❏ Sqoop
-­‐
PMC
Chair
❏ Storm
-­‐
PMC,
CommiFer
❏ MetaModel
-­‐
Mentor
❏ Sentry
-­‐
Mentor
❏ NiFi
-­‐
Mentor
❏ ASF
Member
Previously...
❏ Cloudera
❏ InformaKca
@aprabhakar

3. © 2015 StreamSets, Inc.
Some Background
What is Data Ingestion?
Why do we need Data Ingestion?
❏ Acquiring data from various sources
❏ Storing acquired data where it can be processed
❏ Data is consumed away from where it is produced
❏ Consuming systems are often distributed and remote
❏ Manually, with scripts, with rudimentary automation
❏ Higher level frameworks like Flume, Kafka, etc
How is Data Ingestion Implemented?
Logs
Files
Click
Streams
Sensors
Devices
Database
Logs
Social
Data
Streams
Feeds
Other
Raw
Storage
(HDFS,
S3)
EDW,
NoSQL
(Hive,
Impala,
HBase,
Cassandra,
RedShiY)
Search
(Solr,
ElasKcSearch)
Enterprise
Data
Infrastructure

4. © 2015 StreamSets, Inc.
Data Ingestion Challenge
Ever Increasing data volumes
and rates...
Data sources are physically
distributed and transient...
╳
╳
╳
╳
╳
╳
Data structures and semantics
are constantly changing...

5. © 2015 StreamSets, Inc.
Lot more than moving data!
Data Ingest should be agile
Data Ingest should be safe and reliable
❏ Welcome new data sources as they emerge
❏ Incorporate changes to existing sources as needed
❏ Protect your downstream from silent data corruption
❏ Ensure that there is no data loss in your infrastructure
Data Ingest should scale as needed
❏ Data ingest must never become a bottleneck
❏ Data ingest must scale without significant cost or effort
RELIABLE

6. © 2015 StreamSets, Inc.
»
Design
Wisely
»
Operate
CauKously
»
Update
Liberally
What can you do?
● Pick
the
right
technology
and
toolset
● Instrument
and
monitor
mercilessly
● AnKcipate
and
understand
the
changes
in
your
environment
Here is how...

7. © 2015 StreamSets, Inc.
Picking the right technology
Manual/Scripted
Batch Transport
Micro-batching
Pipelining
Message-Queue
File copying using CLI or GUI interface Cloudera HUE, Hadoop FS client
Ingest Mode Description Example
Bulk data transport using tools Sqoop, DistCp
Transport of small batches of data Sqoop/Sqoop2 (Storm, etc...)
Flow-like transport of event streams Flume, Scribe
Publish-Subscribe like transport of events Kafka, Kinesis

8. © 2015 StreamSets, Inc.
Sqoop
Overview Advantages Disadvantages
❏ Propagates
metadata
❏ Cluster
based
parallel
scaling
capability
❏ Simple
and
easy
to
understand/operate
❏ Rich
set
of
connectors
available
for
use
❏ Supports
popular
formats
like
Avro,
sequence
file
etc.
❏ Not
a
service
❏ Direct
access
to
producKon
data
stores
from
cluster
❏ Requires
access
to
data
store
credenKals
❏ Connector
funcKonality
is
not
consistent
between
different
connectors
❏ CLI
Tool
❏ Oriented
towards
structured
data
stores
❏ Runs
map-­‐only
job
to
transport
data

9. © 2015 StreamSets, Inc.
Sqoop 2
Overview Advantages
❏ Propagates
metadata
❏ Cluster
based
parallel
scaling
capability
❏ Simple
and
easy
to
understand/operate
❏ Supports
popular
formats
like
Avro,
sequence
file
etc.
❏ Consistent
funcKonality
across
connectors
❏ Secure
handling
of
credenKals
with
RBAC
security
❏ Considered
pre-­‐
producKon
quality
before
2.0.0
release.
Currently
at
1.99.6.
❏ May
not
have
connecKvity
at
par
with
Sqoop
1.
❏ Sqoop
Service
with
CLI
and
JSON/REST
interface
❏ Oriented
towards
structured
data
stores
❏ Runs
chained
Map-­‐
Reduce
jobs
for
data
transport
and
conversion
Disadvantages

10. © 2015 StreamSets, Inc.
Flume
Overview Advantages
❏ Guaranteed
delivery
semanKcs
❏ Low-­‐latency
reliable
data
transfer
❏ DeclaraKve
configuraKon
with
no
coding
necessary
for
common
use-­‐cases
❏ Fully
extendable
and
customizable
❏ Integrates
with
most
commonly
used
end-­‐
points
❏ Non-­‐trivial
configuraKon
❏ Complex
topology
configuration
can
be
hard
to
build
and
maintain
❏ Custom
end-­‐point
implementaKon
requires
significant
code
complexity
❏ Distributed
pipeline
system
for
efficient
transport
of
large
volumes
of
data
❏ Built
in
support
for
contextual
rouKng,
filtering,
replicaKon
and
mulKplexing
Disadvantages

11. © 2015 StreamSets, Inc.
Kafka
Overview Advantages
❏ Strong
retenKon
and
ordering
semanKcs
❏ Dynamic
cluster
based
scalability
and
throughput
❏ Low-­‐level
APIs
for
building
consumers
and
producers
❏ Variety
of
open
source
producers
and
consumers
available
on
GitHub
❏ Allows
reprocessing
of
consumed
data
❏ Distributed
and
efficient
publish-­‐subscribe
messaging
system
❏ Used
for
democraKzaKon
of
data
between
applicaKons
Disadvantages
❏ Delivery
guarantee
owned
by
producers
and
consumers
❏ Opaque
pub-­‐sub
design
can
cause
applicaKons
to
be
highly
coupled
❏ Minimal
metadata
support

12. © 2015 StreamSets, Inc.
Typical Examples
For Structured Data
Simple
❏ Sqoop
for
Batch
transport
❏ Sqoop
2
for
micro-­‐batch
transport
Intermediate
❏ Flume
for
Directory
Spooling
Advanced
❏ Custom
Database
Log
Shipping
implementaKon
Simple
❏ Flume
based
AggregaKon
❏ Kaia
based
pub-­‐sub
for
applicaKons
Intermediate
❏ Flume
+
Kaia
based
aggregaKon
and
pub-­‐sub
Advanced
❏ Kaia
+
Storm
for
pub-­‐sub
and
preparaKon
For Streaming Event Data

13. © 2015 StreamSets, Inc.
❏ Apache
Sqoop:
hFp://sqoop.apache.org
❏ Current
Version:
Sqoop1
-­‐
1.4.6;
Sqoop2
-­‐
1.99.6
❏ Apache
Flume:
hFp://flume.apache.org
❏ Current
Version:
Flume
1.6.0
❏ Apache
Kaia:
hFp://kaia.apache.org
❏ Current
Version:
Kaia
0.8.2.1
For more information...

14. © 2015 StreamSets, Inc.
My
Contact
InformaKon:
● Email:
arvind
at
streamsets
dot
com
● TwiFer:
@aprabhakar
● Website:
www.streamsets.com
Thank You!