This document discusses data and process scheduling in Hadoop. It provides examples of loading data from HDFS, Hive, and Avro formats into Pig and querying that data. It also discusses switching file formats from ORC and shows a diagram of data flows from raw to presented data. The document mentions the Apache Falcon project for managing Hadoop data pipelines and some of its adoption and future enhancements.

1. Simplified
Data and Process
Scheduling
in Hadoop

4. Somebody Still Investigates
Do you think
we find the
location and
the owner
of the
“streams”
dataset
today?

5. STREAMS
{trackId:long, userId:long, ts:timestamp, ...}
hdfs://data/core/streams
avro
etl
official=>true, frequency=>hourly
"UserId started to stream trackId at time ts"

7. users = LOAD 'data.user'
USING HCatLoader();
val users = hiveContext.hql(
"FROM data.user SELECT name, country"
)
users = LOAD
'/data/core/user/part-00000.
avro' USING AvroStorage();
Non HCatalog way
in Pig
ID NAME COUNTRY GENDER
1 JOSH US M
2 ADAM PL M

8. [FALCON-790]

9. [FALCON-790]
Email

10. HDFS
HDFS

11. [FALCON-790]

15. Switching to ORC
requires
reimplementing
the Reader Code
in hundreds of
productions jobs...

16. users = LOAD 'data.users' USING HCatLoader();
ORC

17. The picture comes from http://hortonworks.com/blog/introduction-apache-falcon-hadoop. Thanks Hortonworks!

18. Raw Data
Cleansed
Data
Conformed
Data
Presented
Data
Raw Data
Presented
Data

21. Which Elephant Is Your?
A. Elephantus Dirtus
B. Elephantus Cleanus

23. Backup Slides

24. Falcon’s Adoption
■ Top Level Project since December 2014
■ 14 contributors from 3 companies
■ Originated and heavily used at inMobi
● 400+ pipelines and 2000+ data feeds
■ Also used at Expedia and at some undisclosed companies

25. Future Enhancements And Ideas
■ Improved Web UI [FALCON-790]
● More extensive search box, more widgets
● The “today morning” dashboard [FALCON-994]
● Re-running processes
■ Automatic discovery of datasets in HDFS and Hive
■ Streaming feeds and processes e.g. Storm, Spark Streaming
■ Triage of data processing issues [FALCON-796]
■ HDFS snapshots
■ High availability of the Falcon server

26. [FALCON-790]