The document provides an overview of using Python with Hadoop, discussing tools such as Jython, MRJob, and Pig to manage processing large datasets efficiently. It addresses the challenges associated with setting up and debugging distributed systems and emphasizes that while Hadoop facilitates data processing, it does not inherently optimize algorithms. Additionally, it outlines a structured approach for deployment and validation within Python and Hadoop environments.

1. Hadoop, Pig, and Python
PyData NYC 2012

2. Overview
OF THIS SESSION
Why Python on Hadoop?
Fast Hadoop overview
Jython
Python
MrJob
Pig
(How they work, challenges, efficiency,
how to start)

3. Too much data
FOR ONE MACHINE
Data doubles every 18 mo

4. ETL / Munging
Cleanse
Format
Simple calculations

5. Social Graph

6. Predict

7. Detect

8. Genetics

9. Hadoop
RAPID OVERVIEW
MapReduce programming model
from Google
(Jeff Dean and Sanjay Ghemawat)

10. Hadoop
RAPID OVERVIEW

11. Hadoop
RAPID OVERVIEW
Hadoop implements MapReduce (Java)
(Doug Cutting)
Incubated at Yahoo
Indexing, Spam detection, more

12. Hadoop
PROBLEMS
Difficult
Not much Python
Batch only (...or it was)

13. Hadoop
FUTURE
Yarn
MapReduce optional
Generic management + distributed
apps
Impala

14. Hadoop
AND PYTHON

15. Jython
ON HADOOP (MAP)

16. Jython
ON HADOOP (REDUCE; 1ST HALF)

17. Jython
ON HADOOP (REDUCE; 2ND HALF)

18. Jython
ON HADOOP

19. Python
ON HADOOP
Streaming
(Works with any language, not just

20. MrJob (Python)
ON HADOOP
Streaming + local / EMR / your Hadoop

21. MrJob (Python)
ON HADOOP
Multi-step jobs

22. Pig
ON HADOOP
Less code
Expressive code

23. Pig
BRIEF, EXPRESSIVE
(thanks: twitter hadoop world presentation)

24. The Same Script, In
FOR SERIOUS

25. Pig
ON HADOOP
Less code
Expressive code
Compiles to MR
Insulates from API
Popular
(LinkedIn, Twitter,
Salesforce, Yahoo,
Stanford

26. Pig
ON HADOOP
Works with Jython
Not Python
Stream, no types
UDF read stdin
UDF deserialize, no types
Serialize for Pig
Write to stdout
Exceptions

27. Pig + Python
ON HADOOP

30. Hadoop + Python
NOT ACTUALLY MAGIC
Hadoop won’t magically parallelize
your algorithm

31. Hadoop + Python
EFFICIENCY
Don’t stream Java-based languages
• Jython
• Pig + Jython
Streaming has ~30% overhead
• Python
• MrJob
• Pig + Python

32. Hadoop + Python
EXCITED?
Well... 90-95% of time isn’t spent on
algos

33. Hadoop + Python
HARD STUFF: SETUP
Get Hadoop running
Software where it needs to be
Processes communicating
Data available

34. Hadoop + Python
HARD STUFF: DEVELOP
Learn
Project structure, modularity
Dev environment like Production

35. Hadoop + Python
HARD STUFF: VALIDATE
Syntax check
Packages available
Data readable
Data writable
Without long waits for failure

36. Hadoop + Python
HARD STUFF: DEBUG
Distributed execution is hard to debug

37. Hadoop + Python
HARD STUFF: TEST
Data processing is hard to test
But critical

38. Hadoop + Python
HARD STUFF: DEPLOY
Environments identical
Code correctly deployed
Configuration changes
Non-disruptive

39. Hadoop + Python
HARD STUFF: HISTORY
Stats about prior runs
What code was run?
What’s changed?

40. Hadoop + Python
HARD STUFF: LOGS
Distributed logs hard to make sense of
Hadoop logs hard to understand
Ephemeral clusters lose logs

41. Hadoop + Python
HARD STUFF: MORTAR’S APPROACH
Setup: PaaS, pip installation,
connectors
Develop: learning, structure, instant
dev env
Validate: fast validate
Debug: printf, more coming
Test: Rails-like test suites
Deploy: one-button deploy

42. K Young
@kky