Your SlideShare is downloading. ×
Advanced Oozie
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Advanced Oozie

1,750

Published on

Alexey Yakubovich

Alexey Yakubovich

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,750
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
16
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. ADVANCED OOZIE Alexey Yakubovich
  • 2. In Addition to Boris Lublinsky, Kevin T. Smith and Alexey Yakubovich. “Professional Hadoop Solutions”
  • 3. New cases • Use cases • Organizing all data processing steps: up-down • Regular data injection • Regular data transformation • Regular report generation • Extensions • File movement on HDFS (synch. java action) • Data transfer (synch - ftp, synch - ssh) • Logging / monitoring (beyond Oozie console)
  • 4. New & rediscovered Oozie features • 1. JMS notifications (job life cycle, SLA) http://oozie.apache.org/docs/4.0.0/DG_JMSNotifications.html • 2. Overriding the launcher https://github.com/yahoo/oozie/blob/master/examples/src/main/java/org /apache/oozie/example/DemoPigMain.java • Unit testing Oozie with MiniOozie http://oozie.apache.org/docs/4.0.0/ENG_MiniOozie.html
  • 5. JMS notifications “Push” JMS notifications for action status, SLA met and SLA miss Needs “JMS broker” to interprets notifications Apache ActiveMQ Need “JMS notification configuration” in the oozie-site.xml: oozie.services.ext oozie.services.EventHandlerService… oozie.jms.producer.connection.properties (topic) Notification types Job status: start, success, failure, suspended … SLA: start| end| duration && met | miss Message format: javax.jms.TextMessage with Oozie job specific headers
  • 6. Overriding the launcher (cross-cutting concerns) • Regular Pig job launcher – org.apache.oozie.action.hadoop.PigMain Reminder: action executor provides all preparations for submitting action as a hadoop job(s). In particularly the PigMain executor invokes the Pig runtime on an Edge (Gateway) node. public class SpecialPigExec extends PigMain() { e.g. logging, external services (security,, transactions) } • Oozie workflow <action name=“pig-special”> <pig> … <property> <name> oozie.launcher.action.main.class </name> <value> … SpecialPigExec</value>
  • 7. Unit testing Oozie with MiniOozie • MiniOozieTestCase is a junit test class • Allows to test workflow and coordinator applications • Tests workflow directly from IDE (Eclipse for sure) • Does not require access to cluster or running Oozie server • Runs against the local file system • Tested on Linux and Max OS X, configured with Maven (simple) • Needs most (all) Oozie libraries Action choice restricted: java actions is straight forward. others can be “simulated” I can’t tell if possible to combine with PigUnit and Hive standalone mode.

×