Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloud Austin Hadoop Automation Lighting Talk 2014.11.18

393 views

Published on

Hadoop ETL Automation - How to get to the fun part of big data in the shortest amount of time.

Published in: Data & Analytics
  • Be the first to comment

Cloud Austin Hadoop Automation Lighting Talk 2014.11.18

  1. 1. dataFundamentals Hadoop Automation in 15 Minutes Or how to get to the fun stuff before your boss pulls the plug.
  2. 2. ETL is not the Fun Stuff, in Big Data ❖ Analytics ❖ Machine Learning ❖ Spark ❖ [even just Building APIs] But you can’t do the fun stuff until your corporate data is in place to work against. Chicken and egg problem.
  3. 3. Quick! Before your boss turns off the spigot! ❖ Automate your ETL processes. ❖ Automate your server instances.
  4. 4. What kind of code to Automate? ❖ Clean code. Super clean. ❖ Well designed code.
  5. 5. Other pitfalls? ❖ NIH, Not Invented Here
  6. 6. How to get the fun tasks? ❖ 2 week P.O.C. ❖ Your sample data
  7. 7. Code, Content, Contacts ❖ This Slide Deck: http://www.slideshare.net/petecarapetyan/cloud-austin-hadoop-automationlightingtalk141118 ❖ or just remember slideshare.net/datafundamentals ❖ Youtube - 11 minute slide-less version of code demo - https://www.youtube.com/playlist? list=PLO_T9AjxEaYeByfqBqHVCmg4GbLFkYCJe ❖ Dev Code ❖ Carrie (ruby UI and generator) https://github.com/datafundamentals/df_ui_carrie ❖ Avro from delimited https://bitbucket.org/datafundamentals/avro_from_delimited ❖ Camel-Avro https://bitbucket.org/datafundamentals/camel-avro-etl ❖ Ops Code - cookbook recipes ❖ https://github.com/datafundamentals ❖ Contact ❖ pete@datafundamentals.com jeff@datafundamentals.com Jeff Twitter @devopsjeff Pete Twitter @appwritercom Site: datafundamentals.com Be careful! It’s a competitive world out there!

×