Azkaban and Pig at LinkedIn

•Download as PPTX, PDF•

3 likes•2,809 views

Russell Jurney

Description of using Pig with the Azkaban workflow scheduler for Hadoop

Technology

Azkaban and Pig Richard Park, Russell Jurney LinkedIn Search, Network, Analytics

Installing & Running Azkaban wgethttp://github.com/downloads/azkaban/azkaban/azkaban-0.04.tar.gz tar –xvzf azkaban-0.04.tar.gz mkdir /some-dir/azkaban-jobs cd azkaban-0.04 bin/azkaban-server.sh –job-dir /some-dir/azkaban-jobs

Pig Configuration myproject.properties – Global Configuration hadoop.job.ugi=rjurney,hadoop udf.import.list=org.apache.pig.builtin.,com.linkedin.pig.,com.linkedin… cc_0_compute_title_counts.job – Pig Job type=pig pig.script=cc_0_compute_title_counts.pig cc_1_reverse_engineer_durak_3000 – Pig Job with Dependency type=pig pig.script=cc_1_reverse_engineer_durak_3000.pig dependencies=cc_0_compute_title_counts

Running Job > bin/run-jobs.sh –job-dir /some-dir/azkaban-jobs my-job

What's hot

Kapil Thangavelu - Cloud CustodianServerlessConf

Rob Gruhl and Erik Erikson - What We Learned in 18 Serverless Months at Nords...ServerlessConf

Ben Kehoe - Serverless Architecture for the Internet of ThingsServerlessConf

Serverless by examples and case studiesCodeOps Technologies LLP

Era of server less computingBaskar rao Dsn

Akka in Practice: Designing Actor-based ApplicationsNLJUG

CapybaraMona Soni

Apache Zeppelin & ClusterJongyoul Lee

E2E Data Pipeline - Apache Spark/Airflow/LivyRikin Tanna

Reacting to the Isomorphic BuzzBruce Coddington

Going Headless with Craft CMS 3.3JustinHolt20

Serverless Summit - QuizCodeOps Technologies LLP

Era of server less computing finalBaskar rao Dsn

Serverless Architecture Patterns - Manoj Ganapathi - Serverless SummitCodeOps Technologies LLP

Infrastructure as codeNaseath Saly

Monitoring Open Source Databases with IcingaIcinga

Ryan Brown - Open CommunityServerlessConf

How Serverless Changes DevOpsRichard Donkin

How to Build a Big Data Application: Serverless EditionLecole Cole

R2DBC - Good Enough for Production?Olexandra Dmytrenko

What's hot (20)

Kapil Thangavelu - Cloud Custodian

Rob Gruhl and Erik Erikson - What We Learned in 18 Serverless Months at Nords...

Ben Kehoe - Serverless Architecture for the Internet of Things

Serverless by examples and case studies

Era of server less computing

Akka in Practice: Designing Actor-based Applications

Capybara

Apache Zeppelin & Cluster

E2E Data Pipeline - Apache Spark/Airflow/Livy

Reacting to the Isomorphic Buzz

Going Headless with Craft CMS 3.3

Serverless Summit - Quiz

Era of server less computing final

Serverless Architecture Patterns - Manoj Ganapathi - Serverless Summit

Infrastructure as code

Monitoring Open Source Databases with Icinga

Ryan Brown - Open Community

How Serverless Changes DevOps

How to Build a Big Data Application: Serverless Edition

R2DBC - Good Enough for Production?

Viewers also liked

Hadoop ecosystem framework n hadoop in live environmentDelhi/NCR HUG

Interactive workflow management using Azkabandatamantra

Hive vs PigAnatoliy Nikulin

Архитектура продукта Thumbtack RTB BidderAnatoliy Nikulin

Vaadin thinking of u and i. Или как писать Rich Internet Applications, в стар...Anatoliy Nikulin

Куда мы катимся. Анализ многолетних наблюдений омской ИТ отрасли в пяти минутахAnatoliy Nikulin

HBase insideAnatoliy Nikulin

Apache HiveAnatoliy Nikulin

Building a Self-Service Hadoop Platform at Linkedin with AzkabanDataWorks Summit

Конференция Юкон. Процессинг данных на лямбда архитектуре.Anatoliy Nikulin

NoSQL thumbtack experience, Анатолий НикулинAnatoliy Nikulin

俞晨杰：Linked in大数据应用和azkabanhdhappy001

QuartzJini Lee

Hadoop presentationVlad Orlov

Luigi futureErik Bernhardsson

Luigi Presentation at OSCON 2013Erik Bernhardsson

Hadoop Summit 2014: Building a Self-Service Hadoop Platform at LinkedIn with ...David Chen

Networks All Around Us: Extracting networks from your problem domainRussell Jurney

Social Network Analysis in Your Problem DomainRussell Jurney

Viewers also liked (20)

Hadoop ecosystem framework n hadoop in live environment

Interactive workflow management using Azkaban

Hive vs Pig

Архитектура продукта Thumbtack RTB Bidder

Vaadin thinking of u and i. Или как писать Rich Internet Applications, в стар...

Куда мы катимся. Анализ многолетних наблюдений омской ИТ отрасли в пяти минутах

HBase inside

Apache Hive

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

Конференция Юкон. Процессинг данных на лямбда архитектуре.

NoSQL thumbtack experience, Анатолий Никулин

俞晨杰：Linked in大数据应用和azkaban

Quartz

Hadoop presentation

Luigi future

Luigi Presentation at OSCON 2013

Hadoop Summit 2014: Building a Self-Service Hadoop Platform at LinkedIn with ...

Networks All Around Us: Extracting networks from your problem domain

Social Network Analysis in Your Problem Domain

Recently uploaded (20)

Advanced Computer Architecture – An Introduction

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)

Scanning the Internet for External Cloud Exposures via SSL Certs

Advanced Test Driven-Development @ php[tek] 2024

Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost

Story boards and shot lists for my a level piece

Connect Wave/ connectwave Pitch Deck Presentation

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf

Human Factors of XR: Using Human Factors to Design XR Systems

Are Multi-Cloud and Serverless Good or Bad?

From Family Reminiscence to Scholarly Archive .

Developer Data Modeling Mistakes: From Postgres to NoSQL

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

Designing IA for AI - Information Architecture Conference 2024

H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day

Powerpoint exploring the locations used in television show Time Clash

Vertex AI Gemini Prompt Engineering Tips

Dev Dives: Streamline document processing with UiPath Studio Web

SAP Build Work Zone - Overview L2-L3.pptx

Azkaban and Pig at LinkedIn

1. Azkaban and Pig Richard Park, Russell Jurney LinkedIn Search, Network, Analytics

2. Installing & Running Azkaban wgethttp://github.com/downloads/azkaban/azkaban/azkaban-0.04.tar.gz tar –xvzf azkaban-0.04.tar.gz mkdir /some-dir/azkaban-jobs cd azkaban-0.04 bin/azkaban-server.sh –job-dir /some-dir/azkaban-jobs

3. Azkaban @ localhost:8080

4. Pig Configuration myproject.properties – Global Configuration hadoop.job.ugi=rjurney,hadoop udf.import.list=org.apache.pig.builtin.,com.linkedin.pig.,com.linkedin… cc_0_compute_title_counts.job – Pig Job type=pig pig.script=cc_0_compute_title_counts.pig cc_1_reverse_engineer_durak_3000 – Pig Job with Dependency type=pig pig.script=cc_1_reverse_engineer_durak_3000.pig dependencies=cc_0_compute_title_counts

5. Running Job > bin/run-jobs.sh –job-dir /some-dir/azkaban-jobs my-job