Download as PDF, PPTX
Uploaded bySuresh Mandava
Apache pig
This document summarizes a meetup for the BigData Cloud Architects group. It provides details on the weekly meeting including topics, speakers, and structure. Suresh Mandava is introducing Apache Pig, a platform for analyzing large datasets. The meeting will cover what Pig is, how it works, performance advantages over MapReduce, and how to use Pig Latin constructs and write user-defined functions. The goal is to help people master the Hadoop ecosystem step-by-step.
![Unit-5 [Pig] working and architecture.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/unit-5pig-240605082042-8125c633-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache pig
- 1. Welcome to ! BigDataCloud Architects Meetup Group Weekly, Sunday 6:00p - 8:00p Exciting Topic Every Week, Please Register @ ! http://www.meetup.com/BigData-Cloud-Meetup/ 6:00 - 6:15 : Socializing! 6:15 - 6:30 : Meetup Introduction! 6:30 - 7:55 : Topic of the week! 7:55 - 8:00 : Wrap UP Started in Jun 2013! 33 weeks - 3 weeks = 30 weeks
- 2. Suresh Mandava! ! CyberSecurity Cloud! Principal,CSC! ! ! ! 24+ Years of IT Experience in Infrastructure, Platform and Data Management in Mission Critical Enterprise Environments Founder/CTO for Network Security and a Cloud IaaS Product Companies!
- 3.
- 4. Market Need BigData Architect Must-haveTechnical skills and Experience: ! •! 5-7 years of strong system design/development experience in building massively large scale distributed systems and products. ! •! 1 to 2 years of good experience in working with Hadoop and Big Data technologies including core Hadoop (HDFS and MapReduce), Flume and Sqoop, HBase and Zookeeper, Oozie, Hive and Pig. •! Troubleshooting: The candidate must be able to engage in solving complex problems. Programming problems are a good example. •! Databases: Advanced SQL knowledge a must. Good to have advanced data warehousing and MPP knowledge but it's not a must have. •! Linux/Unix and system administration: Advanced Linux knowledge is a must. Understanding of shell, debugging things etc. The candidate should be able to get their way around Linux and get things to work. •! Networking and Hardware: Advanced networking and hardware knowledge required. ! •! Programming: Strong in one programming language. Java would be preferable. If not Java, the candidate must have a good handle on one language and display the ability to pick up a new one if required. Experience on programming infrastructure management or automation tools is required. ! •! Big picture / High level architecture: The candidate must be able to think at a high level about the overall systems and goals of the projects. Jan 9, 2014
- 5. Hadoop 2.0 GARelease Oct 16, 2013
- 6. Apache PIG Suresh Mandava Jan19, 2014 : Session 30
- 7. Handle large datavolume Run queries spanning days/months GB/TB/PBs Structured, Semi and Unstructured data Computationally intensive Deep analytics Machine learning algorithms Hadoop Cluster
- 8.
- 11. But, writing MapReduce! jobsin Java is painful. Let’s see why …!
- 13.
- 14.
- 15. PIG ———-Start ——— A =load ’/app_logs/2012/01/*/' using PigStorage(); ! uLogs = FILTER A BY $0 == ’U'; ! uLogFields = FOREACH uLogs GENERATE $1 as orgId, $2 as userId, ! orgUserGroup = GROUP uLogFields BY (orgId, userId); ! uCount = FOREACH orgUserGroup GENERATE group, COUNT(uLogFields); ! STORE uCount INTO ‘output’; ———— END ———-
- 16. Performance of Pigvs MapReduce
- 17.
- 18. Pigs Eat Everything •Pig can operate on data whether it has metadata or not. It can operate on data that is relational, nested, or unstructured. And it can easily be extended to operate on data beyond files, including key/value stores, databases, etc.
- 19. Pigs Live Everywhere •Pig is intended to be a language for parallel data processing. It is not tied to one particular parallel framework. It has been implemented first on Hadoop, but we do not intend that to be only on Hadoop.
- 20. Pigs are DomesticAnimals • Pig is designed to be easily controlled and modified by its users. • Pig allows integration of user code where ever possible, so it currently supports user defined field transformation functions, user defined aggregates, and user defined conditionals. These functions can be written in Java or scripting languages that can compile down to Java (e.g. Jython). • Pig has an optimizer that rearranges some operations in Pig Latin scripts to give better performance, combines Map Reduce jobs together, etc. However, users can easily turn this optimizer off to prevent it from making changes that do not make sense in their situation.
- 21. Pigs Fly • Pigprocesses data quickly. We want to consistently improve performance, and not implement features in ways that weigh pig down so it can't fly.
- 22. About PIG Large CodeContributions from : Yahoo, Twitter, HortonWorks
- 23.
- 24.
- 25. PIG PROCEDURAL ! VS! SQLDECLARATIVE
- 26.
- 27.
- 28. PIG : DataTypes
- 29.
- 30. PIG : LatinConstructs
- 31.
- 32. Employee > 100Ksalary
- 33. More : PigLatin
- 34. More : PigLatin
- 35. More : PigLatin
- 36. Is that ALL? PIG is Extremely Flexible
- 37.
- 38.
- 39.
- 40. UDF : Howto write (EvalFunc)
- 41. UDF : Howto use custom UDF in PIG
- 42. UDF : Howto execute pig script with custom UDF
- 43.
- 44. Our Goal Master theEco-System one bite at a time, every week
- 45. Appreciate you Feedback! @Group Reviews Thank You.
- 46.
- 47. Thank You andHave a Nice Week! ! Meet you all Next Week