Welcome to YARN Meetup
September 2013

©2013 LinkedIn Corporation. All Rights Reserved.
YARN @ LinkedIn
State of the Art
Mohammad Islam

©2013 LinkedIn Corporation. All Rights Reserved.
Early Adopter
 YARN is good fit for many LinkedIn problems
 Many initiatives by multiple teams
 LI Engineers enjoy the ...
Early Adopter
 Samza : Real-time stream processing
system
– Developed by LinkedIn team
– Apache incubator project
– Use Y...
Early Adopter
 Helix – Generic cluster management
system
– Built and used in LinkedIn
– Apache Incubator project
– Incorp...
Early Adopter
 Not yet open sourced
– Few projects are incubating at LI
– Mostly around custom and near-realtime
executio...
Early Adopter
 Administering YARN:
– One of the pioneers of a 2.1.0-beta prod-like
deployment
– Led by our Ops/Dev team
–...
Early Adopter
 Pig on Tez: Actively working with Pig
community
 Hosted a small “Pig on Tez” dev meeting
– Participants i...
Apache Giraph on YARN

©2013 LinkedIn Corporation. All Rights Reserved.
Overview of Giraph
 A distributed graph processing framework
– Master/slave architecture
– In-memory computation
– Vertex...
Quick History
 HortonWorks/LinkedIn intern (Eli) wrote the
early version of Giraph AM
 Based on 2.0.3
 Since then YARN ...
Giraph on YARN
Node
Manager
Worker

Client

Resource
Manager

Worker

Node
Manager
App
Mstr

ZooKeeper

Worker

Node
Manag...
New Giraph AM
 Girpah AM : Nearly a complete rewrite by LinkedIn
Hadoop dev.
– Used new stable API
– Adopt new asynchrono...
Memory Footprint - Page Rank Algorithm

 Iteration 3



Iteration 27
Reachable
1.5 GB

Reachable
1.5 GB
Unreachable
3 GB...
Challenges in Giraph
 Memory intensive Java based system
 Various (GC) knobs to tune the system and
application
 Depend...
Future Direction
 Option 1: “Worker” in C++
– C++provides direct control over memory management
– No need to rewrite the ...
Final Thoughts on Giraph
 LinkedIn is the 1st player of Giraph on YARN
 Successfully executed full LinkedIn graph run
–
...
Challenges in YARN
 Failover of various components (RM/AM etc.)
 APIs stabilization –almost there!
 Representative exam...
Concluding on YARN
 YARN is the way to go forward!
 Reduce the innovation barrier
 Support non-MR execution platform
 ...
Q& A

Thanks for coming!

©2013 LinkedIn Corporation. All Rights Reserved.
Giraph Architecture
 Master / Workers
 Zookeeper

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Master

Worker...
Upcoming SlideShare
Loading in …5
×

Yarn at LinkedIn

834 views
756 views

Published on

YARN meetup on September 2013 at LinkedIn
How LinkedIn is using YARN?
What is the future of YARN at LinkedIn?
New Giraph AM on YARN and experiences of running LinkedIn Graph.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
834
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
24
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • So what is giraph?Giraph is a distributed graph processing framework. It tries to solve a class of iterative problems that hadoop has problem with, such as pagerank.Graph processing is very improtant to linkedinGiraph is designed with master slave architecture and does all its computation in memory. Meaning, it loads the inputs from HDFS once and writes the output back to HDFS only after finishing its business logic processing.Giraph provides a vertex-centric programming model. All algorithms are implemented from the point of view of a single vertex in the input graph performing a single iteration of the computation.Giraph makes graph algorithm easy to reason about and implement by following the BSP. A bsp computation proceeds in a series of global supersteps. A superstep consists of three components, concurrent computation, communication, and barrier synchronization.
  • Client is nothing but initiating an application for the user. It just asks the resource manager will you launch my application master. From there, the application master is going to do everything.Resource manager just schedules your task (job tracker sort of activity ask the node managers for the containers with the right heap size, and where we could put this task.Application master is sort of master node for your application and is going to launch, manage the life cycle, communicates with health anything to do with your task. New brain of your application.You may wonder what is the difference between the master and application master? The answer is these two components could be combined as one. However, giraph is implemented this way and my focus is on giraph’s performance.
  • byte[] 1.8GBLong/DoubleWritable 2GB----- Meeting Notes (9/3/13 19:49) -----move it upadd GB----- Meeting Notes (9/4/13 11:45) -----move up
  • Step 2. Make them support Java-based applicationsJava interface to write Giraph applications running on C++ Giraph master/workers----- Meeting Notes (9/3/13 19:49) -----No need to rewrite the whole Giraphmore control over memory management----- Meeting Notes (9/4/13 11:45) -----from prev slides, we found out jvm is the killerthat's why we are thinking if c++ is better candidate, we are asking to overhaul the giraph, we only ask the griaph to pice
  • Relative new things in the community.There are a few bugs we fixed to make it work
  • Master – responsible for coordination (load distribution, coordinates synchronization, request checkpoints, collect health status, etc)Worker – responsible for computation within each iteration or superstepZookeeper – responsible for computation state (partition to worker mapping, global state, checkpoints paths, statistics, etc)In the next slide, I’m going to show you how these components work together----- Meeting Notes (9/3/13 19:49) -----add another slide descirbing the computational modelpartition the vertice to distribute the load
  • Yarn at LinkedIn

    1. 1. Welcome to YARN Meetup September 2013 ©2013 LinkedIn Corporation. All Rights Reserved.
    2. 2. YARN @ LinkedIn State of the Art Mohammad Islam ©2013 LinkedIn Corporation. All Rights Reserved.
    3. 3. Early Adopter  YARN is good fit for many LinkedIn problems  Many initiatives by multiple teams  LI Engineers enjoy the fun of emergent technologies ©2013 LinkedIn Corporation. All Rights Reserved.
    4. 4. Early Adopter  Samza : Real-time stream processing system – Developed by LinkedIn team – Apache incubator project – Use YARN and Kafka – Detailed presentation coming later today ©2013 LinkedIn Corporation. All Rights Reserved.
    5. 5. Early Adopter  Helix – Generic cluster management system – Built and used in LinkedIn – Apache Incubator project – Incorporating YARN resource management – Stay tuned to learn more today ©2013 LinkedIn Corporation. All Rights Reserved.
    6. 6. Early Adopter  Not yet open sourced – Few projects are incubating at LI – Mostly around custom and near-realtime execution engine – Status: Some in POC and some are in design state ©2013 LinkedIn Corporation. All Rights Reserved.
    7. 7. Early Adopter  Administering YARN: – One of the pioneers of a 2.1.0-beta prod-like deployment – Led by our Ops/Dev team – Found a lot of issues  Kerberos auth (YARN -621 & others) – Contributing back to Apache to stabilize YARN  Streamlined operational tools (HADOOP9902) ©2013 LinkedIn Corporation. All Rights Reserved.
    8. 8. Early Adopter  Pig on Tez: Actively working with Pig community  Hosted a small “Pig on Tez” dev meeting – Participants include: Yahoo, HortonWorks, Netflix and LinkedIn  Developed a high-level implementation plan ©2013 LinkedIn Corporation. All Rights Reserved.
    9. 9. Apache Giraph on YARN ©2013 LinkedIn Corporation. All Rights Reserved.
    10. 10. Overview of Giraph  A distributed graph processing framework – Master/slave architecture – In-memory computation – Vertex-centric high-level programming model – Based on Bulk Synchronous Parallel (BSP) ©2013 LinkedIn Corporation. All Rights Reserved. 10
    11. 11. Quick History  HortonWorks/LinkedIn intern (Eli) wrote the early version of Giraph AM  Based on 2.0.3  Since then YARN has evolved a lot!  API overhauled Action: Overhaul Giraph onYARN ©2013 LinkedIn Corporation. All Rights Reserved.
    12. 12. Giraph on YARN Node Manager Worker Client Resource Manager Worker Node Manager App Mstr ZooKeeper Worker Node Manager Master ©2013 LinkedIn Corporation. All Rights Reserved. Worker 12
    13. 13. New Giraph AM  Girpah AM : Nearly a complete rewrite by LinkedIn Hadoop dev. – Used new stable API – Adopt new asynchronous/event based model – Status: Patch ready  Client – Used new API – Status: Patch ready  Security – Added Kerberos support for Giraph YARN client and AM – Status: Testing ©2013 LinkedIn Corporation. All Rights Reserved.
    14. 14. Memory Footprint - Page Rank Algorithm  Iteration 3  Iteration 27 Reachable 1.5 GB Reachable 1.5 GB Unreachable 3 GB Unreachable 6 GB ©2013 LinkedIn Corporation. All Rights Reserved.
    15. 15. Challenges in Giraph  Memory intensive Java based system  Various (GC) knobs to tune the system and application  Depends heavily on skillful application developers  Performance degradation from scaling up  Not a good player for multi-tenant system ©2013 LinkedIn Corporation. All Rights Reserved. 15
    16. 16. Future Direction  Option 1: “Worker” in C++ – C++provides direct control over memory management – No need to rewrite the whole Giraph  Issue : Adoption barrier – Writing C++ application – Possible solution: Giraph scripting language  Like Hive or Pig  Option 2: Off-heap memory usage Option 3: Leave it alone! ©2013 LinkedIn Corporation. All Rights Reserved. 16
    17. 17. Final Thoughts on Giraph  LinkedIn is the 1st player of Giraph on YARN  Successfully executed full LinkedIn graph run – – – – Page Rank algorithm 200M+ vertices and XX Billions edges On 40-node cluster with 650GB memory Total time taken: 28 minutes  Ready to go!  Scope for improvements utilizing YARN’s flexibility ©2013 LinkedIn Corporation. All Rights Reserved. 17
    18. 18. Challenges in YARN  Failover of various components (RM/AM etc.)  APIs stabilization –almost there!  Representative examples for quick dev ramp-up  Better documentation – Book on its way!  Operational friendly – Centralized logging – SLA support – timed resource constraint. ©2013 LinkedIn Corporation. All Rights Reserved.
    19. 19. Concluding on YARN  YARN is the way to go forward!  Reduce the innovation barrier  Support non-MR execution platform  Improved utilization/performance – By removing the split of map/reduce slot – Through distribution of JT responsibility ©2013 LinkedIn Corporation. All Rights Reserved.
    20. 20. Q& A Thanks for coming! ©2013 LinkedIn Corporation. All Rights Reserved.
    21. 21. Giraph Architecture  Master / Workers  Zookeeper Worker Worker Worker Worker Worker Worker Worker Master Worker Worker ©2013 LinkedIn Corporation. All Rights Reserved. 21

    ×