This document provides an overview of Giraph, an open source framework for large-scale graph processing on Hadoop. It discusses why graph processing is important at large scales, existing solutions and their limitations, and Giraph's goals of being easily deployable on Hadoop and providing a graph-oriented programming model. The document describes Giraph's design which uses Hadoop and leverages the bulk synchronous parallel computing model, and provides examples of writing Giraph applications and how Giraph jobs interact with Hadoop.
Apache Giraph: start analyzing graph relationships in your bigdata in 45 minu...rhatr
The genesis of Hadoop was in analyzing massive amounts of data with a mapreduce framework. SQL-onHadoop has followed shortly after that, paving a way to the whole schema-on-read notion. Discovering graph relationship in your data is the next logical step. Apache Giraph (modeled on Google’s Pregel) lets you apply the power of BSP approach to the unstructured data. In this talk we will focus on practical advice of how to get up and running with Apache Giraph, start analyzing simple data sets with built-in algorithms and finally how to implement your own graph processing applications using the APIs provided by the project. We will then dive into how Giraph integrates with the Hadoop ecosystem (Hive, HBase, Accumulo, etc.) and will also provide a whirlwind tour of Giraph architecture.
Slides for the tutorial about Apache Giraph for the Data Mining class.
Sapienza, University of Rome.
Master of Science in Engineering in Computer Science
Prof. A. Anagnostopoulos, I. Chatzigiannakis, A. Gionis
Data Mining class
Fall 2016
Apache Giraph: start analyzing graph relationships in your bigdata in 45 minu...rhatr
The genesis of Hadoop was in analyzing massive amounts of data with a mapreduce framework. SQL-onHadoop has followed shortly after that, paving a way to the whole schema-on-read notion. Discovering graph relationship in your data is the next logical step. Apache Giraph (modeled on Google’s Pregel) lets you apply the power of BSP approach to the unstructured data. In this talk we will focus on practical advice of how to get up and running with Apache Giraph, start analyzing simple data sets with built-in algorithms and finally how to implement your own graph processing applications using the APIs provided by the project. We will then dive into how Giraph integrates with the Hadoop ecosystem (Hive, HBase, Accumulo, etc.) and will also provide a whirlwind tour of Giraph architecture.
Slides for the tutorial about Apache Giraph for the Data Mining class.
Sapienza, University of Rome.
Master of Science in Engineering in Computer Science
Prof. A. Anagnostopoulos, I. Chatzigiannakis, A. Gionis
Data Mining class
Fall 2016
Fast, Scalable Graph Processing: Apache Giraph on YARNDataWorks Summit
Apache Giraph performs offline, batch processing of very large graph datasets on top of a Hadoop cluster. Giraph replaces iterative MapReduce-style solutions with Bulk Synchronous Parallel graph processing using in-memory or disk-based data sets, loosely following the model of Google`s Pregel. Many recent advances have left Giraph more robust, efficient, fast, and able to accept a variety of I/O formats typical for graph data in and out of the Hadoop ecosystem. Giraph's recent port to a pure YARN platform offers increased performance, fine-grained resource control, and scalability that Giraph atop Hadoop MRv1 cannot, while paving the way for ports to other platforms like Apache Mesos. Come see whats on the roadmap for Giraph, what Giraph on YARN means, and how Giraph is leveraging the power of YARN to become a more robust, usable, and useful platform for processing Big Graph datasets.
Matrix Factorizations at Scale: a Comparison of Scientific Data Analytics on ...Databricks
Explore the trade-offs of performing linear algebra for data analysis and machine learning using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Apache Spark is designed for data analytics on cluster computing platforms with access to local disks and is optimized for data-parallel tasks.
This session will examine three widely-used and important matrix factorizations: NMF (for physical plausibility), PCA (for its ubiquity) and CX (for data interpretability). Learn how these methods are applied to terabyte-sized problems in particle physics, climate modeling and bioimaging, as use cases where interpretable analytics is of interest. The data matrices are tall-and-skinny, which enable the algorithms to map conveniently into Spark’s data-parallel model. We perform scaling experiments on up to 1600 Cray XC40 nodes, describe the sources of slowdowns and provide tuning guidance to obtain high performance. Based on joint work with Alex Gittens and many others.
• What is MapReduce?
• What are MapReduce implementations?
Facing these questions I have make a personal research, and realize a synthesis, which has help me to clarify some ideas. The attached presentation does not intend to be exhaustive on the subject, but could perhaps bring you some useful insights.
Optimal Execution Of MapReduce Jobs In Cloud - Voices 2015Deanna Kosaraju
Optimal Execution Of MapReduce Jobs In Cloud
Anshul Aggarwal, Software Engineer, Cisco Systems
Session Length: 1 Hour
Tue March 10 21:30 PST
Wed March 11 0:30 EST
Wed March 11 4:30:00 UTC
Wed March 11 10:00 IST
Wed March 11 15:30 Sydney
Voices 2015 www.globaltechwomen.com
We use MapReduce programming paradigm because it lends itself well to most data-intensive analytics jobs run on cloud these days, given its ability to scale-out and leverage several machines to parallel process data. Research has demonstrates that existing approaches to provisioning other applications in the cloud are not immediately relevant to MapReduce -based applications. Provisioning a MapReduce job entails requesting optimum number of resource sets (RS) and configuring MapReduce parameters such that each resource set is maximally utilized.
Each application has a different bottleneck resource (CPU :Disk :Network), and different bottleneck resource utilization, and thus needs to pick a different combination of these parameters based on the job profile such that the bottleneck resource is maximally utilized.
The problem at hand is thus defining a resource provisioning framework for MapReduce jobs running in a cloud keeping in mind performance goals such as Optimal resource utilization with Minimum incurred cost, Lower execution time, Energy Awareness, Automatic handling of node failure and Highly scalable solution.
Pivotal Data Labs - Technology and Tools in our Data Scientist's Arsenal Srivatsan Ramanujam
These slides give an overview of the technology and the tools used by Data Scientists at Pivotal Data Labs. This includes Procedural Languages like PL/Python, PL/R, PL/Java, PL/Perl and the parallel, in-database machine learning library MADlib. The slides also highlight the power and flexibility of the Pivotal platform from embracing open source libraries in Python, R or Java to using new computing paradigms such as Spark on Pivotal HD.
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)Alexey Zinoviev
Alexey Zinoviev presented this paper on the Highload++ conference http://www.highload.ru/2014/abstracts/1516.html
This paper covers next topics: Pregel, Graph Theory, Giraph, Okapi, GraphX, GraphChi, Spark, Shrotest Path Problem, Road Network, Road Graph
In KDD2011, Vijay Narayanan (Yahoo!) and Milind Bhandarkar (Greenplum Labs, EMC) conducted a tutorial on "Modeling with Hadoop". This is the second half of the tutorial.
PyMADlib - A Python wrapper for MADlib : in-database, parallel, machine learn...Srivatsan Ramanujam
These are slides from my talk @ DataDay Texas, in Austin on 30 Mar 2013
(http://2013.datadaytexas.com/schedule)
Favorite and Fork PyMADlib on GitHub: https://github.com/gopivotal/pymadlib
MADlib: http://madlib.net
Best Hadoop Institutes : kelly tecnologies is the best Hadoop training Institute in Bangalore.Providing hadoop courses by realtime faculty in Bangalore.
Fast, Scalable Graph Processing: Apache Giraph on YARNDataWorks Summit
Apache Giraph performs offline, batch processing of very large graph datasets on top of a Hadoop cluster. Giraph replaces iterative MapReduce-style solutions with Bulk Synchronous Parallel graph processing using in-memory or disk-based data sets, loosely following the model of Google`s Pregel. Many recent advances have left Giraph more robust, efficient, fast, and able to accept a variety of I/O formats typical for graph data in and out of the Hadoop ecosystem. Giraph's recent port to a pure YARN platform offers increased performance, fine-grained resource control, and scalability that Giraph atop Hadoop MRv1 cannot, while paving the way for ports to other platforms like Apache Mesos. Come see whats on the roadmap for Giraph, what Giraph on YARN means, and how Giraph is leveraging the power of YARN to become a more robust, usable, and useful platform for processing Big Graph datasets.
Matrix Factorizations at Scale: a Comparison of Scientific Data Analytics on ...Databricks
Explore the trade-offs of performing linear algebra for data analysis and machine learning using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Apache Spark is designed for data analytics on cluster computing platforms with access to local disks and is optimized for data-parallel tasks.
This session will examine three widely-used and important matrix factorizations: NMF (for physical plausibility), PCA (for its ubiquity) and CX (for data interpretability). Learn how these methods are applied to terabyte-sized problems in particle physics, climate modeling and bioimaging, as use cases where interpretable analytics is of interest. The data matrices are tall-and-skinny, which enable the algorithms to map conveniently into Spark’s data-parallel model. We perform scaling experiments on up to 1600 Cray XC40 nodes, describe the sources of slowdowns and provide tuning guidance to obtain high performance. Based on joint work with Alex Gittens and many others.
• What is MapReduce?
• What are MapReduce implementations?
Facing these questions I have make a personal research, and realize a synthesis, which has help me to clarify some ideas. The attached presentation does not intend to be exhaustive on the subject, but could perhaps bring you some useful insights.
Optimal Execution Of MapReduce Jobs In Cloud - Voices 2015Deanna Kosaraju
Optimal Execution Of MapReduce Jobs In Cloud
Anshul Aggarwal, Software Engineer, Cisco Systems
Session Length: 1 Hour
Tue March 10 21:30 PST
Wed March 11 0:30 EST
Wed March 11 4:30:00 UTC
Wed March 11 10:00 IST
Wed March 11 15:30 Sydney
Voices 2015 www.globaltechwomen.com
We use MapReduce programming paradigm because it lends itself well to most data-intensive analytics jobs run on cloud these days, given its ability to scale-out and leverage several machines to parallel process data. Research has demonstrates that existing approaches to provisioning other applications in the cloud are not immediately relevant to MapReduce -based applications. Provisioning a MapReduce job entails requesting optimum number of resource sets (RS) and configuring MapReduce parameters such that each resource set is maximally utilized.
Each application has a different bottleneck resource (CPU :Disk :Network), and different bottleneck resource utilization, and thus needs to pick a different combination of these parameters based on the job profile such that the bottleneck resource is maximally utilized.
The problem at hand is thus defining a resource provisioning framework for MapReduce jobs running in a cloud keeping in mind performance goals such as Optimal resource utilization with Minimum incurred cost, Lower execution time, Energy Awareness, Automatic handling of node failure and Highly scalable solution.
Pivotal Data Labs - Technology and Tools in our Data Scientist's Arsenal Srivatsan Ramanujam
These slides give an overview of the technology and the tools used by Data Scientists at Pivotal Data Labs. This includes Procedural Languages like PL/Python, PL/R, PL/Java, PL/Perl and the parallel, in-database machine learning library MADlib. The slides also highlight the power and flexibility of the Pivotal platform from embracing open source libraries in Python, R or Java to using new computing paradigms such as Spark on Pivotal HD.
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)Alexey Zinoviev
Alexey Zinoviev presented this paper on the Highload++ conference http://www.highload.ru/2014/abstracts/1516.html
This paper covers next topics: Pregel, Graph Theory, Giraph, Okapi, GraphX, GraphChi, Spark, Shrotest Path Problem, Road Network, Road Graph
In KDD2011, Vijay Narayanan (Yahoo!) and Milind Bhandarkar (Greenplum Labs, EMC) conducted a tutorial on "Modeling with Hadoop". This is the second half of the tutorial.
PyMADlib - A Python wrapper for MADlib : in-database, parallel, machine learn...Srivatsan Ramanujam
These are slides from my talk @ DataDay Texas, in Austin on 30 Mar 2013
(http://2013.datadaytexas.com/schedule)
Favorite and Fork PyMADlib on GitHub: https://github.com/gopivotal/pymadlib
MADlib: http://madlib.net
Best Hadoop Institutes : kelly tecnologies is the best Hadoop training Institute in Bangalore.Providing hadoop courses by realtime faculty in Bangalore.
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system.
Introduction into scalable graph analysis with Apache Giraph and Spark GraphXrhatr
Graph relationships are everywhere. In fact, more often than not, analyzing relationships between points in your datasets lets you extract more business value from your data.
Consider social graphs, or relationships of customers to each other and products they purchase, as two of the most common examples. Now, if you think you have a scalability issue just analyzing points in your datasets, imagine what would happen if you wanted to start analyzing the arbitrary relationships between those data points: the amount of potential processing will increase dramatically, and the kind of algorithms you would typically want to run would change as well.
If your Hadoop batch-oriented approach with MapReduce works reasonably well, for scalable graph processing you have to embrace an in-memory, explorative, and iterative approach. One of the best ways to tame this complexity is known as the Bulk synchronous parallel approach. Its two most widely used implementations are available as Hadoop ecosystem projects: Apache Giraph (used at Facebook), and Apache GraphX (as part of a Spark project).
In this talk we will focus on practical advice on how to get up and running with Apache Giraph and GraphX; start analyzing simple datasets with built-in algorithms; and finally how to implement your own graph processing applications using the APIs provided by the projects. We will finally compare and contrast the two, and try to lay out some principles of when to use one vs. the other.
PREGEL a system for large scale graph processingAbolfazl Asudeh
Pregel is a framework for processing Large Scale Graphs efficiently.
The slides are created for presenting the following paper in the class:
http://dl.acm.org/citation.cfm?id=1807184
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15Vasia Kalavri
Apache Flink is a general-purpose platform for batch and streaming distributed data processing. This talk describes how Flink’s powerful APIs, iterative operators and other unique features make it a competitive alternative for large-scale graph processing as well. We take a close look at how one can elegantly express graph analysis tasks, using common Flink operators and how different graph processing models, like vertex-centric, can be easily mapped to Flink dataflows. Next, we get a sneak preview into Flink's upcoming Graph API, Gelly, which further simplifies graph application development in Flink. Finally, we show how to perform end-to-end data analysis, mixing common Flink operators and Gelly, without having to build complex pipelines and combine different systems. We go through a step-by-step example, demonstrating how to perform loading, transformation, filtering, graph creation and analysis, with a single Flink program.
Hadoop became the most common systm to store big data.
With Hadoop, many supporting systems emerged to complete the aspects that are missing in Hadoop itself.
Together they form a big ecosystem.
This presentation covers some of those systems.
While not capable to cover too many in one presentation, I tried to focus on the most famous/popular ones and on the most interesting ones.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
2. Agenda
Why large-scale graph processing?
Existing solutions
Goals
Design and implementation
Bulk synchronous parallel model
Writing an application
Fault-tolerance
Early performance results
Future work / conclusion
2
3. Why scalable graph processing?
Web and social graphs are at immense scale and continuing to grow
› In 2008, Google estimated the number of web pages at 1 trillion
› At the 2011 F8, Facebook announced it has 800 million monthly active users
› In September 2011, Twitter claimed to have over 100 million active monthly users
› In March 2011, LinkedIn said it had over 120 million registered users
Relevant and personalized information for users relies strongly on iterative
graph ranking algorithms (search results, news, ads, etc.)
› In web graphs, page rank and its variants
3
4. Example social graph applications
Popularity rank (page rank)
› Can be personalized for a user or “type” of user
› Determining popular users, news, jobs, etc.
Shortest paths
› Many variants single-source, s-t shortest paths, all-to-all shortest (memory/storage
prohibitive)
› How are users, groups connected?
Clustering, semi-clustering
› Max clique, triangle closure, label propagation algorithms
› Finding related people, groups, interests
4
5. Existing solutions
Sequence of map-reduce jobs in Hadoop
› Classic map-reduce overheads (job startup/shutdown, reloading data from
HDFS, shuffling)
› Map-reduce programming model not a good fit for graph algorithms
Google’s Pregel
› Requires its own computing infrastructure
› Not available (unless you work at Google)
› Master is a SPOF
Message passing interface (MPI)
› Not fault-tolerant
› Too generic
5
6. Giraph goals
Easy deployment on existing big data processing infrastructure
› Maintaining a separate graph processing cluster is a lot of overhead for operations
Dynamic resource management
› Handle failures gracefully
› Integrate new resources when available
Graph-oriented API
› Make graph processing code as simple as possible
Open
› Leverage the community
6
7. Giraph design
Easy deployment on existing big data processing infrastructure
Leverage Hadoop installations around the world for iterative graph processing
› Big data today is processed on Hadoop with the Map-Reduce computing model
› Map-Reduce with Hadoop is widely deployed outside of Yahoo! as well (i.e.
EC2, Cloudera, etc.)
Dynamic resource management
Bulk synchronous parallel (BSP) computing model
Fault-tolerant/dynamic graph processing infrastructure
› Automatically adjust to available resources on the Hadoop grid
› No single point of failure except Hadoop namenode and jobtracker
› Relies on ZooKeeper as a fault-tolerant coordination service
7
8. Giraph design (continued)
Graph-oriented API
Vertex-centric API to do graph processing (similar to the map() method in
Hadoop) in a bulk synchronous parallel computing model
› Inspired by Pregel
Open
Open-sourced to the Apache Incubator
› http://incubator.apache.org/giraph/
8
9. Bulk synchronous parallel model
Sequential computation on a single physical machine restricts the
computational problem domain
BSP is a proposal of a “bridging” model for parallel computation
› High-level languages bridged to parallel machines
3 main attributes
› Components that process and/or provide storage
› Router to deliver point-to-point messages
› Synchronization of all or a subset of components through regular intervals
(supersteps)
Computation is done when all components are done
Only a model, does not describe an implementation
9
10. Why use BSP?
A relatively simple computational model
Parallelization of computation/messaging during a superstep
› Components can only communicate by messages delivered out-of-order in the next
superstep
Fault-tolerant/dynamic resource utilization
› Supersteps are atomic units of parallel computation
› Any superstep can be restarted from a checkpoint (need not be user defined)
› A new superstep provides an opportunity for rebalancing of components among
available resources
10
11. Maximum vertex value example
All vertices find the maximum value in a strongly connected graph
If 1st superstep or the set a new maximum value from incoming messages,
send new value maximum to edges, otherwise vote to halt (gray vertices)
5 5 5 5
1
1 5 5
5
2
2 2 5
5
Superstep 1 Superstep 2 Superstep 3 Superstep 4
11
12. Writing a Giraph application
Every active vertex will call compute() method once during a superstep
› Analogous to map() method in Hadoop for a <key, value> tuple
Users chooses 4 types for their implementation of Vertex (I VertexId, V
VertexValue, E EdgeValue, M MsgValue)
Map Reduce Giraph
public class Mapper< public class Vertex<
KEYIN, I extends WritableComparable,
VALUEIN, V extends Writable,
KEYOUT, E extends Writable,
VALUEOUT> { M extends Writable> {
void map(KEYIN key, void compute(
VALUEIN value, Iterator<M> msgIterator);
Context context) }
throws IOException,
InterruptedException;
}
12
13. Basic Giraph API
Get/set local vertex values Voting
I getVertexId(); void voteToHalt();
V getVertexValue(); boolean isHalted();
void setVertexValue(V vertexValue);
SortedMap<I, Edge<I, E>> getOutEdgeMap();
long getSuperstep();
Graph-wide mutation Messaging
void addVertexRequest(MutableVertex<I, V, E, M> vertex); void sendMsg(I id, M
void removeVertexRequest(I vertexId); msg);
void addEdgeRequest(I sourceVertexId, Edge<I, E> edge); void sentMsgToAllEdges(
void removeEdgeRequest(I sourceVertexId, I M msg);
destVertexId);
Local vertex mutations happen immediately
Vertices “vote” to end the computation
› Once all vertices have voted to end the computation, the application is finished
Graph mutations are processed just prior to the next superstep
Sent messages are available at the next superstep during compute
13
14. Page rank example
public class SimplePageRankVertex extends Vertex<LongWritable, DoubleWritable,
FloatWritable, DoubleWritable> {
public void compute(Iterator<DoubleWritable> msgIterator) {
if (getSuperstep() >= 1) {
double sum = 0;
while (msgIterator.hasNext()) {
sum += msgIterator.next().get();
}
setVertexValue(new DoubleWritable((0.15f / getNumVertices()) + 0.85f * sum);
}
if (getSuperstep() < 30) {
long edges = getOutEdgeIterator().size();
sentMsgToAllEdges(new DoubleWritable(getVertexValue().get() / edges));
} else {
voteToHalt();
}
}
14
15. Hadoop
Open-source implementation of Map-Reduce and GFS (HDFS)
Meant to solve “big data” challenges such as distributed grep, process log
data, sorting, etc.
Includes resource management (JobTracker)
Not good for message passing (every message passing step requires a Map-
Reduce job)
Input Map Intermediate Reduce Output
format tasks files tasks format
Split 0
Output 0
Split 1
Split 2
Split 3 Output 1
15
16. Giraph job from Hadoop’s perspective
Giraph has its own InputFormat that calls the user’s VertexInputFormat
› Hadoop will start up the workers requested by Giraph, not based on the InputSplit
objects generated by the VertexInputFormat
Giraph has its own OutputFormat that will call the user’s VertexOutputFormat
› Giraph’s internal structure will write the data with the user’s VertexWriter
Giraph’s input Map Giraph’s output
format tasks format
Split 0
Output 0
Split 1
Split 2
Split 3 Output 1
16
19. Thread responsibilities
Master
› Only one active master at a time
› Runs the VertexInputFormat getSplits() to get the appropriate number of InputSplit
objects for the application and writes it to ZooKeeper
› Coordinates application
• Synchronizes supersteps, end of application
• Handles changes within supersteps (i.e. vertex movement, change in number of workers, etc.)
Worker
› Reads vertices from InputSplit objects, splitting them into VertexRanges
› Executes the compute() method for every Vertex it is assigned once per superstep
› Buffers the incoming messages to every Vertex it is assigned for the next superstep
ZooKeeper
› Manages a server that Is a part of the ZooKeeper quorum (maintains global
application state)
19
20. Vertex distribution
VertexRange 0
V
Worker V
Worker
Master
V
0 V
0
InputSplit 0
V Workers process V
VertexRange 0,1
the InputSplit
objects and divide
VertexRange 1
V V
them further to Prior to every
V
create V superstep, the
VertexRange master assigns
objects. every worker 0 or
V V
more VertexRange
objects. They are Master
VertexRange 2
V
Worker V
responsible for the
VertexRange 2,3
V
1 V
vertices that fall in
those VertexRange
InputSplit 1
objects.
Master uses the
V V
VertexInputFormat
to divide the input
VertexRange 3
V V
data into InputSplit
objects and
serializes them to V V
ZooKeeper. Worker
V V 1
20
21. Worker phases in a superstep
Execute
Prepare by If desired, compute()
If desired,
registering exchange on each
checkpoint
worker vertex vertex and
vertices
health ranges exchange
messages
Master selects one or more of the available workers to use for the superstep
Users can set the checkpoint frequency
Checkpoints are implemented by Giraph (all types implement Writable)
Users can determine how to distribute vertex ranges on the set of available
workers
BSP model allows for dynamic resource usage
› Every superstep is an atomic unit of computation
› Resources can change between supersteps and used accordingly (shrink or grow)
21
22. Fault tolerance
No single point of failure from Giraph threads
› With multiple master threads, if the current master dies, a new one will automatically
take over.
› If a worker thread dies, the application is rolled back to a previously checkpointed
superstep. The next superstep will begin with the new amount of workers
› If a zookeeper server dies, as long as a quorum remains, the application can
proceed
Hadoop single points of failure still exist
› Namenode, jobtracker
› Restarting manually from a checkpoint is always possible
22
23. Master thread fault tolerance
Before failure of active master 0 After failure of active master 0
“Active” “Active”
Master 0 Master 0
Active Active
Master Master
“Spare” State “Active” State
Master 1 Master 1
“Spare” “Spare”
Master 2 Master 2
One active master, with spare masters taking over in the event of an active
master failure
All active master state is stored in ZooKeeper so that a spare master can
immediately step in when an active master fails
“Active” master implemented as a queue in ZooKeeper
23
24. Worker thread fault tolerance
Superstep i Superstep i+1 Superstep i+2
(no checkpoint) (checkpoint) (no checkpoint)
Worker failure!
Superstep i+1 Superstep i+2 Superstep i+3
(checkpoint) (no checkpoint) (checkpoint)
Worker failure after
checkpoint complete!
Superstep i+3 Application
(no checkpoint) Complete
A single worker failure causes the superstep to fail
In order to disrupt the application, a worker must be registered and chosen for the
current superstep
Application reverts to the last committed superstep automatically
› Master detects worker failure during any superstep with a ZooKeeper “health” znode
› Master chooses the last committed superstep and sends a command through ZooKeeper for all
workers to restart from that superstep
24
25. Optional features
Combiners
› Similar to Map-Reduce combiners
› Users implement a combine() method that can reduce the amount of messages sent
and received
› Run on both the client side and server side
• Client side saves memory and message traffic
• Server side save memory
Aggregators
› Similar to MPI aggregation routines (i.e. max, min, sum, etc.)
› Users can write their own aggregators
› Commutative and associate operations that are performed globally
› Examples include global communication, monitoring, and statistics
25
26. Early Yahoo! customers
Web of Objects
› Currently used for the movie database (10’s of millions of records, run with 400
workers)
› Popularity rank, shared connections, personalized page rank
Web map
› Next generation page-rank related algorithms will use this framework (250 billion web
pages)
› Current graph processing solution uses MPI (no fault-tolerance, customized code)
26
27. Page rank benchmarks
Tiberium Tan
› Almost 4000 nodes, shared among numerous groups in Yahoo!
› Hadoop 0.20.204 (secure Hadoop)
› 2x Quad Core 2.4GHz, 24 GB RAM, 1x 6TB HD
org.apache.giraph.benchmark.PageRankBenchmark
› Generates data, number of edges, number of vertices, # of supersteps
› 1 master/ZooKeeper
› 20 supersteps
› No checkpoints
› 1 random edge per vertex
27
29. Vertex scalability (300 workers)
2500
2000
Total seconds
1500
1000
500
0
0 100 200 300 400 500 600
# of vertices (in 100's of millions)
29
30. Vertex/worker scalability
2500 600
# of vertices (100's of millions)
500
2000
Total seconds
400
1500
300
1000
200
500
100
0 0
0 50 100 150 200 250 300 350
# of workers
30
31. Apache incubation status
9 committers in all from a variety of relevant companies
(LinkedIn, Twitter, Hortonworks, Yahoo!, Facebook, Jybe, etc.)
31 closed issues since incubation, 20 open
92 posts on the user mailing list for the month of September
Probably about a month away from an initial release (guess)
Very interesting tweets (http://twitter.com/#!/search/giraph)
Early project, lots of opportunity to shape the project
› Looking for contributors…hint, hint.
31
32. Future work
Improved fault tolerance testing
› Unit tests exist, but test a limited subset of failures
More performance testing
› Try other edge distributions (map closer to social graphs or web graphs)
Importing some of the currently used graph algorithms into an included library
Storing messages that surpass available memory on disk
› Leverage the programming model, maybe even convert to a native Map-Reduce
application
Better vertex distribution
› Hash and range based distributions
Memory optimization
› Primitive types for special cases, user chosen thread counts
32
33. Conclusion
Giraph is a graph processing infrastructure that runs on existing Hadoop
infrastructure
› Already being used at Yahoo!
› Lots of opportunity for new parallel graph algorithms!
Open source
› http://incubator.apache.org/giraph/
› https://issues.apache.org/jira/browse/GIRAPH
Questions/comments?
› aching@apache.org
33