Apache Spark presentation at HasGeek FifthElelephant
https://fifthelephant.talkfunnel.com/2015/15-processing-large-data-with-apache-spark
Covering Big Data Overview, Spark Overview, Spark Internals and its supported libraries
Apache Spark presentation at HasGeek FifthElelephant
https://fifthelephant.talkfunnel.com/2015/15-processing-large-data-with-apache-spark
Covering Big Data Overview, Spark Overview, Spark Internals and its supported libraries
Apache HBase is the Hadoop opensource, distributed, versioned storage manager well suited for random, realtime read/write access. This talk will give an overview on how HBase achieve random I/O, focusing on the storage layer internals. Starting from how the client interact with Region Servers and Master to go into WAL, MemStore, Compactions and on-disk format details. Looking at how the storage is used by features like snapshots, and how it can be improved to gain flexibility, performance and space efficiency.
Tez is the next generation Hadoop Query Processing framework written on top of YARN. Computation topologies in higher level languages like Pig/Hive can be naturally expressed in the new graph dataflow model exposed by Tez. Multi-stage queries can be expressed as a single Tez job resulting in lower latency for short queries and improved throughput for large scale queries. MapReduce has been the workhorse for Hadoop but its monolithic structure had made innovation slower. YARN separates resource management from application logic and thus enables the creation of Tez, a more flexible and generic new framework for data processing for the benefit of the entire Hadoop query ecosystem.
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang Spark Summit
In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.
Managing Apache Spark Workload and Automatic OptimizingDatabricks
eBay is highly using Spark as one of most significant data engines. In data warehouse domain, there are millions of batch queries running every day against 6000+ key DW tables, which contains over 22PB data (compressed) and still keeps booming every year. In machine learning domain, it is playing a more and more significant role. We have introduced our great achievement in migration work from MPP database to Apache Spark last year in Europe Summit. Furthermore, from the vision of the entire infrastructure, it is still a big challenge on managing workload and efficiency for all Spark jobs upon our data center. Our team is leading the whole infrastructure of big data platform and the management tools upon it, helping our customers -- not only DW engineers and data scientists, but also AI engineers -- to leverage on the same page. In this session, we will introduce how to benefit all of them within a self-service workload management portal/system. First, we will share the basic architecture of this system to illustrate how it collects metrics from multiple data centers and how it detects the abnormal workload real-time. We develop a component called Profiler which is to enhance the current Spark core to support customized metric collection. Next, we will demonstrate some real user stories in eBay to show how the self-service system reduces the efforts both in customer side and infra-team side. That's the highlight part about Spark job analysis and diagnosis. Finally, some incoming advanced features will be introduced to describe an automatic optimizing workflow rather than just alerting.
Speaker: Lantao Jin
Cost-Based Optimizer in Apache Spark 2.2 Databricks
Apache Spark 2.2 ships with a state-of-art cost-based optimization framework that collects and leverages a variety of per-column data statistics (e.g., cardinality, number of distinct values, NULL values, max/min, avg/max length, etc.) to improve the quality of query execution plans. Leveraging these reliable statistics helps Spark to make better decisions in picking the most optimal query plan. Examples of these optimizations include selecting the correct build side in a hash-join, choosing the right join type (broadcast hash-join vs. shuffled hash-join) or adjusting a multi-way join order, among others. In this talk, we’ll take a deep dive into Spark’s cost based optimizer and discuss how we collect/store these statistics, the query optimizations it enables, and its performance impact on TPC-DS benchmark queries.
Apache Spark is a In Memory Data Processing Solution that can work with existing data source like HDFS and can make use of your existing computation infrastructure like YARN/Mesos etc. This talk will cover a basic introduction of Apache Spark with its various components like MLib, Shark, GrpahX and with few examples.
Presented at the MLConf in Seattle, this presentation offers a quick introduction to Apache Spark, followed by an overview of two novel features for data science
GraphFrames: DataFrame-based graphs for Apache® Spark™Databricks
These slides support the GraphFrames: DataFrame-based graphs for Apache Spark webinar. In this webinar, the developers of the GraphFrames package will give an overview, a live demo, and a discussion of design decisions and future plans. This talk will be generally accessible, covering major improvements from GraphX and providing resources for getting started. A running example of analyzing flight delays will be used to explain the range of GraphFrame functionality: simple SQL and graph queries, motif finding, and powerful graph algorithms.
Building a SIMD Supported Vectorized Native Engine for Spark SQLDatabricks
Spark SQL works very well with structured row-based data. Vectorized reader and writer for parquet/orc can make I/O much faster. It also used WholeStageCodeGen to improve the performance by Java JIT code. However Java JIT is usually not working very well on utilizing latest SIMD instructions under complicated queries. Apache Arrow provides columnar in-memory layout and SIMD optimized kernels as well as a LLVM based SQL engine Gandiva. These native based libraries can accelerate Spark SQL by reduce the CPU usage for both I/O and execution.
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Databricks
Parquet is a very popular column based format. Spark can automatically filter useless data using parquet file statistical data by pushdown filters, such as min-max statistics. On the other hand, Spark user can enable Spark parquet vectorized reader to read parquet files by batch. These features improve Spark performance greatly and save both CPU and IO. Parquet is the default data format of data warehouse in Bytedance. In practice, we find that parquet pushdown filters work poorly resulting in reading too much unnecessary data for statistical data has no discrimination across parquet row groups(column data is out of order when writing to parquet files by ETL jobs).
Properly shaping partitions and your jobs to enable powerful optimizations, eliminate skew and maximize cluster utilization. We will explore various Spark Partition shaping methods along with several optimization strategies including join optimizations, aggregate optimizations, salting and multi-dimensional parallelism.
The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...Databricks
So you know you want to write a streaming app, but any non-trivial streaming app developer would have to think about these questions:
– How do I manage offsets?
– How do I manage state?
– How do I make my Spark Streaming job resilient to failures? Can I avoid some failures?
– How do I gracefully shutdown my streaming job?
– How do I monitor and manage my streaming job (i.e. re-try logic)?
– How can I better manage the DAG in my streaming job?
– When do I use checkpointing, and for what? When should I not use checkpointing?
– Do I need a WAL when using a streaming data source? Why? When don’t I need one?
This session will share practices that no one talks about when you start writing your streaming app, but you’ll inevitably need to learn along the way.
Dynamic Partition Pruning in Apache SparkDatabricks
In data analytics frameworks such as Spark it is important to detect and avoid scanning data that is irrelevant to the executed query, an optimization which is known as partition pruning. Dynamic partition pruning occurs when the optimizer is unable to identify at parse time the partitions it has to eliminate. In particular, we consider a star schema which consists of one or multiple fact tables referencing any number of dimension tables. In such join operations, we can prune the partitions the join reads from a fact table by identifying those partitions that result from filtering the dimension tables. In this talk we present a mechanism for performing dynamic partition pruning at runtime by reusing the dimension table broadcast results in hash joins and we show significant improvements for most TPCDS queries.
Apache HBase is the Hadoop opensource, distributed, versioned storage manager well suited for random, realtime read/write access. This talk will give an overview on how HBase achieve random I/O, focusing on the storage layer internals. Starting from how the client interact with Region Servers and Master to go into WAL, MemStore, Compactions and on-disk format details. Looking at how the storage is used by features like snapshots, and how it can be improved to gain flexibility, performance and space efficiency.
Tez is the next generation Hadoop Query Processing framework written on top of YARN. Computation topologies in higher level languages like Pig/Hive can be naturally expressed in the new graph dataflow model exposed by Tez. Multi-stage queries can be expressed as a single Tez job resulting in lower latency for short queries and improved throughput for large scale queries. MapReduce has been the workhorse for Hadoop but its monolithic structure had made innovation slower. YARN separates resource management from application logic and thus enables the creation of Tez, a more flexible and generic new framework for data processing for the benefit of the entire Hadoop query ecosystem.
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang Spark Summit
In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.
Managing Apache Spark Workload and Automatic OptimizingDatabricks
eBay is highly using Spark as one of most significant data engines. In data warehouse domain, there are millions of batch queries running every day against 6000+ key DW tables, which contains over 22PB data (compressed) and still keeps booming every year. In machine learning domain, it is playing a more and more significant role. We have introduced our great achievement in migration work from MPP database to Apache Spark last year in Europe Summit. Furthermore, from the vision of the entire infrastructure, it is still a big challenge on managing workload and efficiency for all Spark jobs upon our data center. Our team is leading the whole infrastructure of big data platform and the management tools upon it, helping our customers -- not only DW engineers and data scientists, but also AI engineers -- to leverage on the same page. In this session, we will introduce how to benefit all of them within a self-service workload management portal/system. First, we will share the basic architecture of this system to illustrate how it collects metrics from multiple data centers and how it detects the abnormal workload real-time. We develop a component called Profiler which is to enhance the current Spark core to support customized metric collection. Next, we will demonstrate some real user stories in eBay to show how the self-service system reduces the efforts both in customer side and infra-team side. That's the highlight part about Spark job analysis and diagnosis. Finally, some incoming advanced features will be introduced to describe an automatic optimizing workflow rather than just alerting.
Speaker: Lantao Jin
Cost-Based Optimizer in Apache Spark 2.2 Databricks
Apache Spark 2.2 ships with a state-of-art cost-based optimization framework that collects and leverages a variety of per-column data statistics (e.g., cardinality, number of distinct values, NULL values, max/min, avg/max length, etc.) to improve the quality of query execution plans. Leveraging these reliable statistics helps Spark to make better decisions in picking the most optimal query plan. Examples of these optimizations include selecting the correct build side in a hash-join, choosing the right join type (broadcast hash-join vs. shuffled hash-join) or adjusting a multi-way join order, among others. In this talk, we’ll take a deep dive into Spark’s cost based optimizer and discuss how we collect/store these statistics, the query optimizations it enables, and its performance impact on TPC-DS benchmark queries.
Apache Spark is a In Memory Data Processing Solution that can work with existing data source like HDFS and can make use of your existing computation infrastructure like YARN/Mesos etc. This talk will cover a basic introduction of Apache Spark with its various components like MLib, Shark, GrpahX and with few examples.
Presented at the MLConf in Seattle, this presentation offers a quick introduction to Apache Spark, followed by an overview of two novel features for data science
GraphFrames: DataFrame-based graphs for Apache® Spark™Databricks
These slides support the GraphFrames: DataFrame-based graphs for Apache Spark webinar. In this webinar, the developers of the GraphFrames package will give an overview, a live demo, and a discussion of design decisions and future plans. This talk will be generally accessible, covering major improvements from GraphX and providing resources for getting started. A running example of analyzing flight delays will be used to explain the range of GraphFrame functionality: simple SQL and graph queries, motif finding, and powerful graph algorithms.
Building a SIMD Supported Vectorized Native Engine for Spark SQLDatabricks
Spark SQL works very well with structured row-based data. Vectorized reader and writer for parquet/orc can make I/O much faster. It also used WholeStageCodeGen to improve the performance by Java JIT code. However Java JIT is usually not working very well on utilizing latest SIMD instructions under complicated queries. Apache Arrow provides columnar in-memory layout and SIMD optimized kernels as well as a LLVM based SQL engine Gandiva. These native based libraries can accelerate Spark SQL by reduce the CPU usage for both I/O and execution.
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Databricks
Parquet is a very popular column based format. Spark can automatically filter useless data using parquet file statistical data by pushdown filters, such as min-max statistics. On the other hand, Spark user can enable Spark parquet vectorized reader to read parquet files by batch. These features improve Spark performance greatly and save both CPU and IO. Parquet is the default data format of data warehouse in Bytedance. In practice, we find that parquet pushdown filters work poorly resulting in reading too much unnecessary data for statistical data has no discrimination across parquet row groups(column data is out of order when writing to parquet files by ETL jobs).
Properly shaping partitions and your jobs to enable powerful optimizations, eliminate skew and maximize cluster utilization. We will explore various Spark Partition shaping methods along with several optimization strategies including join optimizations, aggregate optimizations, salting and multi-dimensional parallelism.
The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...Databricks
So you know you want to write a streaming app, but any non-trivial streaming app developer would have to think about these questions:
– How do I manage offsets?
– How do I manage state?
– How do I make my Spark Streaming job resilient to failures? Can I avoid some failures?
– How do I gracefully shutdown my streaming job?
– How do I monitor and manage my streaming job (i.e. re-try logic)?
– How can I better manage the DAG in my streaming job?
– When do I use checkpointing, and for what? When should I not use checkpointing?
– Do I need a WAL when using a streaming data source? Why? When don’t I need one?
This session will share practices that no one talks about when you start writing your streaming app, but you’ll inevitably need to learn along the way.
Dynamic Partition Pruning in Apache SparkDatabricks
In data analytics frameworks such as Spark it is important to detect and avoid scanning data that is irrelevant to the executed query, an optimization which is known as partition pruning. Dynamic partition pruning occurs when the optimizer is unable to identify at parse time the partitions it has to eliminate. In particular, we consider a star schema which consists of one or multiple fact tables referencing any number of dimension tables. In such join operations, we can prune the partitions the join reads from a fact table by identifying those partitions that result from filtering the dimension tables. In this talk we present a mechanism for performing dynamic partition pruning at runtime by reusing the dimension table broadcast results in hash joins and we show significant improvements for most TPCDS queries.
Reliability Study of wireless corba using Petri net and end to end instanteno...Ahmed Koriem
Reliability Study of wireless corba using Petri net and end to end instantenous reliability measure
master degree seminar 10-2-2015
Al-azhar university- Communication and electronics department
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Elevating Tactical DDD Patterns Through Object Calisthenics
Graph processing - Pregel
1. Pregel: A System for Large-Scale Graph Processing
Amir H. Payberah
amir@sics.se
Amirkabir University of Technology
(Tehran Polytechnic)
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 1 / 40
3. Introduction
Graphs provide a flexible abstraction for describing relationships be-
tween discrete objects.
Many problems can be modeled by graphs and solved with appro-
priate graph algorithms.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 3 / 40
5. Large-Scale Graph Processing
Large graphs need large-scale processing.
A large graph either cannot fit into memory of single computer or
it fits with huge cost.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 5 / 40
6. Question
Can we use platforms like MapReduce or Spark, which are based on data-parallel
model, for large-scale graph proceeding?
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 6 / 40
7. Data-Parallel Model for Large-Scale Graph Processing
The platforms that have worked well for developing parallel applica-
tions are not necessarily effective for large-scale graph problems.
Why?
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 7 / 40
8. Graph Algorithms Characteristics (1/2)
Unstructured problems
• Difficult to extract parallelism based on partitioning of the data: the
irregular structure of graphs.
• Limited scalability: unbalanced computational loads resulting from
poorly partitioned data.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 8 / 40
9. Graph Algorithms Characteristics (1/2)
Unstructured problems
• Difficult to extract parallelism based on partitioning of the data: the
irregular structure of graphs.
• Limited scalability: unbalanced computational loads resulting from
poorly partitioned data.
Data-driven computations
• Difficult to express parallelism based on partitioning of computation:
the structure of computations in the algorithm is not known a priori.
• The computations are dictated by nodes and links of the graph.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 8 / 40
10. Graph Algorithms Characteristics (2/2)
Poor data locality
• The computations and data access patterns do not have much local-
ity: the irregular structure of graphs.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 9 / 40
11. Graph Algorithms Characteristics (2/2)
Poor data locality
• The computations and data access patterns do not have much local-
ity: the irregular structure of graphs.
High data access to computation ratio
• Graph algorithms are often based on exploring the structure of a
graph to perform computations on the graph data.
• Runtime can be dominated by waiting memory fetches: low locality.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 9 / 40
14. Graph-Parallel Processing
Restricts the types of computation.
New techniques to partition and distribute graphs.
Exploit graph structure.
Executes graph algorithms orders-of-magnitude faster than more
general data-parallel systems.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 11 / 40
16. Data-Parallel vs. Graph-Parallel Computation
Data-parallel computation
• Record-centric view of data.
• Parallelism: processing independent data on separate resources.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 13 / 40
17. Data-Parallel vs. Graph-Parallel Computation
Data-parallel computation
• Record-centric view of data.
• Parallelism: processing independent data on separate resources.
Graph-parallel computation
• Vertex-centric view of graphs.
• Parallelism: partitioning graph (dependent) data across processing
resources, and resolving dependencies (along edges) through
iterative computation and communication.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 13 / 40
19. Seven Bridges of K¨onigsberg
Finding a walk through the city that would cross each bridge once
and only once.
Euler proved that the problem has no solution.
Map of K¨onigsberg in Euler’s time, highlighting the river Pregel and the bridges.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 15 / 40
20. Pregel
Large-scale graph-parallel processing platform developed at Google.
Inspired by bulk synchronous parallel (BSP) model.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 16 / 40
21. Bulk Synchronous Parallel (1/2)
It is a parallel programming model.
The model consists of:
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 17 / 40
22. Bulk Synchronous Parallel (1/2)
It is a parallel programming model.
The model consists of:
• A set of processor-memory pairs.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 17 / 40
23. Bulk Synchronous Parallel (1/2)
It is a parallel programming model.
The model consists of:
• A set of processor-memory pairs.
• A communications network that delivers messages in a point-
to-point manner.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 17 / 40
24. Bulk Synchronous Parallel (1/2)
It is a parallel programming model.
The model consists of:
• A set of processor-memory pairs.
• A communications network that delivers messages in a point-
to-point manner.
• A mechanism for the efficient barrier synchronization for all or
a subset of the processes.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 17 / 40
25. Bulk Synchronous Parallel (1/2)
It is a parallel programming model.
The model consists of:
• A set of processor-memory pairs.
• A communications network that delivers messages in a point-
to-point manner.
• A mechanism for the efficient barrier synchronization for all or
a subset of the processes.
• There are no special combining, replicating, or broadcasting fa-
cilities.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 17 / 40
26. Bulk Synchronous Parallel (2/2)
All vertices update in parallel (at the same time).
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 18 / 40
28. Vertex-Centric Programs
Think as a vertex.
Each vertex computes individually its value: in parallel
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 19 / 40
29. Vertex-Centric Programs
Think as a vertex.
Each vertex computes individually its value: in parallel
Each vertex can see its local context, and updates its value accord-
ingly.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 19 / 40
30. Data Model
A directed graph that stores the program state, e.g., the current
value.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 20 / 40
31. Execution Model (1/3)
Applications run in sequence of iterations: supersteps
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 21 / 40
32. Execution Model (1/3)
Applications run in sequence of iterations: supersteps
During a superstep, user-defined functions for each vertex is invoked
(method Compute()): in parallel
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 21 / 40
33. Execution Model (1/3)
Applications run in sequence of iterations: supersteps
During a superstep, user-defined functions for each vertex is invoked
(method Compute()): in parallel
A vertex in superstep S can:
• reads messages sent to it in superstep S-1.
• sends messages to other vertices: receiving at superstep S+1.
• modifies its state.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 21 / 40
34. Execution Model (1/3)
Applications run in sequence of iterations: supersteps
During a superstep, user-defined functions for each vertex is invoked
(method Compute()): in parallel
A vertex in superstep S can:
• reads messages sent to it in superstep S-1.
• sends messages to other vertices: receiving at superstep S+1.
• modifies its state.
Vertices communicate directly with one another by sending mes-
sages.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 21 / 40
35. Execution Model (2/3)
Superstep 0: all vertices are in the active state.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 22 / 40
36. Execution Model (2/3)
Superstep 0: all vertices are in the active state.
A vertex deactivates itself by voting to halt: no further work to do.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 22 / 40
37. Execution Model (2/3)
Superstep 0: all vertices are in the active state.
A vertex deactivates itself by voting to halt: no further work to do.
A halted vertex can be active if it receives a message.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 22 / 40
38. Execution Model (2/3)
Superstep 0: all vertices are in the active state.
A vertex deactivates itself by voting to halt: no further work to do.
A halted vertex can be active if it receives a message.
The whole algorithm terminates when:
• All vertices are simultaneously inactive.
• There are no messages in transit.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 22 / 40
39. Execution Model (3/3)
Aggregation: a mechanism for global communication, monitoring,
and data.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 23 / 40
40. Execution Model (3/3)
Aggregation: a mechanism for global communication, monitoring,
and data.
Runs after each superstep.
Each vertex can provide a value to an aggregator in superstep S.
The system combines those values and the resulting value is made
available to all vertices in superstep S + 1.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 23 / 40
41. Execution Model (3/3)
Aggregation: a mechanism for global communication, monitoring,
and data.
Runs after each superstep.
Each vertex can provide a value to an aggregator in superstep S.
The system combines those values and the resulting value is made
available to all vertices in superstep S + 1.
A number of predefined aggregators, e.g., min, max, sum.
Aggregation operators should be commutative and associative.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 23 / 40
42. Example: Max Value (1/4)
i_val := val
for each message m
if m > val then val := m
if i_val == val then
vote_to_halt
else
for each neighbor v
send_message(v, val)
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 24 / 40
43. Example: Max Value (2/4)
i_val := val
for each message m
if m > val then val := m
if i_val == val then
vote_to_halt
else
for each neighbor v
send_message(v, val)
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 25 / 40
44. Example: Max Value (3/4)
i_val := val
for each message m
if m > val then val := m
if i_val == val then
vote_to_halt
else
for each neighbor v
send_message(v, val)
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 26 / 40
45. Example: Max Value (4/4)
i_val := val
for each message m
if m > val then val := m
if i_val == val then
vote_to_halt
else
for each neighbor v
send_message(v, val)
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 27 / 40
46. Example: PageRank
Update ranks in parallel.
Iterate until convergence.
R[i] = 0.15 +
j∈Nbrs(i)
wjiR[j]
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 28 / 40
47. Example: PageRank
Pregel_PageRank(i, messages):
// receive all the messages
total = 0
foreach(msg in messages):
total = total + msg
// update the rank of this vertex
R[i] = 0.15 + total
// send new messages to neighbors
foreach(j in out_neighbors[i]):
sendmsg(R[i] * wij) to vertex j
R[i] = 0.15 +
j∈Nbrs(i)
wjiR[j]
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 29 / 40
48. Partitioning the Graph
The pregel library divides a graph into a number of partitions.
Each consisting of a set of vertices and all of those vertices’ outgoing
edges.
Vertices are assigned to partitions based on their vertex-ID (e.g.,
hash(ID)).
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 30 / 40
50. Implementation (2/4)
The master is responsible for
• Coordinating workers activity.
• Determining the number of partitions.
Each worker is responsible for
• Maintaining the state of its partitions.
• Executing the user’s Compute() method on its vertices.
• Managing messages to and from other workers.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 32 / 40
51. Implementation (3/4)
The master assigns one or more partitions to each worker.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 33 / 40
52. Implementation (3/4)
The master assigns one or more partitions to each worker.
The master assigns a portion of user input to each worker.
• Set of records containing an arbitrary number of vertices and edges.
• If a worker loads a vertex that belongs to that worker’s partitions,
the appropriate data structures are immediately updated.
• Otherwise the worker enqueues a message to the remote peer that
owns the vertex.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 33 / 40
53. Implementation (4/4)
After the input has finished loading, all vertices are marked as active.
The master instructs each worker to perform a superstep.
After the computation halts, the master may instruct each worker
to save its portion of the graph.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 34 / 40
54. Combiner
Sending a message between workers incurs some overhead: use com-
biner.
This can be reduced in some cases: sometimes vertices only care
about a summary value for the messages it is sent (e.g., min, max,
sum, avg).
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 35 / 40
55. Fault Tolerance (1/2)
Fault tolerance is achieved through checkpointing.
At start of each superstep, master tells workers to save their state:
• Vertex values, edge values, incoming messages
• Saved to persistent storage
Master saves aggregator values (if any).
This is not necessarily done at every superstep: costly
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 36 / 40
56. Fault Tolerance (2/2)
When master detects one or more worker failures:
• All workers revert to last checkpoint.
• Continue from there.
• That is a lot of repeated work.
• At least it is better than redoing the whole job.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 37 / 40
57. Pregel Limitations
Inefficient if different regions of the graph converge at different
speed.
Can suffer if one task is more expensive than the others.
Runtime of each phase is determined by the slowest machine.
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 38 / 40
58. Pregel Summary
Bulk Synchronous Parallel model
Vertex-centric
Superstep: sequence of iterations
Master-worker model
Communication: message passing
Amir H. Payberah (Tehran Polytechnic) Pregel 1393/9/3 39 / 40
