This document discusses programming with MapReduce and proposes a calculational approach for systematic parallel programming over MapReduce. It begins with background on MapReduce and examples of MapReduce programming. It then discusses issues with directly mapping sequential algorithms to MapReduce. The document proposes expressing computations as list homomorphisms, which can be automatically implemented as MapReduce jobs. It presents an interface for defining sequential functions as fold and unfold and discusses implementing list homomorphisms in MapReduce by representing lists and intermediate data. It evaluates the performance of the homomorphism-based approach.
An optimal general type-2 fuzzy controller for Urban Traffic NetworkISA Interchange
Urban traffic network model is illustrated by state-charts and object-diagram. However, they have limitations to show the behavioral perspective of the traffic information flow. Consequently, a state space model is used to calculate the half-value waiting time of vehicles. In this study, a combination of the general type-2 fuzzy logic sets and the modified backtracking search algorithm (MBSA) techniques are used in order to control the traffic signal scheduling and phase succession so as to guarantee a smooth flow of traffic with the least wait times and average queue length. The parameters of input and output membership functions are optimized simultaneously by the novel heuristic algorithm MBSA. A comparison is made between the achieved results with those of optimal and conventional type-1 fuzzy logic controllers.
Scalability has been an essential factor for any kind of computational algorithm while considering its performance. In this Big Data era, gathering of large amounts of data is becoming easy. Data analysis on Big Data is not feasible using the existing Machine Learning (ML) algorithms and it perceives them to perform poorly. This is due to the fact that the computational logic for these algorithms is previously designed in sequential way. MapReduce becomes the solution for handling billions of data efficiently. In this report we discuss the basic building block for the computations behind ML algorithms, two different attempts to parallelize machine learning algorithms using MapReduce and a brief description on the overhead in parallelization of ML algorithms.
The assignment problem is a special type of linear programming problem and it is sub class of transportation problem. Assignment problems are defined with two sets of inputs i.e. set of resources and set of demands. Hungarian algorithm is able to solve assignment problems with precisely defined demands and resources.Nowadays, many organizations and competition companies consider markets of their products. They use many salespersons to improve their organizations marketing. Salespersons travel form one city to another city for their markets. There are some problems in travelling which salespeople should go which city in minimum cost. So, travelling assignment problem is a main process for many business functions. Mie Mie Aung | Yin Yin Cho | Khin Htay | Khin Soe Myint "Minimization of Assignment Problems" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-5 , August 2019, URL: https://www.ijtsrd.com/papers/ijtsrd26712.pdfPaper URL: https://www.ijtsrd.com/computer-science/other/26712/minimization-of-assignment-problems/mie-mie-aung
Accurate Learning of Graph Representations with Graph Multiset PoolingMLAI2
Graph neural networks have been widely used on modeling graph data, achieving impressive results on node classification and link prediction tasks. Yet, obtaining an accurate representation for a graph further requires a pooling function that maps a set of node representations into a compact form. A simple sum or average over all node representations considers all node features equally without consideration of their task relevance, and any structural dependencies among them. Recently proposed hierarchical graph pooling methods, on the other hand, may yield the same representation for two different graphs that are distinguished by the Weisfeiler-Lehman test, as they suboptimally preserve information from the node features. To tackle these limitations of existing graph pooling methods, we first formulate the graph pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose a Graph Multiset Transformer (GMT) which is a multi-head attention based global pooling layer that captures the interaction between nodes according to their structural dependencies. We show that GMT satisfies both injectiveness and permutation invariance, such that it is at most as powerful as the Weisfeiler-Lehman graph isomorphism test. Moreover, our methods can be easily extended to the previous node clustering approaches for hierarchical graph pooling. Our experimental results show that GMT significantly outperforms state-of-the-art graph pooling methods on graph classification benchmarks with high memory and time efficiency, and obtains even larger performance gain on graph reconstruction and generation tasks.
An optimal general type-2 fuzzy controller for Urban Traffic NetworkISA Interchange
Urban traffic network model is illustrated by state-charts and object-diagram. However, they have limitations to show the behavioral perspective of the traffic information flow. Consequently, a state space model is used to calculate the half-value waiting time of vehicles. In this study, a combination of the general type-2 fuzzy logic sets and the modified backtracking search algorithm (MBSA) techniques are used in order to control the traffic signal scheduling and phase succession so as to guarantee a smooth flow of traffic with the least wait times and average queue length. The parameters of input and output membership functions are optimized simultaneously by the novel heuristic algorithm MBSA. A comparison is made between the achieved results with those of optimal and conventional type-1 fuzzy logic controllers.
Scalability has been an essential factor for any kind of computational algorithm while considering its performance. In this Big Data era, gathering of large amounts of data is becoming easy. Data analysis on Big Data is not feasible using the existing Machine Learning (ML) algorithms and it perceives them to perform poorly. This is due to the fact that the computational logic for these algorithms is previously designed in sequential way. MapReduce becomes the solution for handling billions of data efficiently. In this report we discuss the basic building block for the computations behind ML algorithms, two different attempts to parallelize machine learning algorithms using MapReduce and a brief description on the overhead in parallelization of ML algorithms.
The assignment problem is a special type of linear programming problem and it is sub class of transportation problem. Assignment problems are defined with two sets of inputs i.e. set of resources and set of demands. Hungarian algorithm is able to solve assignment problems with precisely defined demands and resources.Nowadays, many organizations and competition companies consider markets of their products. They use many salespersons to improve their organizations marketing. Salespersons travel form one city to another city for their markets. There are some problems in travelling which salespeople should go which city in minimum cost. So, travelling assignment problem is a main process for many business functions. Mie Mie Aung | Yin Yin Cho | Khin Htay | Khin Soe Myint "Minimization of Assignment Problems" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-5 , August 2019, URL: https://www.ijtsrd.com/papers/ijtsrd26712.pdfPaper URL: https://www.ijtsrd.com/computer-science/other/26712/minimization-of-assignment-problems/mie-mie-aung
Accurate Learning of Graph Representations with Graph Multiset PoolingMLAI2
Graph neural networks have been widely used on modeling graph data, achieving impressive results on node classification and link prediction tasks. Yet, obtaining an accurate representation for a graph further requires a pooling function that maps a set of node representations into a compact form. A simple sum or average over all node representations considers all node features equally without consideration of their task relevance, and any structural dependencies among them. Recently proposed hierarchical graph pooling methods, on the other hand, may yield the same representation for two different graphs that are distinguished by the Weisfeiler-Lehman test, as they suboptimally preserve information from the node features. To tackle these limitations of existing graph pooling methods, we first formulate the graph pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose a Graph Multiset Transformer (GMT) which is a multi-head attention based global pooling layer that captures the interaction between nodes according to their structural dependencies. We show that GMT satisfies both injectiveness and permutation invariance, such that it is at most as powerful as the Weisfeiler-Lehman graph isomorphism test. Moreover, our methods can be easily extended to the previous node clustering approaches for hierarchical graph pooling. Our experimental results show that GMT significantly outperforms state-of-the-art graph pooling methods on graph classification benchmarks with high memory and time efficiency, and obtains even larger performance gain on graph reconstruction and generation tasks.
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Pr...MLAI2
Many practical graph problems, such as knowledge graph construction and drug-drug interaction prediction, require to handle multi-relational graphs. However, handling real-world multi-relational graphs with Graph Neural Networks (GNNs) is often challenging due to their evolving nature, as new entities (nodes) can emerge over time. Moreover, newly emerged entities often have few links, which makes the learning even more difficult. Motivated by this challenge, we introduce a realistic problem of few-shot out-of-graph link prediction, where we not only predict the links between the seen and unseen nodes as in a conventional out-of-knowledge link prediction task but also between the unseen nodes, with only few edges per node. We tackle this problem with a novel transductive meta-learning framework which we refer to as Graph Extrapolation Networks (GEN). GEN meta-learns both the node embedding network for inductive inference (seen-to-unseen) and the link prediction network for transductive inference (unseen-to-unseen). For transductive link prediction, we further propose a stochastic embedding layer to model uncertainty in the link prediction between unseen entities. We validate our model on multiple benchmark datasets for knowledge graph completion and drug-drug interaction prediction. The results show that our model significantly outperforms relevant baselines for out-of-graph link prediction tasks.
Using Siamese Graph Neural Networks for Similarity-Based Retrieval in Process...MaximilianHoffmann7
This presentation deals with the application of Siamese GNNs to achieve efficient similarity-based retrieval in POCBR scenarios. It was held at ICCBR 2020 (online conference). Feel free to contact me via E-Mail: hoffmannm@uni-trier.de
Extended Fuzzy C-Means with Random Sampling Techniques for Clustering Large DataAM Publications
Big data are any data that you cannot load into your computer’s primary memory. Clustering is a primary
task in pattern recognition and data mining. We need algorithms that scale well with the data size. The former
implementation, literal Fuzzy C-Means is linear or serialized. FCM algorithm attempts to partition a finite collection
of n elements into collection of c fuzzy clusters. So, given a finite set of data, this algorithm returns a list of c cluster
centers. However it doesn't scale well and slows down with increase in the size of data and is thus impractical and
sometimes undesirable. In this paper, we propose an extended version of fuzzy c-means clustering algorithm by means of various random sampling techniques to study which method scales well for large or very large data.
Edge Representation Learning with HypergraphsMLAI2
Graph neural networks have recently achieved remarkable success in representing graph-structured data, with rapid progress in both the node embedding and graph pooling methods. Yet, they mostly focus on capturing information from the nodes considering their connectivity, and not much work has been done in representing the edges, which are essential components of a graph. However, for tasks such as graph reconstruction and generation, as well as graph classification tasks for which the edges are important for discrimination, accurately representing edges of a given graph is crucial to the success of the graph representation learning. To this end, we propose a novel edge representation learning framework based on Dual Hypergraph Transformation (DHT), which transforms the edges of a graph into the nodes of a hypergraph. This dual hypergraph construction allows us to apply message-passing techniques for node representations to edges. After obtaining edge representations from the hypergraphs, we then cluster or drop edges to obtain holistic graph-level edge representations. We validate our edge representation learning method with hypergraphs on diverse graph datasets for graph representation and generation performance, on which our method largely outperforms existing graph representation learning methods. Moreover, our edge representation learning and pooling method also largely outperforms state-of-theart graph pooling methods on graph classification, not only because of its accurate edge representation learning, but also due to its lossless compression of the nodes and removal of irrelevant edges for effective message-passing. Code is available at https://github.com/harryjo97/EHGNN.
A COMPARISON BETWEEN SWARM INTELLIGENCE ALGORITHMS FOR ROUTING PROBLEMSecij
Travelling salesman problem (TSP) is a most popular combinatorial routing problem, belongs to the class of NP-hard problems. Many approacheshave been proposed for TSP.Among them, swarm intelligence (SI) algorithms can effectively achieve optimal tours with the minimum lengths and attempt to avoid trapping in local minima points. The transcendence of each SI is depended on the nature of the problem. In our studies, there has been yet no any article, which had compared the performance of SI algorithms for TSP perfectly. In this paper,four common SI algorithms are used to solve TSP, in order to compare the performance of SI algorithms for the TSP problem. These algorithms include genetic algorithm, particle swarm optimization, ant colony optimization, and artificial bee colony. For each SI, the various parameters and operators were tested, and the best values were selected for it. Experiments oversome benchmarks fromTSPLIBshow that
artificial bee colony algorithm is the best one among the fourSI-basedmethods to solverouting problems like TSP.
• Process for heuristics optimization
1. The parser of a high-level query generates an initial internal representation;
2. Apply heuristics rules to optimize the internal representation.
3. A query execution plan is generated to execute groups of operations based on the access paths available on the files involved in the query.
Circular Traffic Signs Recognition Using The Number of Peaks AlgorithmCSCJournals
Smart cars nowadays include embedded computers to guide the driver in his trip. An important application that should be added to any car is the detection and recognition of traffic signs. In this paper, we focus on the recognition of a wide set of circular traffic signs using the Number of Peaks Algorithm [1]. After detecting a traffic sign, the algorithm draws three horizontal lines and three vertical lines across the image. The number of peaks (crossing from a black pixel to a white pixel) is calculated for each of the six lines as the image is scanned from right to left (for horizontal lines) or top to bottom (for vertical lines). The resulting numbers of peaks are used by the decision-tree-like search algorithm to distinguish between 51 circular road signs with a mean detection time of 8 milliseconds, 100% detection rate and in a fairly noisy environment.
Contour Line Tracing Algorithm for Digital Topographic MapsCSCJournals
Topographic maps contain information related to roads, contours, landmarks land covers and rivers etc. For any Remote sensing and GIS based project, creating a database using digitization techniques is a tedious and time consuming process especially for contour tracing. Contour line is very important information that these maps provide. They are mainly used for determining slope of the landforms or rivers. These contour lines are also used for generating Digital Elevation Model (DEM) for 3D surface generation from any satellite imagery or aerial photographs. This paper suggests an algorithm that can be used for tracing contour lines automatically from contour maps extracted from the topographical sheets and creating a database. In our approach, we have proposed a modified Moore's Neighbor contour tracing algorithm to trace all contours in the given topographic maps. The proposed approach is tested on several topographic maps and provides satisfactory results and takes less time to trace the contour lines compared with other existing algorithms.
Information Spread in the Context of Evacuation OptimizationDr. Mirko Kämpf
Abstract: Our evacuation simulation tool utilizes established algorithms for the emotional and intelligence driven motion of human beings in addition to a simple lattice gas simulation. We analyze the spread of information inside a restricted geometry of a real building and compare these results with the data from a simulation in the free space. We apply the DFA and the RIS statistic to our simulation dataset to detect phases or phase transitions of the whole system. We study the impact of communication technology by comparison of different update algorithms and exit strategies. These results help us to define basic functional requirements to the underlying communication technology and network topology as well as to the needed sensors.
A unique sorting algorithm with linear time & space complexityeSAT Journals
Abstract Sorting a list means selection of the particular permutation of the members of that list in which the final permutation contains members in increasing or in decreasing order. Sorted list is prerequisite of some optimized operations such as searching an element from a list, locating or removing an element to/ from a list and merging two sorted list in a database etc. As volume of information is growing up day by day in the world around us and these data are unavoidable to manage for real life situations, the efficient and cost effective sorting algorithms are required. There are several numbers of fundamental and problem oriented sorting algorithms but still now sorting a problem has attracted a great deal of research, perhaps due to the complexity of solving it efficiently and effectively despite of its simple and familiar statements. Algorithms having same efficiency to do a same work using different mechanisms must differ their required time and space. For that reason an algorithm is chosen according to one’s need with respect to space complexity and time complexity. Now a day, space (Memory) is available in market comparatively in cheap cost. So, time complexity is a major issue for an algorithm. Here, the presented approach is to sort a list with linear time and space complexity using divide and conquer rule by partitioning a problem into n (input size) number of sub problems then these sub problems are solved recursively. Required time and space for the algorithm is optimized through reducing the height of the recursive tree and reduced height is too small (as compared to the problem size) to evaluate. So, asymptotic efficiency of this algorithm is very high with respect to time and space. Keywords: sorting, searching, permutation, divide and conquer algorithm, asymptotic efficiency, space complexity, time complexity, and recursion.
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Pr...MLAI2
Many practical graph problems, such as knowledge graph construction and drug-drug interaction prediction, require to handle multi-relational graphs. However, handling real-world multi-relational graphs with Graph Neural Networks (GNNs) is often challenging due to their evolving nature, as new entities (nodes) can emerge over time. Moreover, newly emerged entities often have few links, which makes the learning even more difficult. Motivated by this challenge, we introduce a realistic problem of few-shot out-of-graph link prediction, where we not only predict the links between the seen and unseen nodes as in a conventional out-of-knowledge link prediction task but also between the unseen nodes, with only few edges per node. We tackle this problem with a novel transductive meta-learning framework which we refer to as Graph Extrapolation Networks (GEN). GEN meta-learns both the node embedding network for inductive inference (seen-to-unseen) and the link prediction network for transductive inference (unseen-to-unseen). For transductive link prediction, we further propose a stochastic embedding layer to model uncertainty in the link prediction between unseen entities. We validate our model on multiple benchmark datasets for knowledge graph completion and drug-drug interaction prediction. The results show that our model significantly outperforms relevant baselines for out-of-graph link prediction tasks.
Using Siamese Graph Neural Networks for Similarity-Based Retrieval in Process...MaximilianHoffmann7
This presentation deals with the application of Siamese GNNs to achieve efficient similarity-based retrieval in POCBR scenarios. It was held at ICCBR 2020 (online conference). Feel free to contact me via E-Mail: hoffmannm@uni-trier.de
Extended Fuzzy C-Means with Random Sampling Techniques for Clustering Large DataAM Publications
Big data are any data that you cannot load into your computer’s primary memory. Clustering is a primary
task in pattern recognition and data mining. We need algorithms that scale well with the data size. The former
implementation, literal Fuzzy C-Means is linear or serialized. FCM algorithm attempts to partition a finite collection
of n elements into collection of c fuzzy clusters. So, given a finite set of data, this algorithm returns a list of c cluster
centers. However it doesn't scale well and slows down with increase in the size of data and is thus impractical and
sometimes undesirable. In this paper, we propose an extended version of fuzzy c-means clustering algorithm by means of various random sampling techniques to study which method scales well for large or very large data.
Edge Representation Learning with HypergraphsMLAI2
Graph neural networks have recently achieved remarkable success in representing graph-structured data, with rapid progress in both the node embedding and graph pooling methods. Yet, they mostly focus on capturing information from the nodes considering their connectivity, and not much work has been done in representing the edges, which are essential components of a graph. However, for tasks such as graph reconstruction and generation, as well as graph classification tasks for which the edges are important for discrimination, accurately representing edges of a given graph is crucial to the success of the graph representation learning. To this end, we propose a novel edge representation learning framework based on Dual Hypergraph Transformation (DHT), which transforms the edges of a graph into the nodes of a hypergraph. This dual hypergraph construction allows us to apply message-passing techniques for node representations to edges. After obtaining edge representations from the hypergraphs, we then cluster or drop edges to obtain holistic graph-level edge representations. We validate our edge representation learning method with hypergraphs on diverse graph datasets for graph representation and generation performance, on which our method largely outperforms existing graph representation learning methods. Moreover, our edge representation learning and pooling method also largely outperforms state-of-theart graph pooling methods on graph classification, not only because of its accurate edge representation learning, but also due to its lossless compression of the nodes and removal of irrelevant edges for effective message-passing. Code is available at https://github.com/harryjo97/EHGNN.
A COMPARISON BETWEEN SWARM INTELLIGENCE ALGORITHMS FOR ROUTING PROBLEMSecij
Travelling salesman problem (TSP) is a most popular combinatorial routing problem, belongs to the class of NP-hard problems. Many approacheshave been proposed for TSP.Among them, swarm intelligence (SI) algorithms can effectively achieve optimal tours with the minimum lengths and attempt to avoid trapping in local minima points. The transcendence of each SI is depended on the nature of the problem. In our studies, there has been yet no any article, which had compared the performance of SI algorithms for TSP perfectly. In this paper,four common SI algorithms are used to solve TSP, in order to compare the performance of SI algorithms for the TSP problem. These algorithms include genetic algorithm, particle swarm optimization, ant colony optimization, and artificial bee colony. For each SI, the various parameters and operators were tested, and the best values were selected for it. Experiments oversome benchmarks fromTSPLIBshow that
artificial bee colony algorithm is the best one among the fourSI-basedmethods to solverouting problems like TSP.
• Process for heuristics optimization
1. The parser of a high-level query generates an initial internal representation;
2. Apply heuristics rules to optimize the internal representation.
3. A query execution plan is generated to execute groups of operations based on the access paths available on the files involved in the query.
Circular Traffic Signs Recognition Using The Number of Peaks AlgorithmCSCJournals
Smart cars nowadays include embedded computers to guide the driver in his trip. An important application that should be added to any car is the detection and recognition of traffic signs. In this paper, we focus on the recognition of a wide set of circular traffic signs using the Number of Peaks Algorithm [1]. After detecting a traffic sign, the algorithm draws three horizontal lines and three vertical lines across the image. The number of peaks (crossing from a black pixel to a white pixel) is calculated for each of the six lines as the image is scanned from right to left (for horizontal lines) or top to bottom (for vertical lines). The resulting numbers of peaks are used by the decision-tree-like search algorithm to distinguish between 51 circular road signs with a mean detection time of 8 milliseconds, 100% detection rate and in a fairly noisy environment.
Contour Line Tracing Algorithm for Digital Topographic MapsCSCJournals
Topographic maps contain information related to roads, contours, landmarks land covers and rivers etc. For any Remote sensing and GIS based project, creating a database using digitization techniques is a tedious and time consuming process especially for contour tracing. Contour line is very important information that these maps provide. They are mainly used for determining slope of the landforms or rivers. These contour lines are also used for generating Digital Elevation Model (DEM) for 3D surface generation from any satellite imagery or aerial photographs. This paper suggests an algorithm that can be used for tracing contour lines automatically from contour maps extracted from the topographical sheets and creating a database. In our approach, we have proposed a modified Moore's Neighbor contour tracing algorithm to trace all contours in the given topographic maps. The proposed approach is tested on several topographic maps and provides satisfactory results and takes less time to trace the contour lines compared with other existing algorithms.
Information Spread in the Context of Evacuation OptimizationDr. Mirko Kämpf
Abstract: Our evacuation simulation tool utilizes established algorithms for the emotional and intelligence driven motion of human beings in addition to a simple lattice gas simulation. We analyze the spread of information inside a restricted geometry of a real building and compare these results with the data from a simulation in the free space. We apply the DFA and the RIS statistic to our simulation dataset to detect phases or phase transitions of the whole system. We study the impact of communication technology by comparison of different update algorithms and exit strategies. These results help us to define basic functional requirements to the underlying communication technology and network topology as well as to the needed sensors.
A unique sorting algorithm with linear time & space complexityeSAT Journals
Abstract Sorting a list means selection of the particular permutation of the members of that list in which the final permutation contains members in increasing or in decreasing order. Sorted list is prerequisite of some optimized operations such as searching an element from a list, locating or removing an element to/ from a list and merging two sorted list in a database etc. As volume of information is growing up day by day in the world around us and these data are unavoidable to manage for real life situations, the efficient and cost effective sorting algorithms are required. There are several numbers of fundamental and problem oriented sorting algorithms but still now sorting a problem has attracted a great deal of research, perhaps due to the complexity of solving it efficiently and effectively despite of its simple and familiar statements. Algorithms having same efficiency to do a same work using different mechanisms must differ their required time and space. For that reason an algorithm is chosen according to one’s need with respect to space complexity and time complexity. Now a day, space (Memory) is available in market comparatively in cheap cost. So, time complexity is a major issue for an algorithm. Here, the presented approach is to sort a list with linear time and space complexity using divide and conquer rule by partitioning a problem into n (input size) number of sub problems then these sub problems are solved recursively. Required time and space for the algorithm is optimized through reducing the height of the recursive tree and reduced height is too small (as compared to the problem size) to evaluate. So, asymptotic efficiency of this algorithm is very high with respect to time and space. Keywords: sorting, searching, permutation, divide and conquer algorithm, asymptotic efficiency, space complexity, time complexity, and recursion.
Title: Development of integrated rice-azolla-duck-fish farming systems with SRI methods for rice production in the Mekong River region, Vietnam
Presenter: Nghia Nguyen Soil Biology Laboratory, Department of Soil Science, College of Agriculture & Applied Biology, Cantho University, Cantho City, Vietnam
Venue: Cornell University
Date: July 1, 2015
A study of the Behavior of Floating-Point Errorsijpla
The dangers of programs performing floating-point computations are well known. This is due to numerical reliability issues resulting from rounding errors arising during the computations. In general, these round-off errors are neglected because they are small. However, they can be accumulated and propagated and lead to faulty execution and failures. Typically, in critical embedded systems scenario, these faults may cause dramatic damages (eg. failures of Ariane 5 launch and Patriot Rocket mission). The ufp (unit in the first place) and ulp (unit in the last place) functions are used to estimate maximum value of round-off errors. In this paper, the idea consists in studying the behavior of round-off errors, checking their numerical stability using a set of constraints and ensuring that the computation results of round-off errors do not become larger when solving constraints about the ufp and ulp values.
GRAPH MATCHING ALGORITHM FOR TASK ASSIGNMENT PROBLEMIJCSEA Journal
Task assignment is one of the most challenging problems in distributed computing environment. An optimal task assignment guarantees minimum turnaround time for a given architecture. Several approaches of optimal task assignment have been proposed by various researchers ranging from graph partitioning based tools to heuristic graph matching. Using heuristic graph matching, it is often impossible to get optimal task assignment for practical test cases within an acceptable time limit. In this paper, we have parallelized the basic heuristic graph-matching algorithm of task assignment which is suitable only for cases where processors and inter processor links are homogeneous. This proposal is a derivative of the basic task assignment methodology using heuristic graph matching. The results show that near optimal assignments are obtained much faster than the sequential program in all the cases with reasonable speed-up.
Traditional interpreted data-flow analysis is executed on whole plans; however, such whole-program psychoanalysis is not
executable for large or uncompleted plans. We suggest fragment data-flow analysis as a substitute approach which
calculates data-flow information for a particular program fragment. The psychoanalysis is parameterized by the extra
information available about the rest of the program. We depict two frameworks for interracial flow-sensitive fragment
psychoanalysis, the relationship amongst fragment psychoanalysis and whole-program analysis, and the necessities ensuring fragment analysis safety and feasibility. We suggest an application of fragment analysis as a second analysis phase after a cheap flow-insensitive whole-program analysis, in order to obtain better data for important program fragments. We also depict the design of two fragment analyses derived from an already existing whole-program flow- and context-sensitive pointer alias analysis for Computer program and present empirical rating of their cost and precision. Our experiments show evidence of dramatic improves precision gettable at a practical cost.
DETECTION OF CONTROL FLOW ERRORS IN PARALLEL PROGRAMS AT COMPILE TIME ijdpsjournal
This paper describes a general technique to identify control flow errors in parallel programs, which can be automated into a compiler. The compiler builds a system of linear equations that describes the global control flow of the whole program. Solving these equations using standard techniques of linear algebra
can locate a wide range of control flow bugs at compile time. This paper also describes an implementation of this control flow analysis technique in a prototype compiler for a well-known parallel programming language. In contrast to previous research in automated parallel program analysis, our technique is efficient for large programs, and does not limit the range of language features.
An Implementation on Effective Robot Mission under Critical Environemental Co...IJERA Editor
Software engineering is a field of engineering, for designing and writing programs for computers or other electronic devices. A software engineer, or programmer, writes software (or changes existing software) and compiles software using methods that make it better quality. Is the application of engineering to the design, development, implementation, testingand main tenance of software in a systematic method. Now a days the robotics are also plays an important role in present automation concepts. But we have several challenges in that robots when they are operated in some critical environments. Motion planning and task planning are two fundamental problems in robotics that have been addressed from different perspectives. For resolve this there are Temporal logic based approaches that automatically generate controllers have been shown to be useful for mission level planning of motion, surveillance and navigation, among others. These approaches critically rely on the validity of the environment models used for synthesis. Yet simplifying assumptions are inevitable to reduce complexity and provide mission-level guarantees; no plan can guarantee results in a model of a world in which everything can go wrong. In this paper, we show how our approach, which reduces reliance on a single model by introducing a stack of models, can endow systems with incremental guarantees based on increasingly strengthened assumptions, supporting graceful degradation when the environment does not behave as expected, and progressive enhancement when it does.
This article got published in the Software Developer's Journal's February Edition.
It describes the use of MapReduce paradigm to design Clustering algorithms and explain three algorithms using MapReduce.
- K-Means Clustering
- Canopy Clustering
- MinHash Clustering
HyPR: Hybrid Page Ranking on Evolving Graphs (NOTES)Subhajit Sahu
Highlighted notes on A Parallel Algorithm Template for Updating Single-Source Shortest Paths in Large-Scale Dynamic Networks.
While doing research work under Prof. Dip Banerjee, Prof, Kishore Kothapalli.
In Hybrid Pagerank the vertices are divided in 3 groups, V_old, V_border, V_new. Scaling for old, border vertices is N/N_new, and 1/N_new for V_new (i do this too ). Then PR is run only on V_border, V_new.
"V_border which is the set of nodes which have edges in Bi connecting V_old and V_new and is reachable using a breadth first traversal."
Does that mean V_border = V_batch(i) ∩ V_old? BFS from where?
"We can assume that the new batch of updates is topologically sorted since the PR scores of the new nodes in Bi is guaranteed to be lower than those in Co."
Is sum(PR) in V_old > sum(PR) in V_new always?
"For performing the comparisons with GPMA and GPMA+, we configure the experiment to run HyPR on the same platform as used in [1] which is a Intel Xeon CPU connected to a Titan X Pascal GPU, and also the same datasets."
Old GPUs are going to be slower ...
Like we were discussing last time, it is not possible to scale old ranks, and skip the unchanged components (or here V_old). Please check this simple counter example that shows skipping leads to incorrect ranks.
https://github.com/puzzlef/pagerank-levelwise-skip-unchanged-components
Another omission in the paper is that Hybrid PR (just like STICD) wont work for graphs which have dead ends. This is a pre-condition for the algorithm.
Parallel Batch-Dynamic Graphs: Algorithms and Lower BoundsSubhajit Sahu
Highlighted notes on Parallel Batch-Dynamic Graphs: Algorithms and Lower Bounds.
While doing research work under Prof. Kishore Kothapalli.
Laxman Dhulipala, David Durfee, Janardhan Kulkarni, Richard Peng, Saurabh Sawlani, Xiaorui Sun:
Parallel Batch-Dynamic Graphs: Algorithms and Lower Bounds. SODA 2020: 1300-1319
In this paper we study the problem of dynamically maintaining graph properties under batches of edge insertions and deletions in the massively parallel model of computation. In this setting, the graph is stored on a number of machines, each having space strongly sublinear with respect to the number of vertices, that is, n for some constant 0 < < 1. Our goal is to handle batches of updates and queries where the data for each batch fits onto one machine in constant rounds of parallel computation, as well as to reduce the total communication between the machines. This objective corresponds to the gradual buildup of databases over time, while the goal of obtaining constant rounds of communication for problems in the static setting has been elusive for problems as simple as undirected graph connectivity. We give an algorithm for dynamic graph connectivity in this setting with constant communication rounds and communication cost almost linear in terms of the batch size. Our techniques combine a new graph contraction technique, an independent random sample extractor from correlated samples, as well as distributed data structures supporting parallel updates and queries in batches. We also illustrate the power of dynamic algorithms in the MPC model by showing that the batched version of the adaptive connectivity problem is P-complete in the centralized setting, but sub-linear sized batches can be handled in a constant number of rounds. Due to the wide applicability of our approaches, we believe it represents a practically-motivated workaround to the current difficulties in designing more efficient massively parallel static graph algorithms.
Similar to Towards Systematic Parallel Programming over MapReduce (20)
A TPC Benchmark of Hive LLAP and Comparison with PrestoYu Liu
It is a TPC/H/DS benchmark on both Hive (Low Latency Analytical Processing) and Presto, comparing the two popular bigdata query engines.
The results shows significant advantages of Hive LLAP on performance and durability.
Paper introduction to Combinatorial Optimization on Graphs of Bounded TreewidthYu Liu
This slides introduced the paper: H. L. Bodlaender and a. M. C. a. Koster, “Combinatorial Optimization on Graphs of Bounded Treewidth,” Comput. J., vol. 51, no. 3, pp. 255–269, Nov. 2007.
Paper Introduction: Combinatorial Model and Bounds for Target Set SelectionYu Liu
The paper Combinatorial Model and Bounds for Target Set Selection by Eyal Ackerman, Oren Ben-Zwi, Guy Wolfovitz:
1. a combinatorial model for the dynamic activation process of
influential networks;
2. representing Perfect Target Set Selection Problem and its
variants by linear integer programs;
3. combinatorial lower and upper bounds on the size of the
minimum Perfect Target Set
Introduction of A Lightweight Stage-Programming FrameworkYu Liu
The Lightweight Stage-Programming Framework introduced in this slides can be used for making efficient parallel DSL which can be transformed to MapReduce programs. To understand this slides, please firstly read http://www.slideshare.net/YuLiu19/a-generatetestaggregate-parallel-programming-library-on-spark.
On Extending MapReduce - Survey and ExperimentsYu Liu
It talks a survey and my experiments on extending MapReduce programming model. A BSP-based MapReduce interface was implemented and evaluated, which shows dramatically improvement on performance.
Implementing Generate-Test-and-Aggregate Algorithms on HadoopYu Liu
Generate-Test-and-Aggregate is a class of algorithms that can automatically derive efficient MapReduce programs.
MapReduce is a useful and popular programming model for large-scale parallel processing. However, for many complex problems, it is usually not easy to develop the efficient parallel algorithms that match MapReduce paradigm well.
The generator-based parallelization approach has been developed and introduced to simplify parallel programming by its automatic generating and optimizing mechanism. Efficient parallel algorithms can be generated from users' naive but correct programs by making use of generators which exploit knowledge of optimization theorems in the field of skeletal parallel programming. The obtained efficient-parallel algorithms are in the form that very fit for implementation with MapReduce.
By such an approach, a large class of generate-and-test-like computations can be efficiently programmed and computed over MapReduce. Thus a novel programming interface and framework can be built on top of MapReduce, and that would be helpful for resolving the difficulties on programmability and efficiency. In this paper we will introduce a framework that has such a novel programming interface for MapReduce. With this framework, users can just concentrate on making naive correct programs. We will show that a lot of so-called generate-and-test-like computations can be easily and efficiently implemented by this framework over MapReduce.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Towards Systematic Parallel Programming over MapReduce
1. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Towards Systematic Parallel Programming over
MapReduce
Yu Liu1, Zhenjiang Hu2, Kiminori Matsuzaki3
1The Graduate University for Advanced Studies, Japan
2National Institute of Informatics, Japan
3School of Information, Kochi University of Technology, Japan
September 1, 2011
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
2. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
About MapReduce
Processing flow
Computation is mapped to three phases: map, shuffle and reduce ,
and automatically parallelize it.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
3. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Programming with MapReduce
Generally, programmers need to implement the following classes1:
1
This takes the opensource MapReduce framework Hadoop as an example.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
4. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Map and Reduce
map function map :: (k1, v1) → [(k2, v2)]
reduce function reduce :: (k2, [v2]) → v3
getPartition function getPartition :: (k, v) → Int
compare function compare :: (k1 → k2) → (1, 0, −1)
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
5. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
An Example: even-odd sum of squares
Example
Given a set of integers, compute the sum of squares of odd
integers and sum of squares of even integers, respectively.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
6. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
The Parallelism
map phase: fully in parallel
reduce phase: by default there are one reduce task for an
input key but we can resolve it by redefine the partitioner or
use combiners in map phase.
This example can be fully parallelized because there is a
commutative binary operator + to accumulate the values.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
7. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Another Example: maximum prefix sum
Example
Given a list of n integers [a0, a1, ..., an−1] , find the maximum sum
of a prefix among all the sums of prefixes:
y0 = a0
y1 = a0 + a1
y2 = a0 + a1 + a2
yn = a0 + a1 + a2 + ... + an−1
assume that the list is too large to be put in a single node
How to compute the MPS in divide-and-conquer fashion?
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
8. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Another Example: maximum prefix sum (2)
How to resolve MPS problem with MapReduce?
Dived the input list to independent splits...
Sort and shuffle the intermediate data...
Reduce the values...
The problems are:
The order of input list can not be changed
There is no such an binary operator ⊕ that:
mps[a0, a1, ...an]
= mps[a0, ..., ai ] ⊕ mps[ai+1, ..., aj ] ⊕ mps[aj+1, ..., an]
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
9. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
A Myth of MapReduce
Users do not need to consider much about parallelism, just write
sequential programs ...
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
10. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
The Gap between Sequential Algorithms and MapReduce
Mapping a sequential algorithm to MapReduce(parallelization) is
non-trivial. Programmers of MapReduce need
Systematic way to do programming
High level wrappings to get rid of heavy coding
Guarantees for reliability, efficiency, and reusability
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
11. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Calculational Approach for MapReduce
Approach
programmers express their computation by
two sequential programs
the system generate an intermediate format which matches
the divide-and-conquer fashion, and
automatically implement the intermediate format to
MapReduce
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
12. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
List Homomorphism
List Homomorphisms[Bird, 87]2 are well studied from 90’ in
community of parallel programming[Cole,95] 3.
Definition of List Homomorphism
If there is a function f and an associative operator and an
identity element id of , such that for any list x and list y
h [a] = f a
h (x ++ y) = h(x) h(y).
Where ++ is the list concatenation.
2
Bird et al, An introduction to the theory of lists, 1987
3
Cole, Parallel programming with list homomorphisms. etc
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
14. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Programming Interface Definition
In our framework, input sequential functions are defined as:
fold and unfold
fold :: [α] → β
unfold :: β → [α].
The restrictions on fold and unfold are
fold ◦ unfold ◦ fold = fold
There must exist a list homomorphism for fold 4.
4
I.e, fold must be both a leftwards and rightwards function
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
16. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Homomorphism based Calculational Approach
Key points of our approach
Express a sequential algorithm in format of list homomorphism
Implement the list homomorphism by MapReduce
Users’ input programs can be easily tested and reused
Parallelization is automatically done thus high reliability can
be obtained
Combination or fusion of list homomorphisms are still list
homomorphisms thus complex algorithms can be implemented
progressively
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
17. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Derivation of List Homomorphism
By third homomorphism theorem [Gibbons 96]5 and its
corollaries[Morita, PLDI07]6 we can get a list homomorphism by
given a couple of functions:
List homomorphism ([f , ⊕]) defined by a couple functions
Iff there is a list homomorphism for fold and
fold ◦ unfold ◦ fold = fold, then we have:
f a = fold([a])
a ⊕ b = fold(unfold a ++ unfold b)
5
Gibbons, J.: The third homomorphism theorem
6
Morita, et al.: Automatic inversion generates divide-and-conquer parallel
programs.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
18. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Implement a List Homomorphism with MapReduce
To implement list homomorphisms with MapReduce is
straightforward, we only need to consider about two problems:
1 Represent input data as a list
2 Intermediate data should be able to reordered due to the
associativity of binary operator ⊕ of a list homomorphism
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
19. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
The Representation of List in Screwdriver
We use indexed data and sort mechanism to represent a list.
Set of pairs ⇒ list
A list [a, b, c, d, e] can be represented by
{(3, d), (1, b), (2, c), (0, a), (4, e)} and a sortp function.
We can use a preprocessing to produce such pairs form original
input data.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
20. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
The Representation of Intermediate Data in Screwdriver
To extract parallelism, we need to split a long list to multiple
shorter lists and feed them to different Reducers.
Nested pairs ⇒ splited lists
split {(3, d), (1, b), (2, c), (0, a), (4, e)} to:
{((0, 1), b), ((0, 2), c), ((0, 0), a)} and
{((1, 3), d), ((1, 4), e)},
a sortnp function can sort these nested pairs
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
21. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Processing in 2 passes map-reduce (1)
In 1st pass MapReduce
Map phase
(id, val) → ((parId, id), val)
e.g, parId = id ÷ 100000
Shuffle and sort
dispatching all data have same parId to a Reducer, where they are
sorted by the sortnp function
Reduce phase
accumulate values using ⊕ that we have gotten.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
22. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Processing in 2 passes map-reduce (2)
In 2nd MapReduce,
Map phase
map all parId to an identity, e.g, parId = 0
Shuffle and sort
same as 1st pass. Due to same parId, all data are collected by one
Reducer.
Reduce phase
same
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
23. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Case Study: MPS
We can define the fold 7and its right inverse unfold as:
fold [a] = (a ↑ 0, a)
fold (x ++ [a]) = let (m, s) =
fold x in (m ↑ (s + a), s + a)
unfold (m, s) = [m, s − m]
The Java implementation have been showed...
7
It is might be still tricky to define a both leftwards and rightwards function
and find its right inverse but some tools can help.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
24. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Actual Java Program for MPS
You can run your application on a Hadoop client:
:˜$ hadoop jar your.jar input output resource.xml
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
25. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Performance Evaluation
We implemented several programs for three problems on our
framework and Hadoop:
1 the maximum-prefix-sum problem.
MPS-lh is implemented using our framework’ API.
MPS-mr is implemented by Hadoop API.
2 parallel sum of 64-bit integers
SUM-lh1 (using combiner) and SUM-lh2 (not using combiner)
are implemented by our framework’ API.
SUM-mr is implemented by Hadoop API (using combiner).
3 VAR-lh computes the variance of 32-bit floating-point
numbers;
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
26. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Performance Evaluation
Environment: hardware
We configured clusters with 2, 4, 8, 16 and 32 nodes (virtual
machines). Each computing/data node has one CPU (VM, Xeon
E5530@2.4GHz, 1 core), 3 GB memory.
Test data
64-bit integer list of length 108 (593 MB)
32-bit floating-point list of length 108(593 MB)
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
27. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Evaluation Result
The experiment results are summarized :
double the work-nodes, can always get more than 1.8 times speedup
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
28. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
A summary
Our framework introduces slightly overhead
Multi-passes MapReduce processing8
Using and with out using Combiner, the performance are
significant different.
8
When have commutative binary operator for reduce, it can be optimized to
one pass
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
29. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Homomorphism based framework
Performance evaluation and results
Related Work
Parallel programming with list homomorphisms (M.Cole 95)
The Third Homomorphism Theorem(J.Gibbons 96).
Systematic extraction and implementation of
divide-and-conquer parallelism (Gorlatch PLILP96).
Automatic inversion generates divide-and-conquer parallel
programs(Morita et.al., PLDI07).
Google’s Sawzall language 9
9
R. Pike et al., Interpreting the Data: Parallel Analysis with Sawzall
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
30. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Conclusion
The highlight of our approach:
Hiding the technical details of MapReduce(Hadoop)
Parallelization and optimization is automatically done
Automatic list homomorphism generating approach can be
adopted, thus the MapReduce programming can be highly
automated, reliable, and optimized.
Testing and code-reuse for users becomes simpler.
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
31. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
Future Work
Extension and optimization of current framework
Approaches of automatically generating list homomorphisms
Homomorphism and MapReduce on trees and graphs
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce
32. Background
Systematic MapReduce programming
Implementation and evaluation
Conclusion and future work
The project is hosted on
http://screwdriver.googlecode.com
Yu Liu1
, Zhenjiang Hu2
, Kiminori Matsuzaki3
Towards Systematic Parallel Programming over MapReduce