• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Siyapath - A P2P, Gossip based Volunteer Computing Framework

Siyapath - A P2P, Gossip based Volunteer Computing Framework



Slide deck presented at APICTA 2012 - Brunei, Darussalam participating in the tertiary category

Slide deck presented at APICTA 2012 - Brunei, Darussalam participating in the tertiary category



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Siyapath - in my mother tongue means a hundred of petals. A lotus.L&G, we present to you - Siyapath, the P2P VCF based on gossip protocols.This, is a team work of the 4 of us; myself - Nadeeshani, I’ll be introducing you into Siyapath. Next, my colleagues Amila & Malith will take you through the demonstration. Finally, Nufail will add the ending notes.
  • Applications demanding high computational power can be lavishly found today. And the demand has been rising over the time. Simulations,Climate models,Gene sequencers, Biologically inspired algorithms such as ant colonies, genetic algorithmsare some of the instances applied both in research & industry.
  • So how do we address the problem of HPC - High performance computing?Supercomputing, is one approach; this involves a considerable expense.Distributed computing is the next approach, where a set/group of computers collectively get together to solve a single computational problem.Variations exist - clusters, grids, desktop grids & volunteer computing.Let me highlight volunteer computing, the approach Siyapath is based on.It’s the general public who volunteer/donor computational power.Thus the resource pool is large. It is self financing and self maintaining, eliminating the cost burden from the project owner.
  • Siyapath - the unique, innovative solution to address HPC.1. Siyapath possesses a P2P architecture whereas the existing VCFs are client server based. Client server architecture exhibits server bottleneck, when numerous clients request the service of a single server. In case the server crashes, the entire system may become dysfunctional, displaying single point of failure. But in Siyapath’s P2P architecture, a single siyapath node runs a server thread as well as a client thread. Roles played by each node are dynamic. Hence, due to P2P, there is no server bottleneck, no single point of failure.2. Siyapath uses gossip protocol as the group communication protocol. i.e. to communicate between siyapath nodes, discover peers and formulate the network.None of the existing VCFs use gossip for communication. Yet the protocol is proven to be appealing in large scaled decentralized systems. It exhibits well known advantages such as improved scalability, robustness, reliability.3. Siyapath is also very simple to configure, whereas existing VCFs consume lots of time & work including creating projects, databases, and configuring hundreds of parameters. But in Siyapath it’s a matter of downloading the binary distribution and configuring a few parameters.
  • Now lets focus on Siyapath’s features.A job in Siyapath can be defined as a problem demanding high computational power. This problem can be divided into subproblems, each known as a task in Siyapath terminology.Now a user has to chunk his job into several tasks, and submit it to the volunteer computing network. The immediate volunteer receiving the job plays the role of ‘job processor. Once again, roles are dynamic.Received jobs get scheduled at the job processor in a queue, FCFS.Next the tasks are distributed to other volunteer nodes in the network. Each individual task is processed, and results are sent back to the job processor.The user, at his discretion can configure how many nodes run the same task. This is redundancy. It is checked whether all nodes running the same task, return the same result, in order to validate; and discard irregularities.In a volunteer computing system, volunteers can join or leave the network without notice. Siyapath exhibits fault tolerance where a Task-Tracker keeps track of whether a task processor dies, and a backup handler keeps track if a job-processor dies, and resumes from that point.Security is also addressed in Siyapath. We use java sandboxing to implement security.
  • Now L&G, we’ll take you through the demonstration.
  • We have done a load test to test the performance of the Siyapath system.We had a constant number of volunteer nodes and increased the no. of users submitting jobs to the system. So the load on the system was gradually increased while the resources in the system was at a constant.The average response time and the average throughput was measured. The response time shows a gradual increase which is in accordance with the increase in the load. The average throughput is more or less constant and shows no spikes in the graph. Therefore it can be concluded that Siyapath performs well under increasing load and displays scalable properties.
  • To ensure good quality, Siyapath has been developed under industry standards and best practices. Being an open source project, Siyapath achieves source code management and version control using subversion. Apache Maven has been used for build automation. Unit and integration testing was done using junit. Apache log4j framework has been used for all logging purposes. We have a continuous integration system hosted at cloudbees which uses apache jenkins. And a user friendly website is maintained at siyapath.org.
  • Then we'll look at some of the technologies deployed in Siyapath. Apache Thrift is used as the services framework. All communication between nodes are done using service interface provided by thrift. Thrift handles all the implementation details of the network. We have achieved p2p nature by using both thrift client and thrift server in each node.Gossip protocol is used as the group communication protocol in Siyapath. We have our very own implementation of gossip protocol.We have used SIGAR in our system monitoring module. This is used to monitor system resources of volunteer nodes such as the processor and memory available.Implementation of Siyapath has been done using Java Standard Edition.
  • Siyapath won the Gold award in the Tertiary category at national best quality ICT awards 2012. Siyapath was also selected as the best product developed using open source technology at the same award ceremony. A research paper has been accepted for the Engineering Research Unit symposium of university of moratuwa.
  • The binary distribution of Siyapath is accessible through the Siyapath website. Anyone could access the source code of Siyapath through the Google code project page of Siyapath. We have provided user manuals and user API docs for users to get familiar with Siyapath.
  • Siyapath would be helpful to researchers with high performance computing needs. Incountries such as Sri Lanka universities do not have supercomputers. Thereforeresearchers find it difficult to research on subjects that involve high performancecomputing. But they could use siypath and make use of the large number of computersin their university laboratories, which are idle most of the time, to satisfy theircomputing needs.As the product is freely available, siyapath could provide commercial support to gain economical benefits. We could build an economic model based on evalution of the requirements and feasibility, deployment of the system and providing continued production support which provides latest feature upgrades and fixes.

Siyapath - A P2P, Gossip based Volunteer Computing Framework Siyapath - A P2P, Gossip based Volunteer Computing Framework Presentation Transcript

  • SIYAPATHP2P, Gossip Based Volunteer Computing Framework
  • HOW TO ADDRESS? Supercomputing Distributed Computing  Cluster Computing  Grid/Desktop-Grid Computing  Volunteer Computing  Computational power volunteered by general public  Large resource pool3 APICTA 2012 - Brunei Darussalam
  • INTRODUCING SIYAPATHP2P, GOSSIP BASED VOLUNTEER COMPUTING FRAMEWORK P2P Architecture  Absence of server bottleneck  Absence of single point of failure Gossip Protocols  Scalability  Robustness  Reliability Simplicity in Configuring4 APICTA 2012 - Brunei Darussalam
  • FEATURES Job Submission Job Scheduling User Job Processor Task Distribution Task Processing Result Validation Result Delivery Task Task Task Fault Tolerance Processor Processor Processor Security5 APICTA 2012 - Brunei Darussalam
  • DEMONSTRATION6 APICTA 2012 - Brunei Darussalam
  • PERFORMANCE TESTINGTest 1 2 3 4 5Users 5 10 15 20 25Volunteers 30 30 30 30 30Avg. Response Time (s) 52.16 89.63 136.29 211.02 249.25Avg. Throughput (Tasks per 126.76 136.45 130.03 109.49 125.96minute) Average Response Time Average Throughput 300 160 250 140 120 Tasks per minute Time (seconds) 200 100 150 80 100 60 40 50 20 0 0 5 10 15 20 25 5 10 15 20 257 Users APICTA 2012 - Brunei Darussalam Users
  • BEST PRACTICES Version Control System – Subversion Build Automation – Apache Maven Unit / Integrated Testing – JUnit Logging – Apache Log4J Continuous Integration – Apache Jenkins User Friendly Website – siyapath.org8 APICTA 2012 - Brunei Darussalam
  • TECHNOLOGIES Services Framework - Apache Thrift  Communication via a service interface  Addresses implementation details of the network  P2P nature using thrift client/server Gossip Protocols System Monitoring – SIGAR Java Standard Edition9 APICTA 2012 - Brunei Darussalam
  • ACHIEVEMENTSBest Product Developed Using Gold Award Research Paper Open Source Technology Tertiary Category ERU Symposium NBQSA 2012 NBQSA 201210 APICTA 2012 - Brunei Darussalam
  • REACHING THE COMMUNITY  Siyapath distribution available at Siyapath website  http://siyapath.org/download.html  Source code available at Google Code  http://code.google.com/p/siyapath/  User manuals for the framework  User API docs11 APICTA 2012 - Brunei Darussalam
  • COMMERCIALIZATON  Support researchers in conducting high performance computing  Use laboratories of universities  Provide support & consulting  Evaluation  Deployment  Production12 APICTA 2012 - Brunei Darussalam
  • TEAM  Members:  Amila Manoj Silva  Nadeeshani Hewage  Mohamed Nufail  Malith Dhanushka  Supervisors:  Dr. Srinath Perera  Mrs. Vishaka Nanayakkara13 APICTA 2012 - Brunei Darussalam
  • SIYAPATH www.siyapath.org