Kalman Graffi - IEEE HPCS 2013 - Comparative Evaluation of P2P Systems Using PeerfactSim.KOM

263 views

Published on

Kalman Graffi - IEEE HPCS 2013 - Comparative Evaluation of P2P Systems Using PeerfactSim.KOM

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
263
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Kalman Graffi - IEEE HPCS 2013 - Comparative Evaluation of P2P Systems Using PeerfactSim.KOM

  1. 1. Comparative Evaluation of Peer-to-Peer Systems Using PeerfactSim.KOM Kalman Graffi Technology of Social Networks Institute of Computer Science Heinrich Heine University in Düsseldorf eMail: graffi@cs.uni-duesseldorf.de Matthias Feldotto Theory of Distributed Systems Institute of Computer Science University of Paderborn eMail: feldi@mail.upb.de See: www.peerfact.org
  2. 2. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 2 Agenda Background on Evaluation Approaches Requirements on Simulating P2P Systems PeerfactSim.KOM Comparative Evaluation in PeerfactSim.KOM Conclusions
  3. 3. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 3 Analysis – Often simplified model • Homogeneous nodes, probabilistic actions – Leads to proofs (under specific conditions) Example: – Given: Weighted DAG – Results: • Proofs • Complexities - O (log N) Good: General results Weakness: – Details lost in abstraction – Sometimes the constants are important General Evaluation Methods
  4. 4. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 4 Simulation – Advanced and heterogeneous model • Specific node characteristics, capacities, behavior – Investigates emerging behavior – Often focus on quality of service • Response times, induced traffic, specific node load • A response time of 1s to 5s matters! Examples: Given: – 10.000 nodes, capacity distribution X – 70% altruistic nodes, 20% selfish nodes, 10% malicious nodes – Protocol XY, workload Z Results: – Statistics on quality of service over time General Evaluation Methods
  5. 5. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 5 Prototype – in Testbed / in real world – Deployment of code in real testbed (e.g. PlanetLab) – Most adequate models, unpredictable user behaviour – Challenging to coordinate the tests, gather results – Logging and coordination might disturb the results Example Given: – 733 PlanetLab nodes – Full protocol stack: IP, TCP/UDP, middleware, application, virtual users – Deployment in global PlanetLab Results: – Behavior under realistic network conditions • Delays, jitter, node load … General Evaluation Methods
  6. 6. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 6 Agenda Background on Evaluation Approaches PeerfactSim.KOM Requirements on Simulating P2P Systems Comparative Evaluation in PeerfactSim.KOM Conclusions
  7. 7. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 7 PeerfactSim.KOM (see www.peerfact.org) Type – Event-based simulator – Written in Java – Simulations up to 100K peers possible – Focus on simulation of p2p systems on various layers • User • Application • Services: monitoring, replication … • Overlays • Network models Invitation to join the community – Several universities use and extend the simulator actively – Used and heavily extended in the project
  8. 8. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 8 Layered View Layered Architecture – Easy exchange of components – Testing of new applications – Testing of new mechanisms Main idea – Layers have several implementations – Enables testing of individual layer mechanisms • on its own • in combination with other layers See www.peerfact.org Application Overlay User SimulationEngine Network Service Transport Application Overlay User SimulationEngine Network Service Transport
  9. 9. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 9 Layers and Implementations
  10. 10. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 10 Analyzers and Final Graph Generation CSV filesCSV files GraphsGraphs gnuplot
  11. 11. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 11 Agenda Background on Evaluation Approaches PeerfactSim.KOM Requirements on Simulating P2P Systems Comparative Evaluation in PeerfactSim.KOM Conclusions
  12. 12. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 12 Requirements on Simulating P2P Systems For what do we want to use simulators? – Evaluation one or several protocol through – Parameter studies with multiple seeds What we want: specify once – collect all graphs at the end – Specify all setups in the begin • E.g. overlays=pastry,chord,kademlia network=simple,gnp seed=10 • Resulting in 60 simulations: 10 simulations each of pastry-simple, pastry-gnp, chord-simple, chord-gnp, kademlia-simple, kademlia-gnp – Automated execution • Parallelity configurable, individual maximum memory – Automated results processing and plotting • Combine all results of a setup – calculate means and std.deviations • Plot results of all setups in graphs
  13. 13. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 13 Motivation and Essence of the Paper in Summary Before Simulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot results Combine all individual simulation results Combine all individual simulation results Check how well your protocol performed Check how well your protocol performed Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Combine all individual simulation results Combine all individual simulation resultsCombine all individual simulation results Combine all individual simulation resultsCombine all individual simulation results Combine all individual simulation results
  14. 14. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 14 Motivation and Essence of the paper in Summary Now Simulation setup Simulation setup Start simulation Start simulation Check how well your protocol performed Check how well your protocol performed Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Combination of all simulation results Combination of all simulation results Automated plottingAutomated plotting Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim.
  15. 15. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 15 Agenda Background on Evaluation Approaches PeerfactSim.KOM Requirements on Simulating P2P Systems Comparative Evaluation in PeerfactSim.KOM Conclusions
  16. 16. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 16 Approach 1. Common interfaces 2. General analyzers 3. Configurations and automated result compilation
  17. 17. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 17 Analyzers Collecting – Events are monitored and reported to analyzers – Analyzers maintain overview on metrics Hierarchy of output on three different layers: – Comparative evaluation (for all parameter combinations) – Multiple-seed evaluation for one parameter setup – Single evaluation of one setup with one seed Output style: – ordered by the time – grouped by the single peers – grouped by the single peers and sorted by their values
  18. 18. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 18 Analyzers by Layers DefaultAggregationAnalyzer: query events in aggregation nodes DefaultDHTOverlayAnalyzer: lookup and store events in a DHT overlay DefaultKBROverlayAnalyzer: query and routing events in a KBR overlay DefaultUnstructuredAnalyzer: connections and queries in an unstructured overlay DefaultTransAnalyzer: sent and received messages in the transport layer DefaultNetAnalyzer: sent and received messages in the network layer DefaultChurnAnalyzer: session times and inter-session times DefaultConnectivityAnalyzer: on- and offline events of nodes DefaultOperationAnalyzer: operations in the simulator DefaultMemoryUsageAnalyzer: memory used by the simulator DefaultTimeUsageAnalyzer: time used by the simulator
  19. 19. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 19 Sample Evaluations: All Overlays Setup: 1000 nodes Overlays – Unstructured: Gnutella06 and Gia – Structured: Chord, EpiChord, ReChord, CaReChord, Pastry and CAN Seeds: 20 Actions: – 120 min joining phase – 60 min publishing phase – 120 min lookup phase – Publish and lookup files from a file set based on a Zipf distribution Network: Simple network, no churn Interest: Successful lookups/search, bandwidth consumption
  20. 20. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 20 Successful Downloads
  21. 21. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 21 Bandwidth Consumption
  22. 22. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 22 Sample Evaluations: Structured Overlays Setup: 1000 nodes Overlays – Structured: Chord, EpiChord, ReChord, CaReChord, Pastry and CAN Seeds: 20 Actions: – 120 min joining phase – 180 min lookup phase (1 lookup every 30-120 seconds) – Publish and lookup files from a file set based on a Zipf distribution Network: Global Network Positioning, no churn Interest: Hop count, traffic / bandwidth usage
  23. 23. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 23 Hop Count Measure
  24. 24. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 24 Served Messages per Node
  25. 25. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 25 Forwarded Messages
  26. 26. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 26 Bandwidth Consumption
  27. 27. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 27 Sample Evaluations: Chord Variants Setup: 1000 nodes Overlays – Structured: Chord, EpiChord, ReChord, CaReChord Seeds: 20 Actions: – 120 min joining phase – 180 min lookup phase (1 lookup every 30-120 seconds) – Publish and lookup files from a file set based on a Zipf distribution Network: – Modular network layer: measurement-based delay, packet loss, jitter and bandwidth. – Churn: based on measurements in KAD by Steiner et al. Interest: Hop count, traffic / bandwidth usage
  28. 28. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 28 Average Hop Count
  29. 29. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 29 Lookup Duration
  30. 30. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 30 Bandwidth Consumption
  31. 31. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 31 Simulation Time
  32. 32. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 32 Conclusions Simulation – Common approach to evaluate large scale (p2p) networks Requirements – Setup / configure once, collect all results at the end – Comparative evaluation of all parameter combinations Simulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot resultsSimulation setup Simulation setup Start simulation Start simulation Compile and plot results Compile and plot results Combine all individual simulation results Combine all individual simulation results Check how well your protocol performed Check how well your protocol performed Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Simulation setup Simulation setup Start simulation Start simulation Check how well your protocol performed Check how well your protocol performed Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Combination of all simulation results Combination of all simulation results Automated plottingAutomated plotting Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. Sim.Sim. NowNow
  33. 33. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 33 Join PeerfactSim.KOM (see www.peerfact.org) Type – Event-based simulator – Written in Java – Simulations up to 100K peers possible – Focus on simulation of p2p systems on various layers • User • Application • Services: monitoring, replication … • Overlays • Network models Invitation to join the community – Several universities use and extend the simulator actively – Used and heavily extended in the project
  34. 34. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 34 Thank You for Your Attention Jun.-Prof. Dr.-Ing. Kalman Graffi Technology of Social Networks Group Institute of Computer Science Heinrich-Heine-Universität Düsseldorf eMail: graffi@cs.uni-duesseldorf.de Web: www.p2pframework.com Web: www.peerfact.org ??

×