GrenchMark at CCGrid, May 2006.

486 views

Published on

Under the grid computing paradigm, large sets of heterogeneous resources can be aggregated and shared. Grid development and acceptance hinge on proving that grids reliably support real applications, and on creating adequate benchmarks to quantify this support. However, applications of grids (and clouds) are just beginning to emerge, and traditional benchmarks have yet to prove representative in grid environments. To address this chicken-and-egg problem, we propose a middle-way approach: create and run synthetic grid workloads comprised of applications representative for today's grids (and clouds). For this purpose, we have designed and implemented GrenchMark, a framework for synthetic workload generation and submission. The framework greatly facilitates synthetic workload modeling, comes with over 35 synthetic and real applications, and is extensible and flexible. We show how the framework can be used for grid system analysis, functionality testing in grid environments, and for comparing different grid settings, and present the results obtained with GrenchMark in our multi-cluster grid, the DAS.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
486
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Solution: Build synthetic grid workloads using real and synthetic applications, and run them in real or simulated environments.
  • GrenchMark at CCGrid, May 2006.

    1. 1. GrenchMark: A Framework forAnalyzing, Testing, and Comparing GridsA. Iosup, D.H.J. EpemaPDS Group, ST/EWI, TU Delft April 4, 2012 CCGrid 2006 1
    2. 2. Outline• Introduction and Motivation• The GrenchMark Framework• Past and Current Experience with GrenchMark• A GrenchMark Success Story• Future Work• Conclusions April 4, 2012 2
    3. 3. The Generic Problem ofAnalyzing, Testing, and Comparing Grids• Use cases for automatically analyzing, testing, and comparing Grids • Comparisons for system design and procurement • Functionality testing and system tuning • Performance testing/analysis of grid applications • …• For grids, this problem is hard ! • Testing in real environments is difficult • Grids change rapidly • Validity of tests • … April 4, 2012 3
    4. 4. A Generic Solution toAnalyzing, Testing, and Comparing Grids• “ Generate and run synthetic grid workloads, based on real and synthetic applications “• Current alternatives (not covering all problems) • Benchmarking with real/synthetic applications (representative?) • User-defined test management (statistically sound?)• Advantages of using synthetic grid workloads • Statistically sound composition of benchmarks • Statistically sound test management • Generic: cover the use cases’ broad spectrum (to be shown) April 4, 2012 4
    5. 5. Outline• Introduction and Motivation• The GrenchMark Framework• Past and Current Experience with GrenchMark• A GrenchMark Success Story• Future Work• Conclusions April 4, 2012 5
    6. 6. GrenchMark: a Framework forAnalyzing, Testing, and Comparing grids• What’s in a name? grid benchmark → working towards a generic tool for the whole community: help standardizing the testing procedures, but benchmarks are too early; we use synthetic grid workloads instead• What’s it about? A systematic approach to analyzing, testing, and comparing grid settings, based on synthetic workloads • A set of metrics for analyzing grid settings • A set of representative grid applications • Both real and synthetic • Easy-to-use tools to create synthetic grid workloads • Flexible, extensible framework April 4, 2012 6
    7. 7. GrenchMark: Iterative Research Roadmap April 4, 2012 7
    8. 8. GrenchMark: Iterative Research Roadmap Simple functional system A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID IW, Nov 2005. April 4, 2012 8
    9. 9. GrenchMark: Iterative Research Roadmap Open-GrenchMarkCommunity Effort Complex extensible system This work April 4, 2012 9
    10. 10. GrenchMark Overview:Generate and Run Synthetic Workloads April 4, 2012 10
    11. 11. GrenchMark Overview: Easy toGenerate and Run Synthetic Workloads April 4, 2012 11
    12. 12. … but More Complicated Than You Think• Workload structure • Measurement methods • User-defined and • Long workloads statistical models • Saturated / non-saturated system • Dynamic jobs arrival • Start-up, production, and • Burstiness and self-similarity cool-down scenarios • Feedback, background load • Scaling workload to system • Machine usage assumptions • Applications • Users, VOs • Synthetic• Metrics • Real • A(W) Run/Wait/Resp. Time • Workload definition language • Efficiency, MakeSpan • Base language layer • Failure rate [!] • Extended language layer• (Grid) notions • Other • Co-allocation, interactive jobs, • Can use the same workload for malleable, moldable, … both simulations and real environments April 4, 2012 12
    13. 13. GrenchMark Overview:Unitary and Composite Applications• Unitary applications • sequential, MPI, Java RMI, Ibis, …• Composite applications • Bag of tasks • Chain of jobs • Direct Acyclic Graph-based (Standard Task Graph Archive) April 4, 2012 13
    14. 14. GrenchMark Overview: Workload Description Files • Format:Number of jobs and application and Composition Co-allocation types Language extensions Inter-arrival and number of componentsCombining four workloads into one start time April 4, 2012 14
    15. 15. Using GrenchMark:Grid System Analysis• Performance testing: test the performance of an application (for sequential, MPI, Ibis applications) • Report runtimes, waiting times, grid middleware overhead • Automatic results analysis• What-if analysis: evaluate potential situations • System change • Grid inter-operability • Special situations: spikes in demand April 4, 2012 15
    16. 16. Using GrenchMark:Functionality Testing in Grid Environments• System functionality testing: show the ability of the system to run various types of applications • Report failure rates [ arguably, functionality in grids is even more important than performance !  10% job failure rate in a controlled system like the DAS ]• Periodic system testing: evaluate the current state of the grid • Replay workloads April 4, 2012 16
    17. 17. Using GrenchMark:Comparing Grid Settings• Single-site vs. co-allocated jobs: compare the success rate of single-site and co-allocated jobs, in a system without reservation capabilities • Single-site jobs 20% better vs. small co-allocated jobs (<32 CPUs), 30% better vs. large co-allocated jobs [setting and workload-dependent !]• Unitary vs. composite jobs: compare the success rate of unitary and composite jobs, with and without failure handling mechanisms • Both 100% with simple retry mechanism [setting and workload-dependent !] April 4, 2012 17
    18. 18. A GrenchMark Success Story:Releasing the Koala Grid Scheduler on the DAS• Koala [ http://www.st.ewi.tudelft.nl/koala/ ] • Grid Scheduler with co-allocation capabilities• DAS: The Dutch Grid, ~200 researchers• Initially • Koala, a tested (!) scheduler, pre-release version• Test specifics • 3 different job submission modules • Workloads with different jobs requirements, inter-arrival rates, co-allocated v. single site jobs… • Evaluate: job success rate, Koala overhead and bottlenecks• Results • 5,000+ jobs successfully run (all workloads); functionality tests • 2 major bugs first day, 10+ bugs overall (all fixed) • KOALA is now officially released on the DAS (full credit to KOALA developers, 10x for testing with GrenchMark) April 4, 2012 18
    19. 19. GrenchMark’s Current Status:pre-”Open-GrenchMark”• Already done in Python [http://www.python.org] • Workload Generator • Generic Workload Submitter (Koala, Globus GRAM, option to extend for JSDL, Condor, PBS, LSF, SGE, …) • Applications • Unitary, 3 types: sequential, MPI, Ibis (Java) • +35 real and synthetic applications • Composite applications: DAG-based• Extending modeling capabilitiesA.Iosup, D.H.J.Epema (TU Delft), C. Franke, A. Papaspyrou,L. Schley, B. Song, R. Yahyapour (U Dortmund), On modelingsynthetic workloads for Grid performance evaluation, 12thWorkshop on Job Scheduling Strategies for ParallelProcessing (JSSPP), held April 4, 2012 in conjunction with SIGMETRICS2006, Saint Malo, France, June 2006 (accepted). 19
    20. 20. Towards Open-GrenchMark: Grid traces, Simulators, Benchmarks • Distributed testing • Integrate with DiPerF (C. Dumitrescu, I. Raicu, M. Ripeanu) • Grid traces analysis • Automatic tools for grid traces analysisA. Iosup, C. Dumitrescu, D.H.J. Epema (TU Delft), H. Li,L. Wolters (U Leiden), How are Real Grids Used? The Analysisof Four Grid Traces and Its Implications, (submitted). • Use in conjunction with simulators • Ability to generate workloads which can be used in simulated environments (e.g., GangSim, GridSim, …) • Grid benchmarks • Analyze the requirements for domain-specific grid benchmarks April 4, 2012 20
    21. 21. Conclusion• GrenchMark generates diverse grid workloads easy-to-use, flexible, portable, extensible, …• Experience used GrenchMark to test KOALA’s functionality and performance. used GrenchMark to analyze, test, and compare grid settings. 15,000+ jobs generated and run … and counting.• (more) advertisement Have specific grid setting you would like to test? Test with GrenchMark! April 4, 2012 21
    22. 22. Thank you! Jason Maassen to Hashim Mohamed (Koala), Many thanks and Rob van Nieuwpoort (Ibis). Questions? Remarks? Observations? All welcome! GrenchMark http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo] Alexandru IOSUP TU Delft A.Iosup@tudelft.nl http://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”] April 4, 2012 22
    23. 23. April 4, 2012 23
    24. 24. Outline• Introduction [here]• The GrenchMark framework• Experience with GrenchMark• Extending GrenchMark• Conclusions April 4, 2012 24
    25. 25. Representative Grid applications (1/4)• Unitary applications • Just one scheduling unit (otherwise recursive definition) • Examples: Sequential, MPI, Java RMI, Ibis, …• Composite applications • Composed of several unitary or composite applications • Examples: Parameter sweeps, chains of tasks, DAGs, workflows, … April 4, 2012 25
    26. 26. Representative Grid applications (2/4) Unitary: synthetic sequential <<single site/single node/single stdin processor>> stdout[I] read I1 data elements from stdin N+1:I1,N*I2 N+2:O1,N*O2,O3[O] write O1 data elements to stdout System reqs Processor (float*N*S*C)for each N steps (i) # superstep Memory (float*(M+1)*S) [I] read I2 data elements from stdin for each M memory locations (j) # computation step [M] get item j of size S [P] compute C computation units per memory item (fmul), and store results into temp memory location [M] put values from temp to location j [O] write O2 data elements to stdout[O] write O3 data elements to stdout Also version with computation and I/O April 4, 2012 26
    27. 27. Representative Grid applications (2/4) Unitary: synthetic parallel <<MPI system/single site>> <<single node/single processor>> <<single node/single processor>>[I] read I1 data from in$ProcID <<single node/single processor>> app-input.i app-output.i[O] write O1 data to out$ProcIDfor each N steps (i) # superstep N+1: I1,N*I2 Process i N+2: O1,N*O2,O3 [B] synchronize with barrier [I] read I2 data from in$ProcID for each M memory locations (j) Machine i [M] get item j of size S Processor i (float*N*S*C) k [P] compute C fmul/mem Memory (float*N*2S) [M] put values from temp to j [C] communicate to all other processes X values and receive X values from every other processor [O] write O2 data elements to out$ProcID[O] write O3 data elements to out$ProcID April 4, 2012 27
    28. 28. Representative Grid applications (3/4)Unitary: Ibis jobs• No modeling, just real applications [ http://www.cs.vu.nl/ibis ] • physical simulation • parallel rendering • computational mathematics • state space search • bioinformatics • data compression • grid methods • optimization April 4, 2012 28
    29. 29. Representative Grid applications (3/4)Composite: DAG-based• DAG-based applications • Real DAG • Chain of tools• Try to model real or predicted (use) cases perf1.dat <<final>> param2.in some-list.in param1.in out_1-2.dat l1p.dat <<final>> out2.res App1 out_1-1.dat Linker1 huge-data.out App2 <<single>> <<link>> <<single>> perf2.dat Linker Input Identity (one task’s output = other’s input, unmodified) User task Output Final result April 4, 2012 29

    ×