An Integrated Framework for Parameter-based Optimization of Scientific Workflows


Published on

Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them. Performance optimization of these workflows can be viewed as a search for a set of optimal values in a multi-dimensional parameter space. While some performance parameters such as grouping of workflow components and their mapping to machines do not affect the accuracy of the output, others may dictate trading the output quality of individual components (and of the whole workflow) for performance. This paper describes an integrated framework which is capable of supporting performance optimizations along multiple dimensions of the parameter space. Using two real-world applications in the spatial data analysis domain, we present an experimental evaluation of the proposed framework.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

An Integrated Framework for Parameter-based Optimization of Scientific Workflows

  1. 1. An Integrated Framework for Parameter-based Optimization of Scientific Workflows Vijay S Kumar, G. Mehta, K. Vahi, V. Ratnakar, P. Sadayappan Jihie Kim, Ewa Deelman, Yolanda Gil Mary Hall Tahsin Kurc, Joel Saltz 15 June 2009 HPDC 2009 1
  2. 2. Motivations • Performance of data analysis applications is influenced by parameters – optimization search for optimal values in a multi-dimensional parameter space • A systematic approach to: – enable the tuning of performance parameters (i.e., select optimal parameter values given an application execution context) – support optimizations arising from performance-quality trade-offs 15 June 2009 HPDC 2009 2
  3. 3. Contributions of this paper • No auto-tuning yet (work in progress) • Core framework that can – support workflow execution (with application-level QoS) in distributed heterogeneous environments – enable manually tuning of parameters simultaneously – allow application developers and users to express applications semantically – leverage semantic descriptions to achieve performance optimizations • customized data-driven scheduling within Condor 15 June 2009 HPDC 2009 3
  4. 4. Application characteristics • Workflows: Directed Acyclic Graphs with well- defined data flow dependencies – mix of sequential, pleasingly parallelizable and complex parallel components – flexible execution in distributed environments • Multidimensional data analysis – data partitioned into chunks for analysis – dataset elements bear spatial relationships, constraints – data has an inherent notion of quality applications can trade accuracy of analysis output for performance • End-user queries supplemented with application- level 2009 requirements 2009 15 June QoS HPDC 4
  5. 5. Application scenario 1: No quality trade-offs • Minimize makespan while preserving highest output quality • Scale execution to handle terabyte-sized image data 15 June 2009 HPDC 2009 5
  6. 6. Application scenario 2: Trade quality for performance • Support queries with application-level QoS requirements – “Minimize time to classify image regions with 60% accuracy” – “Maximize classification accuracy of overall image within 30 minutes” 15 June 2009 HPDC 2009 6
  7. 7. Performance optimization decisions • What algorithm to use for • Where to map each this component? workflow component? • What data-chunking • Which components to strategy to adopt? merge into meta-components? • What is the quality of input data to this component • Which components need to ? perform at lower accuracy levels? • What is the processing order of the chunks? 15 June 2009 each View decision as a parameter that can be tuned HPDC 2009 7
  8. 8. Conventional Approach workflow design datasets Application workflow Application workflow Workflow Description Workflow Execution Semantic representation • clusters, the Grid or SOA • component discovery • task-based / services-based • workflow composition • batch mode / interactive • workflow validation 15 June 2009 HPDC 2009 8
  9. 9. Proposed approach: extensions workflow design Analysis requests, queries with QoS Analysis requests, queries with QoS:: “Maximize accuracy within tttime units” “Maximize accuracy within time units” Application workflow Application workflow Trade-off module • map high-level queries to metadata low-level execution strategies datasets • select appropriate values for performance parameters Description module Execution module Semantic representation Hierarchical execution: • search for components • map workflow components onto • workflow composition Grid sites • workflow validation • fine-grain dataflow execution of • performance parameters 15 June 2009 components on clusters HPDC 2009 9
  10. 10. An instance of our proposed framework WINGS Description module P (Workflow INstance Generation and Selection) A R A Pegasus WMS M DataCutter Execution module E T Condor, DAGMan E R S Trade-off module Interacts with the description and execution modules 15 June 2009 HPDC 2009 10
  11. 11. Description Module: WINGS (Workflow Instance Generation and Selection) • Layered workflow (1) refinement • Workflow Template: – abstract description Conceptual – dataset-independent workflow sketch – resource-independent • Compact workflow (2) Instance: Workflow template – contains mappings to actual datasets – resource-independent (3) • Expanded workflow to Execution instance module 15 June 2009 HPDC 2009 11 Workflow instance
  12. 12. Extensions to WINGS data ontology Extensions for Extensions for Application- “Core” data ontology “Core” data ontology multidimensional data Application- multidimensional data specific specific analysis analysis File ChunkFile Chunk hasCreationMetadata hasNXtiles, hasNYtiles, ProjectedChunk hasFormatMetadata hasChunksizeX, hasChunksizeY, NormalizedChunk hasDescriptionFile hasChunkIDX, hasChunkIDY, StitchedChunk hasContentTemplate hasChunkIDZ, hasOverlap, Collection StackFile Stack hasFileType hasStartZ, hasEndZ Slice hasN_items SliceFile ProjectedSlice hasFiles hasSliceIDZ, hasNXChunks, hasNYChunks NormalizedSlice CollOfCollections • Relations between entities, constraints on metadata • Automatic description, naming of intermediate data products 15 June 2009 HPDC 2009 12
  13. 13. Execution Module Pegasus WMS ( • Coarse-grain mapping of workflow tasks onto Grid sites • Submits sub-workflows to DAG schedulers at each site • Automatic data transfer between sites (via GridFTP) DataCutter ( • Fine-grain mapping of components onto clusters • Filter-stream model, asynchronous delivery • Each filter executes as a thread (could be C++/Java/Python) • Pipelined dataflow execution: Combined task- and data- parallelism • MPI-based version ( Condor ( can now execute DataCutter jobs within its “parallel universe” 15 June 2009 HPDC 2009 13
  14. 14. Quality-preserving parameters Data Chunking strategy [W, H ] Image1 Image1 Partition Partition Chunks C1 C2 C3 … Cn A A1 A2 A3 … An … … • algorithmic variant of a component • component placement • grouping components into meta-components • task-parallelism and data streaming within meta-component 15 June 2009 HPDC 2009 14
  15. 15. Quality-trading Parameters • Data approximation – e.g. spatial resolution of chunk – higher resolutions greater execution times, but does not imply higher accuracy of output • Processing order of chunks – the order in which data chunks are operated upon by a component collection – can process “favorable” chunks ahead of other chunks 15 June 2009 HPDC 2009 15
  16. 16. Processing order processing order A A1 A2 A3 … An • Tasks within a component collection treated as a batch – Condor: executes them in FIFO order • Implemented a priority-queue based heuristic for reordering task execution for a component collection – “favorable” chunks are processed ahead of other chunks – different QoS requirements change the insertion scheme • Can the execution of the bag-of-tasks be reordered dynamically? – condor_prio alone is not suitable 15 June 2009 HPDC 2009 16
  17. 17. Customized scheduling in Condor processing PQ order A A1 A2 A3 … An • Customized job scheduling within Condor to support performance-quality trade-offs for application-level quality-of-service (QoS) – implements the priority queue scheme (overrides the FIFO scheme) – executes within Condor’s “scheduler” universe • Associates tasks with the spatial coordinates of the respective chunks that are being processed – uses the automated naming of data products (metadata propagation) brought about by semantic descriptions 15 June 2009 HPDC 2009 17
  18. 18. Experimental setup: Test bed • RII-MEMORY – 64 node Linux cluster – Dual-processor 2.4 GHz Opteron nodes – 8GB RAM, 437 GB local RAID0 volume – Gigabit Ethernet • RII-COMPUTE – 32 node Linux cluster – 3.6 GHz Intel Xeon processors – 2GB RAM, 10 GB local disk – Gigabit Ethernet and Infiniband • Wide-area 10 Gbps connection 15 June 2009 HPDC 2009 18
  19. 19. Performance Evaluation • Focus on performance-quality trade-offs • Neuroblastoma Classification workflow: – “Maximize overall confidence of classification within t time units” – “Maximize number of data chunks processed within t time units” • How to tune quality-trading parameters to achieve high performance? – Data resolution – Processing order of chunks 15 June 2009 HPDC 2009 19
  20. 20. Parameters: resolution, processing order Custom scheduling in Condor helps trade quality for performance better than default scheduling for QoS requirement type 1 • 32 nodes, 21 GB image, confidence threshold = 0.25 • “Maximize overall classification confidence within time t units” 15 June 2009 HPDC 2009 20
  21. 21. Parameters: resolution, processing order Custom scheduling in Condor helps trade quality for performance better than default scheduling for QoS requirement type 1 Custom scheduling in Condor can improve throughput for QoS requirement type 2 • 32 nodes, 21 GB image, confidence threshold = 0.25 • “Maximize data chunks processed within t time units” 15 June 2009 HPDC 2009 21
  22. 22. Conclusions • Performance optimization for workflows: search for values in a multidimensional parameter space • Instance of our proposed framework allows users to manually express values for many performance parameters (simultaneously): – quality-preserving & quality-trading • Semantic representations of domain data and performance parameters can be leveraged – Data chunking strategy and data approximation can help restructure workflow for a given resource configuration – Customized job scheduling within Condor can scalably support application-level QoS 15 June 2009 HPDC 2009 22
  23. 23. Current and Future work • Use semantic representations to map high- level queries onto low-level execution strategies • Techniques to efficiently navigate the parameter space – Assume high data cardinality Uniformity of application context over time – Use information from sample runs to build statistical models 15 June 2009 HPDC 2009 23
  24. 24. An Integrated Framework for Parameter-based Optimization of Scientific Workflows Thanks! Vijay S Kumar (, P. Sadayappan Tahsin Kurc, Joel Saltz ( Ewa Deelman, Yolanda Gil ( 15 June 2009 HPDC 2009 24