ZigZag: The Meandring Language

Slide 1: ZigZag: The Meandring Language Xavier Llorà1,2" 1Data-Intensive Technologies and Applications, ! National Center for Supercomputing Applications, ! University of Illinois at Urbana-Champaign! xllora@ncsa.uiuc.edu The SEASR project and its Meandre infrastructure! are sponsored by The Andrew W. Mellon Foundation

Slide 2: Outline •  Motivation •  Do we need yet-another language? •  ZigZag: A language for describing DG data-intensive flows •  Syntax •  Compile and run Powered by 2 Xavier Llorà

Slide 3: Motivation •  Meadre success relies on the ability to easily assemble data-intensive flows •  Meandre App (MA) is: –  An admin tool –  Not build (although it might help) ensemble data-intensive flows •  Meandre Workbench (MW) is: –  A visual tool for ensembling data-intensive flows –  A tool useful for non-technical practitioner –  Requires a high degree of interactivity –  Does not allow easy orchestration Powered by 3 Xavier Llorà

Slide 4: Motivation •  Speed up the flow development cycle –  Data-intensive flows are mainly assembled on the MW –  Takes a long time from you start building the flow till you run it •  The programmatic API is currently designed for Java components and requires a heavy setup •  Jobs do not have a basic unit structure •  Large-scale job orchestration: –  It is not feasible with the current tools –  Meandre does not provide any scripting Powered by 4 Xavier Llorà

Slide 5: Why Do We Need Yet-Another Language?" •  Provide a fast-lane data-intensive flow development cycle: –  Describe a data-intensive flow –  Compile a data-intensive flow into a self-contained task –  Run the task •  Large-scale task requires basic movable units •  Streamline the tools and the process for hard-core developers •  Prepare the stage for distributed data-intensive flow execution. Powered by 5 Xavier Llorà

Slide 6: ZigZag: A Language for Describing " DG Data-Intensive Flows •  A simple language for describe a data-intensive flow •  Modeled after Python simplicity •  ZigZag is declarative language for expressing the directed graphs (DG) that describe a data-intensive flow •  A compiler is provided to transform a ZigZag program (.zz) into a Meandre self-contained task •  Meandre self-contained task—or Meandre archive unit (.mau) •  Maus can then be executed by a Meandre engine •  Command-line tools allow to compile and execute Powered by 6 Xavier Llorà

Slide 7: ZigZag Streamlined Process ZigZag .zz Compiler .mau Meandre Mau STDOUT .mau Executor Powered by 7 Xavier Llorà

Slide 8: ZigZag Syntax •  Pythonic case-sensitive language •  Four basic instructions 1.  Component discovering and aliasing (CDA) 2.  Component instantiation (CI) 3.  Component modification (CM) 4.  Instance invocation (II) •  Instructions are group in a .zz files •  Each .zz file describes a single data-intensive flow •  Comments start at the beginning of a line using # Powered by 8 Xavier Llorà

Slide 9: ZigZag: Component Discovery and Aliasing •  CDK instructions: –  Retrieve a components from a repository –  Allow to create convenient alias name for components •  The basic retrieval presents two forms –  import <URL_to_the_repository> –  from <UR_ to_the_repository> import <component_URI> •  CDK alias instructions also present two basic formats –  alias <component_URI> as COMPONENT_ALIAS_NAME –  from <UR_ to_the_repository> import <component_URI> 
 as COMPONENT_ALIAS_NAME Powered by 9 Xavier Llorà

Slide 10: ZigZag: Component Discovery and Aliasing •  CDA examples: import <http://some_host/some_path/repository.ttl> alias <http://test.org/component/push_string> as PUSH alias <http://test.org/component/concatenate-strings> as CONCAT alias <http://test.org/component/print-object> as PRINT from <http://some_host/some_path/repository.ttl> 
 import <http://test.org/component/push_string> as PUSH from <http://some_host/some_path/repository.ttl> 
 import <http://test.org/component/conctenate_strings> as CONCAT from <http://some_host/some_path/repository.ttl> 
 import <http://test.org/component/print_object> as PRINT Powered by 10 Xavier Llorà

Slide 11: ZigZag: Component Instantiation •  CI instructions define component instances" –  instance_name = COMPONENT_ALIAS_NAME ()
 •  Supports multiple assignments" –  Ins1, ins2 = 
 COMPONENT_ALIAS_NAME (), 
 COMPONENT_ALIAS_NAME() Powered by 11 Xavier Llorà

Slide 12: ZigZag: Component Instantiation •  CI examples: push_hello = PUSH() push_world = PUSH() concat = CONCAT() print = PRINT() push_hello, push_world, concat, print = PUSH(), PUSH(), CONCAT(), PRINT() Powered by 12 Xavier Llorà

Slide 13: ZigZag: Component Modification •  CM allow to modify component properties" –  instance_name.property_name = value –  instance_name.property_name = another_instance.property_name
 •  Supports multiple assignments" –  instance_name.property_name1, instance_name.property_name2 , instance_name.property_name3 = 
 value1, another_instance_name.property_name Powered by 13 Xavier Llorà

Slide 14: ZigZag: Component Modification •  CM examples: push_hello.message = “Hello ” push_world.message = “world!” push_hello.message = push_world.message push_hello.message, push_world.message = “Hello ”, “world!” Powered by 14 Xavier Llorà

Slide 15: ZigZag: Instance Invocation •  II instructions: –  Describe instance execution invocation –  Define the DG graph of the data-intensive flow –  Use the object analogy to define data port interconnection –  The semantic parsing guaranties the constrains of data-intensive flows •  Allows single and multiple assignations •  The left-hand side of the assignment is a pseudo-object •  The right-hand side invokes and link ports of instances Powered by 15 Xavier Llorà

Slide 16: ZigZag: Instance Invocation •  Dictionary/Keyword-based invocations allows to implicitly define the DG structure of the data-intensive flow" –  @pseudo_object = COMPONENT_ALIAS_NAME (
 in_port_name_1: 
 pseudo_object.output_port_name_1,
 pseudo_object.output_port_name_k; 
 ...
 in_port_name_n: pseudo_object.output_port_name_n
 ) Powered by 16 Xavier Llorà

Slide 17: ZigZag: Instance Invocation •  II examples: @phres = push_hello() @pwres = push_world() @ccres = concat( string_1: phres.message; string_2: pwres.message ) print( object: ccres.message ) @phres, @pwres = push_hello(), push_world() @ccres = concat( string_1: phres.message; string_2: pwres.message ) print( object: ccres.message ) Powered by 17 Xavier Llorà

Slide 18: ZigZag: All Together import <http://some_host/some_path/repository.ttl> alias <http://test.org/component/push_string> as PUSH alias <http://test.org/component/concatenate-strings> as CONCAT alias <http://test.org/component/print-object> as PRINT push_hello, push_world, concat, print = PUSH(), PUSH(), CONCAT(), PRINT() push_hello.message, push_world.message = “Hello ”, “world!” @phres, @pwres = push_hello(), push_world() @ccres = concat( string_1: phres.message; string_2: pwres.message ) print( object: cres.object ) Powered by 18 Xavier Llorà

Slide 19: Compiling ZigZag Programs •  Command line utility for compiling –  java zzc.jar data-intensive-flow.zz •  Warnings and errors are reported to the STDERR •  If no error is found data-intensive-flow.mau is generated •  Mau are RDF files with the components (with embedded contexts), the describe flow, all zipped together. •  Command line utility for running mau unites –  java zzre.jar data-intensive-flow.mau Powered by 19 Xavier Llorà

Slide 20: ZigZag: The Meandring Language Xavier Llorà1,2" 1Data-Intensive Technologies and Applications, ! National Center for Supercomputing Applications, ! University of Illinois at Urbana-Champaign! xllora@ncsa.uiuc.edu The SEASR project and its Meandre infrastructure! are sponsored by The Andrew W. Mellon Foundation