SlideShare a Scribd company logo
1 of 15
Download to read offline
YesWorkflow:	
  A	
  User-­‐Oriented,	
  Language-­‐
Independent	
  Tool	
  for	
  Recovering	
  Workflow	
  
Informa9on	
  from	
  Scripts	
  
Timothy	
  McPhillips1,	
  Tianhong	
  Song2,	
  Tyler	
  Kolisnik3,	
  	
  
Steve	
  Aulenbach4,	
  Khalid	
  Belhajjame5,	
  Kyle	
  Bocinsky6,	
  Yang	
  Cao1,	
  
Fernando	
  Chiriga97,	
  Saumen	
  Dey2,	
  Juliana	
  Freire7,	
  Deborah	
  Huntzinger11,	
  
Christopher	
  Jones8,	
  David	
  Koop9,	
  Paolo	
  Missier10,	
  Mark	
  Schildhauer8,	
  
Christopher	
  Schwalm11,	
  Yaxing	
  Wei12,	
  James	
  Cheney13,	
  Mark	
  Bieda3,	
  
Bertram	
  Ludäscher1,	
  14	
  
10th	
  Intl.	
  Digital	
  CuraRon	
  Conference	
  (IDCC)	
  
30	
  Euston	
  Square,	
  London,	
  UK	
  
Overview	
  
•  Enter	
  	
  ScienRfic	
  Workflows	
  …	
  	
  
–  Kepler,	
  Taverna,	
  VisTrails,	
  …	
  	
  
•  ... Exeunt	

•  Enter Scripts	
  …	
  	
  
–  Python,	
  R,	
  Matlab,	
  …	
  
•  Enter YesWorkflow	
  
–  Complements	
  noWorkflow	
  
•  …	
  run$me	
  (=	
  retrospec)ve)	
  provenance	
  from	
  scripts	
  
–  Combines	
  the	
  best	
  of	
  both	
  worlds:	
  
•  Scripts	
  +	
  Comments	
  
	
  =>	
  Workflow	
  Views	
  (prospec$ve	
  provenance)	
  from	
  scripts	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   2	
  
Scientific Workflows: ASAP!
•  Automation
–  wfs to automate computational aspects of science
•  Scaling (exploit and optimize machine cycles)
–  wfs should make use of parallel compute resources
–  wfs should be able handle large data
•  Abstraction, Evolution, Reuse (human cycles)
–  wfs should be easy to (re-)use, evolve, share
•  Provenance
–  wfs should capture processing history, data lineage
è traceable data- and wf-evolution
è  Reproducible Science
Science	
  Example:	
  Paleoclimate	
  ReconstrucRon	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   4	
  
•  Kohler	
  &	
  Bocinsky:	
  study	
  rain-­‐fed	
  maize	
  of	
  Ancestral	
  Pueblo,	
  Anasazi	
  	
  
–  Four	
  Corners;	
  AD	
  600–1500	
  
–  Climate	
  change	
  influenced	
  Mesa	
  Verde	
  MigraRons;	
  late	
  13th	
  century	
  AD.	
  
–  Uses	
  network	
  of	
  tree-­‐ring	
  chronologies	
  to	
  reconstruct	
  a	
  spaRo-­‐temporal	
  
climate	
  field	
  at	
  a	
  fairly	
  high	
  resolu9on	
  (~800	
  m)	
  from	
  AD	
  1–2000	
  	
  
–  Algorithm	
  es9mates	
  joint	
  informa9on	
  in	
  tree-­‐rings	
  and	
  a	
  climate	
  signal	
  to	
  
iden9fy	
  “best”	
  	
  tree-­‐ring	
  chronologies	
  for	
  reconstruc9ng	
  climate	
  at	
  a	
  given	
  
9me	
  and	
  place.	
  
K.	
  Bocinsky,	
  T.	
  Kohler,	
  A	
  2000-­‐year	
  reconstruc9on	
  of	
  the	
  rain-­‐fed	
  
maize	
  agricultural	
  niche	
  in	
  the	
  US	
  Southwest.	
  Nature	
  
Communica$ons.	
  doi:10.1038/ncomms6618	
  	
  
… implemented as an R Script …
ERRT	
  @	
  GSLIS	
   5	
  
K.	
  Bocinsky,	
  T.	
  Kohler,	
  A	
  2000-­‐year	
  reconstruc9on	
  of	
  the	
  rain-­‐fed	
  maize	
  agricultural	
  niche	
  in	
  
the	
  US	
  Southwest.	
  Nature	
  Communica$ons.	
  doi:10.1038/ncomms6618	
  	
  
…	
  Paleoclimate	
  	
  
ReconstrucRon	
  …	
  	
  
Map	
  showing	
  the	
  "selected"	
  trees	
  for	
  reconstruc)ng	
  
precipita)on	
  at	
  four	
  sites	
  in	
  our	
  CAR	
  regression	
  
approach	
  (Correla)on-­‐Adjusted	
  corRela)on).	
  
Reconstruc)ons	
  
for	
  AD	
  1247	
  	
  
YesWorkflow	
  =	
  Scripts	
  +	
  Comments	
  
•  Scripts	
  can	
  be	
  hard	
  to	
  digest,	
  communicate	
  
•  Idea:	
  	
  
– Add	
  structured	
  comments	
  (cf.	
  JavaDoc)	
  
=>	
  	
  reveal	
  workflow	
  structure	
  and	
  dataflow	
  
	
  =>	
  	
  obtain	
  some	
  scien9fic	
  workflow	
  benefits	
  
•  	
  …	
  ASAP	
  …	
  	
  
6	
  




 

 
 
Get	
  3	
  views	
  for	
  the	
  price	
  of	
  1!	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   7	
  

 

 





 
 
   


Process	
  view	
  
Data	
  view	
  
Combined	
  view	
  
User	
  Comments:	
  YW	
  AnnotaRons	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   8	
  
@begin GO_Analysis
@in hgCutoff
@in …
@out BP_Summl_file
@out …
@end GO_Analysis
...	
  
Paleoclimate	
  ReconstrucRon	
  …	
  	
  	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   9	
  
GetModernClimate
PRISM_annual_growing_season_precipitation
SubsetAllData
dendro_series_for_calibration
dendro_series_for_reconstruction CAR_Analysis_unique
cellwise_unique_selected_linear_models
CAR_Analysis_union
cellwise_union_selected_linear_models
CAR_Reconstruction_union
raster_brick_spatial_reconstruction raster_brick_spatial_reconstruction_errors
CAR_Reconstruction_union_output
ZuniCibola_PRISM_grow_prcp_ols_loocv_union_recons.tif ZuniCibola_PRISM_grow_prcp_ols_loocv_union_errors.tif
master_data_directory prism_directory
tree_ring_datacalibration_years retrodiction_years
•  …	
  explained	
  using	
  YesWorkflow	
  
Kyle	
  B.,	
  (computa9onal)	
  archeologist:	
  	
  
"It	
  took	
  me	
  about	
  20	
  minutes	
  to	
  comment.	
  Less	
  
than	
  an	
  hour	
  to	
  learn	
  and	
  YW-­‐annotate,	
  all-­‐told."	
  
YesWorkflow	
  Architecture:	
  KISS!	
  
•  YW-­‐Extract	
  
– …	
  structured	
  comments	
  	
  
•  YW-­‐Model	
  
– Program	
  Block,	
  Workflow	
  
– Port	
  (data,	
  parameters)	
  
– Channels	
  (dataflow)	
  
	
  
•  YW-­‐Graph	
  
– ...	
  using	
  GraphViz/DOT	
  files	
  
•  YW-­‐Query,	
  YW-­‐Validate,	
  YW-­‐CLI	
  	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   10	
  

























Figure 4: Process workflow view of an A↵ymetrix analysis script (in R).
4 YesWorkflow Examples
In the following we show YesWorkflow views extracted from real-world scientific use cases.
The scripts were annoted with YW tags by scientists and script authors, using a very
modest training and mark-up e↵ort.1
Due to lack of space, the actual MATLAB and R
scripts with their YW markup are not included here. However, they are all available
from the yw-idcc-15 repository on the YW GitHub site [Yes15].
Gene	
  Expression	
  Microarray	
  Data	
  Analysis	
  
•  [Normalize]	
  	
  
–  Normaliza9on	
  of	
  data	
  across	
  microarray	
  datasets	
  	
  
•  [SelectDEGs]	
  	
  
–  Selec9on	
  of	
  differen9ally	
  expressed	
  genes	
  between	
  condi9ons	
  	
  
•  [GO	
  Analysis]	
  	
  
–  determina9on	
  of	
  gene	
  ontology	
  sta9s9cs	
  for	
  the	
  resul9ng	
  datasets	
  	
  
•  [MakeHeatmap]	
  	
  
–  crea9on	
  of	
  a	
  heatmap	
  of	
  the	
  differen9ally	
  expressed	
  genes.	
  	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   11	
  
Tyler	
  Kolisnik,	
  Mark	
  Bieda	
  
MulR-­‐Scale	
  Synthesis	
  and	
  Terrestrial	
  Model	
  
Intercomparison	
  Project	
  (MsTMIP)	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   12	
  
fetch_drought_variable
drought_variable_1
fetch_effect_variable
effect_variable_1
convert_effect_variable_units
effect_variable_2
create_land_water_mask
land_water_mask
init_data_variables
predrought_effect_variable_1 drought_value_variable_1 recovery_time_variable_1 drought_number_variable_1
define_droughts
sigma_dv_event month_dv_length
detrend_deseasonalize_effect_variable
effect_variable_3
calculate_data_variables
recovery_time_variable_2 drought_value_variable_2 predrought_effect_variable_2 drought_number_variable_2
export_recovery_time_figure
output_recovery_time_figure
export_drought_value_variable_figure
output_drought_value_variable_figure
export_predrought_effect_variable_figure
output_predrought_effect_variable_figure
export_drought_number_variable_figure
output_drought_number_figure
input_drough_variable
input_effect_variable
Christopher	
  Schwalm,	
  
Yaxing	
  Wei	
  
Summary:	
  ScienRfic	
  Workflows	
  
ScienRfic	
  Workflows	
  
•  [+]	
  Automa9on	
  
•  [+]	
  Scalability	
  
•  [+]	
  Abstrac9on	
  
•  [+]	
  Provenance	
  
•  …	
  
•  [+/0]	
  Easy	
  to	
  use	
  	
  
–  [0]	
  learning	
  a	
  new	
  paradigm	
  
•  [-­‐]	
  Teaching	
  resources	
  	
  
•  [-­‐]	
  Special	
  exper9se	
  needed	
  for	
  deep	
  changes	
  
	
  e.g.	
  new	
  Java	
  actors,	
  shims,	
  …	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   13	
  
Summary:	
  Scripts	
  +	
  YesWorkflow	
  
Scripts:	
  [+]	
  Automa9on,	
  [0]	
  Scalability,	
  	
  [-­‐]	
  Abstrac9on,	
  [0/-­‐]	
  Provenance	
  
Now:	
  Scripts	
  +	
  YesWorkflow	
  AnnotaRons	
  
•  [+]	
  AbstracRon	
  
–  explain	
  your	
  methods	
  to	
  mere	
  mortals	
  
=>	
  encourage	
  (re-­‐)use	
  
•  [+]	
  Provenance:	
  
–  noWorkflow	
  (retrospec$ve	
  provenance)	
  
–  YesWorkflow	
  (prospec$ve	
  provenance)	
  
•  [+]	
  Language	
  independent	
  (R,	
  Matlab,	
  Python,	
  …)	
  	
  
•  [+]	
  Empower	
  tool	
  makers	
  (script	
  programmers):	
  give	
  them	
  …	
  
–  …	
  some	
  immediate	
  benefits	
  (workflow	
  views)	
  
–  …	
  some	
  medium	
  term	
  improvements	
  (provenance	
  integraRon)	
  
–  …	
  some	
  long	
  term	
  benefits:	
  think	
  about	
  your	
  methods	
  differently	
  	
  
=>	
  dataflow	
  programming	
  =>	
  [+]	
  Scalability	
  	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   14	
  
YesWorkflow:	
  Acknowledgements	
  
•  With	
  support	
  from	
  NSF	
  awards	
  DBI-­‐1356751	
  (Kurator),	
  
ACI-­‐0830944	
  (DataONE),	
  SMA-­‐1439603	
  (SKOPE).	
  	
  
	
  
•  Thanks	
  to	
  the	
  IDCC	
  organizers	
  for	
  choice	
  of	
  venues!	
  
B.	
  Ludäscher	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  YesWorkflow:	
  Workflow	
  Views	
  from	
  Scripts.	
  IDCC'15,	
  London	
  	
   15	
  

More Related Content

Viewers also liked

Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...
Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...
Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...Bertram Ludäscher
 
YesWorkflow: Retrospective Provenance Without a Runtime Provenance Recorder
YesWorkflow: Retrospective Provenance Without a Runtime Provenance RecorderYesWorkflow: Retrospective Provenance Without a Runtime Provenance Recorder
YesWorkflow: Retrospective Provenance Without a Runtime Provenance RecorderBertram Ludäscher
 
Yin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflow
Yin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflowYin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflow
Yin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflowBertram Ludäscher
 
Provenance in Databases and Scientific Workflows: Part II (Databases)
Provenance in Databases and Scientific Workflows: Part II (Databases)Provenance in Databases and Scientific Workflows: Part II (Databases)
Provenance in Databases and Scientific Workflows: Part II (Databases)Bertram Ludäscher
 
Theory & Practice of Data Cleaning: Introduction to OpenRefine
Theory & Practice of Data Cleaning: Introduction to OpenRefineTheory & Practice of Data Cleaning: Introduction to OpenRefine
Theory & Practice of Data Cleaning: Introduction to OpenRefineBertram Ludäscher
 
From Data to Knowledge with Workflows & Provenance
From Data to Knowledge with Workflows & ProvenanceFrom Data to Knowledge with Workflows & Provenance
From Data to Knowledge with Workflows & ProvenanceBertram Ludäscher
 
A Sightseeing Tour of Provenance in Databases & Workflows
A Sightseeing Tour of Provenance in Databases & WorkflowsA Sightseeing Tour of Provenance in Databases & Workflows
A Sightseeing Tour of Provenance in Databases & WorkflowsBertram Ludäscher
 
DAIS Seminar: The Many Faces of Provenance in Databases and Workflows
DAIS Seminar: The Many Faces of Provenance in Databases and WorkflowsDAIS Seminar: The Many Faces of Provenance in Databases and Workflows
DAIS Seminar: The Many Faces of Provenance in Databases and WorkflowsBertram Ludäscher
 
Querying Provenance Information: Basic Notions and an Example from Paleoclima...
Querying Provenance Information: Basic Notions and an Example from Paleoclima...Querying Provenance Information: Basic Notions and an Example from Paleoclima...
Querying Provenance Information: Basic Notions and an Example from Paleoclima...Bertram Ludäscher
 

Viewers also liked (10)

Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...
Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...
Introducing the Whole Tale Project: Merging Science and Cyberinfrastructure P...
 
Works 2015-provenance-mileage
Works 2015-provenance-mileageWorks 2015-provenance-mileage
Works 2015-provenance-mileage
 
YesWorkflow: Retrospective Provenance Without a Runtime Provenance Recorder
YesWorkflow: Retrospective Provenance Without a Runtime Provenance RecorderYesWorkflow: Retrospective Provenance Without a Runtime Provenance Recorder
YesWorkflow: Retrospective Provenance Without a Runtime Provenance Recorder
 
Yin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflow
Yin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflowYin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflow
Yin & Yang: Demonstrating complementary provenance from noWorkflow & YesWorkflow
 
Provenance in Databases and Scientific Workflows: Part II (Databases)
Provenance in Databases and Scientific Workflows: Part II (Databases)Provenance in Databases and Scientific Workflows: Part II (Databases)
Provenance in Databases and Scientific Workflows: Part II (Databases)
 
Theory & Practice of Data Cleaning: Introduction to OpenRefine
Theory & Practice of Data Cleaning: Introduction to OpenRefineTheory & Practice of Data Cleaning: Introduction to OpenRefine
Theory & Practice of Data Cleaning: Introduction to OpenRefine
 
From Data to Knowledge with Workflows & Provenance
From Data to Knowledge with Workflows & ProvenanceFrom Data to Knowledge with Workflows & Provenance
From Data to Knowledge with Workflows & Provenance
 
A Sightseeing Tour of Provenance in Databases & Workflows
A Sightseeing Tour of Provenance in Databases & WorkflowsA Sightseeing Tour of Provenance in Databases & Workflows
A Sightseeing Tour of Provenance in Databases & Workflows
 
DAIS Seminar: The Many Faces of Provenance in Databases and Workflows
DAIS Seminar: The Many Faces of Provenance in Databases and WorkflowsDAIS Seminar: The Many Faces of Provenance in Databases and Workflows
DAIS Seminar: The Many Faces of Provenance in Databases and Workflows
 
Querying Provenance Information: Basic Notions and an Example from Paleoclima...
Querying Provenance Information: Basic Notions and an Example from Paleoclima...Querying Provenance Information: Basic Notions and an Example from Paleoclima...
Querying Provenance Information: Basic Notions and an Example from Paleoclima...
 

Similar to YesWorkflow: How to render a script as a workflow in half an hour!

Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1
Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1
Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1Bertram Ludäscher
 
Is there a free lunch for cloud-based evolutionary algorithms?
Is there a free lunch for cloud-based evolutionary algorithms?Is there a free lunch for cloud-based evolutionary algorithms?
Is there a free lunch for cloud-based evolutionary algorithms?Mario Garcia Valdez
 
From Workflows to Transparent Research Objects and Reproducible Science Tales
From Workflows to Transparent Research Objects and Reproducible Science TalesFrom Workflows to Transparent Research Objects and Reproducible Science Tales
From Workflows to Transparent Research Objects and Reproducible Science TalesBertram Ludäscher
 
Real-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL Databases Real-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL Databases MongoDB
 
Real-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL DatabasesReal-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL DatabasesEugene Dvorkin
 
Polygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugPolygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugChris Richardson
 
Scientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & SociologyScientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & SociologyNeil Chue Hong
 
Kotlin+MicroProfile: Ensinando 20 anos para uma linguagem nova
Kotlin+MicroProfile: Ensinando 20 anos para uma linguagem novaKotlin+MicroProfile: Ensinando 20 anos para uma linguagem nova
Kotlin+MicroProfile: Ensinando 20 anos para uma linguagem novaVíctor Leonel Orozco López
 
Cape2013 scilab-workshop-19Oct13
Cape2013 scilab-workshop-19Oct13Cape2013 scilab-workshop-19Oct13
Cape2013 scilab-workshop-19Oct13Naren P.R.
 
On the need for a W3C community group on RDF Stream Processing
On the need for a W3C community group on RDF Stream ProcessingOn the need for a W3C community group on RDF Stream Processing
On the need for a W3C community group on RDF Stream ProcessingPlanetData Network of Excellence
 
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...Oscar Corcho
 
Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...Anubhav Jain
 
OSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchainOSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchainCARLOS III UNIVERSITY OF MADRID
 
Static Energy Prediction in Software: A Worst-Case Scenario Approach
Static Energy Prediction in Software: A Worst-Case Scenario ApproachStatic Energy Prediction in Software: A Worst-Case Scenario Approach
Static Energy Prediction in Software: A Worst-Case Scenario ApproachGreenLabAtDI
 
ScalaDays 2013 Keynote Speech by Martin Odersky
ScalaDays 2013 Keynote Speech by Martin OderskyScalaDays 2013 Keynote Speech by Martin Odersky
ScalaDays 2013 Keynote Speech by Martin OderskyTypesafe
 
Scilab: Computing Tool For Engineers
Scilab: Computing Tool For EngineersScilab: Computing Tool For Engineers
Scilab: Computing Tool For EngineersNaren P.R.
 
Automatic and Interpretable Machine Learning with H2O and LIME
Automatic and Interpretable Machine Learning with H2O and LIMEAutomatic and Interpretable Machine Learning with H2O and LIME
Automatic and Interpretable Machine Learning with H2O and LIMEJo-fai Chow
 
2015 10-7-11am-reproducible research
2015 10-7-11am-reproducible research2015 10-7-11am-reproducible research
2015 10-7-11am-reproducible researchYannick Wurm
 
Analytics of analytics pipelines: from optimising re-execution to general Dat...
Analytics of analytics pipelines:from optimising re-execution to general Dat...Analytics of analytics pipelines:from optimising re-execution to general Dat...
Analytics of analytics pipelines: from optimising re-execution to general Dat...Paolo Missier
 
Products go Green: Worst-Case Energy Consumption in Software Product Lines
Products go Green: Worst-Case Energy Consumption in Software Product LinesProducts go Green: Worst-Case Energy Consumption in Software Product Lines
Products go Green: Worst-Case Energy Consumption in Software Product LinesGreenLabAtDI
 

Similar to YesWorkflow: How to render a script as a workflow in half an hour! (20)

Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1
Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1
Data and Workflow Provenance: (From Theory to Hands-­‐on Practice)^­‐1
 
Is there a free lunch for cloud-based evolutionary algorithms?
Is there a free lunch for cloud-based evolutionary algorithms?Is there a free lunch for cloud-based evolutionary algorithms?
Is there a free lunch for cloud-based evolutionary algorithms?
 
From Workflows to Transparent Research Objects and Reproducible Science Tales
From Workflows to Transparent Research Objects and Reproducible Science TalesFrom Workflows to Transparent Research Objects and Reproducible Science Tales
From Workflows to Transparent Research Objects and Reproducible Science Tales
 
Real-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL Databases Real-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL Databases
 
Real-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL DatabasesReal-Time Integration Between MongoDB and SQL Databases
Real-Time Integration Between MongoDB and SQL Databases
 
Polygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugPolygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @Oakjug
 
Scientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & SociologyScientific Software: Sustainability, Skills & Sociology
Scientific Software: Sustainability, Skills & Sociology
 
Kotlin+MicroProfile: Ensinando 20 anos para uma linguagem nova
Kotlin+MicroProfile: Ensinando 20 anos para uma linguagem novaKotlin+MicroProfile: Ensinando 20 anos para uma linguagem nova
Kotlin+MicroProfile: Ensinando 20 anos para uma linguagem nova
 
Cape2013 scilab-workshop-19Oct13
Cape2013 scilab-workshop-19Oct13Cape2013 scilab-workshop-19Oct13
Cape2013 scilab-workshop-19Oct13
 
On the need for a W3C community group on RDF Stream Processing
On the need for a W3C community group on RDF Stream ProcessingOn the need for a W3C community group on RDF Stream Processing
On the need for a W3C community group on RDF Stream Processing
 
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
 
Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...
 
OSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchainOSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchain
 
Static Energy Prediction in Software: A Worst-Case Scenario Approach
Static Energy Prediction in Software: A Worst-Case Scenario ApproachStatic Energy Prediction in Software: A Worst-Case Scenario Approach
Static Energy Prediction in Software: A Worst-Case Scenario Approach
 
ScalaDays 2013 Keynote Speech by Martin Odersky
ScalaDays 2013 Keynote Speech by Martin OderskyScalaDays 2013 Keynote Speech by Martin Odersky
ScalaDays 2013 Keynote Speech by Martin Odersky
 
Scilab: Computing Tool For Engineers
Scilab: Computing Tool For EngineersScilab: Computing Tool For Engineers
Scilab: Computing Tool For Engineers
 
Automatic and Interpretable Machine Learning with H2O and LIME
Automatic and Interpretable Machine Learning with H2O and LIMEAutomatic and Interpretable Machine Learning with H2O and LIME
Automatic and Interpretable Machine Learning with H2O and LIME
 
2015 10-7-11am-reproducible research
2015 10-7-11am-reproducible research2015 10-7-11am-reproducible research
2015 10-7-11am-reproducible research
 
Analytics of analytics pipelines: from optimising re-execution to general Dat...
Analytics of analytics pipelines:from optimising re-execution to general Dat...Analytics of analytics pipelines:from optimising re-execution to general Dat...
Analytics of analytics pipelines: from optimising re-execution to general Dat...
 
Products go Green: Worst-Case Energy Consumption in Software Product Lines
Products go Green: Worst-Case Energy Consumption in Software Product LinesProducts go Green: Worst-Case Energy Consumption in Software Product Lines
Products go Green: Worst-Case Energy Consumption in Software Product Lines
 

More from Bertram Ludäscher

Reconciling Conflicting Data Curation Actions: Transparency Through Argument...
Reconciling Conflicting Data Curation Actions:  Transparency Through Argument...Reconciling Conflicting Data Curation Actions:  Transparency Through Argument...
Reconciling Conflicting Data Curation Actions: Transparency Through Argument...Bertram Ludäscher
 
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion
Games, Queries, and Argumentation Frameworks: Time for a Family ReunionGames, Queries, and Argumentation Frameworks: Time for a Family Reunion
Games, Queries, and Argumentation Frameworks: Time for a Family ReunionBertram Ludäscher
 
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!Bertram Ludäscher
 
[Flashback] Integration of Active and Deductive Database Rules
[Flashback] Integration of Active and Deductive Database Rules[Flashback] Integration of Active and Deductive Database Rules
[Flashback] Integration of Active and Deductive Database RulesBertram Ludäscher
 
[Flashback] Statelog: Integration of Active & Deductive Database Rules
[Flashback] Statelog: Integration of Active & Deductive Database Rules[Flashback] Statelog: Integration of Active & Deductive Database Rules
[Flashback] Statelog: Integration of Active & Deductive Database RulesBertram Ludäscher
 
Answering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query PatternsAnswering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query PatternsBertram Ludäscher
 
Computational Reproducibility vs. Transparency: Is It FAIR Enough?
Computational Reproducibility vs. Transparency: Is It FAIR Enough?Computational Reproducibility vs. Transparency: Is It FAIR Enough?
Computational Reproducibility vs. Transparency: Is It FAIR Enough?Bertram Ludäscher
 
Which Model Does Not Belong: A Dialogue
Which Model Does Not Belong: A DialogueWhich Model Does Not Belong: A Dialogue
Which Model Does Not Belong: A DialogueBertram Ludäscher
 
From Research Objects to Reproducible Science Tales
From Research Objects to Reproducible Science TalesFrom Research Objects to Reproducible Science Tales
From Research Objects to Reproducible Science TalesBertram Ludäscher
 
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of Us
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of UsPossible Worlds Explorer: Datalog & Answer Set Programming for the Rest of Us
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of UsBertram Ludäscher
 
Deduktive Datenbanken & Logische Programme: Eine kleine Zeitreise
Deduktive Datenbanken & Logische Programme: Eine kleine ZeitreiseDeduktive Datenbanken & Logische Programme: Eine kleine Zeitreise
Deduktive Datenbanken & Logische Programme: Eine kleine ZeitreiseBertram Ludäscher
 
[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...
[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...
[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...Bertram Ludäscher
 
Dissecting Reproducibility: A case study with ecological niche models in th...
Dissecting Reproducibility:  A case study with ecological niche models  in th...Dissecting Reproducibility:  A case study with ecological niche models  in th...
Dissecting Reproducibility: A case study with ecological niche models in th...Bertram Ludäscher
 
Incremental Recomputation: Those who cannot remember the past are condemned ...
Incremental Recomputation:  Those who cannot remember the past are condemned ...Incremental Recomputation:  Those who cannot remember the past are condemned ...
Incremental Recomputation: Those who cannot remember the past are condemned ...Bertram Ludäscher
 
Validation and Inference of Schema-Level Workflow Data-Dependency Annotations
Validation and Inference of Schema-Level Workflow Data-Dependency AnnotationsValidation and Inference of Schema-Level Workflow Data-Dependency Annotations
Validation and Inference of Schema-Level Workflow Data-Dependency AnnotationsBertram Ludäscher
 
An ontology-driven framework for data transformation in scientific workflows
An ontology-driven framework for data transformation in scientific workflowsAn ontology-driven framework for data transformation in scientific workflows
An ontology-driven framework for data transformation in scientific workflowsBertram Ludäscher
 
Knowledge Representation & Reasoning and the Hierarchy-of-Hypotheses Approach
Knowledge Representation & Reasoning and the Hierarchy-of-Hypotheses ApproachKnowledge Representation & Reasoning and the Hierarchy-of-Hypotheses Approach
Knowledge Representation & Reasoning and the Hierarchy-of-Hypotheses ApproachBertram Ludäscher
 
Whole-Tale: The Experience of Research
Whole-Tale: The Experience of ResearchWhole-Tale: The Experience of Research
Whole-Tale: The Experience of ResearchBertram Ludäscher
 
ETC & Authors in the Driver's Seat
ETC & Authors in the Driver's SeatETC & Authors in the Driver's Seat
ETC & Authors in the Driver's SeatBertram Ludäscher
 
From Provenance Standards and Tools to Queries and Actionable Provenance
From Provenance Standards and Tools to Queries and Actionable ProvenanceFrom Provenance Standards and Tools to Queries and Actionable Provenance
From Provenance Standards and Tools to Queries and Actionable ProvenanceBertram Ludäscher
 

More from Bertram Ludäscher (20)

Reconciling Conflicting Data Curation Actions: Transparency Through Argument...
Reconciling Conflicting Data Curation Actions:  Transparency Through Argument...Reconciling Conflicting Data Curation Actions:  Transparency Through Argument...
Reconciling Conflicting Data Curation Actions: Transparency Through Argument...
 
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion
Games, Queries, and Argumentation Frameworks: Time for a Family ReunionGames, Queries, and Argumentation Frameworks: Time for a Family Reunion
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion
 
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!
 
[Flashback] Integration of Active and Deductive Database Rules
[Flashback] Integration of Active and Deductive Database Rules[Flashback] Integration of Active and Deductive Database Rules
[Flashback] Integration of Active and Deductive Database Rules
 
[Flashback] Statelog: Integration of Active & Deductive Database Rules
[Flashback] Statelog: Integration of Active & Deductive Database Rules[Flashback] Statelog: Integration of Active & Deductive Database Rules
[Flashback] Statelog: Integration of Active & Deductive Database Rules
 
Answering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query PatternsAnswering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query Patterns
 
Computational Reproducibility vs. Transparency: Is It FAIR Enough?
Computational Reproducibility vs. Transparency: Is It FAIR Enough?Computational Reproducibility vs. Transparency: Is It FAIR Enough?
Computational Reproducibility vs. Transparency: Is It FAIR Enough?
 
Which Model Does Not Belong: A Dialogue
Which Model Does Not Belong: A DialogueWhich Model Does Not Belong: A Dialogue
Which Model Does Not Belong: A Dialogue
 
From Research Objects to Reproducible Science Tales
From Research Objects to Reproducible Science TalesFrom Research Objects to Reproducible Science Tales
From Research Objects to Reproducible Science Tales
 
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of Us
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of UsPossible Worlds Explorer: Datalog & Answer Set Programming for the Rest of Us
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of Us
 
Deduktive Datenbanken & Logische Programme: Eine kleine Zeitreise
Deduktive Datenbanken & Logische Programme: Eine kleine ZeitreiseDeduktive Datenbanken & Logische Programme: Eine kleine Zeitreise
Deduktive Datenbanken & Logische Programme: Eine kleine Zeitreise
 
[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...
[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...
[Flashback 2005] Managing Scientific Data: From Data Integration to Scientifi...
 
Dissecting Reproducibility: A case study with ecological niche models in th...
Dissecting Reproducibility:  A case study with ecological niche models  in th...Dissecting Reproducibility:  A case study with ecological niche models  in th...
Dissecting Reproducibility: A case study with ecological niche models in th...
 
Incremental Recomputation: Those who cannot remember the past are condemned ...
Incremental Recomputation:  Those who cannot remember the past are condemned ...Incremental Recomputation:  Those who cannot remember the past are condemned ...
Incremental Recomputation: Those who cannot remember the past are condemned ...
 
Validation and Inference of Schema-Level Workflow Data-Dependency Annotations
Validation and Inference of Schema-Level Workflow Data-Dependency AnnotationsValidation and Inference of Schema-Level Workflow Data-Dependency Annotations
Validation and Inference of Schema-Level Workflow Data-Dependency Annotations
 
An ontology-driven framework for data transformation in scientific workflows
An ontology-driven framework for data transformation in scientific workflowsAn ontology-driven framework for data transformation in scientific workflows
An ontology-driven framework for data transformation in scientific workflows
 
Knowledge Representation & Reasoning and the Hierarchy-of-Hypotheses Approach
Knowledge Representation & Reasoning and the Hierarchy-of-Hypotheses ApproachKnowledge Representation & Reasoning and the Hierarchy-of-Hypotheses Approach
Knowledge Representation & Reasoning and the Hierarchy-of-Hypotheses Approach
 
Whole-Tale: The Experience of Research
Whole-Tale: The Experience of ResearchWhole-Tale: The Experience of Research
Whole-Tale: The Experience of Research
 
ETC & Authors in the Driver's Seat
ETC & Authors in the Driver's SeatETC & Authors in the Driver's Seat
ETC & Authors in the Driver's Seat
 
From Provenance Standards and Tools to Queries and Actionable Provenance
From Provenance Standards and Tools to Queries and Actionable ProvenanceFrom Provenance Standards and Tools to Queries and Actionable Provenance
From Provenance Standards and Tools to Queries and Actionable Provenance
 

Recently uploaded

Data Analysis Project Presentation : NYC Shooting Cluster Analysis
Data Analysis Project Presentation : NYC Shooting Cluster AnalysisData Analysis Project Presentation : NYC Shooting Cluster Analysis
Data Analysis Project Presentation : NYC Shooting Cluster AnalysisBoston Institute of Analytics
 
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证acoha1
 
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam DunksNOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam Dunksgmuir1066
 
What is Insertion Sort. Its basic information
What is Insertion Sort. Its basic informationWhat is Insertion Sort. Its basic information
What is Insertion Sort. Its basic informationmuqadasqasim10
 
MATERI MANAJEMEN OF PENYAKIT TETANUS.ppt
MATERI  MANAJEMEN OF PENYAKIT TETANUS.pptMATERI  MANAJEMEN OF PENYAKIT TETANUS.ppt
MATERI MANAJEMEN OF PENYAKIT TETANUS.pptRachmaGhifari
 
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...ThinkInnovation
 
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证pwgnohujw
 
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjSCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjadimosmejiaslendon
 
Digital Marketing Demystified: Expert Tips from Samantha Rae Coolbeth
Digital Marketing Demystified: Expert Tips from Samantha Rae CoolbethDigital Marketing Demystified: Expert Tips from Samantha Rae Coolbeth
Digital Marketing Demystified: Expert Tips from Samantha Rae CoolbethSamantha Rae Coolbeth
 
Seven tools of quality control.slideshare
Seven tools of quality control.slideshareSeven tools of quality control.slideshare
Seven tools of quality control.slideshareraiaryan448
 
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...ThinkInnovation
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfRobertoOcampo24
 
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证dq9vz1isj
 
Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024patrickdtherriault
 
如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一
如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一
如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一fztigerwe
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...Amil baba
 
如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证
如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证
如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证ju0dztxtn
 
Predictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting TechniquesPredictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting TechniquesBoston Institute of Analytics
 
Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...
Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...
Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...ssuserf63bd7
 
obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...
obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...
obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...yulianti213969
 

Recently uploaded (20)

Data Analysis Project Presentation : NYC Shooting Cluster Analysis
Data Analysis Project Presentation : NYC Shooting Cluster AnalysisData Analysis Project Presentation : NYC Shooting Cluster Analysis
Data Analysis Project Presentation : NYC Shooting Cluster Analysis
 
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
 
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam DunksNOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
 
What is Insertion Sort. Its basic information
What is Insertion Sort. Its basic informationWhat is Insertion Sort. Its basic information
What is Insertion Sort. Its basic information
 
MATERI MANAJEMEN OF PENYAKIT TETANUS.ppt
MATERI  MANAJEMEN OF PENYAKIT TETANUS.pptMATERI  MANAJEMEN OF PENYAKIT TETANUS.ppt
MATERI MANAJEMEN OF PENYAKIT TETANUS.ppt
 
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
 
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
 
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjSCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
 
Digital Marketing Demystified: Expert Tips from Samantha Rae Coolbeth
Digital Marketing Demystified: Expert Tips from Samantha Rae CoolbethDigital Marketing Demystified: Expert Tips from Samantha Rae Coolbeth
Digital Marketing Demystified: Expert Tips from Samantha Rae Coolbeth
 
Seven tools of quality control.slideshare
Seven tools of quality control.slideshareSeven tools of quality control.slideshare
Seven tools of quality control.slideshare
 
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdf
 
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
 
Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024
 
如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一
如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一
如何办理哥伦比亚大学毕业证(Columbia毕业证)成绩单原版一比一
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证
如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证
如何办理英国卡迪夫大学毕业证(Cardiff毕业证书)成绩单留信学历认证
 
Predictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting TechniquesPredictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting Techniques
 
Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...
Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...
Statistics Informed Decisions Using Data 5th edition by Michael Sullivan solu...
 
obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...
obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...
obat aborsi Tarakan wa 081336238223 jual obat aborsi cytotec asli di Tarakan9...
 

YesWorkflow: How to render a script as a workflow in half an hour!

  • 1. YesWorkflow:  A  User-­‐Oriented,  Language-­‐ Independent  Tool  for  Recovering  Workflow   Informa9on  from  Scripts   Timothy  McPhillips1,  Tianhong  Song2,  Tyler  Kolisnik3,     Steve  Aulenbach4,  Khalid  Belhajjame5,  Kyle  Bocinsky6,  Yang  Cao1,   Fernando  Chiriga97,  Saumen  Dey2,  Juliana  Freire7,  Deborah  Huntzinger11,   Christopher  Jones8,  David  Koop9,  Paolo  Missier10,  Mark  Schildhauer8,   Christopher  Schwalm11,  Yaxing  Wei12,  James  Cheney13,  Mark  Bieda3,   Bertram  Ludäscher1,  14   10th  Intl.  Digital  CuraRon  Conference  (IDCC)   30  Euston  Square,  London,  UK  
  • 2. Overview   •  Enter    ScienRfic  Workflows  …     –  Kepler,  Taverna,  VisTrails,  …     •  ... Exeunt •  Enter Scripts  …     –  Python,  R,  Matlab,  …   •  Enter YesWorkflow   –  Complements  noWorkflow   •  …  run$me  (=  retrospec)ve)  provenance  from  scripts   –  Combines  the  best  of  both  worlds:   •  Scripts  +  Comments    =>  Workflow  Views  (prospec$ve  provenance)  from  scripts   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     2  
  • 3. Scientific Workflows: ASAP! •  Automation –  wfs to automate computational aspects of science •  Scaling (exploit and optimize machine cycles) –  wfs should make use of parallel compute resources –  wfs should be able handle large data •  Abstraction, Evolution, Reuse (human cycles) –  wfs should be easy to (re-)use, evolve, share •  Provenance –  wfs should capture processing history, data lineage è traceable data- and wf-evolution è  Reproducible Science
  • 4. Science  Example:  Paleoclimate  ReconstrucRon   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     4   •  Kohler  &  Bocinsky:  study  rain-­‐fed  maize  of  Ancestral  Pueblo,  Anasazi     –  Four  Corners;  AD  600–1500   –  Climate  change  influenced  Mesa  Verde  MigraRons;  late  13th  century  AD.   –  Uses  network  of  tree-­‐ring  chronologies  to  reconstruct  a  spaRo-­‐temporal   climate  field  at  a  fairly  high  resolu9on  (~800  m)  from  AD  1–2000     –  Algorithm  es9mates  joint  informa9on  in  tree-­‐rings  and  a  climate  signal  to   iden9fy  “best”    tree-­‐ring  chronologies  for  reconstruc9ng  climate  at  a  given   9me  and  place.   K.  Bocinsky,  T.  Kohler,  A  2000-­‐year  reconstruc9on  of  the  rain-­‐fed   maize  agricultural  niche  in  the  US  Southwest.  Nature   Communica$ons.  doi:10.1038/ncomms6618     … implemented as an R Script …
  • 5. ERRT  @  GSLIS   5   K.  Bocinsky,  T.  Kohler,  A  2000-­‐year  reconstruc9on  of  the  rain-­‐fed  maize  agricultural  niche  in   the  US  Southwest.  Nature  Communica$ons.  doi:10.1038/ncomms6618     …  Paleoclimate     ReconstrucRon  …     Map  showing  the  "selected"  trees  for  reconstruc)ng   precipita)on  at  four  sites  in  our  CAR  regression   approach  (Correla)on-­‐Adjusted  corRela)on).   Reconstruc)ons   for  AD  1247    
  • 6. YesWorkflow  =  Scripts  +  Comments   •  Scripts  can  be  hard  to  digest,  communicate   •  Idea:     – Add  structured  comments  (cf.  JavaDoc)   =>    reveal  workflow  structure  and  dataflow    =>    obtain  some  scien9fic  workflow  benefits   •   …  ASAP  …     6  
  • 7.            Get  3  views  for  the  price  of  1!   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     7                        Process  view   Data  view   Combined  view  
  • 8. User  Comments:  YW  AnnotaRons   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     8   @begin GO_Analysis @in hgCutoff @in … @out BP_Summl_file @out … @end GO_Analysis ...  
  • 9. Paleoclimate  ReconstrucRon  …       B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     9   GetModernClimate PRISM_annual_growing_season_precipitation SubsetAllData dendro_series_for_calibration dendro_series_for_reconstruction CAR_Analysis_unique cellwise_unique_selected_linear_models CAR_Analysis_union cellwise_union_selected_linear_models CAR_Reconstruction_union raster_brick_spatial_reconstruction raster_brick_spatial_reconstruction_errors CAR_Reconstruction_union_output ZuniCibola_PRISM_grow_prcp_ols_loocv_union_recons.tif ZuniCibola_PRISM_grow_prcp_ols_loocv_union_errors.tif master_data_directory prism_directory tree_ring_datacalibration_years retrodiction_years •  …  explained  using  YesWorkflow   Kyle  B.,  (computa9onal)  archeologist:     "It  took  me  about  20  minutes  to  comment.  Less   than  an  hour  to  learn  and  YW-­‐annotate,  all-­‐told."  
  • 10. YesWorkflow  Architecture:  KISS!   •  YW-­‐Extract   – …  structured  comments     •  YW-­‐Model   – Program  Block,  Workflow   – Port  (data,  parameters)   – Channels  (dataflow)     •  YW-­‐Graph   – ...  using  GraphViz/DOT  files   •  YW-­‐Query,  YW-­‐Validate,  YW-­‐CLI     B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     10  
  • 11.                          Figure 4: Process workflow view of an A↵ymetrix analysis script (in R). 4 YesWorkflow Examples In the following we show YesWorkflow views extracted from real-world scientific use cases. The scripts were annoted with YW tags by scientists and script authors, using a very modest training and mark-up e↵ort.1 Due to lack of space, the actual MATLAB and R scripts with their YW markup are not included here. However, they are all available from the yw-idcc-15 repository on the YW GitHub site [Yes15]. Gene  Expression  Microarray  Data  Analysis   •  [Normalize]     –  Normaliza9on  of  data  across  microarray  datasets     •  [SelectDEGs]     –  Selec9on  of  differen9ally  expressed  genes  between  condi9ons     •  [GO  Analysis]     –  determina9on  of  gene  ontology  sta9s9cs  for  the  resul9ng  datasets     •  [MakeHeatmap]     –  crea9on  of  a  heatmap  of  the  differen9ally  expressed  genes.     B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     11   Tyler  Kolisnik,  Mark  Bieda  
  • 12. MulR-­‐Scale  Synthesis  and  Terrestrial  Model   Intercomparison  Project  (MsTMIP)   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     12   fetch_drought_variable drought_variable_1 fetch_effect_variable effect_variable_1 convert_effect_variable_units effect_variable_2 create_land_water_mask land_water_mask init_data_variables predrought_effect_variable_1 drought_value_variable_1 recovery_time_variable_1 drought_number_variable_1 define_droughts sigma_dv_event month_dv_length detrend_deseasonalize_effect_variable effect_variable_3 calculate_data_variables recovery_time_variable_2 drought_value_variable_2 predrought_effect_variable_2 drought_number_variable_2 export_recovery_time_figure output_recovery_time_figure export_drought_value_variable_figure output_drought_value_variable_figure export_predrought_effect_variable_figure output_predrought_effect_variable_figure export_drought_number_variable_figure output_drought_number_figure input_drough_variable input_effect_variable Christopher  Schwalm,   Yaxing  Wei  
  • 13. Summary:  ScienRfic  Workflows   ScienRfic  Workflows   •  [+]  Automa9on   •  [+]  Scalability   •  [+]  Abstrac9on   •  [+]  Provenance   •  …   •  [+/0]  Easy  to  use     –  [0]  learning  a  new  paradigm   •  [-­‐]  Teaching  resources     •  [-­‐]  Special  exper9se  needed  for  deep  changes    e.g.  new  Java  actors,  shims,  …   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     13  
  • 14. Summary:  Scripts  +  YesWorkflow   Scripts:  [+]  Automa9on,  [0]  Scalability,    [-­‐]  Abstrac9on,  [0/-­‐]  Provenance   Now:  Scripts  +  YesWorkflow  AnnotaRons   •  [+]  AbstracRon   –  explain  your  methods  to  mere  mortals   =>  encourage  (re-­‐)use   •  [+]  Provenance:   –  noWorkflow  (retrospec$ve  provenance)   –  YesWorkflow  (prospec$ve  provenance)   •  [+]  Language  independent  (R,  Matlab,  Python,  …)     •  [+]  Empower  tool  makers  (script  programmers):  give  them  …   –  …  some  immediate  benefits  (workflow  views)   –  …  some  medium  term  improvements  (provenance  integraRon)   –  …  some  long  term  benefits:  think  about  your  methods  differently     =>  dataflow  programming  =>  [+]  Scalability     B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     14  
  • 15. YesWorkflow:  Acknowledgements   •  With  support  from  NSF  awards  DBI-­‐1356751  (Kurator),   ACI-­‐0830944  (DataONE),  SMA-­‐1439603  (SKOPE).       •  Thanks  to  the  IDCC  organizers  for  choice  of  venues!   B.  Ludäscher                                                                                                  YesWorkflow:  Workflow  Views  from  Scripts.  IDCC'15,  London     15