This document discusses a project that aims to develop graphical data workflows and cross-platform optimization. It allows users to design data processing workflows with minimal programming. The project also optimizes workflows to run efficiently across multiple computing clusters. Example use cases discussed are studying drug synergies in cancer research, predicting financial market behaviors, and maritime situational awareness. The document outlines the graphical editor, streaming operations, and how an optimization component can split workflows across clusters for better performance.

1. DavidArnu
RapidMiner
Graphical Data Analytic Workflows and
Cross-Platform Optimization
Graphical Data Workflows and Cross-Platform Optimization 1

2. Project Overview
Graphical Data Workflows and Cross-Platform Optimization 2
• Graphically design data processing workflows and data
analytics tasks with minimal or no programming overhead
• Real-time, interactive machine learning and data mining
tools
• Distributed Complex Event Forecasting

3. Life Sciences Use Case
Studying the effect of drug synergiesin cancer
from in-silico simulations to in-vivo experimentsand back
• Challenges:
• Huge CPU, memory requirements + output data to be processed
• Too manysimulations, too few promising ones
• Train a ML model to classify promising simulations and kill non-promising ones
• Learn which genes drive evolution of other genes and which to monitor
Graphical Data Workflows and Cross-Platform Optimization 3

4. Financial Use Case
Goal: Train ML models that extract valid rules used to perform:
• Real-time suggestion and forecast of investmentopportunities
• Systemic risk (i.e., great linkage between major market participants)
prediction
• Forecast price swings
PredictingPrice Swings,SystemicRisk and
ForecastingInvestmentOpportunities
Graphical Data Workflows and Cross-Platform Optimization 4

5. Maritime Use Case
Maritime Situational Awareness (MSA),
Monitoring Ship Movement and Detecting Illegal
Activities
Graphical Data Workflows and Cross-Platform Optimization 5
Challenges:
• Large amountof ships to monitor
• Many different data sources are
available
• Complex event classification
(patterns of movement or other
behavior)

6. Graphical Data Analytic Workflows
Graphical Data Workflows and Cross-Platform Optimization 6

7. Graphical Editor
7
• Simply design Streaming Analytics Workflows
• Upon execution one job is created and deployed
to the connected streaming cluster
Graphical Data Workflows and Cross-Platform Optimization

8. Inner Workings
8Graphical Data Workflows and Cross-Platform Optimization

9. Supported Clusters:
Apache Flink
Apache Spark (structured)Streaming
Available Operations
Streaming analytics operations
Synopsis Data Engine
Custom Online Machine Learning engines (running on Flink
and AKKA)
• Connections to financial service providers
Capabilities
9Graphical Data Workflows and Cross-Platform Optimization

10. Code-Free development
Platform and back-end independent
Pluggable connection management
Easy to share and collaborate
Benefits
10Graphical Data Workflows and Cross-Platform Optimization

11. https://youtu.be/9SKcM70Bi2U
Link to Demo Video
Graphical Data Workflows and Cross-Platform Optimization 11

12. Cross-Platform Optimization
Graphical Data Workflows and Cross-Platform Optimization 12
Optimizer Component

13. In a multi-cluster set-up the optimal stream execution
can depend on
Available resourcesper cluster
Data location
Software performance and implementation details
An optimized process layout can greatly enhance the
performance
Optimizer Component
13Graphical Data Workflows and Cross-Platform Optimization

14. Streaming Optimization Operator
14Graphical Data Workflows and Cross-Platform Optimization

15. Optimizer Response
15
Split connection
(process flow
control)
Split connection
(Streaming Graph)
Graphical Data Workflows and Cross-Platform Optimization

16. Streaming Optimization Operator
16Graphical Data Workflows and Cross-Platform Optimization

17. Optimized Workflow
17Graphical Data Workflows and Cross-Platform Optimization

18. Optimized Workflow
18
Split connection
(process flow
control)
Split connection (Streaming Graph)
Graphical Data Workflows and Cross-Platform Optimization

19. Conclusion
• What we have seen:
• Project use cases and goals
• Graphical editor
• Cross-Platform optimization
• What‘s next:
• Job and Data Monitoring
• Better integration of HPC systems
• Refinements and deployment of the use cases
Graphical Data Workflows and Cross-Platform Optimization 19

20. Thank you!
http://www.infore-project.eu
Graphical Data Workflows and Cross-Platform Optimization 20