Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kamal Hakimzadeh – Reproducible Distributed Experiments

6,166 views

Published on

Flink Forward 2015

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Kamal Hakimzadeh – Reproducible Distributed Experiments

  1. 1. Reproducible Distributed Experiments Kamal Hakimzadeh PhD Student mahh@kth.se 1 Jim Dowling Associate Professor jdowling@kth.se www.karamel.io
  2. 2. Agenda 2 • Motivation • Reproducibility • Demo: Simple experiment 30-40 min • Karamel Rep. • Karamel Engine • Orchestration • Challenges www.karamel.io
  3. 3. Motivation 3 Analytical vs Empirical proof DS supports many scientific advancements Scheduling, fault tolerant, scalability … Extremely complex www.karamel.io
  4. 4. Reproducible vs. Replicable 4 1. Laboratory 2. Experimenter 3. Apparatus Reproducible Replicable Computational Reproducibility: Infrastructure, software, experiment and data www.karamel.io
  5. 5. Demo : Word Count 5 Text Generator Text Generator Text Generator Word Count www.karamel.io
  6. 6. Karamel: Rep. in different layers 6 Bare Metal Google Compute Engine Virtual Machine is and abstract entity Software is defined in Chef It is publicly available in Github www.karamel.io
  7. 7. Karamel Engine 7 DSL Service Cloud Clients Karamel Engine Physical Mapping Orchestrator www.karamel.io
  8. 8. Orchestration – queuing model 8www.karamel.io
  9. 9. Result 9www.karamel.io
  10. 10. Challenges and future work 10 Scalability Fault Recovery Model Elasticity – Handle Churn Instrumentation Recommendation System Language Support Load generators Scheduling Container base machines Result Management Debugging www.karamel.io
  11. 11. Team members 11 Kamal Hakimzadeh PhD Student at KTH mahh@kth.se Alberto Lorente Leal Software Developer at Comeon a.lorenteleal@gmail.com Jim Dowling Associate Professor at KTH jdowling@kth.se Hooman Peiro Sajjad PhD Student at KTH shps@kth.se Abhimanyu Babbar Backend Developer at Wrap abhimanyu.babbar88@gmail.comwww.karamel.io
  12. 12. 12www.karamel.io

×