Download to read offline
![o Software market rapidly shifting to Big Data
32% compound annual growth rate in EU through 2016
35% Big data projects are successful [CapGemini 2015]
o ICT-9 call focused on SW quality assurance (QA)
ISTAG: call to define environments “for understanding the
consequences of different implementation alternatives (e.g.
quality, robustness, performance, maintenance, evolvability, ...)”
o QA evolving too slowly compared to the technology
trends (Big data, Cloud, DevOps ...)
DICE aims at closing the gap
DICE RIA - Overview
Motivation
3©DICE 9/19/2015](https://image.slidesharecdn.com/imp-dice-overview-150919092730-lva1-app6892/85/Towards-Quality-Aware-Development-of-Big-Data-Applications-with-DICE-3-320.jpg)
![DICE RIA - Overview
Year 1 Milestones
16©DICE 9/19/2015
Milestone Deliverables
Baseline and
Requirements -
July 2015
[COMPLETED]
• State of the art analysis
• Requirement specification
• Dissemination, communication,
collaboration and standardisation report
• Data management plan
Architecture
Definition -
January 2016
• Design and quality abstractions
• DICE simulation tools
• DICE verification tools
• Monitoring and data warehousing tools
• DICE delivery tools
• Architecture definition and integration plan
• Exploitation plan](https://image.slidesharecdn.com/imp-dice-overview-150919092730-lva1-app6892/85/Towards-Quality-Aware-Development-of-Big-Data-Applications-with-DICE-16-320.jpg)
Uploaded byPooyan Jamshidi
Towards Quality-Aware Development of Big Data Applications with DICE
The document summarizes the DICE Horizon 2020 project, which aims to improve quality-aware development of big data applications. The 3-year project involves 9 partners across 7 EU countries. It seeks to shorten development times and reduce costs and quality incidents for big data projects through model-driven engineering and DevOps approaches. The project will demonstrate its techniques on three big data case studies and has milestones to define requirements, provide tools, and define its integrated architecture.
More Related Content
Similar to Towards Quality-Aware Development of Big Data Applications with DICE
Towards Quality-Aware Development of Big Data Applications with DICE
- 1. DICE Horizon 2020Project Grant Agreement no. 644869 http://www.dice-h2020.eu Funded by the Horizon 2020 Framework Programme of the European Union Towards Quality-Aware Development of Big Data Applications with DICE Pooyan Jamshidi Imperial College London, UK Project Coordinator: Giuliano Casale Imperial College London, UK
- 2. DICE RIA -Overview DICE Project o Horizon 2020 Research & Innovation Action (RIA) Quality-Aware Development for Big Data applications Feb 2015 - Jan 2018, 4M Euros budget 9 partners (Academia & SMEs), 7 EU countries 2©DICE 9/19/2015
- 3. o Software marketrapidly shifting to Big Data 32% compound annual growth rate in EU through 2016 35% Big data projects are successful [CapGemini 2015] o ICT-9 call focused on SW quality assurance (QA) ISTAG: call to define environments “for understanding the consequences of different implementation alternatives (e.g. quality, robustness, performance, maintenance, evolvability, ...)” o QA evolving too slowly compared to the technology trends (Big data, Cloud, DevOps ...) DICE aims at closing the gap DICE RIA - Overview Motivation 3©DICE 9/19/2015
- 4. o Reliability o Efficiency oSafety & Privacy DICE RIA - Overview Quality Dimensions 4 Availability Fault-tolerance Performance Costs ©DICE 9/19/2015 Verification (e.g., deadlines) Data protection
- 5. DICE RIA -Overview High-Level Objectives o Tackling skill shortage and steep learning curves Data-aware methods, models, and OSS tools o Shorter time to market for Big Data applications Cost reduction, without sacrificing product quality o Decrease development and testing costs Select optimal architectures that can meet SLAs o Reduce number and severity of quality incidents Iterative refinement of application design 5©DICE 9/19/2015
- 6. DICE RIA -Overview Some Challenges in Big Data… o Lack of quality-aware development for Big Data o How to described in MDE Big Data technologies o Spark, Hadoop/MapReduce, Storm, Cassandra, ... oCloud storage, auto-scaling, private/public/hybrid, ... o Today no QA toolchain can help reasoning on data-intensive applications o What if I double memory? o What if I parallelize more the application? 6©DICE 9/19/2015
- 7. DICE RIA -Overview … in a DevOps fashion o Software development methods are evolving o DevOps closes the gap between Dev and Ops From agile development to agile delivery Lean release cycles with automated tests and tools Deep modelling of systems is the key to automation 7©DICE 9/19/2015 Agile Development DevOps Business Dev Ops
- 8. DICE RIA -Overview Demonstrators 8©DICE 9/19/2015 Case study Domain Features & Challenges Distributed data- intensive media system (ATC) • News & Media • Social media • Large-scale software • Data velocities • Data volumes • Data granularity • Multiple data sources and channels • Privacy Big Data for e- Government (Netfective) • E-Gov application • Data volumes • Legacy data • Data consolidation • Data stores • Privacy • Forecasting and data analysis Geo-fencing (Prodevelop) • Maritime sector • Vessels movements • Safety requirements • Streaming & CEP • Geographical information
- 9. Bringing QA andDevOps together 9/19/2015 9 Requirements SLAs Compare Alternatives Load testing Cost Tradeoffs Monitoring Capacity Management Incident Analysis Deployment ProfilingSPE Regression Bottleneck Identification Root Cause Analysis DICE User behaviour Adaptation DICE RIA - Overview©DICE 9/19/2015
- 10. DevOps in DICE:Measurement 10 MySQL NoSQL S3 DIA Node 1 DIA Node 2Users Dev jenkins chef monitoring and incident report release Ops incident report (performance unit tests) Deployment & CI DICE RIA - Overview©DICE 9/19/2015
- 11. 11 MySQL NoSQL S3 DIA Node 1 DIANode 2Users Dev jenkins chef monitoring and incident report early-stage quality assessment Ops incident report release (performance unit tests) DevOps in DICE: Early-stage MDE Deployment & CI DICE RIA - Overview©DICE 9/19/2015
- 12. Quality-Aware MDE • UMLMARTE profile, UML DAM profile, Palladio, … 9/19/2015 G. Casale – Performance Aware DevOps 12 Failure Probability Usage Profile System Behaviour
- 13. Platform-Indep. Model Domain Models Quality-Aware MDE 13 QA Models Architecture Model Platform-Specific Model Code stub generation Platform Description MARTE SimulationTools Cost Optimization Tools Data Intensive Application DICE RIA - Overview©DICE 9/19/2015
- 14. 14 MySQL NoSQL S3 DIA Node 1 DIANode 2Users Dev jenkins chef incident report & model correlation continuous quality engineering (“shared system view” via MDE) Ops incident report continuous monitoring and enhancement release (performance unit tests) DevOps in DICE: Enhancement Deployment & CI DICE RIA - Overview©DICE 9/19/2015
- 15. Platform-Indep. Model Domain Models DICE Integrated Solution 15 Continuous Enhancement Continuous Monitoring Data Awareness Architecture Model Platform-Specific Model Platform Description DICEMARTE Deployment & Continuous Integration DICE IDE QA Models Data Intensive Application DICE RIA - Overview©DICE 9/19/2015
- 16. DICE RIA -Overview Year 1 Milestones 16©DICE 9/19/2015 Milestone Deliverables Baseline and Requirements - July 2015 [COMPLETED] • State of the art analysis • Requirement specification • Dissemination, communication, collaboration and standardisation report • Data management plan Architecture Definition - January 2016 • Design and quality abstractions • DICE simulation tools • DICE verification tools • Monitoring and data warehousing tools • DICE delivery tools • Architecture definition and integration plan • Exploitation plan
- 17. DICE RIA -Overview Thanks www.dice-h2020.eu 17©DICE
Editor's Notes
- #2 - consortium info objectives motivation innovation concepts running case approach expected achievements at M12 impact assessment project structure technical architecture case studies