In the Big Data systems, the sheer volume of incoming data is itself a problem to be solved, as well as the problem of how to process that data for the end user. As a French leader in the domain of UML modelling, Softeam then saw the opportunity to improve its commercial offer of UML based modelling environments. This work was funded by the EU commission as part of the JUNIPER FP7 project .
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Model driven engineering for big data management systems
1. www.modeliosoft.com
Model driven engineering for big data
management systems
Marcos ALMEIDA marcos.almeida@softeam.fr
Sarah DAHAB sarah.dahab@telecom-sudparis.eu
Andrey SADOVYKH andrey.sadovykh@softeam.fr
1
3. 20 ME
2006
17,5 ME
2005
70 ME
2013
Paris
Rennes
Nantes
Sophia
SOFTEAM – a French IT services / Software vendor
•SOFTEAM, a growing
company
25 years’ experience
900 experts
Regular growth
• Specialist in OO technologies,
new architectures,
methodologies
• Banking, Defense, Telecom, …
www.modeliosoft.com 3
23 ME
2008
4. Modelio for Software
and System Engineering
• UML editor with 20 years’ history
o CloudML
o SysML
o MARTE
o Code generation
o Documentation
o Teamwork
www.modeliosoft.com 4
• Available under open
source at Modelio.org
6. It is all about models … Starting with UML
www.modeliosoft.com 6
Requirements
UML Use
Cases
Architecture
UML
Components
and Classes
Design
Refined
Classes
or Domain
Specific
Language
Implementation
Code
generation
Java, C++,
Frameworks
8. Typical example: Control system for a frigate
• 800+ components
• Developed by 100+ engineers
• 1M+ LOC
• MDD fosters Productivity and Quality with
o Code generation
o Components reuse
o Tracing
o Automation
www.modeliosoft.com 8
9. Curious DSL example: Ruby on Rails
Haml HTML
%br{:clear => left’} <br clear=”left”/>
%p.foo Hello <p class=”foo”>Hello</p>
%p#foo Hello <p id=”foo”>Hello</p>
.foo <div class=”foo”>...</div>
#foo.bar <div id=”foo” class=”bar”>...</div>
www.modeliosoft.com 9
Feature: User can manually add movie
Scenario: Add a movie
Given I am on the RottenPotatoes home page
When I follow "Add new movie"
Then I should be on the Create New Movie page
When I fill in "Title" with "Men In Black"
And I should see "Men In Black"
Cucumber
and Capybara
HAML
10. What do we get from MDD?
Pros
• Design once, deploy
everywhere!
• Write your
transformation once,
transform anything!
Cons
• Transformations are
hard to write…
• How to make sure they
are CORRECT? i.e.
– Is there any
data/semantic loss?
www.modeliosoft.com 10
12. Volume, variety, velocity
1. @-mails sent
every second : 2,9
million
2. Video uploaded to
YouTube every
minute: 25 hours
3. Data processed by
Google every day:
24 petabytes
4. Tweets per day:
50 million
5. Products ordered
on Amazon per
second: 73 items
www.modeliosoft.com 12
13. Only 0,5 % of data is analyzed
• In 2012, 2 837EB generated
- just 0,5% actually
analyzed.
That still amounts to 14EB
(or 14.185 million
terabytes)
Source: IDC & EMC
www.modeliosoft.com 13
14. The main problem is Heterogeneity!
• Many different database management systems
o Ex:
• MySQL (www.mysql.com/),
• Big Table (http://research.google.com/archive/bigtable.html)
• SimpleDB (http://aws.amazon.com/simpledb/)
• Memcached (http://memcached.org/)
• …
• Many underlying data representation paradigms
o Ex:
• Relational Databases
• Key-value Stores
• Object-oriented Databases
• Big Tables
• …
www.modeliosoft.com 14
15. The basis of our solution is MDE… Why?
• Separating the problem from the solution
o In JUNIPER we model the solution
• Fostering automation
o Analysis
o Code generation
www.modeliosoft.com 15
Business
Objects Transformation
HDFS
MySQL
MongoDB
Abstract Models
Specific Models / code
Transformation
Transformation
16. Understanding the problem… Why is it so
HARD? (1/2)
• Target Technologies based on different paradigms
• Example:
www.modeliosoft.com 16
A
B
JPA
@Entity
public class A {
@Basic
public B getB(){
…
}
…
}
SQL
create table A (…)
create table B (…)
create table A_B (…)
17. Understanding the problem… Why is it so
HARD? (2/2)
• Target structure is variable
• Example:
www.modeliosoft.com 17
A
B
ER
NoSQL
A
BAB
Here A and B
are
independent
entities
Here, for
performance
reasons, B is
embedded in A
A
B
19. Our solution: a component based approach to NoSQL
heterogeneity
• Generic model transformation chain
oIntegrated to other Juniper tools
•Audit rules
•Model to model transformations
•Code generators
• Database specific instantiations
oApplication architecture modelling
oData modelling
oHardware architecture (deployment) modelling
www.modeliosoft.com 19
24. Mapping Programming Model, UML and MARTE
www.modeliosoft.com 24
JUNIPER
Program
Channel
Cloud Node
Programming
Model
UML MARTE
25. Modelling the application and real-time constraints
www.modeliosoft.com 25
Real-time constrains
- response time
- bandwidth
Big Data flow
JUNIPER
Programs
26. Modelling the hardware infrastructure at a high level
www.modeliosoft.com 26
Cloud Node
CPU with 4 cores Hard drive
31. Approach taken for dealing with heterogeneity in JUNIPER
1. Define a generic template for Modelio modules to provide support
for big data management systems
2. Instantiate the template for MongoDB and PostgreSQL
www.modeliosoft.com 31
40. In short…
• Challenge:
o Big data applications How should we handle heterogeneous data ??
• Juniper response:
o Model driven solution for designing real-time big data systems
o Component based solution to heterogeneity
• General business objects + big data concepts modelling
• Database specific concepts
– Modelling
– Model transformations
– Code generation
www.modeliosoft.com 40
41. … and Perspectives / Exploitation
• Source code and documentation available on our website
o http://forge.modelio.org/projects/juniper
o http://forge.modelio.org/projects/mongodb-modeler
o http://forge.modelio.org/projects/postgresql-modeler
• Tutorial + Dissemination on our forum
www.modeliosoft.com 41
42. Questions?
Marcos Almeida
SOFTEAM | ModelioSoft
{name.surname}@softeam.fr
SOFTEAM R&D Web Site:
http://rd.softeam.com
Modelio Web Site :
http://www.modelio.org
http://forge.modelio.org/projects/juniper
JUNIPER Web Site :
http://www.juniper-project.org
www.modeliosoft.com 42
*
*for your questions