Presentation at Innovate-Data 2017 conference on Near real-time big-data processing for data driven applications

1. 1
Near real-time big-data
processing for data driven
applications
Jānis Kampars, Jānis Grabis
Institute of Information Technology
Riga Technical University, Riga, Latvia
janis.kampars@rtu.lv, grabis@rtu.lv

2. 22
 Background and objectives
 Conceptual model
 Architecture and technologies
 Sample use case
 Conclusion
Outline
2

3. 33
 Development of context-aware adaptive
applications
 Context as business process execution driver
Background
3
FP7 project
Capability
Driven
Development
(CaaS)

4. 44
 To develop a platform for context-
dependent adaption of data-driven
applications
– Externalized context processing and adaption
logics
– Model-driven and horizontally scalable
Objective
4

5. 55
General Approah
5

6. 66
Conceptual Model
6
Entity
Context Provider
Measurable Property
Dimenssion ValueSchema
Archiving Specification Context Element
Context Calculation Adjustment Adjustment Trigger
Context Element Range
0..*
defines
1
1
1
1..* 1..*
11
2
relates
.*
0..*
measures
0..*
1
0..* 0..*
uses
0..*
0..*
uses
0..* 1
Calculates
1
1..*
trigers
1
1
takes value from
0..*
1..*
uses
1..*

7. 77
Key Elements
7
• Data affecting process
execution
Context elements
• Data items characterizing
the domain
Entities
• Capturing of physical
context
Context providers
and measurable
properties
• Adaptive actions due to
context change
Adjustments

8. 88
Entity model
Stream processing
Clustering
Persistence
Architecture and Technology
8

9. 99
Overview of Architecture
9
CDP
Cassandra cluster
Cassandra 2 Cassandra N
Spark cluster
Spark 2 Spark NSpark 1
Cassandra 1
MP archiving job
CEcalculation
job
Adjustment
triggering job
Data-drivensystem
Adjustmentengine
Adjustment1Adjustment2AdjustmentN
Kafka proxy cluster
Proxy 1 Proxy 2 Proxy N
Kafka cluster
Kafka 1 Kafka 2 Kafka N
Raw MP data
Raw MP data 63
AggregatedMPdata
4
CEdata
7
CEdata 8 Trigger
adjustment
95
Trigger
adjustment
10
Performadjustment
11
Raw MP data
1
2
ASAPCS core + UI

10. 1010
 Spark jobs
– Aggregation and archiving of measurable properties
– Context element calculation
– Adjustment triggering
 Jobs are created according to the entity model
Spark integration
10
Entity
model
Compu-
tations
Measurab
le
properties
Context
elements
Adjustme
nt triggers
Docker
contain
ers

11. 1111
 Adjustments are placed and executed in
dedicated Docker containers
Docker Integration
11
Entity model
Adjustment
specification
Docker
container

12. 12
Sample Use Case
12

13. 13
 Data storage problem
– Data is stored on disks that
are located on data nodes
– Data centers belong to
specific geographic regions
– Disk health is measured by
write errors, read errors,
temperature and bad sectors
– Data center region safety is
measured by nature hazards,
security incidents or terrorist
attacks
 Additional data replication is
required to deal with security risks
Identification of context dependent
variations in the data-driven application
Specification of potential context providers
Definition of relevant entities and
measurable properties
Creation of context elements and their
calculations
Implementation of adjustments associated
with the context elements defined
Deployment of the solution
Operation (context data integration and
execution of adjustments)
13
Model-based Infrastructure
Management

14. 1414
Entity Model
14

15. 1515
Specification of Computations
15

16. 1616
 Efficient use of computational resources
depending on applications
 Use cases
– Data center management
– Adaptation of enterprise application on the
basis of log processing
Conclusion
16