What did I sell yesterday and how much of my plan did I fulfill today? How do my clients use our offer? What configuration combinations are in demand and what trends are emerging? How can I improve the user experience? These and other questions are frequently asked by board members and stakeholders and must be answered within a short period of time. Especially in companies that provide configurable products, it is important to support the product and pricing managers in short-term and competition-related matters with all the important data in a timely manner. In our use case, Cassandra, Kafka and Flink will take up this challenge. In this session, we will present a reference architecture based on selected use cases and demonstrate what applications arise for companies. We also take a closer look to information privacy and give some words about data visualisation. About the Speakers Alexandra Klimova Big Data Architect, Allianz Deutschland AG Alexandra has 10 years of experience in both programing and operations. For the last 4 years she has focused on design and integration of Big Data Systems into enterprise platforms. She is working on data processing pipelines, distributed systems, realtime processing and data science. Alexandra holds a degree in Computer Science from the Technical University Munich. She is certified Hortonworks Hadoop Trainer and Big Data Architect at metafinanz. Dominique Ronde Big Data Architect, Allianz Deutschland AG Dominique Ronde is Big Data Architect at Allianz Deutschland AG and focused on the cassandra plattform. He also enjoys the part of data analytics with Flink and Spark. As a real java nerd since 2002 Dominique is familiar with the programming part, too. He is certified DataStax Solution Architect

1. Dominique Rondé (@talk2nerd)
Alexandra Klimova (@aklimova)
Real Time Business Intelligence with Cassandra, Kafka and Hadoop
A real story @ Allianz Deutschland AG

2. © Copyright Allianz

3. © Copyright Allianz
We don‘t have an agenda
-
We have some checklists!
4. Oktober 2016 3
Agenda

4. © Copyright Allianz
Security
Instructions
4. Oktober 2016 4

5. © Copyright Allianz
Checklist
Before Engine Start
Define the destination
4. Oktober 2016 5

6. © Copyright Allianz 4. Oktober 2016 6
Real Time Reporting
• Sold items for the current day
• Open tickets during the day
• Response Time on consumer
requests
• Sold items grouped by type
• Current Errors

7. © Copyright Allianz 4. Oktober 2016 7
Fraud Protection
• Prevent „Fake Accounts“
• Figure out „data grabber“
• Detect fraud pattern

8. © Copyright Allianz 4. Oktober 2016 8
Helping decision makers to understand the market
• Risk Specialists
• Product Designers
• Marketing Experts

9. © Copyright Allianz 4. Oktober 2016 9
Our destination
TTD
Reduce the Time – To – Data

10. © Copyright Allianz
Time to Data is the time which is required until a requester received the data he / she needs to do his / her
job.
Time to
• find the source of required data
• get the needed aggregation
• clean up the data
• write the statistical scripts
• execute and refine these scripts
• get a visualized result
4. Oktober 2016 10
Definition of TTD

11. © Copyright Allianz
Checklist
Before Taxi
Check if we know all we need
4. Oktober 2016 11

12. © Copyright Allianz
• Decoupled from all other development work
Changes in analytics should not require additional work in all other applications
• Allow fast deployments
Learn through the data and bring improvements fast into production
• High available
No Event should get lost after it was fired
• Very accurate
Make sure that every Event processed
• Horizontal scalable
Start small and grow with the data
4. Oktober 2016 12
Define functional requirements

13. © Copyright Allianz
• Data Privacy
• Data Security
• Data Protection
4. Oktober 2016 13
Define legal requirements

14. © Copyright Allianz
Checklist
Before Take Off
Do the first steps
4. Oktober 2016 14

15. © Copyright Allianz
Picking Measuring points
• Implement servlet filters to stay informed about http headers i.e. error-code, referrer
• Implement interceptors for the or-mapper to store the history of entities
• Instrument the web ui to send events about user interactions i.e. changes between pages
• Instrument the java code to send events with additional data at some points i.e. create a document

16. © Copyright Allianz
Each transfer object holds at least the
• current sessionId
• timestamp when this event occurs
• unique identifier of this event
• version identifier
In some cases
• current authenticated user
4. Oktober 2016 16
Create some transfer objects

17. © Copyright Allianz 4. Oktober 2016 17
Find an architecture
WebApplication
Reports
Dashboards
R-Scripts

18. © Copyright Allianz 4. Oktober 2016 18
Design you first CF
Design
conceptual
model
Specify
access
pattern
Choose a
logical
model
Configure
physical
model
Write a cql
script

19. © Copyright Allianz
Checklist
During Take-Off
Run everything up
4. Oktober 2016 19

20. © Copyright Allianz
But mention the difference
Start small
Add
nodes
Grow
up

21. © Copyright Allianz
Checklist
During Climb Out
Fill your speed-layer
4. Oktober 2016 21

22. © Copyright Allianz
Monitor the Instruments

23. © Copyright Allianz
Consume
• DataStream<String> messageStream =
• env.addSource(new FlinkKafkaConsumer09<>(parameterTool.getRequired("topicName"), new SimpleStringSchema(), properties));
Map
• DataStream<Tuple3<String,Date,Double>> clickMessageStream = messageStream.map(new ClickEventMapper());
Aggregate
• DataStream<Tuple2<Date,Double,String>> aggregatedClickMessageStream = clickMessageStream.map(new
KeyStreamMapper()).keyBy("f1").timeWindow(Time.minutes(2)).apply(new KeyWindowFunktion())
Store
• CassandraSink.addSink(clickMessageStream).setQuery("INSERT INTO itemssale_by_product (eventtime, price, product) values (?, ?,?);")
• .setClusterBuilder(new ClusterBuilder() {
• public Cluster buildCluster(Cluster.Builder builder) {
• return builder.addContactPoint(„csn-node1.development.allianz.de").build();
• }
• })
• .build();

24. © Copyright Allianz
Use the cassandra connector coming with Apache Flink since v. 1.1.0
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-cassandra_2.11</artifactId>
<version>1.1.1</version>
</dependency>
Write aggregated data

25. © Copyright Allianz
@Table(keyspace= "allianz", name = "itemssale_by_product")
public class MyCustomSalesEvent implements Serializable {
private static final long serialVersionUID = 1L;
@Column(name = "product")
private String product;
@Column(name = "eventdate")
private Date eventdate;
@Column(name = "price")
private double price;
//Getter and Setter
}
4. Oktober 2016 25
Write aggregated data

26. © Copyright Allianz
DataStream<MyCustomSalesEvent> clickMessageStream = messageStream.map(new
ClickEventMapper());
CassandraSink.addSink(clickMessageStream)
.setClusterBuilder(new ClusterBuilder() {
@Override
public Cluster buildCluster(Cluster.Builder builder) {
return builder.addContactPoint(„csn-node1.development.allianz.de").build();
}
})
.build();
4. Oktober 2016 26
Write aggregated data

27. © Copyright Allianz
Checklist
At 10.000 Feet
Make it safe and fancy
4. Oktober 2016 27

28. © Copyright Allianz 4. Oktober 2016 28
Privacy
WebApplication
Reports
Dashboards
R-Scripts

29. © Copyright Allianz 4. Oktober 2016 29
Single gateway to the data
AdHoc Queries
Proof of Thesis
Quick Lookups
Periodic
Reports
Web-based
Dashboard
3rd Partie
Reportings
Expert
Systems

30. © Copyright Allianz 4. Oktober 2016 30
Encryption
DC 1
Node
1
Node
3
Node
5
DC 2
Node
0
Node
4
Node
2

31. © Copyright Allianz
server_encryption_options:
internode_encryption: all
keystore: nasmount/conf/keystore.node0
keystore_password: changeme
truststore: nasmount/conf/truststore.node0
truststore_password: changeme
require_client_auth: true
4. Oktober 2016 31
Encryption – Just easy to enable
all
none
dc: Cassandra encrypts the traffic between the data centers.
rack: Cassandra encrypts the traffic between the racks.

32. © Copyright Allianz
CREATE TABLE zzz …
with compression_parameters:sstable_compression = 'Encryptor'
... and compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding'
... and compression_parameters:secret_key_strength = 128;
4. Oktober 2016 32
Encryption – With DSE

33. © Copyright Allianz
• Zeppelin
Is ok as developer or data scientist tool
Not suitable for C-Level reports
• MicroStrategy
Only support of Cassandra 2.x
Needs write permissions for the Column family (?)
• Tablaeu
Access Cassandra via Spark (?)
4. Oktober 2016 33
Hard to find a visualization solution

34. © Copyright Allianz
• D3.js
Is great to visualize and has stunning features
Needs an AngularJS developer to create a new report
• R
Provides simple visualization
Needs knowledge in R
4. Oktober 2016 34
Hard to find a visualization solution

35. © Copyright Allianz
CREATE ROLE flink;
CREATE ROLE productsales;
CREATE ROLE riskanalyst;
GRANT SELECT ON allianz.solditems TO productsales;
GRANT SELECT ON allianz.riskdata TO riskanalyst;
GRANT MODIFY ON KEYSPACE allianz TO flink;
4. Oktober 2016 35
Limit read / write access

36. © Copyright Allianz
The maximum period to store some detailed information is limited by law
We have to ensure that me meet this requirement
TTL in cassandra does this job well
INSERT INTO proposal (id,date,product,price) VALUES (‘p-4711’, ‘09.09.2016’,’product-1’,50.00);
UPDATE proposal USING TTL 86400 SET firstname = ‘Joe’ WHERE id = ‘p-4711’;
UPDATE proposal USING TTL 86400 SET lastname = ‘Doe’ WHERE id = ‘p-4711’;
UPDATE proposal USING TTL 172800 SET city = ‘Berlin’ WHERE id = ‘p-4711’;
4. Oktober 2016 36
Remove outdated events

37. © Copyright Allianz
Checklist
At cruising altitude
Work with it
4. Oktober 2016 37

38. © Copyright Allianz
Circle of data
Meet the
experts
Extract and
Enrich data
Aggregate
data
Analyse
the data
Visualize
Test
Hypothesis
Discuss
Actions

39. © Copyright Allianz 4. Oktober 2016 39
Recalculate the
Speed-Layer
WebApplication

40. © Copyright Allianz
#Load RJDBC
library(RJDBC)
#Load in the Cassandra-JDBC diver
cassdrv <- JDBC("org.apache.cassandra.cql.jdbc.CassandraDriver",
list.files(„/opt/cassandra/lib/",pattern="jar$",full.names=T))
#Connect to Cassandra node and Keyspace
casscon <- dbConnect(cassdrv, "jdbc:cassandra://localhost:9160/allianz")
4. Oktober 2016 40
Bring the Data to R

41. © Copyright Allianz
#Query timeseries data
res <- dbGetQuery(casscon, "select * from solditems limit 10")
#Transpose
tres <- t(res[2:10])
#Plot
boxplot(tres,names=res$KEY,col=topo.colors(length(res$KEY)))
title("BoxPlot of 10 Sold Items prices Historie")
4. Oktober 2016 41
Bring the Data to R