Will Bleker and Gary Stewart of ING discuss pipelining data using Kafka and Neo4j Graph. They describe shifting from a traditional database approach of managing "trains" to a NoSQL approach of treating data as disposable "cattle". This allows rebuilding the data pipeline frequently for more agility. They provide an example architecture and lessons learned about simplifying operations and increasing resilience by treating data as immutable and designing systems to rebuild frequently.

1. Pipelining the Heroes
with Kafka and Graph
Will Bleker
Gary Stewart
Amsterdam, 12 November 2018

2. 2

3. Graph - Nodes and Relationships
3

4. Graph - Add properties
4

5. About us
5
EMPLOYED_BY
EMPLOYED_BY
Role: Chapter Lead
Role: Platform Architect
Gary Stewart
Will Bleker

6. Market leaders Benelux
Growth markets
Commercial Banking
Challengers
6
About ING
Over 40 countries
52,000+ employees

7. Concepts
7
• Concepts
• Pipeline
• Use case
• Learnings from NoSQL thinking

8. A little history …
8
Pets
• Unique and indispensable
• Often long lived
• Scale-up
• Active-passive architecture, pairs of servers, etc
Cattle
• Disposable, one of a herd
• Often short lived
• Scale-out
• Active-active architecture
• ‘Cattle’ datastores include
• Apache Cassandra
• Apache Kafka

9. Databases are much like the transport industry
• Undergone massive transformations
• Various types suitable for different needs
• Trains
• Relatively expensive
• Built on-demand
• Longer lifespan
• Cars
• Manual to mass production
• Almost consumable
• Often cheaper to rebuild, recycle than repair
Our twist – trains versus cars
9

10. Our journey to NoSQL
10
A few years back we started with Cassandra
• Exotic key/value store (column family)
• Distributed active-active architecture
• Tuneable consistency
• Paradigm shift
• Nights and weekends are now free (including LCM)
• However now a liquid expectation!
RDBMSNoSQL

11. Adding graph to the landscape
11
Gap in our DB landscape - connected data
• Another NoSQL database!
Interesting observation
• Key/Value easy to mess up a partition
• RDBMS easy to mess up a table
• Graph easy to mess up the database!
?
RDBMSNoSQL Graph

12. Typical architecture for caching data
12
SOR
Challenges include
• Initial load is usually once-off or manual
• Lifecycle management (LCM) is hard and per component
• Resilience patterns need to be applied everywhere
• Cross DC resilience pattern also required
Real-Time
Sync
Batch
Loader
Cache
Cache

13. Cache ‘Cattle’ Paradigm
13
SOR
Real-Time
Sync
Single
Materialized ViewCommit Log
Challenges include
• Initial load Integrated by design
• Lifecycle management (LCM) Simplified
• Resilience Simplified
• Cross DC Simplified
Batch
Loader

14. Cache ‘Cattle’ Paradigm – Taking it further
14
SOR
Real-Time
Sync
Single
Materialized ViewCommit Log
E.g. Apache Kafka supports compacting topics
• Ideal for rebuilding cache use cases
Batch
Loader

15. Cache ‘Cattle’ Paradigm – Going all the way
15
Real-Time
Sync
Single
Materialized ViewCommit Log
Batch
Loader
SOR

16. Pipeline
16
• Concepts
• Pipeline
• Use case
• Learnings from NoSQL thinking

17. Pipelining the ‘Materialized View’
17
Real-Time
Sync
Batch
Loader
Materialized View
SOR

18. Block ports
Get stuff
• Artifacts
Pre Processing
• Convert data
• neo4j-admin import
Start Neo4j
Post processing
Create Indexes
Complex calculations
Start applications in desired order
Unblock ports
Switch - Opening for client requests
Pipelining the ‘Materialized View’
18
Real-Time
Sync
Batch
Loader
Materialized View
SOR

19. Neo4j-admin import
19

20. Neo4j-admin import
20
IMPORT DONE in 4m 17s 923ms. Imported:
11 759 853 nodes
11 774 714 relationships
35 332 411 properties
IMPORT DONE in 1m 41s 941ms.Imported:
26 444 678 nodes
26 444 020 relationships
66 129 151 properties

21. Use case
21
• Concepts
• Pipeline
• Use case
• Learnings from NoSQL thinking

22. • Customer Teams
• Platform Teams
• Components
Challenges
• Rapid growth
• Too many dependencies
• Too many changes
• Often converging
• Too much data (metrics)
• Heroes
• …
Change is needed!
Managing our Cassandra Platform
22

23. Observation
• Ops person ‘pulls’ information when needed (requires logging on)
• Full shifting left is particularly challenging for multi-tenant platforms
Pipeline everything
• We don’t value initial bursts of energy anymore
• Pipeline the heroes J
• Design to rebuild
‘Personal’ goal
• Learn Graph technology
Understanding the challenges ahead
23

24. Architecture
24
• Simple and effective
• Executes ‘Ops’ commands:
• ps –ef
• netstat
• cat config.json
• Raw output sent as message
• All parsing rules for raw messages
• Store in databases
• Complex rules and analysis

25. Architecture
Commit LogProducer Materialized view
Consumer
App
25

26. Consuming message (data pipeline)
• Parse
• Store as Key/Value
• Calculate
• Observe
• Recommend
Principles
• Compare to inventory not ‘self discovery’
• Define agreements in code
• Design to be idempotent as possible
• No data migrations allowed (simply rebuild!)
Consumer - Graph Model
26

27. Graph proven to be a VERY good fit for this use case!
• Already producing 50+ commands
• Heroes have a place to ‘query’
• Various unexpected findings
• Misconfigured tenants
• 100+ recommendations made
Architecture now helps with
• Allow us to be ‘in’ control without ‘having’ control
• Remaining agile
• Shifted left our operational analysis
300+ cars were destroyed in our 8 month journey to learn graph!
So what happened?
27

28. Learnings from
NoSQL thinking
28
• Concepts
• Pipeline
• Use case
• Learnings from NoSQL thinking

29. Parse each message into key/value
WITH {a: "val 1", b: "val 2", c: "val 3"} AS data
CREATE (n:MyNode)
SET n += data;
Added 1 label, created 1 node, set 3 properties …
Benefits
• Easier to build insights with existing data
• ‘Raw’ data remains unchanged
• However if parsing rules change
Modify code
à Simply rebuild and …
Breaking it down to Key/Value
29

30. 30

31. Simple model
• (:Source)-[:HAS_MSG]->(:Message)
Lots of nodes
31

32. Consider traversing to reduce density
• (:Source)-[:LAST]->(:Message)->[:PREV]->(:Message)
Modify code
à Simply rebuild and J
Partitioning in graph
32

33. 33

34. MATCH …
OPTIONAL MATCH
WHERE …
MERGE
MATCH …
WHERE …
OPTIONAL MATCH
MERGE
Modify code
à Simply rebuild and J
Attempted to migrate data anyway
34

35. 35

36. Kafka is not just a ‘messaging’ product
Actually it is a distributed commit log!
Like Graph we learnt Kafka too!!
What about Kafka?
36

37. Conclusions
37

38. Summary
38
Shifting left
• Requires initial extra effort
• Design for rebuild instead of migrations
• Creates a learning environment for new technology
Ensures agility
• Adding new features
• Implement learnings quickly
• Easier to experiment
Ensures control
• Easier to secure
• Data quality actually improves due to frequent rebuilds
• Safer and cheaper to build systems to not last long

39. Pipeline the heroes
• Create a data pipeline
• Automate data collection
• Automate analysis
Shift left your operational analysis!
In other words …
39

40. Thank you
Wilhelmus.Bleker@ing.com - @WBleker
Gary.Stewart@ing.com - @Gaz_GandA
We are hiring!
40

41. Follow us to stay a step ahead
ING.com
YouTube.com/ING
SlideShare.net/ING@ING_News LinkedIn.com/company/ING
Flickr.com/INGGroupFacebook.com/ING

42. ING Group’s Annual Accounts are prepared in accordance with
International Financial Reporting Standards as adopted by the
European Union (‘IFRS-EU’).
In preparing the financial information in this document, the same
accounting principles are applied as in the 2014 ING Group Annual
Accounts. All figures in this document are unaudited. Small
differences are possible in the tables due to rounding.
Certain of the statements contained herein are not historical facts,
including, without limitation, certain statements made of future
expectations and other forward-looking statements that are based
on management’s current views and assumptions and involve
known and unknown risks and uncertainties that could cause actual
results, performance or events to differ materially from those
expressed or implied in such statements. Actual results,
performance or events may differ materially from those in such
statements due to, without limitation: (1) changes in general
economic conditions, in particular economic conditions in ING’s core
markets, (2) changes in performance of financial markets, including
developing markets, (3) consequences of a potential (partial) break-
up of the euro, (4) the implementation of ING’s restructuring plan to
separate banking and insurance operations, (5) changes in the
availability of, and costs associated with, sources of liquidity such as
interbank funding, as well as conditions in the credit markets
generally, including changes in borrower and counterparty
creditworthiness, (6) the frequency and severity of insured loss
events, (7) changes affecting mortality and
morbidity levels and trends,(8) changes affecting persistency levels,
(9) changes affecting interest rate levels, (10) changes affecting
currency exchange rates, (11) changes in investor, customer and
policyholder behaviour, (12) changes in general competitive factors,
(13) changes in laws and regulations, (14) changes in the policies of
governments and/or regulatory authorities, (15) conclusions with
regard to purchase accounting assumptions and methodologies,
(16) changes in ownership that could affect the future availability to
us of net operating loss, net capital and built-in loss carry forwards,
(17) changes in credit ratings, (18) ING’s ability to achieve projected
operational synergies and (19) the other risks and uncertainties
detailed in the Risk Factors section contained in the most recent
annual report of ING Groep N.V. Any forward-looking statements
made by or on behalf of ING speak only as of the date they are
made, and, ING assumes no obligation to publicly update or revise
any forward-looking statements, whether as a result of new
information or for any other reason.
This document does not constitute an offer to sell, or a solicitation
of an offer to purchase, any securities in the United States or any
other jurisdiction. The securities of NN Group have not been and will
not be registered under the U.S. Securities Act of 1933, as amended
(the “Securities Act”), and may not be offered or sold within the
United States absent registration or an applicable exemption from
the registration requirements of the Securities Act.
www.ing.com
Disclaimer
42