How Apache Ignite adds speed, scale & agility to databases, Hadoop & analytics. By Glenn Wiebe, GridGain Systems senior solution architect

1. IT Modernization in Practice
How Apache Ignite adds speed, scale & agility to databases,
Hadoop & analytics.
Glenn Wiebe
March 2019
2019 © GridGain Systems
Sr. Solution Architect
Phil Hunt
Account Executive

2. 2019 © GridGain Systems GridGain Company Confidential
Agenda
• The Memory Centric solution to IT Modernization
• 4 Modernization Use Cases
– Existing Databases & Applications
– Real-time & Streaming Analytics
– Low Latency Hadoop Performance
– Machine & Deep Learning
• Demo

3. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
10-100x
Queries and
Transactions
50x
Data Storage
(Big Data)
10-1000x
Faster Analytics
(Hours to Sec)
Application Layer
Web SaaS SocialMobile IoT
Mainframe NoSQL Hadoop
Data Layer
RDBMS
The Memory Centric Solution to IT Modernization
Public Sector Challenges in the Last Decade

4. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
OLAP and OLTP Converge - The Emergence of HTAP
Hybrid Analytical/Transactional Processing
(HTAP)
Application Layer
Web SaaS SocialMobile IoT
Mainframe NoSQL Hadoop
Data Layer
RDBMS

5. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
In-Memory Computing
In-Memory Computing
Application Layer
Web SaaS SocialMobile IoT
Mainframe NoSQL Hadoop
Data Layer
RDBMS

6. 2019 © GridGain Systems GridGain Company Confidential
Apache Ignite – Top 5 overall of Apache Top Line Projects
– Now at ~2 Million Downloads per Year
5
Top 5 Developer
Mailing Lists
Top 5 User
Mailing Lists
• Ignite
• Kafka
• Tomcat
• Beam
• James
• Lucene-Solr
• Ignite
• Flink
• Kafka
• Cassandra
Top 5 in Commits
last two years
• Hadoop
• Ambari
• Camel
• Ignite
• Beam

7. 2019 © GridGain Systems GridGain Company Confidential
Typical Implementations/Use Cases
• New Digital Transformation
– FRTB - xVA/CVA and compliance
– High speed trading, fraud, anti-money laundering
– Geospatial/Image Processing
– Real time analytics (HTAP) and risk analytics
– Real time cybersecurity and attack prevention
– Hadoop/data lake acceleration (Fast data layer/stream processing for data mart’s & reporting)
– IoT
• IT Modernization
– Data center consolidation
– Database and web acceleration (database scaling)
– Mainframe offload
– Basic caching
Relativecomplexity

8. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
9
• Existing Databases & Applications
• Real-time & Streaming Analytics
• Low Latency Hadoop/Data Lake Performance
• Machine & Deep Learning

9. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
Adding Speed & Scale to Existing Databases
11
Ignite as an In-Memory Data Grid (IMDG)
• Slides in-between apps and RDBMSs
with no rip and replace
– ANSI-99 SQL compliant
– Support for ACID transactions
• Accelerates existing app performance
• Offload new data and computing
requirements (real-time auditing
and compliance, analytics, computations)
In-Memory
Database
Streaming
Analytics
Continuous
Learning Framework
In-Memory
Data Grid
Compute and Service Grid
ACID TransactionsANSI-99 SQLKey-Value
In-Memory Data Store
Mainframe NoSQL Hadoop
Data Layer
RDBMS

10. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
Adding Speed & Scale to Existing Databases – cont.
12
Ignite as an In-Memory Database (IMDB)
• Memory-centric storage
– From 100% in-memory to 100% disk
– Leverages any combination of RAM,
Flash, SSD, Intel 3D Xpoint and disk
– Low cost, disk-based reliable persistence
– Immediate restart during recovery
• Highest read+write performance
– In-memory with unlimited linear,
scale-out on commodity servers
– SQL and NoSQL (multi-model)
– Always-on availability
• Single data access layer for ALL data
• Extensible compute grid
In-Memory
Data Grid
Streaming
Analytics
Continuous
Learning Framework
In-Memory
Database
Persistent Store
Compute and Service Grid
ACID TransactionsANSI-99 SQLKey-Value
In-Memory Data Store
Mainframe NoSQL HadoopRDBMS GridGain
Data Layer

11. 2019 © GridGain Systems GridGain Company Confidential
ING – Next-Generation Banking
13
• Problem
– To deliver new competitive customer services fast
– High cost of running on mainframe infrastructure
– Transaction consistency over multiple geo-locations
• GridGain Solution
– Powers core solution for delivering new services
– Aggregates data for APIs across multiple sources
– Supports 25% annual growth in mobile traffic
– Reduced end-to-end latency to below 100ms
– Helped ING be first to market for PSD2, SEPA, STET
Dutch Multinational Banking and Financial
Services Firm Headquartered in Amsterdam
Front-End APIs
Payments SecuritiesAccounts Credits Clients
GridGain In-Memory Computing Platform
In-Memory
Data Grid
In-Memory
Database
Streaming
Analytics
Continuous
Learning Framework
Mainframe Cassandra
Multi-Datacenter Infrastructure
RDBMS

13. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
Performing Real-time & Streaming Analytics
16
Ignite for Stream Ingestion, Processing and Analytics
• Native support for stream ingestion
– Built-in support for high speed ingestion
from Apache Camel, Flink, Flume, Spark,
Storm, JMS, Kafka and MQTT
– Combines streams with data-at-rest
– Co-located data processing across all data,
including optimized SQL querying
• Continuous Queries
– Subscribe queries to cache changes
• Broadest in-memory support for Apache Spark
– Native in-memory RDD, DataFrame support
– Shares state in memory across Spark jobs
– Native access to ANY data across Ignite cluster
– Optimizes SparkSQL using distributed SQL and indexing
In-Memory
Data Grid
In-Memory
Database
Continuous
Learning Framework
Streaming
Analytics
Persistent Store
Compute and Service Grid
EventsStream ProcessingMessaging
In-Memory Data Store
ACID TransactionsANSI-99 SQLKey-Value
Mainframe NoSQL HadoopGridGain
Data Layer
RDBMS

14. 2019 © GridGain Systems GridGain Company Confidential
Ignite for Spark
Broadest In-Memory Support for Apache Spark
17

15. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
Streaming: American Express
Payment Processing Modernization
18
Leading multinational financial services company
with nearly 60M cardholders worldwide
• Problem
– Reduce time to pay merchants, from days to hours
– Required migration from mainframe to more
modern scalable and scalable architecture
• Ignite Solution
– Offered unified API to bridge disparate technologies
– Enabled a multi-step migration effort for lagging
applications – add new nodes for non-grid aware
applications as they become ready for migration
– Increased performance on batch jobs for
reconciliation for Merchant Payment
PDSPDS PDS
VSAM
Cobol
App
Java
App
Client
JCICS API
JCICS API
Ignite API
Ignite
Streaming
API
Use for
Disaster
Recovery
DB2

16. 2019 © GridGain Systems GridGain Company Confidential
Wellington - Next Generation, Real-time IBOR
A top 20 worldwide asset management firm
with over $1 trillion under management
• Problem
– Current systems no longer scaled to handle the volumes
– Didn’t comply with new regulations following financial crisis
– Needed to introduce new asset classes faster
• GridGain Solution
– Investment Book of Record (IBOR), a single real-time
version of the truth for positions, exposure, valuations
and performance for all customers, teams and trades,
Streamed in real-time.
– 10x performance gains, linear horizontal scalability
– Support for SQL and ACID transactions, and for
existing systems and skillsets
– Enabled transactions and analytics on a single platform
– Co-located computing scales complex calculations, analytics
Trading
Systems
GridGain In-Memory Computing Platform
In-Memory
Data Grid
In-Memory
Database
Streaming
Analytics
Continuous
Learning Framework
Accounting
System
Other
Back Office
Portfolio
Management
Risk
Management
Regulatory &
Compliance
Investment
Book of
Record (IBOR)
Oracle RAC

17. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
Boosting Hadoop Performance for Low Latency SQL Queries
20

18. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
Enhancing Machine & Deep Learning
21
Continuous Learning Framework for
Machine and Deep Learning
• Real-time performance on petabytes of data
– No ETL (runs learning in place)
– In-memory performance
– Horizontal, linear scalability
• Machine learning
– Linear, multi-linear regression
– K-means clustering
– Decision trees
– K-NN classification and regression
• Deep Learning
– TensorFlow integration
Machine and Deep Learning
In-Memory
Data Grid
In-Memory
Database
Streaming
Analytics
Continuous
Learning Framework
Persistent Store
Compute and Service Grid
EventsStream ProcessingMessaging
In-Memory Data Store
ACID TransactionsANSI-99 SQLKey-Value
Mainframe NoSQL HadoopGridGain
Data Layer
RDBMS

19. 2019 © GridGain Systems GridGain Company Confidential
4 Modernization Use Cases
Enhancing Machine & Deep Learning
22

20. 2019 © GridGain Systems GridGain Company Confidential
Hadoop Acceleration with ML – Federal Department
Slow Analytics from Data Lake
23
• Problems
– Query and reporting times for fraud
analytics too slow due to slow Hadoop
(HIVE) performance
– Desire to modernize database (DB2)
– New need for Machine Learning
• Ignite Solution
– In-memory computing for fraud
analytics that eliminated performance
bottlenecks
– Supports future machine learning
needs
Web Portal
GridGain In-Memory Computing Platform
In-Memory
Data Grid
In-Memory
Database
Streaming
Analytics
Continuous
Learning Framework
Data Infrastructure
IBM DB2 Hortonworks
ETL
Data Load
Analytics

21. 2019 © GridGain Systems GridGain Company Confidential
RBC Article – January, 2016
“The new Sberbank IT plan is to create a platform that enables
the bank to introduce new products in hours, not weeks. The
platform will have virtually unlimited performance and very high
reliability. It will be much cheaper and will significantly reduce
human interaction during customer transactions. The system
will use machine-learning, flexible pricing, and artificial
intelligence,” said German Gref, head of Sberbank.
“The new system will use technology from GridGain, which
won the tender from Oracle, IBM and others, and turned out to
deliver an order of magnitude higher performance than those
of the largest companies,” he added.
German Gref
CEO & Chairman
Sberbank

22. 2019 © GridGain Systems GridGain Company Confidential
DEMO
25