How we broke Apache Ignite by adding persistence
Report
Stephen DarlingtonFollow
Jan. 3, 2020•0 likes•90 views
How we broke Apache Ignite by adding persistence
Jan. 3, 2020•0 likes•90 views
Download to read offline
Report
Software
Apache Ignite™ is a rapidly changing platform: if you were to look at 3 years ago, you would see a completely different product. In this talk, we will follow the path that led Apache Ignite™ from a compute grid and data grid product to a distributed database and In-memory computing platform. We will examine technical tasks and decisions that were driving the transformations (as an example - how we added native persistence to Apache Ignite™) and will wrap up the talk with the outstanding problems that are going to be solved for Apache Ignite™.
Stephen DarlingtonFollow
Recommended
Apache Spark and Apache Ignite: Where Fast Data Meets the IoTDenis Magda
Apache Ignite: In-Memory Hammer for Your Data Science ToolkitDenis Magda
In-Memory Computing Essentials for Architects and EngineersDenis Magda
Google Cloud Platform as a Backend Solution for your ProductSergey Smetanin
Google Cloud Platform Data StorageJoseph Holbrook, Chief Learning Officer (CLO)
Apache Ignite - Distributed SQL Database CapabilitiesDenis Magda
More Related Content
What's hot
20210427 azure lille_meetup_azure_data_stackAlexandre BERGERE
Google Cloud Storage | Google Cloud Platform Tutorial | Google Cloud Architec...Edureka!
AWS Summit Singapore 2019 | Snowflake: Your Data. No LimitsAWS Summits
GCP Cloud Storage SecurityJoseph Holbrook, Chief Learning Officer (CLO)
node.js on Google Compute EngineArun Nagarajan
Understanding cloud with Google Cloud PlatformDr. Ketan Parmar
![code lab live Google Cloud Endpoints [DevFest 2015 Bari]](https://cdn.slidesharecdn.com/ss_thumbnails/nickcodelablivegooglecloudendpoints-150509100715-lva1-app6891-thumbnail.jpg?width=2048&height=1162&fit=bounds)
Similar to How we broke Apache Ignite by adding persistence
OSDC 2017 - Christos Erotocritou - Apache ignite in-memory data fabricNETWAYS
On Cloud Nine: How to be happy migrating your in-memory computing platform to...Stephen Darlington
In-Memory Computing Essentials for Software EngineersDenis Magda
Data Summer Conf 2018, “Apache Ignite + Apache Spark RDDs and DataFrames inte...Provectus
“Building consistent and highly available distributed systems with Apache Ign...Tom Diederich
An Introduction to Apache Ignite - Mandhir Gidda - Codemotion Rome 2017Codemotion
Recently uploaded
How to fix JavaScript SEO problems on a scaleSerge Bezborodov
La strada verso il successo con i database a grafo, la Graph Data Science e l...Neo4j
Top Benefits of Web based Systems. sarasiva4
Transform Your Digital Reality with Metaverse Development SolutionsFlexsin
Kubernetes with Cilium in AWS - Experience Report!QAware GmbH
Advanced Object- Oriented Design MoocESUG
How we broke Apache Ignite by adding persistence
- 1. How we broke Apache Ignite by adding persistence Stephen Darlington 16 December, 2019 2018 © GridGain Systems
- 2. 2019 © GridGain Systems GridGain Company Confidential2 (spoiler: already fixed)
- 3. 2019 © GridGain Systems GridGain Company Confidential What is Ignite? Distributed memory-centric storage Combines the performance and scale of in- memory computing together with the disk durability and strong consistency in one system Co-located Computations Brings the computations to the servers where the data actually resides, eliminating need to move data over the network Distributed Key-Value Read, write and transact with fast key-value APIs Distributed SQL ACID Transactions Machine and Deep Learning Horizontally, fault-tolerant distributed SQL database that treats memory and disk as active storage tiers Supports distributed ACID transactions for key-value as well as SQL operations Set of simple, scalable and efficient tools that allow building predictive machine learning models without costly data transfers (ETL)
- 4. 2019 © GridGain Systems GridGain Company Confidential Apache Ignite In-Memory Computing Platform Mainframe NoSQL HadoopIgnite Persistence Persistent Layer RDBMS Machine and Deep Learning EventsStreamingMessaging Transactio ns SQLKey-Value Service GridCompute Grid Application Layer Web SaaS SocialMobile IoT In-Memory Data Store
- 5. 2019 © GridGain Systems GridGain Company Confidential Apache Ignite’s History 5 Data Grid Local Store Transactional Persistence ?
- 6. 2019 © GridGain Systems GridGain Company Confidential Circa 2011 6 Local Store Transactional Persistence ?Data Grid
- 7. 2019 © GridGain Systems GridGain Company Confidential Circa 2014 7 Transactional Persistence ?Data Grid Local Store
- 8. 2019 © GridGain Systems GridGain Company Confidential Circa 2017 • Start time does not depend on the data volume • Can store more data than memory • Crash recovery • Single in-memory & native persistence architecture
- 9. 2019 © GridGain Systems GridGain Company Confidential Ignite 2.0: What we wanted “We will just save everything to disk”
- 10. 2019 © GridGain Systems GridGain Company Confidential Circa 2017 10 ?Data Grid Local Store Transactional Persistence
- 11. 2019 © GridGain Systems GridGain Company Confidential Beginning: Durable Memory
- 12. 2019 © GridGain Systems GridGain Company Confidential Beginning: Durable Memory • ARIES Architecture • Page-based • Write-ahead log (when persistence is enabled) • Everything is off heap
- 13. 2019 © GridGain Systems GridGain Company Confidential Beginning: Durable Memory • PK Index: how to replace a HashMap • Concurrent B+ Tree: a well-known data structure • Separate PK Index per each partition • Compare key hash first • Bonus: guaranteed iteration order in a hash map
- 14. 2019 © GridGain Systems GridGain Company Confidential Baseline Topology 16 • [16:21:01] Ignite node started OK (id=326bab44) • [16:21:01] >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate. • [16:21:01] Topology snapshot [ver=1, locNode=326bab44, servers=1, clients=0, state=INACTIVE, CPUs=8, offheap=3.2GB, heap=3.6GB] • [16:21:01] ^-- Baseline [id=11, size=3, online=1, offline=2] • [16:21:01] ^-- 2 nodes left for auto-activation [6213b7af-23bb-4c8d-a045-157d7f2d7718, db969788- fc01-41f4-a91c-c03f2d201f76] • [16:21:19] Joining node doesn't have encryption data [node=89b6ef6c-1055-4678-bcfa-00fb222208ce] • [16:21:19] Topology snapshot [ver=2, locNode=326bab44, servers=2, clients=0, state=INACTIVE, CPUs=8, offheap=6.4GB, heap=7.1GB] • [16:21:19] ^-- Baseline [id=11, size=3, online=2, offline=1] • [16:21:19] ^-- 1 nodes left for auto-activation [6213b7af-23bb-4c8d-a045-157d7f2d7718] • [16:21:37] Joining node doesn't have encryption data [node=dd55ff24-da61-42cd-bbaf-c7940fab07d3] • [16:21:37] Topology snapshot [ver=3, locNode=326bab44, servers=3, clients=0, state=INACTIVE, CPUs=8, offheap=9.6GB, heap=11.0GB] • [16:21:37] ^-- Baseline [id=11, size=3, online=3, offline=0] • [16:21:37] ^-- All baseline nodes are online, will start auto-activation
- 15. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • Disks are slow (even NVMe) • At peak load naïve implementation steps on it’s tail easily • Sudden performance drops to 0
- 16. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • So, we need to… make Ignite slower • Throttle input load depending on • How fast we produce “dirty” pages • How fast we write to disk • How free the Copy-On-Write buffer is
- 17. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • Page cache: what can go wrong • We already have one-page cache in Ignite (durable memory) • OS-level page cache • Effectively doubles the memory consumption
- 18. 2019 © GridGain Systems GridGain Company Confidential Disk. Predictable access speed • Page cache: solution is Direct IO • Available in Java 10, but we build on Java 8 • Need to have native/platform specific calls or Java- dependent module
- 19. 2019 © GridGain Systems GridGain Company Confidential The future? 21 Data Grid Local Store Transactional Persistence ?
- 20. 2019 © GridGain Systems GridGain Company Confidential22 Questions?
- 21. 2019 © GridGain Systems GridGain Company Confidential More information 23 • Main landing page: https://ignite.apache.org • Documentation: https://apacheignite.readme.io/docs • Please complete our survey on how Apache Ignite should evolve: https://docs.google.com/forms/d/e/1FAIpQLSdUveEVXer3lpkyiqfFw4175T vZzGHUOS4snPfnkO0NDku0eQ/viewform • Realtime data loading: https://www.imcsummit.org/2019/us/session/best- practices-loading-real-time-data-distributed-systems-change-data-capture
- 22. 2019 © GridGain Systems GridGain Company Confidential24 Stephen Darlington Senior Consultant, GridGain Systems @sdarlington
Editor's Notes
- First, I’m not Alexey Second… we didn’t break it
- Alternate title: How and why Apache Ignite™ is Changing from an In-Memory Data Grid into an In-Memory Database. In this talk, we will follow the path that led Apache Ignite™ from a compute grid and data grid product to a distributed database and an in-memory computing platform. We will examine technical tasks and decisions that were driving the transformations. Step back: who knows about Ignite?
- Traditional databases don’t scale. Buy bigger and bigger boxes until you run out of money. Traditional compute grids have to copy data across the network, which at modern scale is just impractical. Ignite scales horizontally and sends compute to the data rather than the other way around. In memory for speed. Disk persistence for volume.
- Memory data grid Optionally backed by the “Cache Store” SQL works as fork-join – SQL executed on every node, results joined together Resuts correct results in limited use cases SQL only in-memory Restart means loading all data Centralised store – scalablity issues
- LocalStore designed to allow faster restarts Local Store is an append-only structure with periodical compactions Still need to load data from local store to memory (on datasets of 100s of GBs warmup could take 3+ hours) Keys are still residing in memory, so the data set size is limited SQL functionality is improved (two-step queries like plain ORDER BY, GROUP BY, some cases of distributed joins are implemented, but still there are restrictions, not all queries return correct results, a user must check collocation rules) So better but still missing stuff. What are our requirements for The Best system?
- Algorithms for Recovery and Isolation Exploiting Semantics Ignite 1.x had options for on- and off-heap storage. 2.x only off-heap Unlike LocalStore (and competition), it uses a WAL… like legacy databases. Crash recovery
- ABA problem -- thread synchronization (https://en.wikipedia.org/wiki/ABA_problem)
- We’ve mostly talked about the Good Stuff.. But there are complications What’s happening here? Baseline topology
- … and the discussion about Java 10 brings us to … the futur
- Improve SQL What’s wrong with H2? H2 is not a distributed database with a very simple planner. Ignite uses H2 internal APIs and hacks to integrate with H2. No query execution graph. A query is effectively executed based on AST with minimum transformations H2 planner does not know about distributed SQL, it is almost impossible to execute complex SQL queries (subqueries, complex aggregations, non-colocated tables) effectively No memory control Currently investigating using Apache Calcite as a query optimizer + custom execution engine. See Apache Developers list (+IEP) Improve Split-Brain handling out-of-the-box Require users to implement TopologyValidator Should work without users writing a single line of code Modularisation don’t download everything Dependencies for stuff like Spark / Kafka with different release schedules Even “internal” stuff like ML