Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

In memory grids IMDG


Published on

Published in: Technology, Business
  • Be the first to comment

In memory grids IMDG

  1. 1. 100K times faster apps.In Memory GridsPrateek Jain
  2. 2. Agenda• In Memory Grids– 10,000 foot view.– Present scenario– Why– Why now• Use Cases• Types of In Memory Grids– Compute Grid– Data Grid• Reference Architecture• Sample application demo• Further Resources• Questions & Feedback
  3. 3. In Memory Grids• 10,000 foot viewBreaking your problem to solve it using multiple resources on network.Using main memory instead of Disk to do file I/O.
  4. 4. BigData landscapeTraditional App Associated ChallengesRDBMS -- Used to run many analyticssystemsPerformance ( Not Real time), Scaling,Cost++CEP -- Designed to correlate data in realtimeScaling (often necessary to aggregateevents into a centralized source ), notdesigned for historical data.Hadoop -- Designed for batch analytics andcomplex correlationNot designed for Real time.NoSQL -- Designed to handle large datavolumes at low costProcessing capability: Sheer amount ofdata can be challenging.IMDG -- Fast for storing and processingdataStoring vast amounts of information in-memory doesn’t scale, in terms of bothsystem scaling and costDifferent problems, so are the solutions.
  5. 5. Why ?• Speed matters– Citi : 100ms == $1 M– Google : 500ms == 20% traffic drop• Disk up to 107 times slower than RAM.
  6. 6. In Memory Grids• Why now?– Hardware, ability++ and cost--• 1TB RAM & 48 core cluster (can hold full week tweets) ~ $40KData Growth, PB DRAM Cost, $BigData tech. plannedData is growing exponentially 30% drop each 12-18 months
  7. 7. Use Cases• Trading Systems– Handle large volume of transactions• Real time risk analytics– Analysis of trading positions and risk• Online gaming– Online real-time backbone for gaming• Geo Mapping– Real-time geographical route and traffic information• Bio Informatics– Real-time DNA sequencing and matching
  8. 8. In Memory Compute Grid(IMCG)
  9. 9. In Memory Grids1. In Memory -- Compute Grid.Compute Grids allow you to take a computation, optionally split it into multiple parts, andexecute them on different grid nodes in parallel.
  10. 10. Functionality• Distributed Execution Models - map-reduce, StreamingProcessing & CEP, MPP, MPI style• Distributed Execution Management Services – taskdistribution, failover, load balancing, collision resolution,job stealing, redundant mapping support, task scheduling,asynchronous reduction, task checkpoints• Distributed Deployment & Provisioning.• Distributed Resources Management - Automatic discovery
  11. 11. In Memory Data Grid(IMDG)
  12. 12. In Memory Grids2. In Memory -- Data Grid. (aka, Distributed data caching )Provides applications with ability to keep data in memory for high availability rather thanconstantly fetching it from slower storage elsewhere, like RDBMS or shared file systems.
  13. 13. IMDG ?• Several JVMs sharing in-memory partitioned data.• Provides extremely low latency access to,and high availability of, application data by keeping it inmemory and to do so in a highly parallelized way.• Support most of the Big Data processing requirements.
  14. 14. Common Features• Distributed maps• Caching , Evictions• Code execution (executor service, map-reduce)• Listeners• Queries (SQL like)• Pluggable indexing• Hibernate L 2 cache (optional)• ACID Transactions• MapStore (write-behind, write-through, read-through)• Optimized Serialization
  15. 15. Common Features• The same object your business logic is using can be kept in the data grid.• No extra step of marshaling and un-marshaling.• Embeddable (optional)
  16. 16. Reference Architecture
  17. 17. IMDG is not a• NoSQL database• In Memory Database (IMDB)• How?• Support for true distributed ACID transactions with highly optimized 2PC protocol implementation.• Scalable Data Partitioning across a cluster including both partitioned or fully replicated scenarios• Ability to work directly with application domain objects rather than with primitive types or “documents”• Tight integration with In-Memory Compute Grid (IMCG)• Pluggable segmentation (a.k.a. "brain split" problem) resolution• Pluggable expiration policies• Pluggable indexing support
  18. 18. Further Reading•••••••