Battery Framework

734 views

Published on

eXtremene Transaction Processing Framework description

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
734
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Battery Framework

  1. 1. © 2009 Grid Dynamics — Scaling Mission Critical Systems<br />Battery Frameworkextreme transaction processing<br />Eugene Steinberg<br />CTO<br />
  2. 2. Extreme transaction processing<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />2<br />
  3. 3. XTP: reducing latency Strategies<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />3<br />
  4. 4. Data management approaches<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />4<br />
  5. 5. MANAGING DATA IN RAM: Products<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />5<br />RDBMS features<br />Both products support:<br />Reliability (via redundancy)<br />Failover<br />Data durability (via async persistence)<br />Oracle TimesTen<br />SQL and analytic queries<br />JDBC<br />Oracle Coherence<br />Selects, filters, aggregators<br />SMP<br />Cluster<br />Grid<br />Horizontal scalability<br />
  6. 6. XTP: scalability and reliability<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />6<br />
  7. 7. High level architecture<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />7<br /><ul><li> Data management in RAM
  8. 8. Data partitioning
  9. 9. Load balancing
  10. 10. In-place data processing
  11. 11. Asynchronous persistence
  12. 12. Horizontal scalability</li></ul>RDBMS<br />
  13. 13. Physics of distributed DATA<br />CAP theorem (Eric Brewer, 2000)<br />You may have at most two of Consistency, Availability, and Partition Tolerancein any shared-data system<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />8<br />C<br />Data grids<br />Quorum-based consistency<br />P<br />A<br />Eventually consistent databases<br />
  14. 14. In Memory data grid technology<br />Storing and managing data in main memory (RAM)<br />Automated data partitioning<br />Automated clustering and dynamic node membership<br />Built-in data redundancy<br />Strong data consistency and availability<br />Multiple ways to access data: Map interface, Data Filters, Queries, Template queries, etc…<br />Designed to scale out<br />Proven and reliable technology (10 years on market, major players)<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />9<br />
  15. 15. IMDG benefits for xtp<br />Fastest possible access to data (in-memory)<br />Optimized network interactions (peer-to-peer)<br />Built-in load balancing<br />Consistent data management<br />Reliable in-memory transactions <br />Transparent scale out<br />Continuity of operations (redundancy of data)<br />Automated failover and re-balancing<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />10<br />
  16. 16. COHERENCE PLATFORM<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />11<br />
  17. 17. STAGED EVENT DRIVEN ARCHITECTURE<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />12<br />
  18. 18. PROCESSING UNITS<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />13<br />Generic<br />Generic processing unit, base for all hierarchy<br /> Transformer<br />Consumes inbound message and produces outbound message <br />Router<br />Routes inbound message to one of several outbound queues<br />Duplicates inbound message to several outbound queues<br />Fork<br />Consumes several inbound messages and produces outbound message. Synchronization point<br />Junction<br />
  19. 19. processing network<br />Application model: network of processing units interconnected with queues<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />14<br />Fraud detection<br />Payment<br />Validation<br />Receipt<br />Processing<br />Order<br />service<br />Validated<br />Execution<br />Shipping<br />Product availability<br />Failed order<br />Legend<br />Junction<br />Transformator<br />Generic<br />Fork<br />Router<br />
  20. 20. Bringing processing closer to data<br />IMDG + SEDA forms uniform processing fabric<br />Data and algorithms are in the same processes.<br />Leveraging data affinity and data aware routing allows to bring algorithms close to the data<br />Same server or even same process algorithm execution<br />reduce latency,<br />reduce network traffic,<br />reduce CPU utilization (no encoding/decoding of binary data format)<br />We do not need to change business logic to achieve this!<br />Done on distributed system configuration level.<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />15<br />
  21. 21. Managing transactions<br />Low latency distributed transactions are challenging<br />What is provided by our basic technologies:<br />IMDG provides as fast and reliable in-memory data transactions<br />SEDA simplifies modeling of complex business transactions<br />Strategies to reduce transaction latency without compromising consistency:<br />optimistic locking(multi version concurrency control)<br />coalescence of transactions,<br />exploiting idempotent operations.<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />16<br />
  22. 22. Maintainability and integration<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />17<br />
  23. 23. November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />18<br />Application of Battery Framework:core of high performance billing system<br />
  24. 24. BILLING SYSTEM<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />19<br />Telecommunication services<br />Payment system<br />Billing System<br />Events<br />Events<br />Service provided logs<br />Payments<br />User data<br />Tariffing of services<br />Billing rules<br />CRM<br />Balance management<br />Balance<br />Users<br />Rules<br />Service enable/disable<br />Balance data<br />Balance<br />...<br />
  25. 25. BILLING SYSTEM EVENTS<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />20<br />Payments<br />Events<br /><ul><li>Service consumption
  26. 26. Payments</li></ul>Balance access<br /><ul><li>Tariffing
  27. 27. Write-off
  28. 28. Queries from network
  29. 29. Queries from CRM</li></ul>Issues<br /><ul><li>Reading load
  30. 30. Writing load
  31. 31. Transactions</li></ul>Billing engine<br />Tariffing<br />Balance management<br />Events<br />Events<br />Billing system data<br />Queries<br />Queries<br />Balance<br />
  32. 32. BOTTLENECK localization<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />21<br />Requirements (20 millions users)<br /><ul><li> ~ 109 Objects (counters)
  33. 33. 1000 events/sec
  34. 34. 50 ms latency </li></ul>Data<br /><ul><li>Users
  35. 35. Balance
  36. 36. Rules (wholesales, group bonuses and such)</li></ul>Billing engine<br />Events<br />Events<br />Balance management<br />Tariffing<br />Write-off<br />payment<br />Balance<br />Queries<br />Batches<br />Balance management system is a bottleneck<br />
  37. 37. BILLING processing network<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />22<br />Inbound event<br />Outbound event<br />Search for relevant counters<br />Write-off check<br />Tariffing<br />Junction<br />Rules processing<br />Fork<br />Changes fixation<br />Asynchronous RDBMS replication<br />Router<br />
  38. 38. SCALING processing networks<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />23<br />Horizontal scaling<br />
  39. 39. PERFORMANCE AND SCALABILITY<br />November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />24<br />Data scalability<br />Users ( millions)<br />Hardware scalability<br />Servers<br />Events/sec<br />Events/sec<br />Servers characteristics<br /><ul><li>CPU 2.5 GHz Quad core
  40. 40. RAM 32 GB</li></li></ul><li>November 12, 2009<br />© 2009 Grid Dynamics — Scaling Mission Critical Systems<br />25<br />Thanks!<br />Eugene Steinberg<br />CTO<br />

×