In-Memory Data Grids Essentials. Oracle Coherence

1,185 views

Published on

Published in: Software, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,185
On SlideShare
0
From Embeds
0
Number of Embeds
187
Actions
Shares
0
Downloads
34
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Macys:
    сотни тысяч продуктов
    Кохеренс
  • Macys:
    сотни тысяч продуктов
    Кохеренс
  • кох – ведущий, широкий функционал, широко применяется, я его знаю
  • Говорить план след слайдов
  • DHT
  • Showcase node leave
    Reference to real data amount:
  • IMDG -> Coh

    CA – one node leave
    AP – brain split
  • Filter-Map-Reduce
  • Write-behind batching didn’t work in 3.5
  • In-Memory Data Grids Essentials. Oracle Coherence

    1. 1. Scalable eCommerce Platform Solutions Scalable eCommerce Platform Solutions In-Memory Data Grids Essentials. Oracle Coherence Vladimir Kondraschenko Max Myslyvtsev
    2. 2. Scalable eCommerce Platform Solutions About me Vladimir Kondraschenko • Lead Java engineer • More than 10 years of hands on experience in – software design and coding – technical leadership – development management vkondraschenko@griddynamics.com
    3. 3. Scalable eCommerce Platform Solutions Agenda • Preconditions of using IMDG • Architecture overview – topology – distributed cache layer – local cache layer • Cache operations • Persistence integration • Tips
    4. 4. Scalable eCommerce Platform Solutions Different Solutions Coherence
    5. 5. Scalable eCommerce Platform Solutions Preconditions of Using IMDG • Big amounts of data (10-100 GB) • Low latency • High availability • Distributed calculations
    6. 6. Scalable eCommerce Platform Solutions Transport Local Data Architecture Layers Transport Local Data Transport Local Data Shared Data
    7. 7. Scalable eCommerce Platform SolutionsScalable eCommerce Platform Solutions Transport Layer Discovery and communication
    8. 8. Scalable eCommerce Platform Solutions IMDG Topology
    9. 9. Scalable eCommerce Platform Solutions Discovery • Multicast • Well-known address
    10. 10. Scalable eCommerce Platform Solutions Discovery • Multicast • Well-known address New
    11. 11. Scalable eCommerce Platform Solutions Discovery • Multicast • Well-known address WKA WKA New
    12. 12. Scalable eCommerce Platform Solutions Communication • Custom protocols over TCP or UDP • Senior node • Heartbeats – Cluster heartbeats – Node heartbeats Senior
    13. 13. Scalable eCommerce Platform Solutions Communication • Custom protocols over TCP or UDP • Senior node • Heartbeats – Cluster heartbeats – Node heartbeats Senior
    14. 14. Scalable eCommerce Platform Solutions Communication • Custom protocols over TCP or UDP • Senior node • Heartbeats – Cluster heartbeats – Node heartbeats Senior
    15. 15. Scalable eCommerce Platform Solutions Node Departure • TCP Ring • IP monitor • Timeout
    16. 16. Scalable eCommerce Platform Solutions Node Departure • TCP Ring • IP monitor • Timeouts
    17. 17. Scalable eCommerce Platform Solutions Node Departure • TCP Ring • IP monitor • Timeouts Box1: IP 177.230.14.5 IP ping
    18. 18. Scalable eCommerce Platform Solutions Node Departure • TCP Ring • IP monitor • Timeout
    19. 19. Scalable eCommerce Platform SolutionsScalable eCommerce Platform Solutions Shared Data Layer Cluster-wide data
    20. 20. Scalable eCommerce Platform Solutions Distributed Cache Cluster Client Node Cache Interface Storage Node Data Storage Cache Interface
    21. 21. Scalable eCommerce Platform Solutions Cache Topology • Replicated Storage 1 A D B E C F Storage 2 A D B E C F Storage 3 A D B E C F Storage 1 A D B E Storage 2 C A D F Storage 3 E B F C • Partitioned
    22. 22. Scalable eCommerce Platform Solutions Storage 3 Repartitioning Storage 1 A D B E Storage 3 E B F CE Storage 2 C A D FF FCEB
    23. 23. Scalable eCommerce Platform Solutions CAP Theorem C A P Coherence Coherence
    24. 24. Scalable eCommerce Platform SolutionsScalable eCommerce Platform Solutions Get/Put/Remove, Query, Invocation, Map-Reduce Cache operations
    25. 25. Scalable eCommerce Platform Solutions Key hashing Cache Operations: Put/Get Storage 1 Storage 2 Client Primary A A B C D E F • Client knows responsible storage – Key hash is used to find it • Automatic backups Backup A Backup F Primary F
    26. 26. Scalable eCommerce Platform Solutions Storage 2Storage 2Storage 1 Cache Operations: Query • Broadcast request – Unless Query is Key-Associated • All entries are evaluated – Unless Indexes are used field1=10 or field2>70field1=10 and key=A key: field1: field2: D 70 80 key: field1: field2: C 50 60 key: field1: field2: B 30 40 key: field1: field2: D 70 80 key: field1: field2: C 50 60 key: field1: field2: B 30 40 key: field1: field2: A 10 20
    27. 27. Scalable eCommerce Platform Solutions Cache Operations: Invocation/Entry Processor • Task is serialized and distributed – May contain additional data • Configurable execution scope – Specific nodes – All nodes – Nodes bound to data Node 1 Node 2 Task Task Client Task
    28. 28. Scalable eCommerce Platform Solutions • Data-bound mapping – By keys – By Query • Parallel execution Storage 2Storage 1 key: field1: field2: A 10 20 key: field1: field2: B 30 40 key: field1: field2: C 50 60 key: field1: field2: D 70 80 Cache Operations: Map-Reduce sum(field2) where field1>20 40 60 80 180 key: field1: field2: A 10 20 140
    29. 29. Scalable eCommerce Platform SolutionsScalable eCommerce Platform Solutions Persistence integration
    30. 30. Scalable eCommerce Platform Solutions Synchronous Persistence Integration • Read-through • Write-through StorageClient Durable Storage Data Grid Client Durable Storage Data Grid Storage
    31. 31. Scalable eCommerce Platform Solutions • Refresh-ahead • Write-behind Asynchronous Persistence Integration Storage QueueClient Durable Storage Data Grid Storage QueueClient Durable Storage Data Grid
    32. 32. Scalable eCommerce Platform SolutionsScalable eCommerce Platform Solutions Local Data Layer Node-specific data
    33. 33. Scalable eCommerce Platform Solutions Storage Storage Node Near Caching • Local access by primary key • Lazy population • Invalidation – On entry change – By timeout Near Cache Distributed Cache A A B
    34. 34. Scalable eCommerce Platform Solutions Continuous Caching • Local access by any Query • Eager population • Near real-time data Storage Storage Node Continuous Cache Distributed Cache A B A B
    35. 35. Scalable eCommerce Platform SolutionsScalable eCommerce Platform Solutions Tips Using Oracle Coherence in practice
    36. 36. Scalable eCommerce Platform Solutions Before integrating Oracle Coherence • Do I really need IMDG or just a cache? • How large my cluster should be? • How much heap memory should be configured for each node? • What GC configuration to use? • Do I have a relevant environment to test my application?
    37. 37. Scalable eCommerce Platform Solutions Memory usage Limit cache size • By binary size (per node) • By number of objects in cache (per node) Estimate cache object memory consumption • Use BinaryMemoryCalculator • Remember about backup copies • Cache listeners (near cache, custom)
    38. 38. Scalable eCommerce Platform Solutions Investigation & troubleshooting • Coherence Management Framework (JMX) – view cluster and nodes settings – list caches (front and back) on each node – view cache dynamic metrics • Coherence command-line tool – manage cache services – manipulate cache data • Profiling and heap analysis – analyze heap dump – analyze GC logs
    39. 39. Scalable eCommerce Platform Solutions Scalable eCommerce Platform Solutions Thank you! Questions & Answers
    40. 40. Scalable eCommerce Platform Solutions References • Workshop – https://github.com/mmyslyvtsev/imdg-workshop • Coherence Knowledge Base – http://coherence.oracle.com • Developers Guide – http://docs.oracle.com/cd/E18686_01/coh.37/e18677/toc.htm • Book – http://www.amazon.com/Oracle-Coherence-3-5-Aleksandar- Seovic/dp/1847196128

    ×