Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
Use your current wisely
Harvey Raja
Oracle
Chris Neal
Pegasus
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
§  Senior Engineer
§  Oracle Coherence
Introductio...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4
§  Systems Architect
§  Pegasus Solutions
Introduc...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5 5 Copyright © 2011, Oracle and/or its affiliates. Al...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6
What’s all this about?
§  Big Data / Big Memory on ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7
Heap me up
§  JVM manages our objects
§  Understan...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8
Heap me up
§  Two distinct regions of data locality...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9
Heap me up
§  All memory allocations are against th...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10
Heap me up
JVM
100%
0%
0%
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11
Applications & Their Objects
§  Every application ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12
Application Object Profiles
§  Ye olde faithful… P...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13
Application Object Profiles
§  Foreign Exchange Po...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14
What we know about applications?
§  The applicatio...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15
Can the JVM just make the right choice?
§  The JVM...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16
Municipality
§  The application deems its own usag...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17
CPU Architectures
§  As latency increases so does
...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18
CPU Architectures
Speed Capacity
Registry 1ns 0.00
...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19
Blank Canvas
£600pcm
£1800pcm
£2400pcm
£100000pcm
£...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20
How do you choose the right property?
§  How do yo...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21
Translated to memory
USD -> GBP
USD-> JMD
Heap RAM
...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22
Translated to memory
§  Objects held on heap are:
...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23
Translated to memory
CPU
Heap
NIO
Flash
Mechanical ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24
Brainstorm Summary
§  Cherish your high commodity ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25
Challenges
§  Serialization cost
§  Generally int...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26
Elastic Data
§  A Feature in Coherence
§  Store b...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27
Elastic Data
§  Its simply writing a number of byt...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28
Elastic Data
§  Require a compact structure to hol...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29
Elastic Data
FlashJournal
RAMJournal
Collector
Over...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30
Elastic Data
FlashJournal
RAMJournal
Collector
Over...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31
Elastic Data
Elastic Data
100%
100%
100%
Utilization
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32
Elastic Data
§  RAM Journal can be used with NIO B...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33
Elastic Data
§  Several platters & reading / writi...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34
Elastic Data
§  NAND gates
§  MLC
§  Write in pa...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35
Elastic Data
JVM
handle *
handle *
handle *
handle ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36
Elastic Data
H
AR
RIET VEY
IL
ARY TON
HARRIET
HARVE...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37
Elastic Data
§  Writes are journalled
§  Erase is...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38
Elastic Data
§  Require a Journal Garbage Collecto...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39
Elastic Data
§  Dedicated threads to unblock write...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40
Who am I?
§  Chris Neal, Systems Architect
§  Sta...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.41
Who is Pegasus Solutions?
§  Founded in 1988
§  P...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42
Distribution Services
–  Connects hotel systems wit...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.43
Why do we need a cache?
§  In travel agent days, “...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.44
Distribution Services and Coherence
Physical deploy...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.45
The problem…
§  The cache is too small. Empty to f...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.46
The Challenge….
§  Increase the cache size on the ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.47
The Process…
§  “spend as little as possible” mean...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.48
iozone
§  Reads and writes a file to a filesystem ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.49
SSD benchmark results
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.50
SATA benchmark results, part 1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.51
SATA benchmark results, part 2
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.52
Elastic Data Hardware configuration
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.53
What we observe
§  At production volumes in RAM:
–...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.54
Hurdles We Have Overcome
§  Configuring the heap s...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.55
Configuring Coherence for Elastic Data
§  Operatio...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.56
Configuring Coherence for Elastic Data
§  Cache-co...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.57
Garbage Collection settings
argv[12]: -XX:+UseG1GC
...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.58
Did we meet our goals
§  Goal: 500M cached objects...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.59
Graphic Section Divider
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.60
Upcoming SlideShare
Loading in …5
×

JavaOne-2013: Save Scarce Resources by Managing Terabytes of Objects off-heap or Even on Disk

382 views

Published on

Presented at JavaOne 2013. A description of Coherence's Elastic Data feature and the reasons to want such a solution that allows storing data in other resource other than heap-based memory.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
382
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

JavaOne-2013: Save Scarce Resources by Managing Terabytes of Objects off-heap or Even on Disk

  1. 1. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
  2. 2. Use your current wisely Harvey Raja Oracle Chris Neal Pegasus
  3. 3. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3 §  Senior Engineer §  Oracle Coherence Introduction Harvey Raja
  4. 4. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4 §  Systems Architect §  Pegasus Solutions Introduction Chris Neal
  5. 5. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5 5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.
  6. 6. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6 What’s all this about? §  Big Data / Big Memory on a transistor diet §  Applications and conceived concerns §  Object Profiling §  Elastic Data §  Improvements
  7. 7. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7 Heap me up §  JVM manages our objects §  Understands Live Data –  References –  Free Lists new Object()
  8. 8. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8 Heap me up §  Two distinct regions of data locality –  Young Generation –  Old Generation §  Allows conscious distinction between: –  Long living objects –  Short lived objects
  9. 9. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9 Heap me up §  All memory allocations are against the same resource §  Why would it be any different? §  Provides means to access –  off-heap memory –  File Descriptors a.k.a. any resource
  10. 10. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10 Heap me up JVM 100% 0% 0%
  11. 11. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11 Applications & Their Objects §  Every application has very different uses of objects –  Size –  Scope §  Structures / Containers –  Structures {Containers} –  Containers {Structures}
  12. 12. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12 Application Object Profiles §  Ye olde faithful… Pet Store §  Short-lived objects –  Search Results §  Long-living objects –  Popular items JEE Pet Store JEE Pet Store JEE Pet Store JEE Pet Store Pet Store
  13. 13. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13 Application Object Profiles §  Foreign Exchange Position Keeping §  Aggregate Trades Values per currency pair §  Some currencies are a ‘busier’ §  Currencies may have varying SLAs FX Position Keeping T T T T USD-GBP USD-EUR T T T T T T T T T T T T
  14. 14. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14 What we know about applications? §  The application understands more about each object than the JVM –  Frequency –  Size §  Keeping everything in RAM is possible but is it efficient? §  Huge Leap between Object on heap & stored in DB -  Recency -  Custom characteristics of the object
  15. 15. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15 Can the JVM just make the right choice? §  The JVM would have to span multiple devices §  Non-Heap must be serialized §  Applications are diverse therefore to make generic decisions on object usage would likely lead to false-positives §  Down to the application or a layer above the JVM allowing users to define resource assignment policies
  16. 16. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16 Municipality §  The application deems its own usage of each resource §  JVM provides primitives to load & store to these devices §  May be useful to have an API that performs this storage appropriate to the device: –  Routing stores to appropriate device –  Handle concerns of multiple applications on the same JVM
  17. 17. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17 CPU Architectures §  As latency increases so does capacity §  Data fetched as required by instructions §  Data is demoted as well as writes to shared data rippling through the caches
  18. 18. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18 CPU Architectures Speed Capacity Registry 1ns 0.00 L1 Cache 2-5ns 2x32KB L2 Cache 5ns 256KB L3 Cache 20ns 8MB Main Memory 20-60ns 16GB Nehalem2GHZ processor Mayfair Kensington Camden Wimbledon Manchester §  Perhaps we should charge our objects rental premiums?
  19. 19. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19 Blank Canvas £600pcm £1800pcm £2400pcm £100000pcm £100 Per TimeUnit £500 Per TimeUnit £250 Per TimeUnit £1000 Per TimeUnit
  20. 20. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20 How do you choose the right property? §  How do you select the right property: –  How often are you in the office? –  How long does it take you to get into the office? –  How much space do you occupy in the office? –  How do you get to the office? –  MFU –  Device latency –  Object size –  Every device has its own quirks
  21. 21. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21 Translated to memory USD -> GBP USD-> JMD Heap RAM (off heap) Disk GBP -> MUR Usage Pet store example Pedigree Chum Orijen Dog Ugg boots
  22. 22. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22 Translated to memory §  Objects held on heap are: –  Structures (Containers) –  Containers (Structures) §  Similar to a file descriptor, each object has its own metadata: –  Access Time –  Modified Time §  With ((Map) Containers) we already have a location to store metadata -  Size -  Touch Count
  23. 23. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23 Translated to memory CPU Heap NIO Flash Mechanical Disk §  Each object has metadata §  Some policy can manage these objects §  Demoted & promoted to the various media types §  Big jump from heap to NIO
  24. 24. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24 Brainstorm Summary §  Cherish your high commodity investments §  Reduce the regularity of going to a highly contended foreign resource §  Would be ideal to have objects float between high latency resources using a telepathic API –  Having some metadata could drive our decision for data locality §  Map provides a nice abstraction for objects that should ripple
  25. 25. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25 Challenges §  Serialization cost §  Generally interactions are performed against the object form §  False Positives §  Device type peculiarities §  A handle to the object (key) and metadata must be held on heap
  26. 26. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26 Elastic Data §  A Feature in Coherence §  Store binary key and value objects in RAM or Disk §  Overflow from RAM to Disk §  RAM can be configured as NIO store(byte[] key, byte[] val) load(byte[] key) erase(byte[] key) RAM Flash
  27. 27. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27 Elastic Data §  Its simply writing a number of bytes to some stream? §  How do you maintain handles? §  Need a pointer to the written data? §  How about updates, seek & replace? Easy Peasy
  28. 28. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28 Elastic Data §  Require a compact structure to hold handles (keys) to device pointers §  Journal writes to the file system §  Consistent API regardless of write to RAM, NIO, Flash or Mechanical Disk §  Buffer writes with thread dedicated to writing Implementation Details
  29. 29. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29 Elastic Data FlashJournal RAMJournal Collector Overflow Preparer Writer Binary Key Pointer 00110101001 11001001011 ... ... Serialize 110101 LFU Collector Store Index Object deemed unworthy of Heap store(key, value)Pointer returned Create
  30. 30. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30 Elastic Data FlashJournal RAMJournal Collector Overflow Preparer Writer Binary Key Pointer 00110101001 11001001011 ... ... Collector Store Index Deserialize 110101 Deserialized object returned load(pointer)binary value The binary key provides the physical location of the stored item Read
  31. 31. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31 Elastic Data Elastic Data 100% 100% 100% Utilization
  32. 32. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32 Elastic Data §  RAM Journal can be used with NIO ByteBuffers –  Memory managed using same mechanisms between RAM & Flash §  Consider device specifics prior to use and design components / interactions accordingly §  Multi-threaded clients §  Flash vs Mechanical More Features
  33. 33. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33 Elastic Data §  Several platters & reading / writing heads §  Faraday & Lorentz §  Seek time + Rotational Latency = L §  Can not get to the same speeds as the disk controllers Mechanical Media
  34. 34. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34 Elastic Data §  NAND gates §  MLC §  Write in pages erase in blocks §  No Seek Time or Rotational Latency Flash Media
  35. 35. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35 Elastic Data JVM handle * handle * handle * handle * handle * 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 0111011100110110101 0111010110100110101 handle *(disk pointer) HARRIET * HARVEY * HILARY * HILTON * Handles are stored in process Number of handles constrains amount of data that can be stored Key Management
  36. 36. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36 Elastic Data H AR RIET VEY IL ARY TON HARRIET HARVEY HILARY HILTON 1 2 4 8Tickets: §  Data structure to hold handles is a Binary Radix Tree –  allows sharing of common denominators §  Handles (keys) are stored in serialized form §  Benefit increases as common bytes increase and less heap memory is used
  37. 37. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37 Elastic Data §  Writes are journalled §  Erase is a logical removal §  Update = erase + write §  Avoids seek time or cascading pointer changes write write write APPEND APPEND APPEND erase 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 010111001 101111010 100111010 100111100 File 1 File 2 File 3
  38. 38. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38 Elastic Data §  Require a Journal Garbage Collector –  Reclaim unreferenced memory §  Evacuation process for each file §  Eviction logically removes from Journal File §  May enter an exhaustive mode –  Synonymous to Full GC 1001110 100111100 010111001 101111010 100111101 110011000 1001110 100111100 010111001 101111010 100111101 110011000 1001110 100111100 010111001 101111010 100111101 110011000 1001110 100111100 010111001 101111010 100111101 110011000 1001110 100111100 010111001 101111010 100111101 110011000 1001110 100111100 010111001 101111010 100111101 110011000 1001110 100111100 010111001 101111010 100111101 110011000 Journal Collector sort evacuate (exhaustive- mode)
  39. 39. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39 Elastic Data §  Dedicated threads to unblock writes –  Tuned to device type §  Client write appears to be as fast as heap write §  Overwhelming number of writes will result in push back to the client Preparer Buffer Writer
  40. 40. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40 Who am I? §  Chris Neal, Systems Architect §  Started with Pegasus in 1994 §  Worked with Coherence since 3.3.1 in 2007 §  Participate in Coherence CAB
  41. 41. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.41 Who is Pegasus Solutions? §  Founded in 1988 §  Provide technology and services to hotels and travel distributors §  Three main service areas: –  Representation Services –  Distribution Services –  Central Reservation Services (CRS)
  42. 42. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42 Distribution Services –  Connects hotel systems with distribution partners. –  100,000 hotels connected to all major distributors (Expedia, Orbitz, Hotwire, Travelocity, etc) –  Cheaper and easier than a direct connect –  If you book a hotel online, chances are your transaction goes through Pegasus. –  Pegasus processes roughly 8 billion transactions per month… sustained ~3000TPS @200ms latency or less.
  43. 43. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.43 Why do we need a cache? §  In travel agent days, “Look to book” ratios were 3:1 §  At internet scale, they are roughly 4200:1 –  Travel aggregators like Sidestep, Kayak, Mobissimo, etc burst this to >100,000:1 §  Looks are the most expensive transaction from systems processing perspective. §  We make no money on “looks”, so saving resources by not processing these transactions is important to both Pegasus and the downstream hotel systems. §  …Hello Coherence….
  44. 44. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.44 Distribution Services and Coherence Physical deployment: –  Production cluster consists of 6 servers, 144GB RAM each. –  Each server runs 3 storage enabled JVMs –  Servers run Solaris x64 –  Each Hotspot JVM is 32GB –  Using CMS collector, and having no GC pauses –  The vast majority of the storage space is for AvailibilityCache (22GB) to service the “looks” Client Applications –  ~120 storage disabled clients either in containers or stand alone –  Backing store is so large that NearCache is disabled
  45. 45. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.45 The problem… §  The cache is too small. Empty to full in ~3 hours. §  Evicting valuable, usable data §  Cache hit rate is too low
  46. 46. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.46 The Challenge…. §  Increase the cache size on the current servers from 20M to 200M §  Spend as little money as possible §  Do it by EOY
  47. 47. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.47 The Process… §  “spend as little as possible” means adding servers to reach 200M is not possible, which means no more RAM. §  Enter ElasticData §  In terms of $, RAM > SSD > SATA, but is SATA fast enough?
  48. 48. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.48 iozone §  Reads and writes a file to a filesystem as fast as it can §  Compare SSD to SATA with regards to throughput §  We know SSD is faster, but will SATA do? www.iozone.org - Filesystem benchmark tool
  49. 49. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.49 SSD benchmark results
  50. 50. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.50 SATA benchmark results, part 1
  51. 51. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.51 SATA benchmark results, part 2
  52. 52. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.52 Elastic Data Hardware configuration
  53. 53. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.53 What we observe §  At production volumes in RAM: –  Avg get/put times ~2ms §  At production volumes on SATA: –  Avg put times ~3ms –  Avg get times ~10ms Benchmark data through the application
  54. 54. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.54 Hurdles We Have Overcome §  Configuring the heap size: –  Enough room to store keys for 1.6TB of objects, so that Full GCs do not occur –  Enough room to store puts() while partitions are being evicted –  64GB (up to 72GB with G1) §  Eviction killing throughput. Eviction process was reading the values from disk at it evicted (for indexes and listeners). Behavior was changed with BlindCompactSerializationCache. §  Stopping a JVM: Transferring full partitions. Instead, drop the data, then transfer (DropContentPartitionListener) §  Starting a JVM: Rebalancing partitions to the new JVM. (DropContentPartitionListener)
  55. 55. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.55 Configuring Coherence for Elastic Data §  Operational overrides: <journaling-config> <ramjournal-manager> <minimum-load-factor>.4</minimum-load-factor> <maximum-size>8GB</maximum-size> </ramjournal-manager> <flashjournal-manager> <minimum-load-factor>.7</minimum-load-factor> <!-- 3.6TB filesystem size total / 2 JVMs is 1843GB each VM --> <!-- That gives 511 files @ 3690MB each per JVM --> <maximum-file-size>3690MB</maximum-file-size> <collector-timeout>30m</collector-timeout> <!-- 1600GB to force a more aggressive prune (same as high-units) --> <high-journal-size>1600GB</high-journal-size> </flashjournal-manager> </journaling-config>
  56. 56. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.56 Configuring Coherence for Elastic Data §  Cache-config.xml <distributed-scheme> <backup-count>0</backup-count> <partition-listener> <class-name>com.tangosol.net.partition.DropContentPartitionListener</class-name> </partition-listener> <backing-map-scheme> <ramjournal-scheme> <class-name>com.tangosol.net.cache.BlindCompactSerializationCache</class-name> <high-units>1600KB</high-units> <low-units>1400KB</low-units> <unit-calculator>Binary</unit-calculator> <unit-factor>1048576</unit-factor> </ramjournal-scheme> </backing-map-scheme> </distributed-scheme>
  57. 57. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.57 Garbage Collection settings argv[12]: -XX:+UseG1GC argv[13]: -XX:MaxGCPauseMillis=800 argv[14]: -XX:ConcGCThreads=10 argv[15]: -XX:ParallelGCThreads=10 argv[16]: -XX:InitiatingHeapOccupancyPercent=25 argv[17]: -XX:NewRatio=16
  58. 58. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.58 Did we meet our goals §  Goal: 500M cached objects –  Actual: 1.6B cached objects §  Goal: Spend as little as possible –  Actual: Spent 1400/machine (8640 total), 84x more objects §  Goal: Do it by EOY –  Actual: On track for production release before EOY So far, so good…
  59. 59. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.59 Graphic Section Divider
  60. 60. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.60

×