A fully optimized HBase cluster could easily hit the limit of the underlying storage device’s capability, which is beyond the reach of software optimization alone. To get around this constraint, we need a new design that brings data processing and data storage closer together. In this presentation, we will look at how persistent memory will change the way large datasets are stored. We will review the hardware characteristics of 3D XPoint™, a new persistent memory technology with low latency and high capacity. We will also discuss opportunities for further improvement within the HBase framework using persistent memory.

1. Breaking the Sound Barrier with Persistent Memory
Liqi Yi
Shylaja Kokoori

2. Legal Disclaimer
2
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of
any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for
conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this
information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published
specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual
performance. Consult other sources of information to evaluate performance as you consider your purchase.
For more complete information about performance and benchmark results, visit http://www.intel.com/performance.
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or
software design or configuration may affect actual performance.
Results have been simulated and are provided for informational purposes only. Results were derived using simulations run on an architecture simulator or
model. Any difference in system hardware or software design or configuration may affect actual performance.
Intel does not control or audit the design or implementation of third party benchmark data or Web sites referenced in this document. Intel encourages all of its
customers to visit the referenced Web sites or others where similar performance benchmark data are reported and confirm whether the referenced
benchmark data are accurate and reflect performance of systems available for purchase.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2016 Intel Corporation. All rights reserved.

3. Problems at Hand
 Disk writes in burst fashion
 High bandwidth in flush, compaction
 Disk bandwidth inflation
 Write: Key/Value (KV) pairs written to disk many times in consecutive
compactions
 Read: All KVs bring back to memory when HFile block was hit
 Data format changing
 Write: “serialize” maps to HFile blocks
 Read: linear scan within HFile blocks
3

4. Problems at Hand
 Disk writes in burst fashion
 High bandwidth in flush, compaction
 Disk bandwidth inflation
 Write: Key/Value (KV) pairs written to disk many times in consecutive
compactions
 Read: All KVs bring back to memory when HFile block was hit
 Data format changing
 Write: “serialize” maps to HFile blocks
 Read: linear scan within HFile blocks
4
These problems come with
Mem+Disk hardware architecture

5. Addressing
 Disk writes in burst fashion
 Faster drives: PCIe SSDs
 Disk bandwidth inflation
 Larger DRAM
 Data format changing
 None
5

6. Addressing
 Disk writes in burst fashion
 Faster drives: PCIe SSDs
 Disk bandwidth inflation
 Larger DRAM
 Data format changing
 None
6
Expensive, still not fast enough

7. Addressing
 Disk writes in burst fashion
 Faster drives: PCIe SSDs
 Disk bandwidth inflation
 Larger DRAM
 Data format changing
 None
7
Expensive, still not fast enough
More expensive, small, volatile

8. Addressing
 Disk writes in burst fashion
 Faster drives: PCIe SSDs
 Disk bandwidth inflation
 Larger DRAM
 Data format changing
 None
8
Expensive, still not fast enough
More expensive, small, volatile
Do we have to persist on block device?

9. Addressing with Persistent Memory
 Disk writes in burst fashion
 Disk bandwidth inflation
 Data format changing
9
High bandwidth, low latency
High bandwidth, non-volatile
Could be eliminated

10. Persistent Memory
10
CPU caches (L1-L3)
Register
DRAM
Persistent Memory
NAND SSD
HDD and other block devices
CPU caches (L1-L3)
Register
DRAM
NAND SSD
HDD and other block devices
Bandwidt
h
Latency
Size
Byte Addressable
Performance Gap

11. Experiment on BucketCache
• BucketCache on persistent memory
• Code change in HBase
• Introduce new IOEngine for BucketCache
• Use libraries from http://pmem.io for persistent memory operations
• Experiment
• Persistent memory emulation with configurable latencies
• Focus on performance impact
11

12. Experiment Design
• Basic setup
• HBase 2.0.0-SNAPSHOT, YCSB 0.6.0, Hadoop 2.5.2
• Preloaded table, 10 fields per row, 100 Bytes per field
• 100% un-throttled uniform read, measures after BlockCache is filled
• Experiments
• Baseline: DRAM_LRU_BlockCache only
• PM runs: DRAM_LRU_BlockCache + different_size_PM_BucketCache
• Measure the throughput/latency impact
12

13. 1.00 1.06 1.13 1.23
1.44
1.65
1.98
2.44
3.13
4.22
5.95
0
1
2
3
4
5
6
7
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Normalizedspeedup
Size of extra PM BucketCache
Speed up with extra BucketCache
Result: Throughput
5x
13

14. 1.00 0.98
0.92
0.83
0.73
0.63
0.52
0.42
0.33
0.23
0.14
0
0.2
0.4
0.6
0.8
1
1.2
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Normalized95%latencyvs.baseline
Size of extra PM BucketCache (% of test table)
95% latency with extra BucketCache
Result: 95% Latency
Reduced by
85%
14

15. Summary
• Significant performance improvement with persistent memory (~6x throughput,
latency reduced by 85%)
• Offers new possible solution from architecture side
• Persist partially or completely on persistent memory
• No more format changing overhead
• Faster recovery
15

16. Acknowledging
Anoop S John, Ramkrishna S Vasudevan
16