SlideShare a Scribd company logo
1 of 11
Download to read offline
HBaseConAsia2018
August 17,2018
Gehua New Century Hotel Beijing,China
HBase On Persistent Memory
Anoop Sam John, Ramkrishna S Vasudevan
hosted by
hosted by
Content
01
02
04
03
HBase Present Model
Region Replica
Persistent Memory Technology
HBase On Persistent Memory
05 Performance Numbers
hosted by
Apache HBase Present Model
 Accumulate data in Memory
o Sorted Map
o Cell data bytes in Local Allocation Buffers (LABs) in DRAM
o LABs with size 2 MB in RAM.
 Also write to Write Ahead Log (WAL)
o Cell data in Volatile RAM.
o To recover from server crash
o HDFS interaction adding more latency
hsync vs hflush - HBASE-19024
 Flushes as files to HDFS on reaching memstores size
o 128 MB default flush size
 Replay WAL on server crash
o Data unavailable till replay completes
o Large Mean Time To Recover (MTTR)
o Takes several minutes on large cluster (Complaint from many
users like Alibaba – HBaseConAsia , Huawei)
hosted by
Apache HBase Present Model
Region Replica for better availability (?)
 Read only replica regions on other RSs
o Refer to same HFiles in HDFS
o Memstore data also can be replicated
o Using WAL read replication path
o Eventual consistency
o Can read from replica regions when primary is down
o No strong consistency guarantees.
o Only when Scan/Get says TIMELINE Consistency, replica read happens
(not by default)
o Better availability but only for selected use cases !!! – No strong
Consistency
hosted by
Persistent Memory Technology
Persistent Memory
o Get back data even after power cycles
o Accessed using memory APIs
o Processor load and store instructions
3DXPoint
o New NVM technology
o Stackable cross-gridded data access array
Intel Apache Pass (AEP)
o Persistent Memory (pmem)
o Big, affordable and persistent
o Accessed like volatile memory, using processor load
and store instructions
Library
o NVML (Now called PMDK)
o Java wrappers around it
o Apache Mnemonic – (used for PoC)
o https://github.com/pmem/pcj
hosted by
Apache HBase On Persistent Memory
 Accumulate data in Memory
o Sorted Map
o Cell data bytes in Local Allocation Buffers with size 2 MB
 Region Replica in other servers
o Replica regions feature already in place.
o Synchronous replication to replica regions
 No need to write to Write Ahead Log (WAL)
o Cell data in non volatile AEP.
 Server crash
o Fast switch to replica regions
o Consistent data in replica regions
o Full cluster down – Data in non volatile area. Fast way to recreate in memory Map.
 HBASE-20003
hosted by
Apache HBase On Persistent Memory
 More memory size available - DRAM 100s of GBs. AEP even more
 More and More Data in Memory
o Large memstore size and global memstore size
• More data in memory
• Less flushes and compactions = Less IO
• Subsequent reads can get data from memory mostly (?)
• More memstores size => More Java heap size => Larger GC pause issues.
• More cell entries to CSLM. More compares for ordering => Lower Throughput
• Server down - more data in live WAL files for replay => Higher MTTR
o More Java heap size –> Off heap writes using Off heap memstores
o More cell entries to CSLM –> Compacting Memstore work by Yahoo , New faster CSLM implementation by Alibaba
o More data in live WALs –> HBase on AEP with no WALs. Persistent memstores LABs. Instant rebuild of memstores CSLM.
hosted by
Performance Results
 PerformanceEvaluation Tool
o Write only workload
o 4 Node cluster
o 100 client thread
o 250 GB Total data
o Single column per row
o WAL – 3 Replicas (HDFS replicas)
o WALLess – Primary and 2 replica regions
Average throughput is > 2x compared to With WAL cases.
Latency is consistent through out for the WALLess case. In WAL
case the latency varies as the amount of data to ‘sync’ increases.
(Here again with ‘fsync’ latency is more than ‘hflush’).
hosted by
Performance Results
 PerformanceEvaluation Tool
o Random Reads
o 4 Node cluster
o 100 client thread
o 5 secs/ 30 sec ZK session time outs
o One RS node crash in between PE run
o
 Max latency is 44x larger with WAL (ZK session time out = 5 sec)
 Max latency is 104x larger with WAL (ZK session time out = 30 sec)
hosted by
Apache HBase On Persistent Memory
 HBASE-20003
 https://docs.google.com/document/d/1sYJS9lMZa_EMhTTOJ7y_KzVUjXdBgdKdPWmsN5p2lH0/
 Write path, read path changes done in PoC
 Pending – WAL based features – Inter cluster replication, backup
o Similar issue as that in HBASE-20951 (Ratis LogService backed WALs)
o Work with this project - HBase improvements for Cloud
 Testing – Full cluster restart/ Rolling restart scenarios
 Load balancer , AM stabilization.
o Specially with Region replicas. Many bugs. Solving…
 http://pmem.io/
Project Status
Thanks

More Related Content

What's hot

What's hot (20)

Accordion HBaseCon 2017
Accordion HBaseCon 2017Accordion HBaseCon 2017
Accordion HBaseCon 2017
 
Kafka on ZFS: Better Living Through Filesystems
Kafka on ZFS: Better Living Through Filesystems Kafka on ZFS: Better Living Through Filesystems
Kafka on ZFS: Better Living Through Filesystems
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
 
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environmentHBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environment
 
HBaseCon 2013: Scalable Network Designs for Apache HBase
HBaseCon 2013: Scalable Network Designs for Apache HBaseHBaseCon 2013: Scalable Network Designs for Apache HBase
HBaseCon 2013: Scalable Network Designs for Apache HBase
 
TeraCache: Efficient Caching Over Fast Storage Devices
TeraCache: Efficient Caching Over Fast Storage DevicesTeraCache: Efficient Caching Over Fast Storage Devices
TeraCache: Efficient Caching Over Fast Storage Devices
 
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaHBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
 
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBaseHBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
 
Hadoop Hardware @Twitter: Size does matter!
Hadoop Hardware @Twitter: Size does matter!Hadoop Hardware @Twitter: Size does matter!
Hadoop Hardware @Twitter: Size does matter!
 
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase UpdateHBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
 
Rolling Out Apache HBase for Mobile Offerings at Visa
Rolling Out Apache HBase for Mobile Offerings at Visa Rolling Out Apache HBase for Mobile Offerings at Visa
Rolling Out Apache HBase for Mobile Offerings at Visa
 
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
HBaseCon 2015: HBase at Scale in an Online and  High-Demand EnvironmentHBaseCon 2015: HBase at Scale in an Online and  High-Demand Environment
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
 
Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path
 
HBase: How to get MTTR below 1 minute
HBase: How to get MTTR below 1 minuteHBase: How to get MTTR below 1 minute
HBase: How to get MTTR below 1 minute
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce
 
HBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a FlurryHBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a Flurry
 
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
 
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQLHBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
 
HBaseCon 2013: How to Get the MTTR Below 1 Minute and More
HBaseCon 2013: How to Get the MTTR Below 1 Minute and MoreHBaseCon 2013: How to Get the MTTR Below 1 Minute and More
HBaseCon 2013: How to Get the MTTR Below 1 Minute and More
 

Similar to HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices

[B4]deview 2012-hdfs
[B4]deview 2012-hdfs[B4]deview 2012-hdfs
[B4]deview 2012-hdfs
NAVER D2
 
Hug Hbase Presentation.
Hug Hbase Presentation.Hug Hbase Presentation.
Hug Hbase Presentation.
Jack Levin
 
02.28.13 WANdisco ApacheCon 2013
02.28.13 WANdisco ApacheCon 201302.28.13 WANdisco ApacheCon 2013
02.28.13 WANdisco ApacheCon 2013
WANdisco Plc
 
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Databricks
 
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Databricks
 

Similar to HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices (20)

[B4]deview 2012-hdfs
[B4]deview 2012-hdfs[B4]deview 2012-hdfs
[B4]deview 2012-hdfs
 
HBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and CompactionHBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and Compaction
 
Hug Hbase Presentation.
Hug Hbase Presentation.Hug Hbase Presentation.
Hug Hbase Presentation.
 
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
 
Redis on NVMe SSD - Zvika Guz, Samsung
 Redis on NVMe SSD - Zvika Guz, Samsung Redis on NVMe SSD - Zvika Guz, Samsung
Redis on NVMe SSD - Zvika Guz, Samsung
 
02.28.13 WANdisco ApacheCon 2013
02.28.13 WANdisco ApacheCon 201302.28.13 WANdisco ApacheCon 2013
02.28.13 WANdisco ApacheCon 2013
 
Hadoop Architecture_Cluster_Cap_Plan
Hadoop Architecture_Cluster_Cap_PlanHadoop Architecture_Cluster_Cap_Plan
Hadoop Architecture_Cluster_Cap_Plan
 
HBaseCon2017 Accordion: Apache HBase Beathes with In-Memory Compaction
HBaseCon2017 Accordion: Apache HBase Beathes with In-Memory CompactionHBaseCon2017 Accordion: Apache HBase Beathes with In-Memory Compaction
HBaseCon2017 Accordion: Apache HBase Beathes with In-Memory Compaction
 
Red Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep DiveRed Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep Dive
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
 
Ceph at salesforce ceph day external presentation
Ceph at salesforce   ceph day external presentationCeph at salesforce   ceph day external presentation
Ceph at salesforce ceph day external presentation
 
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance Tuning
 
Hbase 20141003
Hbase 20141003Hbase 20141003
Hbase 20141003
 
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
 
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
 
VMWare Performance Tuning by Virtera (Jan 2009)
VMWare Performance Tuning by  Virtera (Jan 2009)VMWare Performance Tuning by  Virtera (Jan 2009)
VMWare Performance Tuning by Virtera (Jan 2009)
 
Ceph Day San Jose - Ceph at Salesforce
Ceph Day San Jose - Ceph at Salesforce Ceph Day San Jose - Ceph at Salesforce
Ceph Day San Jose - Ceph at Salesforce
 
Hadoop Cluster With High Availability
Hadoop Cluster With High AvailabilityHadoop Cluster With High Availability
Hadoop Cluster With High Availability
 
HBase Application Performance Improvement
HBase Application Performance ImprovementHBase Application Performance Improvement
HBase Application Performance Improvement
 

More from Michael Stack

More from Michael Stack (20)

hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloudhbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
 
hbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinteresthbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinterest
 
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltdhbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
 
hbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didihbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didi
 
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
 
hbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencenthbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencent
 
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
 
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
 
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Componenthbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
 
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibabahbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
 
hbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomihbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomi
 
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Sparkhbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
 
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBasehbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
 
hbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solutionhbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solution
 
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memoryhbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
 
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACLhbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
 
hbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBasehbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBase
 
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
 
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
 
HBaseConAsia2019 Keynote
HBaseConAsia2019 KeynoteHBaseConAsia2019 Keynote
HBaseConAsia2019 Keynote
 

Recently uploaded

audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
lolsDocherty
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
ChloeMeadows1
 

Recently uploaded (16)

How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?
 
Bug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideBug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's Guide
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirt
 
Premier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfPremier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdf
 
GOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdfGOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdf
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of apps
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
The Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case StudyThe Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case Study
 
Thank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsThank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirts
 
Pvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdfPvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdf
 
Reggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirts
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
 
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
 
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresenceCyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
 
Statistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfStatistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdf
 

HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices

  • 1. HBaseConAsia2018 August 17,2018 Gehua New Century Hotel Beijing,China HBase On Persistent Memory Anoop Sam John, Ramkrishna S Vasudevan hosted by
  • 2. hosted by Content 01 02 04 03 HBase Present Model Region Replica Persistent Memory Technology HBase On Persistent Memory 05 Performance Numbers
  • 3. hosted by Apache HBase Present Model  Accumulate data in Memory o Sorted Map o Cell data bytes in Local Allocation Buffers (LABs) in DRAM o LABs with size 2 MB in RAM.  Also write to Write Ahead Log (WAL) o Cell data in Volatile RAM. o To recover from server crash o HDFS interaction adding more latency hsync vs hflush - HBASE-19024  Flushes as files to HDFS on reaching memstores size o 128 MB default flush size  Replay WAL on server crash o Data unavailable till replay completes o Large Mean Time To Recover (MTTR) o Takes several minutes on large cluster (Complaint from many users like Alibaba – HBaseConAsia , Huawei)
  • 4. hosted by Apache HBase Present Model Region Replica for better availability (?)  Read only replica regions on other RSs o Refer to same HFiles in HDFS o Memstore data also can be replicated o Using WAL read replication path o Eventual consistency o Can read from replica regions when primary is down o No strong consistency guarantees. o Only when Scan/Get says TIMELINE Consistency, replica read happens (not by default) o Better availability but only for selected use cases !!! – No strong Consistency
  • 5. hosted by Persistent Memory Technology Persistent Memory o Get back data even after power cycles o Accessed using memory APIs o Processor load and store instructions 3DXPoint o New NVM technology o Stackable cross-gridded data access array Intel Apache Pass (AEP) o Persistent Memory (pmem) o Big, affordable and persistent o Accessed like volatile memory, using processor load and store instructions Library o NVML (Now called PMDK) o Java wrappers around it o Apache Mnemonic – (used for PoC) o https://github.com/pmem/pcj
  • 6. hosted by Apache HBase On Persistent Memory  Accumulate data in Memory o Sorted Map o Cell data bytes in Local Allocation Buffers with size 2 MB  Region Replica in other servers o Replica regions feature already in place. o Synchronous replication to replica regions  No need to write to Write Ahead Log (WAL) o Cell data in non volatile AEP.  Server crash o Fast switch to replica regions o Consistent data in replica regions o Full cluster down – Data in non volatile area. Fast way to recreate in memory Map.  HBASE-20003
  • 7. hosted by Apache HBase On Persistent Memory  More memory size available - DRAM 100s of GBs. AEP even more  More and More Data in Memory o Large memstore size and global memstore size • More data in memory • Less flushes and compactions = Less IO • Subsequent reads can get data from memory mostly (?) • More memstores size => More Java heap size => Larger GC pause issues. • More cell entries to CSLM. More compares for ordering => Lower Throughput • Server down - more data in live WAL files for replay => Higher MTTR o More Java heap size –> Off heap writes using Off heap memstores o More cell entries to CSLM –> Compacting Memstore work by Yahoo , New faster CSLM implementation by Alibaba o More data in live WALs –> HBase on AEP with no WALs. Persistent memstores LABs. Instant rebuild of memstores CSLM.
  • 8. hosted by Performance Results  PerformanceEvaluation Tool o Write only workload o 4 Node cluster o 100 client thread o 250 GB Total data o Single column per row o WAL – 3 Replicas (HDFS replicas) o WALLess – Primary and 2 replica regions Average throughput is > 2x compared to With WAL cases. Latency is consistent through out for the WALLess case. In WAL case the latency varies as the amount of data to ‘sync’ increases. (Here again with ‘fsync’ latency is more than ‘hflush’).
  • 9. hosted by Performance Results  PerformanceEvaluation Tool o Random Reads o 4 Node cluster o 100 client thread o 5 secs/ 30 sec ZK session time outs o One RS node crash in between PE run o  Max latency is 44x larger with WAL (ZK session time out = 5 sec)  Max latency is 104x larger with WAL (ZK session time out = 30 sec)
  • 10. hosted by Apache HBase On Persistent Memory  HBASE-20003  https://docs.google.com/document/d/1sYJS9lMZa_EMhTTOJ7y_KzVUjXdBgdKdPWmsN5p2lH0/  Write path, read path changes done in PoC  Pending – WAL based features – Inter cluster replication, backup o Similar issue as that in HBASE-20951 (Ratis LogService backed WALs) o Work with this project - HBase improvements for Cloud  Testing – Full cluster restart/ Rolling restart scenarios  Load balancer , AM stabilization. o Specially with Region replicas. Many bugs. Solving…  http://pmem.io/ Project Status