Date-tiered Compaction Policy for Time-series Data
•
1 like•1,053 views
Clara Xiong (Flurry/Yahoo!) With petabytes of data on thousands of nodes replicated across multiple data centers, growing at an accelerating rate, we have been running a workload at scale with a bottleneck of IO bandwidth. This talk covers a new compaction policy to improve efficiency for time-range scans of various look-back windows by structuring and maintaining a date-tiered store file layout for time-series data with infrequent updates and deletes.
Report
Share
Report
Share
Recommended
Apache HBase, Accelerated: In-Memory Flush and Compaction
Eshcar Hillel and Anastasia Braginsky (Yahoo!)
Real-time HBase application performance depends critically on the amount of I/O in the datapath. Here we’ll describe an optimization of HBase for high-churn applications that frequently insert/update/delete the same keys, such as for high-speed queuing and e-commerce.
Breaking the Sound Barrier with Persistent Memory
Liqi Yi and Shylaja Kokoori (Intel)
A fully optimized HBase cluster could easily hit the limit of the underlying storage device’s capability, which is beyond the reach of software optimization alone. To get around this constraint, we need a new design that brings data processing and data storage closer together. In this presentation, we will look at how persistent memory will change the way large datasets are stored. We will review the hardware characteristics of 3D XPoint™, a new persistent memory technology with low latency and high capacity. We will also discuss opportunities for further improvement within the HBase framework using persistent memory.
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
Pinterest runs 38 different HBase clusters in production, doing a lot of different types of work—with some doing up to 5 million operations per second. In this talk, you'll get details about how we do capacity planning, maintenance tasks such as online automated rolling compaction, configuration management, and monitoring.
hbaseconasia2017: hbase-2.0.0
HBase-2.0.0 has been a couple of years in the making. It is chock-a-block full of a long list of new features and fixes. In this session, the 2.0.0 release manager will perform the impossible, describing the release content inside the session time bounds.
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
The document summarizes a presentation on using persistent memory technology with Apache HBase. It describes the current HBase model using volatile memory and write-ahead logs, as well as persistent memory technology capabilities. The presentation proposes using persistent memory with HBase to store data non-volatility, eliminate write-ahead logs for faster performance, and increase availability through replica regions in memory. It shows performance test results finding over 2x higher throughput and up to 104x lower maximum latency without write-ahead logs when using persistent memory with HBase.
Redis on NVMe SSD - Zvika Guz, Samsung
In this talk we report on our experience with Redis-on-Flash (RoF)—a recently introduced product that uses SSDs as a RAM extension to dramatically increase the effective dataset capacity that can be stored on a single server. This talk provides the first in-depth RoF system performance characterization: we consider different use cases (varying both RAM-to-disk access ratio and object size), and compare SATA-based RoF, NVMe-based RoF, and all-RAM Redis deployments. We show that the superior performance of NVMe drives in terms of both latency and peak bandwidth makes them a particularly good fit for RoF use cases. Specifically, we show that backing RoF with NVMe drives can deliver more than 2 million operations per second with sub-millisecond latency on a single server.
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
If you’re running an HBase cluster in production, you’ve probably noticed that HBase shares a number of useful metrics about everything from your block cache performance to your HDFS latencies over JMX (or Ganglia, or just a file). The problem is that it’s sometimes hard to know what these metrics mean to you and your users. Should you be worried if your memstore SizeMB is 1.5GB? What if your regionservers have a hundred stores each? This talk will explain how to understand and interpret the metrics HBase exports. Along the way we’ll cover some high-level background on HBase’s internals, and share some battle tested rules-of-thumb about how to interpret and react to metrics you might see.
Accordion HBaseCon 2017
The document describes Accordion, a novel in-memory compaction algorithm for HBase that improves write performance. Accordion applies the log-structured merge tree design used in LSM databases to HBase's memory data structure. This transforms random writes to sequential writes, keeping more data in memory for longer. As a result, Accordion achieves higher write throughput, lower disk I/O, and reduced read latency compared to HBase's previous approach. Accordion has been integrated into HBase 2.0 as the default memory compaction implementation.
Recommended
Apache HBase, Accelerated: In-Memory Flush and Compaction
Eshcar Hillel and Anastasia Braginsky (Yahoo!)
Real-time HBase application performance depends critically on the amount of I/O in the datapath. Here we’ll describe an optimization of HBase for high-churn applications that frequently insert/update/delete the same keys, such as for high-speed queuing and e-commerce.
Breaking the Sound Barrier with Persistent Memory
Liqi Yi and Shylaja Kokoori (Intel)
A fully optimized HBase cluster could easily hit the limit of the underlying storage device’s capability, which is beyond the reach of software optimization alone. To get around this constraint, we need a new design that brings data processing and data storage closer together. In this presentation, we will look at how persistent memory will change the way large datasets are stored. We will review the hardware characteristics of 3D XPoint™, a new persistent memory technology with low latency and high capacity. We will also discuss opportunities for further improvement within the HBase framework using persistent memory.
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
Pinterest runs 38 different HBase clusters in production, doing a lot of different types of work—with some doing up to 5 million operations per second. In this talk, you'll get details about how we do capacity planning, maintenance tasks such as online automated rolling compaction, configuration management, and monitoring.
hbaseconasia2017: hbase-2.0.0
HBase-2.0.0 has been a couple of years in the making. It is chock-a-block full of a long list of new features and fixes. In this session, the 2.0.0 release manager will perform the impossible, describing the release content inside the session time bounds.
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
The document summarizes a presentation on using persistent memory technology with Apache HBase. It describes the current HBase model using volatile memory and write-ahead logs, as well as persistent memory technology capabilities. The presentation proposes using persistent memory with HBase to store data non-volatility, eliminate write-ahead logs for faster performance, and increase availability through replica regions in memory. It shows performance test results finding over 2x higher throughput and up to 104x lower maximum latency without write-ahead logs when using persistent memory with HBase.
Redis on NVMe SSD - Zvika Guz, Samsung
In this talk we report on our experience with Redis-on-Flash (RoF)—a recently introduced product that uses SSDs as a RAM extension to dramatically increase the effective dataset capacity that can be stored on a single server. This talk provides the first in-depth RoF system performance characterization: we consider different use cases (varying both RAM-to-disk access ratio and object size), and compare SATA-based RoF, NVMe-based RoF, and all-RAM Redis deployments. We show that the superior performance of NVMe drives in terms of both latency and peak bandwidth makes them a particularly good fit for RoF use cases. Specifically, we show that backing RoF with NVMe drives can deliver more than 2 million operations per second with sub-millisecond latency on a single server.
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
If you’re running an HBase cluster in production, you’ve probably noticed that HBase shares a number of useful metrics about everything from your block cache performance to your HDFS latencies over JMX (or Ganglia, or just a file). The problem is that it’s sometimes hard to know what these metrics mean to you and your users. Should you be worried if your memstore SizeMB is 1.5GB? What if your regionservers have a hundred stores each? This talk will explain how to understand and interpret the metrics HBase exports. Along the way we’ll cover some high-level background on HBase’s internals, and share some battle tested rules-of-thumb about how to interpret and react to metrics you might see.
Accordion HBaseCon 2017
The document describes Accordion, a novel in-memory compaction algorithm for HBase that improves write performance. Accordion applies the log-structured merge tree design used in LSM databases to HBase's memory data structure. This transforms random writes to sequential writes, keeping more data in memory for longer. As a result, Accordion achieves higher write throughput, lower disk I/O, and reduced read latency compared to HBase's previous approach. Accordion has been integrated into HBase 2.0 as the default memory compaction implementation.
HBaseConAsia2018 Track1-7: HDFS optimizations for HBase at Xiaomi
This document discusses optimizations made to HDFS for Hbase at XiaoMi. It addresses issues like shared memory allocation causing full GC in datanodes, listen drops on SSD clusters causing delays, peer cache bucket adjustment, and connection timeouts. Changes like preallocating shared memory, increasing socket backlog, reducing client and datanode timeouts, and adjusting the datanode dead node detection help improve performance and availability. The overall goal is to maintain local data, return fast responses from HDFS to Hbase, and reduce GC overhead from both systems.
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
In this presentation, we will introduce Hotspot's Garbage First collector (G1GC) as the most suitable collector for latency-sensitive applications running with large memory environments. We will first discuss G1GC internal operations and tuning opportunities, and also cover tuning flags that set desired GC pause targets, change adaptive GC thresholds, and adjust GC activities at runtime. We will provide several HBase case studies using Java heaps as large as 100GB that show how to best tune applications to remove unpredicted, protracted GC pauses.
Tales from Taming the Long Tail
Deepankar Reddy and Ishan Chhabra (Rocket Fuel)
Rocket Fuel is a marketing technology company that participates in 120+ billion real-time bidding auctions daily to show the right ad to the right user at the right time for our clients. In this talk, we discuss our efforts to systematically identify causes of, and how to decrease, long-tail read latencies.
Real-time HBase: Lessons from the Cloud
Speaker: Bryan Beaudreault (HubSpot)
Running HBase in real time in the cloud provides an interesting and ever-changing set of challenges -- instance types are not ideal, neighbors can degrade your performance, and instances can randomly die in unanticipated ways. This talk will cover what HubSpot has learned about running in production on Amazon EC2, how it handle DR and redundancy, and the tooling the team has found to be the most helpful.
IMC Summit 2016 Breakout - Andy Pavlo - What Non-Volatile Memory Means for th...
IMC Summit 2016 Breakout - Andy Pavlo - What Non-Volatile Memory Means for th...In-Memory Computing Summit
The advent of non-volatile memory (NVM) will fundamentally change the dichotomy between memory and durable storage in database management systems (DBMSs). These new NVM devices are almost as fast as DRAM, but all writes to it are potentially persistent even after power loss. Existing DBMSs are unable to take full advantage of this technology because their internal architectures are predicated on the assumption that memory is volatile. That means when NVM finally arrives, just like when you finally passed that kidney stone after three weeks, everyone will be relieved but the transition will be painful. Many of the components of legacy DBMSs will become unnecessary and will degrade the performance of data intensive applications.Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bull...
Packets per second (PPS) is an often overlooked value within an environment. Most network concerns are around throughput and interface speed, but what happens when this value becomes the bottleneck due to large-scale hosting providers (AWS, Azure, etc.) with rigid standards? This talk covers what a Redis packet looks like and how the out-of-the-box configuration can drastically affect
packet per second overhead. From here we’ll deep dive into specific configuration values which help lower PPS numbers, as well as different Redis master/slave relationships that can be utilized to keep PPS below inflexible network thresholds.
hbaseconasia2017: HBase在Hulu的使用和实践
Qianxi Zhang
1. Hulu是美国最受欢迎的在线视频网站之一，Hulu Beijing是Hulu第二大研发中心。北京大数据基础架构团队负责整个公司的大数据基础架构的研发和运维。
2. HBase在Hulu的概况
3. HBase在Hulu的使用
4. 用户画像系统，存放所有用户的基本信息，用户行为，第三方DMP数据和机器学习结果标签(几十万个Qualifier)，Spark和Spark Streaming读写HBase数据，运行各种机器学习模型，为公司的视频推荐，精准广告和Marketing团队服务
5. HBase在Hulu的优化
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
Володимир Цап "Constraint driven infrastructure - scale or tune?"
Volodymyr Tsap discusses how to save money on infrastructure through constraint driven design. He provides examples of hardware configurations on AWS, bare metal servers, and PaaS platforms to demonstrate how costs can be optimized. Tsap also outlines ways to reduce software costs through choices in operating system, virtualization, databases, and orchestration. Infrastructure support costs depend on the complexity of the environment, with basic setups costing $500-800 per month while more advanced architectures are $4,000-6,000 per month. The overall message is that money saved through optimization can be invested in people.
hbaseconasia2017: Large scale data near-line loading method and architecture
This document proposes a read-write split near-line data loading method and architecture to:
- Increase data loading performance by separating write operations from read operations. A WriteServer handles write requests and loads data to HDFS to be read from by RegionServers.
- Control resources used by write operations to ensure read operations are not starved of resources like CPU, network, disk I/O, and handlers.
- Provide an architecture corresponding to Kafka and HDFS for streaming data from Kafka to HDFS to be loaded into HBase in a delayed manner.
- Include optimizations like task balancing across WriteServer slaves, prioritized compaction of small files, and customizable storage engines.
- Report test results showing one Write
hbaseconasia2017: Apache HBase at Netease
This document summarizes Netease's use of Apache HBase for big data. It discusses Netease operating 7 HBase clusters with 200+ RegionServers and hundreds of terabytes of data across more than 40 applications. It outlines key practices for Linux system configuration, HBase schema design, garbage collection, and request queueing at the table level. Ongoing work includes region server grouping, inverted indexes, and improving high availability of HBase.
Off-heaping the Apache HBase Read Path
Anoop Sam John and Ramkrishna Vasudevan (Intel)
HBase provides an LRU based on heap cache but its size (and so the total data size that can be cached) is limited by Java’s max heap space. This talk highlights our work under HBASE-11425 to allow the HBase read path to work directly from the off-heap area.
Development to Production with Sharded MongoDB Clusters
Severalnines presentation at MongoDB Stockholm Conference.
Presentation covers:
- mongoDB sharding/clustering concepts
- recommended dev/test/prod setups
- how to verify your deployment
- how to avoid downtime
- what MongoDB metrics to watch
- when to scale
HBase at Flurry
This document summarizes Dave Latham's experience operating a large HBase cluster for Flurry Analytics. Some key points include:
- Flurry runs two HBase clusters with 1000 slave nodes each storing over 400TB of data across 30 tables and 250k regions.
- They faced bottlenecks with the HMaster, NameNode, and ZooKeeper. The cluster also had too many small regions which caused problems.
- Lessons learned include the need to understand system limitations, monitor performance, have a load-aware design, and know your workload characteristics.
- Future work includes testing HDFS HA, snapshots at large scale, separating workloads, and testing larger region and HDFS block sizes.
IMC Summit 2016 Breakout - Per Minoborg - Work with Multiple Hot Terabytes in...
IMC Summit 2016 Breakout - Per Minoborg - Work with Multiple Hot Terabytes in...In-Memory Computing Summit
Speedment SQL Reflector is a software solution that allows applications to get automatically updated data in real time. The SQL Reflector loads data from your existing SQL database and feeds it into an in-memory data grid e.g. GridGain. When started, the SQL reflector will load your selected existing relational data into your map cluster. Also, any subsequent changes that are made to the relational database (regardless how, via your application, script, SQL commands or even stored procedures) are then continuously fed to your GridGain nodes. Even SQL-transactions are preserved so that your maps will always reflect a valid state of the underlying SQL database.BigTable PreReading
This document outlines the key concepts of Google's Bigtable distributed database system. It discusses Bigtable's data model, APIs, implementation details including its use of GFS and Chubby, refinements to improve performance, and lessons learned. The document poses many questions about Bigtable's design and implementation for further discussion.
Argus Production Monitoring at Salesforce
We’ll present details about Argus, a time-series monitoring and alerting platform developed at Salesforce to provide insight into the health of infrastructure as an alternative to systems such as Graphite and Seyren.
HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...
Gap Inc Direct, the online division for Gap Inc., uses HBase to serve, in real-time, apparel catalog for all its brands’ and markets’ web sites. This case study will review the business case as well as key decisions regarding schema selection and cluster configurations. We will also discuss implementation challenges and insights that were learned.
HBaseCon 2015: HBase Operations at Xiaomi
In this session, you will learn the work Xiaomi has done to improve the availability and stability of our HBase clusters, including cross-site data and service backup and a coordinated compaction framework. You'll also learn about the Themis framework, which supports cross-row transactions on HBase based on Google's percolator algorithm, and its usage in Xiaomi's applications.
HBaseConAsia2018 Track1-1: Use CCSMap to improve HBase YGC time
CCSMap is a new data structure introduced by Alibaba to improve the performance of HBase. It aims to reduce the overhead of the default Java ConcurrentSkipListMap (CSLM) data structure and improve young garbage collection times. CCSMap chunks data into fixed size blocks for better memory management and uses direct pointers between nodes for faster access. It also provides various configuration options. Alibaba has achieved significant performance gains using CCSMap in HBase, including reduced young GC times, and it continues working to integrate CCSMap further and add new features.
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency.
Real world capacity
The document discusses capacity planning considerations for running Cassandra in a production environment based on the author's experience. Some key points include starting with a small number of nodes and low replication factor initially, using tools like Puppet from the beginning, upgrading nodes quickly to the same version, ensuring sufficient disk space, memory, and fast disks, avoiding upgrading right before capacity is reached, and understanding read/write patterns and latency needs. Different strategies are needed for real-time versus batch workloads.
From HDFS to S3: Migrate Pinterest Apache Spark Clusters
The document discusses Pinterest migrating their Apache Spark clusters from HDFS to S3 storage. Some key points:
1) Migrating to S3 provided significantly better performance due to the higher IOPS of modern EC2 instances compared to their older HDFS nodes. Jobs saw 25-35% improvements on average.
2) S3 is eventually consistent while HDFS is strongly consistent, so they implemented the S3Committer to handle output consistency issues during job failures.
3) Metadata operations like file moves were very slow in S3, so they optimized jobs to reduce unnecessary moves using techniques like multipart uploads to S3.
More Related Content
What's hot
HBaseConAsia2018 Track1-7: HDFS optimizations for HBase at Xiaomi
This document discusses optimizations made to HDFS for Hbase at XiaoMi. It addresses issues like shared memory allocation causing full GC in datanodes, listen drops on SSD clusters causing delays, peer cache bucket adjustment, and connection timeouts. Changes like preallocating shared memory, increasing socket backlog, reducing client and datanode timeouts, and adjusting the datanode dead node detection help improve performance and availability. The overall goal is to maintain local data, return fast responses from HDFS to Hbase, and reduce GC overhead from both systems.
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
In this presentation, we will introduce Hotspot's Garbage First collector (G1GC) as the most suitable collector for latency-sensitive applications running with large memory environments. We will first discuss G1GC internal operations and tuning opportunities, and also cover tuning flags that set desired GC pause targets, change adaptive GC thresholds, and adjust GC activities at runtime. We will provide several HBase case studies using Java heaps as large as 100GB that show how to best tune applications to remove unpredicted, protracted GC pauses.
Tales from Taming the Long Tail
Deepankar Reddy and Ishan Chhabra (Rocket Fuel)
Rocket Fuel is a marketing technology company that participates in 120+ billion real-time bidding auctions daily to show the right ad to the right user at the right time for our clients. In this talk, we discuss our efforts to systematically identify causes of, and how to decrease, long-tail read latencies.
Real-time HBase: Lessons from the Cloud
Speaker: Bryan Beaudreault (HubSpot)
Running HBase in real time in the cloud provides an interesting and ever-changing set of challenges -- instance types are not ideal, neighbors can degrade your performance, and instances can randomly die in unanticipated ways. This talk will cover what HubSpot has learned about running in production on Amazon EC2, how it handle DR and redundancy, and the tooling the team has found to be the most helpful.
IMC Summit 2016 Breakout - Andy Pavlo - What Non-Volatile Memory Means for th...
IMC Summit 2016 Breakout - Andy Pavlo - What Non-Volatile Memory Means for th...In-Memory Computing Summit
The advent of non-volatile memory (NVM) will fundamentally change the dichotomy between memory and durable storage in database management systems (DBMSs). These new NVM devices are almost as fast as DRAM, but all writes to it are potentially persistent even after power loss. Existing DBMSs are unable to take full advantage of this technology because their internal architectures are predicated on the assumption that memory is volatile. That means when NVM finally arrives, just like when you finally passed that kidney stone after three weeks, everyone will be relieved but the transition will be painful. Many of the components of legacy DBMSs will become unnecessary and will degrade the performance of data intensive applications.Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bull...
Packets per second (PPS) is an often overlooked value within an environment. Most network concerns are around throughput and interface speed, but what happens when this value becomes the bottleneck due to large-scale hosting providers (AWS, Azure, etc.) with rigid standards? This talk covers what a Redis packet looks like and how the out-of-the-box configuration can drastically affect
packet per second overhead. From here we’ll deep dive into specific configuration values which help lower PPS numbers, as well as different Redis master/slave relationships that can be utilized to keep PPS below inflexible network thresholds.
hbaseconasia2017: HBase在Hulu的使用和实践
Qianxi Zhang
1. Hulu是美国最受欢迎的在线视频网站之一，Hulu Beijing是Hulu第二大研发中心。北京大数据基础架构团队负责整个公司的大数据基础架构的研发和运维。
2. HBase在Hulu的概况
3. HBase在Hulu的使用
4. 用户画像系统，存放所有用户的基本信息，用户行为，第三方DMP数据和机器学习结果标签(几十万个Qualifier)，Spark和Spark Streaming读写HBase数据，运行各种机器学习模型，为公司的视频推荐，精准广告和Marketing团队服务
5. HBase在Hulu的优化
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
Володимир Цап "Constraint driven infrastructure - scale or tune?"
Volodymyr Tsap discusses how to save money on infrastructure through constraint driven design. He provides examples of hardware configurations on AWS, bare metal servers, and PaaS platforms to demonstrate how costs can be optimized. Tsap also outlines ways to reduce software costs through choices in operating system, virtualization, databases, and orchestration. Infrastructure support costs depend on the complexity of the environment, with basic setups costing $500-800 per month while more advanced architectures are $4,000-6,000 per month. The overall message is that money saved through optimization can be invested in people.
hbaseconasia2017: Large scale data near-line loading method and architecture
This document proposes a read-write split near-line data loading method and architecture to:
- Increase data loading performance by separating write operations from read operations. A WriteServer handles write requests and loads data to HDFS to be read from by RegionServers.
- Control resources used by write operations to ensure read operations are not starved of resources like CPU, network, disk I/O, and handlers.
- Provide an architecture corresponding to Kafka and HDFS for streaming data from Kafka to HDFS to be loaded into HBase in a delayed manner.
- Include optimizations like task balancing across WriteServer slaves, prioritized compaction of small files, and customizable storage engines.
- Report test results showing one Write
hbaseconasia2017: Apache HBase at Netease
This document summarizes Netease's use of Apache HBase for big data. It discusses Netease operating 7 HBase clusters with 200+ RegionServers and hundreds of terabytes of data across more than 40 applications. It outlines key practices for Linux system configuration, HBase schema design, garbage collection, and request queueing at the table level. Ongoing work includes region server grouping, inverted indexes, and improving high availability of HBase.
Off-heaping the Apache HBase Read Path
Anoop Sam John and Ramkrishna Vasudevan (Intel)
HBase provides an LRU based on heap cache but its size (and so the total data size that can be cached) is limited by Java’s max heap space. This talk highlights our work under HBASE-11425 to allow the HBase read path to work directly from the off-heap area.
Development to Production with Sharded MongoDB Clusters
Severalnines presentation at MongoDB Stockholm Conference.
Presentation covers:
- mongoDB sharding/clustering concepts
- recommended dev/test/prod setups
- how to verify your deployment
- how to avoid downtime
- what MongoDB metrics to watch
- when to scale
HBase at Flurry
This document summarizes Dave Latham's experience operating a large HBase cluster for Flurry Analytics. Some key points include:
- Flurry runs two HBase clusters with 1000 slave nodes each storing over 400TB of data across 30 tables and 250k regions.
- They faced bottlenecks with the HMaster, NameNode, and ZooKeeper. The cluster also had too many small regions which caused problems.
- Lessons learned include the need to understand system limitations, monitor performance, have a load-aware design, and know your workload characteristics.
- Future work includes testing HDFS HA, snapshots at large scale, separating workloads, and testing larger region and HDFS block sizes.
IMC Summit 2016 Breakout - Per Minoborg - Work with Multiple Hot Terabytes in...
IMC Summit 2016 Breakout - Per Minoborg - Work with Multiple Hot Terabytes in...In-Memory Computing Summit
Speedment SQL Reflector is a software solution that allows applications to get automatically updated data in real time. The SQL Reflector loads data from your existing SQL database and feeds it into an in-memory data grid e.g. GridGain. When started, the SQL reflector will load your selected existing relational data into your map cluster. Also, any subsequent changes that are made to the relational database (regardless how, via your application, script, SQL commands or even stored procedures) are then continuously fed to your GridGain nodes. Even SQL-transactions are preserved so that your maps will always reflect a valid state of the underlying SQL database.BigTable PreReading
This document outlines the key concepts of Google's Bigtable distributed database system. It discusses Bigtable's data model, APIs, implementation details including its use of GFS and Chubby, refinements to improve performance, and lessons learned. The document poses many questions about Bigtable's design and implementation for further discussion.
Argus Production Monitoring at Salesforce
We’ll present details about Argus, a time-series monitoring and alerting platform developed at Salesforce to provide insight into the health of infrastructure as an alternative to systems such as Graphite and Seyren.
HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...
Gap Inc Direct, the online division for Gap Inc., uses HBase to serve, in real-time, apparel catalog for all its brands’ and markets’ web sites. This case study will review the business case as well as key decisions regarding schema selection and cluster configurations. We will also discuss implementation challenges and insights that were learned.
HBaseCon 2015: HBase Operations at Xiaomi
In this session, you will learn the work Xiaomi has done to improve the availability and stability of our HBase clusters, including cross-site data and service backup and a coordinated compaction framework. You'll also learn about the Themis framework, which supports cross-row transactions on HBase based on Google's percolator algorithm, and its usage in Xiaomi's applications.
HBaseConAsia2018 Track1-1: Use CCSMap to improve HBase YGC time
CCSMap is a new data structure introduced by Alibaba to improve the performance of HBase. It aims to reduce the overhead of the default Java ConcurrentSkipListMap (CSLM) data structure and improve young garbage collection times. CCSMap chunks data into fixed size blocks for better memory management and uses direct pointers between nodes for faster access. It also provides various configuration options. Alibaba has achieved significant performance gains using CCSMap in HBase, including reduced young GC times, and it continues working to integrate CCSMap further and add new features.
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency.
What's hot (20)
HBaseConAsia2018 Track1-7: HDFS optimizations for HBase at Xiaomi
HBaseConAsia2018 Track1-7: HDFS optimizations for HBase at Xiaomi
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
IMC Summit 2016 Breakout - Andy Pavlo - What Non-Volatile Memory Means for th...
IMC Summit 2016 Breakout - Andy Pavlo - What Non-Volatile Memory Means for th...
Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bull...
Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bull...
Володимир Цап "Constraint driven infrastructure - scale or tune?"
Володимир Цап "Constraint driven infrastructure - scale or tune?"
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
Development to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB Clusters
IMC Summit 2016 Breakout - Per Minoborg - Work with Multiple Hot Terabytes in...
IMC Summit 2016 Breakout - Per Minoborg - Work with Multiple Hot Terabytes in...
HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...
HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...
HBaseConAsia2018 Track1-1: Use CCSMap to improve HBase YGC time
HBaseConAsia2018 Track1-1: Use CCSMap to improve HBase YGC time
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
Similar to Date-tiered Compaction Policy for Time-series Data
Real world capacity
The document discusses capacity planning considerations for running Cassandra in a production environment based on the author's experience. Some key points include starting with a small number of nodes and low replication factor initially, using tools like Puppet from the beginning, upgrading nodes quickly to the same version, ensuring sufficient disk space, memory, and fast disks, avoiding upgrading right before capacity is reached, and understanding read/write patterns and latency needs. Different strategies are needed for real-time versus batch workloads.
From HDFS to S3: Migrate Pinterest Apache Spark Clusters
The document discusses Pinterest migrating their Apache Spark clusters from HDFS to S3 storage. Some key points:
1) Migrating to S3 provided significantly better performance due to the higher IOPS of modern EC2 instances compared to their older HDFS nodes. Jobs saw 25-35% improvements on average.
2) S3 is eventually consistent while HDFS is strongly consistent, so they implemented the S3Committer to handle output consistency issues during job failures.
3) Metadata operations like file moves were very slow in S3, so they optimized jobs to reduce unnecessary moves using techniques like multipart uploads to S3.
Healthcare Claim Reimbursement using Apache Spark
The document discusses rewriting a claims reimbursement system using Spark. It describes how Spark provides better performance, scalability and cost savings compared to the previous Oracle-based system. Key points include using Spark for ETL to load data into a Delta Lake data lake, implementing the business logic in a reusable Java library, and seeing significant increases in processing volumes and speeds compared to the prior system. Challenges and tips for adoption are also provided.
Maaz Anjum - IOUG Collaborate 2013 - An Insight into Space Realization on ODA...
The document provides an overview of Maaz Anjum, a solutions architect specializing in Oracle products like OEM12c, Golden Gate, and Engineered Systems. It lists his email, blog, and experience using Oracle products since 2001. It also provides details about Bias Corporation, the company he works for, including its founding date, certifications, expertise, customers, and implementations.
Using Time Window Compaction Strategy For Time Series Workloads
Cassandra is a great fit for high write use cases, which makes it a popular choice for storing time series and sensor-collection workloads. At Crowdstrike, we've been using Cassandra for just that purpose, collecting petabytes of expiring time series data. In this talk, I'll discuss compaction in time series workloads, and the TimeWindowCompactionStrategy we developed specifically for this purpose. I'll detail TWCS specific configuration properties, some lesser known compaction sub-properties that apply to all compaction strategies, and also cover other general tricks and tuning that are useful for very large time-series workloads.
Cassandra admin
The document provides information on tools for monitoring and administering Cassandra clusters. It discusses Cassandra-specific tools like nodetool for monitoring metrics and performing administrative tasks. It also lists system monitoring tools for metrics like CPU usage, disk I/O, network activity, and more. Finally, it gives best practices for hardware selection with Cassandra including recommendations for memory, CPU, and disk space.
Kafka on ZFS: Better Living Through Filesystems
(Hugh O'Brien, Jet.com) Kafka Summit SF 2018
You’re doing disk IO wrong, let ZFS show you the way. ZFS on Linux is now stable. Say goodbye to JBOD, to directories in your reassignment plans, to unevenly used disks. Instead, have 8K Cloud IOPS for $25, SSD speed reads on spinning disks, in-kernel LZ4 compression and the smartest page cache on the planet. (Fear compactions no more!)
Learn how Jet’s Kafka clusters squeeze every drop of disk performance out of Azure, all completely transparent to Kafka.
-Striping cheap disks to maximize instance IOPS
-Block compression to reduce disk usage by ~80% (JSON data)
-Instance SSD as the secondary read cache (storing compressed data), eliminating >99% of disk reads and safe across host redeployments
-Upcoming features: Compressed blocks in memory, potentially quadrupling your page cache (RAM) for free
We’ll cover:
-Basic Principles
-Adapting ZFS for cloud instances (gotchas)
-Performance tuning for Kafka
-Benchmarks
The Parquet Format and Performance Optimization Opportunities
The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads.
As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general.
This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
Mastering Data Management: Leveraging FME for Cloud Native Databases
Facing challenges with data management in the cloud environment? Tackling issues like inefficient data management, integration difficulties, scalability constraints, and security concerns? You're not alone. In this Part 2 of our database webinar series, we'll offer you the tools to overcome these challenges, streamline workflows, boost performance, and implement industry-leading cloud data management practices -- fueling your organization to extract maximum value from your data.
In this hour-long interactive session, our experts will walk you through:
The different formats and benefits of using cloud-native databases with FME
Strategies for deploying FME in cloud environments to ensure seamless integration and scalability
Best practices for managing and scaling data effectively in a cloud environment
Exclusive tips on data processing and transformation specifically for cloud-native databases
To cap it off, our experts will host a live Q&A session to answer any questions you may have. Don't let this opportunity slip by to optimize your cloud-native data workflows. Register now and secure your place at the forefront of data management innovation. Your data's untapped potential awaits.
Taking Splunk to the Next Level - Architecture Breakout Session
This document provides an overview and agenda for taking a Splunk deployment to the next level by addressing scaling needs and high availability requirements. It discusses growing use cases and data volumes, making Splunk mission critical through clustering, and supporting global deployments. The agenda covers scaling strategies like indexer clustering, search head clustering, and hybrid cloud deployments. It also promotes justifying increased spending by mapping dependencies and costs of failures across an organization's systems.
Why does my choice of storage matter with cassandra?
The document discusses how the choice of storage is critical for Cassandra deployments. It summarizes that SSDs are generally the best choice as they have no moving parts, resulting in much faster performance compared to HDDs. Specifically, SSDs can eliminate issues caused by disk seeks and allow the use of compaction strategies like leveled compaction that require lower seek times. The document provides measurements showing SSDs are up to 100x faster than HDDs for read/write speeds and latency. It recommends choosing local SSD storage in a JBOD configuration when possible for best performance and manageability.
How to scale your data collection on the cloud like a champ
When we met with WalkMe, a company which offers helpful in-app walkthroughs (we use it for our app and it’s great), our meeting took a surprising turn. We expected a discussion about crunching big data. They already had a data collection mechanism in place, but they had a problem that preceded any sort of crunching. They had a problem scaling the data collection process. Here's how they solved it.
Accelerating Apache Spark Shuffle for Data Analytics on the Cloud with Remote...
The increasing challenge to serve ever-growing data driven by AI and analytics workloads makes disaggregated storage and compute more attractive as it enables companies to scale their storage and compute capacity independently to match data & compute growth rate. Cloud based big data services is gaining momentum as it provides simplified management, elasticity, and pay-as-you-go model.
Tachyon memory centric, fault tolerance storage for cluster framworks
Tachyon is a memory-centric storage system that provides reliable data sharing across different cluster frameworks and jobs at memory speeds. It uses a lineage-based approach to track the sequence of jobs and tasks that create output files to enable fault tolerance through recomputation rather than replication. Tachyon consists of a lineage layer to capture metadata and deliver high throughput I/O, and a persistence layer that takes asynchronous checkpoints of hot files to bound recomputation costs following failures while avoiding impacting system performance. Evaluation results showed Tachyon was over 100x faster than disk-based systems and reduced network traffic by up to 50% while keeping recomputation overhead below 1.6%.
Cassandra in Operation
Some vignettes and advice based on prior experience with Cassandra clusters in live environments. Includes some material from other operational slides.
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
This document discusses Apache Kudu, an open source columnar storage system for analytics workloads on Hadoop. Kudu is designed to enable both fast analytics queries as well as real-time updates on fast changing data. It aims to fill gaps in the current Hadoop storage landscape by supporting simultaneous high throughput scans, low latency reads/writes, and ACID transactions. An example use case described is for real-time fraud detection on streaming financial data.
Building Data Pipelines with SMACK: Designing Storage Strategies for Scale an...
Learn how to build an effective storage layer for a variety of workloads. With changing trends in system and storage hardware, understanding design trade-offs can be a challenge. This webinar will focus on cutting through the noise and diving into the choices that matter when designing for scale and performance.
Video: https://youtu.be/uEL8vyVSIis
Building a High Performance Analytics Platform
The document discusses using flash memory to build a high performance data platform. It notes that flash memory is faster than disk storage and cheaper than RAM. The platform utilizes NVMe flash drives connected via PCIe for high speed performance. This allows it to provide in-memory database speeds at the cost and density of solid state drives. It can scale independently by adding compute nodes or storage nodes. The platform offers a unified database for both real-time and analytical workloads through common APIs.
Veeam webinar - Deduplication best practices
This document discusses best practices for using data deduplication with Veeam Backup & Replication 6.5 and Windows Server 2012. It recommends using data deduplication for backups with long retention periods of over 60 days to reduce storage costs. It provides guidance on planning and configuring deduplication, including sizing estimates, optimizing the backup repository, using forward incremental backups, and enabling inline and compression deduplication. It also demonstrates how Windows Server 2012 provides global deduplication across backup jobs and volumes.
Creating an 86,000 Hour Speech Dataset with Apache Spark and TPUs
This document summarizes Daniel Galvez's presentation on creating The People's Speech Dataset using Apache Spark and TPUs. The key points are:
1) The dataset aims to provide 86,000 hours of speech data with forced alignments between audio and transcripts in order to be challenging, free to use, and have a commercial license.
2) The conceptual workload is to take hour-long audio files, split them into 15 second segments, and use a pretrained speech recognition model to discover when each word in the transcript was said.
3) Creating the dataset encountered limitations with accelerator-aware scheduling in Spark, memory issues with PySpark UDFs, crashes in TPUs, and the need to reorder data by
Similar to Date-tiered Compaction Policy for Time-series Data (20)
From HDFS to S3: Migrate Pinterest Apache Spark Clusters
From HDFS to S3: Migrate Pinterest Apache Spark Clusters
Maaz Anjum - IOUG Collaborate 2013 - An Insight into Space Realization on ODA...
Maaz Anjum - IOUG Collaborate 2013 - An Insight into Space Realization on ODA...
Using Time Window Compaction Strategy For Time Series Workloads
Using Time Window Compaction Strategy For Time Series Workloads
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Mastering Data Management: Leveraging FME for Cloud Native Databases
Mastering Data Management: Leveraging FME for Cloud Native Databases
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout Session
Why does my choice of storage matter with cassandra?
Why does my choice of storage matter with cassandra?
How to scale your data collection on the cloud like a champ
How to scale your data collection on the cloud like a champ
Accelerating Apache Spark Shuffle for Data Analytics on the Cloud with Remote...
Accelerating Apache Spark Shuffle for Data Analytics on the Cloud with Remote...
Tachyon memory centric, fault tolerance storage for cluster framworks
Tachyon memory centric, fault tolerance storage for cluster framworks
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
Building Data Pipelines with SMACK: Designing Storage Strategies for Scale an...
Building Data Pipelines with SMACK: Designing Storage Strategies for Scale an...
Creating an 86,000 Hour Speech Dataset with Apache Spark and TPUs
Creating an 86,000 Hour Speech Dataset with Apache Spark and TPUs
More from HBaseCon
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
Zhiyong Bai
As a high performance and scalable key value database, Zhihu use HBase to provide online data store system along with Mysql and Redis. Zhihu’s platform team had accumulated some experience in technology of container, and this time, based on Kubernetes, we build flexible platform of online HBase system, create multiple logic isolated HBase clusters on the shared physical cluster with fast rapid，and provide customized service for different business needs. Combined with Consul and DNS server, we implement high available access of HBase using client mainly written with Python. This presentation is mainly shared the architecture of online HBase platform in Zhihu and some practical experience in production environment.
hbaseconasia2017 hbasecon hbase
hbaseconasia2017: HBase on Beam
Jingcheng Du
Apache Beam is an open source and unified programming model for defining batch and streaming jobs that run on many execution engines, HBase on Beam is a connector that allows Beam to use HBase as a bounded data source and target data store for both batch and streaming data sets. With this connector HBase can work with many batch and streaming engines directly, for example Spark, Flink, Google Cloud Dataflow, etc. In this session, I will introduce Apache Beam, and the current implementation of HBase on Beam and the future plan on this.
hbaseconasia2017 hbasecon hbase
https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
Ashish Singhi
HBase Disaster recovery solution aims to maintain high availability of HBase service in case of disaster of one HBase cluster with very minimal user intervention. This session will introduce the HBase disaster recovery use cases and the various solutions adopted at Huawei like.
a) Cluster Read-Write mode
b) DDL operations synchronization with standby cluster
c) Mutation and bulk loaded data replication
d) Further challenges and pending work
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
Tianying Chang
HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency.
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程
Mon-Fong Mike Jiang, Kuan-Yu Hubert Fan-Chiang and Tienyu Rebecca Lin
自2011年起，我們就開始使用HBase作為結構化大數據的儲存工具，主要是做為半導體製造設備參數的分析。為了有效進行數據查詢，我們開發Standard Query Language(SQL)的整合介面，最早的方式是(1)自行開發GUI操作介面及(2)透過自行定義SQL語法的方式進行，但是這樣會衍生出很多額外的工作，特別是SQL Parser與對應的HBase API的連結。
為了解決此問題，我們解析了Hive QL Parser作為主要的核心，將此部分的原始碼整合進HareDB HBase Client之中，另外，也整合了HBase Coprocessor，可以加速查詢的進行，這個架構我們實際使用在數個半導體製造廠的大數據系統中，也展現了高查詢效率。
除此之外，透過整合Kafka來處理串流數據的匯入，同時對於數據分析的呈現也加上Cube建立工具，這些都是實際開發大數據系統時陸續面對的問題與解決方法，我們將分享這一連串的系統開發過程。
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
hbaseconasia2017: 基于HBase的企业级大数据平台
Xinyu Zhang, Xueliang Chen and Zheng Fan
基于HBase的大数据平台已经成为中国人寿新一代综合业务处理系统中非常重要的基础性数据平台。目前基于该平台已经整合了上百TB的数据，并将几亿客户的客户、业务、接触数据整合到一个统一的数据模型中，并基于此形成了上千个客户标签。同时，基于该平台为客户、营销员和内部管理人员提供了销售支持、客户服务、运营支持等多类应用。通过APP、网页等形式提供了多种信息的检索和查询，并通过深度学习模型提供了反欺诈等方面的数据应用。
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
hbaseconasia2017: HBase at JD.com
Xingbo Peng, Nan Zhang and Bang Wen
1.规模现状
HBase在京东CTO体系中经历了数年的发展，集群规模已经达到3000+台，支持了京东600+业务系统，京东CTO体系的HBase集群，已经经历了多次618和双11的考验。京东CTO体系是HBase的重要用户。
2.应用的业务场景
介绍HBase在京东的典型应用的业务，包括监控、风控、推荐、广告等
3.高可用改进
介绍我们在HBase集群高可用方面做的一些工作，包括跨机房容灾、多租户-资源分组、集群安全等
4.运维实践
主要介绍我们在HBase集群运维上的一些实践，包括：HBase集群监控系统Mummut、报警系统、HBase集群与大数据平台结合、业务运营及数据迁移等
5.未来展望
介绍我们正在基于HBase做的及未来要做的一些工作，包括：kylin、phoenix和容器化部署等
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
CTBase is a lightweight HBase client designed for structured data use cases. It provides features like schematized tables, global secondary indexes, cluster tables for joins, and online schema changes. Tagram is a distributed bitmap index implementation on HBase that supports ad-hoc queries on low-cardinality attributes with millisecond latency. CloudTable Service offers HBase as a managed service on Huawei Cloud with features including easy maintenance, security, high performance, service level agreements, high availability and low cost.
hbaseconasia2017: HBase Practice At XiaoMi
Zheng Hu
We'll share some HBase experience at XiaoMi:
1. How did we tuning G1GC for HBase Clusters.
2. Development and performance of Async HBase Client.
hbaseconasia2017 hbasecon hbase xiaomi https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
HBaseCon2017 Democratizing HBase
As HBase and Hadoop continue to become routine across enterprises, these enterprises inevitably shift priorities from effective deployments to cost-efficient operations. Consolidation of infrastructure, the sum of hardware, software, and system-administrator effort, is the most common strategy to reduce costs. As a company grows, the number of business organizations, development teams, and individuals accessing HBase grows commensurately, creating a not-so-simple requirement: HBase must effectively service many users, each with a variety of use-cases. This is problem is known as multi-tenancy. While multi-tenancy isn’t a new problem, it also isn’t a solved one, in HBase or otherwise. This talk will present a high-level view of the common issues organizations face when multiple users and teams share a single HBase instance and how certain HBase features were designed specifically to mitigate the issues created by the sharing of finite resources.
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
Hundreds of millions of people use Quora to find accurate, informative, and trustworthy answers to their questions. As it so happens, counting things at scale is both an important and a difficult problem to solve.
In this talk, we will be talking about Quanta, Quora's counting system built on top of HBase that powers our high-volume near-realtime analytics that serves many applications like ads, content views, and many dashboards. In addition to regular counting, Quanta supports count propagation along the edges of an arbitrary DAG. HBase is the underlying data store for both the counting data and the graph data.
We will describe the high-level architecture of Quanta and share our design goals, constraints, and choices that enabled us to build Quanta very quickly on top of our existing infrastructure systems.
HBaseCon2017 Transactions in HBase
In the age of NoSQL, big data storage engines such as HBase have given up ACID semantics of traditional relational databases, in exchange for high scalability and availability. However, it turns out that in practice, many applications require consistency guarantees to protect data from concurrent modification in a massively parallel environment. In the past few years, several transaction engines have been proposed as add-ons to HBase; three different engines, namely Omid, Tephra, and Trafodion were open-sourced in Apache alone. In this talk, we will introduce and compare the different approaches from various perspectives including scalability, efficiency, operability and portability, and make recommendations pertaining to different use cases.
HBaseCon2017 Highly-Available HBase
In order to effectively predict and prevent online fraud in real time, Sift Science stores hundreds of terabytes of data in HBase—and needs it to be always available. This talk will cover how we used circuit-breaking, cluster failover, monitoring, and automated recovery procedures to improve our HBase uptime from 99.7% to 99.99% on top of unreliable cloud hardware and networks.
HBaseCon2017 Apache HBase at Didi
In DiDi Chuxing Company, which is China’s most popular ride-sharing company. we use HBase to serve when we have a bigdata problem.
We run three clusters which serve different business needs. We backported the Region Grouping feature back to our internal HBase version so we could isolate the different use cases.
We built the Didi HBase Service platform which is popular amongst engineers at our company. It includes a workflow and project management function as well as a user monitoring view.
Internally we recommend users use Phoenix to simplify access.even more,we used row timestamp;multidimensional table schema to slove muti dimension query problems
C++, Go, Python, and PHP clients get to HBase via thrift2 proxies and QueryServer.
We run many important buisness applications out of our HBase cluster such as ETA/GPS/History Order/API metrics monitoring/ and Traffic in the Cloud. If you are interested in any aspects listed above, please come to our talk. We would like to share our experiences with you.
HBaseCon2017 gohbase: Pure Go HBase Client
gohbase is an implementation of an HBase client in pure Go: https://github.com/tsuna/gohbase. In this presentation we'll talk about its architecture and compare its performance against the native Java HBase client as well as AsyncHBase (http://opentsdb.github.io/asynchbase/) and some nice characteristics of golang that resulted in a simpler implementation.
HBaseCon2017 Improving HBase availability in a multi tenant environment
The document discusses improvements made by Hubspot's Big Data Team to increase the availability of HBase in a multi-tenant environment. It outlines reducing the cost of region server failures by improving mean time to recovery, addressing issues that slowed recovery, and optimizing the load balancer. It also details eliminating workload-driven failures through service limits and improving hardware monitoring to reduce impacts of failures. The changes resulted in 8-10x faster balancing, reduced recovery times from 90 to 30 seconds, and consistently achieving 99.99% availability across clusters.
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
Both Spark and HBase are widely used, but how to use them together with high performance and simplicity is a very hard topic. Spark HBase Connector(SHC) provides feature rich and efficient access to HBase through Spark SQL. It bridges the gap between the simple HBase key value store and complex relational SQL queries and enables users to perform complex data analytics on top of HBase using Spark.
SHC implements the standard Spark data source APIs, and leverages the Spark catalyst engine for query optimization. To achieve high performance, SHC constructs the RDD from scratch instead of using the standard HadoopRDD. With the customized RDD, all critical techniques can be applied and fully implemented, such as partition pruning, column pruning, predicate pushdown and data locality. The design makes the maintenance very easy, while achieving a good tradeoff between performance and simplicity.
Also, SHC has supported Phoenix data as input to HBase in addition to Avro data. Defaulting to a simple native binary encoding seems susceptible to future changes and is a risk for users who write data from SHC into HBase. For example, with SHC going forward, backwards compatibility needs to be properly handled. So the default, SHC needs to support a more standard and well tested format like Phoenix.
In this talk, we will demo how SHC works, how to use SHC in secure/non-secure clusters, how SHC works with multi-HBase clusters, etc. This talk will also benefit people who use Spark and other data sources (besides HBase) as it inspires them with ideas of how to support high performance data source access at the Spark DataFrame level.
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
In this talk we introduce Apache Beam, a unified model to create efficient and portable data processing pipelines. Beam uses a single set of abstractions to implement both batch and streaming computations that can be executed in different environments, e.g. Apache Spark, Apache Flink and Google Dataflow. Beam not only does data processing, but can be used as a tool to ingest/extract data to/from different data stores including HBase. We will present interaction scenarios between HBase and Beam and explore Beam's Input/Output (IO) model and how we leverage it to provide support for HBase.
HBaseCon2017 HBase at Xiaomi
Our team is responsible for storage at Xiaomi and we provide storage services for dozens of businesses, such as personal cloud storage for smart phones and user profile data. So we will share some practices and improvements of HBase at Xiaomi：
1: We upgraded most of our cluster from 0.94 to 0.98 in the last year and will share some experience about upgrading.
2: We encountered some problems and made some improvements on replication.
3: We fixed or still fixing some confusing behavior from client side.
4: We introduced some improvements on scan to make users easy to use and reduce the time of RPC requests.
5: We implement an asynchronous hbase client which is an important feature for HBase 2.0.
HBaseCon2017 HBase/Phoenix @ Scale @ Salesforce
This document summarizes Salesforce's use of HBase and Phoenix for storing and querying large amounts of unstructured data at scale. Some key details include:
- Salesforce uses over 100 HBase clusters to store both customer and internal data, handling over 4 billion write requests and 600 million read requests per day.
- This includes storing login data, archived relational data, user activity, machine metrics and more, totaling over 80 terabytes written and 500 gigabytes read daily.
- An internal metrics database collects data from over 80,000 machines, storing 11.4 trillion metrics and growing, with 2.8 trillion metrics added in the last 6 months alone.
More from HBaseCon (20)
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
Recently uploaded
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
When it is all about ERP solutions, companies typically meet their needs with common ERP solutions like SAP, Oracle, and Microsoft Dynamics. These big players have demonstrated that ERP systems can be either simple or highly comprehensive. This remains true today, but there are new factors to consider, including a promising new contender in the market that’s Odoo. This blog compares Odoo ERP with traditional ERP systems and explains why many companies now see Odoo ERP as the best choice.
What are ERP Systems?
An ERP, or Enterprise Resource Planning, system provides your company with valuable information to help you make better decisions and boost your ROI. You should choose an ERP system based on your company’s specific needs. For instance, if you run a manufacturing or retail business, you will need an ERP system that efficiently manages inventory. A consulting firm, on the other hand, would benefit from an ERP system that enhances daily operations. Similarly, eCommerce stores would select an ERP system tailored to their needs.
Because different businesses have different requirements, ERP system functionalities can vary. Among the various ERP systems available, Odoo ERP is considered one of the best in the ERp market with more than 12 million global users today.
Odoo is an open-source ERP system initially designed for small to medium-sized businesses but now suitable for a wide range of companies. Odoo offers a scalable and configurable point-of-sale management solution and allows you to create customised modules for specific industries. Odoo is gaining more popularity because it is built in a way that allows easy customisation, has a user-friendly interface, and is affordable. Here, you will cover the main differences and get to know why Odoo is gaining attention despite the many other ERP systems available in the market.
Mobile app Development Services | Drona Infotech
Drona Infotech is one of the Best Mobile App Development Company In Noida Maintenance and ongoing support. mobile app development Services can help you maintain and support your app after it has been launched. This includes fixing bugs, adding new features, and keeping your app up-to-date with the latest
Visit Us For :
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Requirement Traceability in Xen Functional Safety
Using OpenFastTrace to trace requirements in Xen functional safety
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech is a premier mobile app development company in Noida, providing cutting-edge solutions for businesses.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
Unveiling the Advantages of Agile Software Development.pdf
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
Malibou Pitch Deck For Its €3M Seed Round
French start-up Malibou raised a €3 million Seed Round to develop its payroll and human resources
management platform for VSEs and SMEs. The financing round was led by investors Breega, Y Combinator, and FCVC.
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Charla impartida en el evento de "KuberTENes Birthday Bash Guadalajara" para celebrar el 10mo. aniversario de Kubernetes #kuberTENes #celebr8k8s #k8s
How to write a program in any programming language
How to write a program in any programming language
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar's Aviation Industry Q1 Incident Report is out now!
The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers.
SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.
Microservice Teams - How the cloud changes the way we work
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
原版一模一样【微信：741003700 】【加拿大英属哥伦比亚大学毕业证本科学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Top 9 Trends in Cybersecurity for 2024.pptx
Security and risk management (SRM) leaders face disruptions on technological, organizational, and human fronts. Preparation and pragmatic execution are key for dealing with these disruptions and providing the right cybersecurity program.
Hand Rolled Applicative User Validation Code Kata
Could you use a simple piece of Scala validation code (granted, a very simplistic one too!) that you can rewrite, now and again, to refresh your basic understanding of Applicative operators <*>, <*, *>?
The goal is not to write perfect code showcasing validation, but rather, to provide a small, rough-and ready exercise to reinforce your muscle-memory.
Despite its grandiose-sounding title, this deck consists of just three slides showing the Scala 3 code to be rewritten whenever the details of the operators begin to fade away.
The code is my rough and ready translation of a Haskell user-validation program found in a book called Finding Success (and Failure) in Haskell - Fall in love with applicative functors.
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Presentation about a VSCode plugin from Dario Jurisic at the GSD Community Stage meetup
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
UMN硕士毕业证成绩单【微信95270640】购买（明尼苏达大学毕业证成绩单硕士学历）Q微信95270640代办UMN学历认证留信网伪造明尼苏达大学学位证书精仿明尼苏达大学本科/硕士文凭证书补办明尼苏达大学 diplomaoffer,Transcript购买明尼苏达大学毕业证成绩单购买UMN假毕业证学位证书购买伪造明尼苏达大学文凭证书学位证书,专业办理雅思、托福成绩单，学生ID卡，在读证明，海外各大学offer录取通知书，毕业证书，成绩单，文凭等材料:1:1完美还原毕业证、offer录取通知书、学生卡等各种在读或毕业材料的防伪工艺（包括 烫金、烫银、钢印、底纹、凹凸版、水印、防伪光标、热敏防伪、文字图案浮雕，激光镭射，紫外荧光，温感光标）学校原版上有的工艺我们一样不会少，不论是老版本还是最新版本，都能保证最高程度还原，力争完美以求让所有同学都能享受到完美的品质服务。
#毕业证成绩单 #毕业証 #成绩单 #學生卡 #OFFER录取通知书 #雅思#托福等……
国外大学明尼苏达大学明尼苏达大学毕业证offer制作方法（一对一专业服务）
1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）
— — 制作工艺 【高仿真】— —
凭借多年的制作经验本公司制作明尼苏达大学明尼苏达大学毕业证offer《激光》《水印》《钢印》《烫金》《紫外线》凹凸版uv版等防伪技术一流高精仿度几乎跟学校100%相同！让您绝对满意。
— — -公司理念 【诚信为主】— — —
我們以質量求生存.以服务求发展有雄厚的实力专业的团队咨询顾问为您细心解答可详谈是真是假眼见为实让您真正放心平凡人生,尽我所能助您一臂之力让我們携手圆您梦想!
此贴长年有效【贴心专线/微-信: 95270640】敬请保留此联系方式以备用！如有不在线请给我们留言！我们将在第一时间给您回复!上散发着一抹抹的光晕而这每处自然形成的细节融合在一起浑然天成的美实在令人心生愉悦小道的周边无秩序的生长着几株艳丽的野花红的粉的紫的虽混乱无章却给这幅美景更增添一份性感夹杂着一份纯洁的妖娆毫无违和感实在给人带来一份悠然幸福的心情如果说现在的审美已经断然拒绝了无声的话那么在树林间飞掠而过的小鸟叽叽咋咋的叫声是否就是这最后的点睛之笔悠然走在林间的小路上宁静与清香一丝丝的盛夏气息吸入身体昔日生活里的繁忙多
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Artificia Intellicence and XPath Extension Functions
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
Recently uploaded (20)
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
How to write a program in any programming language
How to write a program in any programming language
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Date-tiered Compaction Policy for Time-series Data
- 1. Date Tiered Compaction Clara Xiong
- 2. ▪Not a time-series system(row key is independent of time) but similar read/write pattern. ▪Exponential data growth pushes disk IO to the limit. ▪Date tiered compaction partitions the storage by time. ▪Resulted in drastically lower latency and massive reduction of disk IO consumption. Why Date Tiered Compaction for Flurry?
- 3. ▪ Inspired by Cassandra’s Date Tiered Compaction. ▪When new window appears, older windows get merged into exponentially larger windows. What is Date Tiered Compaction? Base window = 1 hour, windows per tier = 4 From https://labs.spotify.com/2014/12/18/date-tiered-compaction/
- 4. ▪Exponential sizing balances the tradeoffs between optimal scan performance of recent data and total file count. ▪Out of range time windows will no longer be merged for efficient data retention management What is Date Tiered Compaction? (cont’d)
- 5. ▪Use maxTimestamp for file selection to favor late-arriving data. ▪Normalize timestamp by the order of sequence id. ▪Plugged-in per- window compaction policy to reduce wasteful compaction. File Selection
- 6. A Date Tiered Layout in Production 0 50 100 150 200 250 300 350 400 450 File Size (20MB) Window (Days)
- 7. ▪HDFS Read Bytes ▪Mapper Run Time Minutes Days After DTC Was On 0.00E+00 2.00E+14 4.00E+14 6.00E+14 8.00E+14 1.00E+15 1.20E+15 1.40E+15 1.60E+15 1.80E+15 2.00E+15 0 2 4 6 8 10 12 7-day job 1-day job 0 10 20 30 40 50 60 70 0 2 4 6 8 10 12 7-day job(avg) 7-day job(max) 1-day job(avg) 1-day job(max)
- 8. Reduction on HDFS IO (%) Notes 1-day look back job 79% 7-day look back job 97% Minor Compaction 85% Still provisional IO Reduction Summary
- 9. ▪ Date Tiered Compactor (Duo Zhang @xiaomi) ● Outputs store files for given time boundaries. ● Allows us to split the data by window boundaries ▪ Date tiered major compaction ● Maintains tiered layout while processing deletes, updates and locality. ● Restores layout skewed by out-of-order writes, bulk load and etc. ▪ Window factory and Minor compaction for archiving (Duo Zhang) Enhancements
- 10. ▪What will benefit most? ●Most writes have cell timestamp near the write time and most reads are for limited recent data out of the full data set ▪What will not? ●Random GET without a time range ●Frequent out of order data writes creating long tails, especially writes with future timestamps ●Frequent bulk load of heavily overlapping time-ranges Is Date Tiered Compaction for You?
- 11. ▪ Don’t turn it on cluster wide: Meta should not be on Date Tiered Compaction. Limit all the related configuration changes by table /column family ▪ Increase blocking file count and max compaction file count settings generously ▪ Put in pressure aware compaction throughput throttler to limit the impact of synchronized large minor compaction Best Practice Recommendations
- 12. ▪ Design specification https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLK IqL6T-oHoRLZ323MG_uy8 ▪ Email claraxiong@yahoo-inc.com Questions?