Keynote: Apache HBase at Yahoo! Scale

Hive + Tez: A Performance Deep DiveDataWorks Summit

Oracle RAC 19c with Standard Edition (SE) 2 - Support Update

Markus Michalewicz

MyRocks Deep Dive

Yoshinori Matsunobu

SeaweedFS introduction

chrislusf

Scaling HBase for Big Data

Salesforce Engineering

Apache Ranger

Rommel Garcia

Data in Hadoop is getting bigger every day, consumers of the data are growing, organizations are now looking at making their Hadoop cluster compliant to federal regulations and commercial demands. Apache Ranger simplifies the management of security policies across all components in Hadoop. Ranger provides granular access controls to data. The deck describes what security tools are available in Hadoop and their purpose then it moves on to discuss in detail Apache Ranger.

Consumer offset management in Kafka

Joel Koshy

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

confluent

MQ, ETL and ESB middleware are often used as integration backbone between legacy applications, modern microservices and cloud services. This introduces several challenges and complexities like point-to-point integration or non-scalable architectures. This session discusses how to build a completely event-driven streaming platform leveraging Apache Kafka’s open source messaging, integration and streaming components to leverage distributed processing, fault-tolerance, rolling upgrades and the ability to reprocess events. Learn the differences between a event-driven streaming platform leveraging Apache Kafka and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson

Tuning and Debugging in Apache Spark

Patrick Wendell

Video: https://www.youtube.com/watch?v=kkOG_aJ9KjQ This talk gives details about Spark internals and an explanation of the runtime behavior of a Spark application. It explains how high level user programs are compiled into physical execution plans in Spark. It then reviews common performance bottlenecks encountered by Spark users, along with tips for diagnosing performance problems in a production application.

Internal Hive

Recruit Technologies

[오픈소스컨설팅]Day #1 MySQL 엔진소개, 튜닝, 백업 및 복구, 업그레이드방법

Ji-Woong Choi

Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...

GetInData

Did you like it? Check out our E-book: Apache NiFi - A Complete Guide https://ebook.getindata.com/apache-nifi-complete-guide Apache NiFi is one of the most popular services for running ETL pipelines otherwise it’s not the youngest technology. During the talk, there are described all details about migrating pipelines from the old Hadoop platform to the Kubernetes, managing everything as the code, monitoring all corner cases of NiFi and making it a robust solution that is user-friendly even for non-programmers. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale

Colin Charles

As proxies (and database routers) go, the first one I ever used was the now deprecated MySQL Proxy. Since then, I've managed to use MariaDB MaxScale quite a bit (including its fork AirBnB MaxScale), played around with ProxySQL in recent time, and also started taking a look at MySQL Router. In this quick 20-minute overview, we'll discuss why these three exist, a feature comparison, and reasons when to use the right tool for the job.

Running MariaDB in multiple data centers

MariaDB plc

MariaDB is often deployed in multiple data centers for high availability and/or disaster recovery. Tim Tadeo, Senior Sales Engineer, walks through real-world use cases and the topologies customers have created to leverage multiple data centers. He also discusses important considerations and how to address them, as well as more advanced options such as dedicated binlog servers for cross–data center replication.

HBase in Practice

larsgeorge

From: DataWorks Summit 2017 - Munich - 20170406 HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.

Apache HBase - Just the Basics

Keynote: The Future of Apache HBase

NTT DATA Technology & Innovation

What's hot

Apache Bigtop3.2 (仮)（Open Source Conference 2022 Online/Hiroshima 発表資料）

Floating on a RAFT: HBase Durability with Apache Ratis

Mastering PostgreSQL Administration

EDB

Apache Tez – Present and Future

Hive + Tez: A Performance Deep DiveDataWorks Summit

Oracle RAC 19c with Standard Edition (SE) 2 - Support Update

Markus Michalewicz

MyRocks Deep Dive

Yoshinori Matsunobu

SeaweedFS introduction

chrislusf

Scaling HBase for Big Data

Salesforce Engineering

Apache Ranger

Rommel Garcia

Consumer offset management in Kafka

Joel Koshy

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

confluent

Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson

Tuning and Debugging in Apache Spark

Patrick Wendell

Internal Hive

Recruit Technologies

[오픈소스컨설팅]Day #1 MySQL 엔진소개, 튜닝, 백업 및 복구, 업그레이드방법

Ji-Woong Choi

Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...

GetInData

The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale

Colin Charles

Running MariaDB in multiple data centers

MariaDB plc

HBase in Practice

larsgeorge

What's hot (20)

Apache Bigtop3.2 (仮)（Open Source Conference 2022 Online/Hiroshima 発表資料）

Floating on a RAFT: HBase Durability with Apache Ratis

Mastering PostgreSQL Administration

Apache Tez – Present and Future

Hive + Tez: A Performance Deep Dive

Oracle RAC 19c with Standard Edition (SE) 2 - Support Update

MyRocks Deep Dive

SeaweedFS introduction

Scaling HBase for Big Data

Apache Ranger

Consumer offset management in Kafka

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC

Tuning and Debugging in Apache Spark

Internal Hive

[오픈소스컨설팅]Day #1 MySQL 엔진소개, 튜닝, 백업 및 복구, 업그레이드방법

Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...

The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale

Running MariaDB in multiple data centers

HBase in Practice

Viewers also liked

Apache HBase - Just the Basics

Keynote: The Future of Apache HBase

Keynote: Welcome Message/State of Apache HBase

Apache HBase at Airbnb

Improvements to Apache HBase and Its Applications in Alibaba Search

Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight

Nitin Verma, Pravin Mittal, and Maxim Lukiyanov (Microsoft) This session presents our success story of enabling a big internal customer on Microsoft Azure’s HBase service along with the methodology and tools used to meet high-throughput goals. We will also present how new features in HBase (like BucketCache and MultiWAL) are helping our customers in the medium-latency/high-bandwidth cloud-storage scenario.

Hw09 Practical HBase Getting The Most From Your H Base InstallCloudera, Inc.

Chicago Data Summit: Apache HBase: An Introduction

Cloudera, Inc.

Apache HBase is an open source distributed data-store capable of managing billions of rows of semi-structured data across large clusters of commodity hardware. HBase provides real-time random read-write access as well as integration with Hadoop MapReduce, Hive, and Pig for batch analysis. In this talk, Todd will provide an introduction to the capabilities and characteristics of HBase, comparing and contrasting it with traditional database systems. He will also introduce its architecture and data model, and present some example use cases.

HBase Read High Availability Using Timeline Consistent Region Replicas

enissoz

Hourglass: a Library for Incremental Processing on Hadoop

Matthew Hayes

Hadoop enables processing of large data sets through its relatively easy-to-use semantics. However, jobs are often written inefficiently for tasks that could be computed incrementally due to the burdensome incremental state management for the programmer. This paper introduces Hourglass, a library for developing incremental monoid computations on Hadoop. It runs on unmodified Hadoop and provides an accumulator-based interface for programmers to store and use state across successive runs; the framework ensures that only the necessary subcomputations are performed. It is successfully used at LinkedIn, one of the largest online social networks, for many use cases in dashboarding and machine learning. Hourglass is open source and freely available.

HBaseCon 2015: HBase Operations at Xiaomi

Argus Production Monitoring at Salesforce

Apache Mesos at Twitter (Texas LinuxFest 2014)

Chris Aniszczyk

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

Yahoo Developer Network

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we'll look at how SDC's "intent-driven" approach keeps the data flowing, whether you're processing data 'off-cluster', in Spark, or in MapReduce. StreamSets software delivers performance management for data flows that feed the next generation of big data applications. Its mission is to bring operational excellence to the management of data in motion, so that data arrives on time and with quality, accelerating analysis and decision making. StreamSets Data Collector is in use at hundreds of companies where it brings unprecedented visibility into and control over data as it moves between an expanding variety of sources and destinations. Speakers: Pat Patterson has been working with Internet technologies since 1997, building software and working with communities at Sun Microsystems, Huawei, Salesforce and StreamSets. At Sun, Pat was the community lead for the OpenSSO open source project, while at Huawei he developed cloud storage infrastructure software. Part of the developer evangelism team at Salesforce, Pat focused on identity, integration and the Internet of Things. Now community champion at StreamSets, Pat is responsible for the care and feeding of the StreamSets open source community.

HBaseCon 2015: HBase @ Flipboard

Flipboard services over 100 million users using heterogenous results including user generated content, interest profile, algorithmically generated content, social firehose, friends graph, ads, and web/rss crawlers. To personalize and serve these results in real time, Flipboard employs a variety of data models, access patterns and configuration. This talk will present how some of these strategies are implemented using HBase.

Tales from Taming the Long Tail

Rolling Out Apache HBase for Mobile Offerings at Visa

Partha Saha and CW Chung (Visa) Visa has embarked on an ambitious multi-year redesign of its entire data platform that powers its business. As part of this plan, the Apache Hadoop ecosystem, including HBase, will now become a staple in many of its solutions. Here, we will describe our journey in rolling out a high-availability NoSQL solution based on HBase behind some of our prominent mobile offerings.

Breaking the Sound Barrier with Persistent Memory

Liqi Yi and Shylaja Kokoori (Intel) A fully optimized HBase cluster could easily hit the limit of the underlying storage device’s capability, which is beyond the reach of software optimization alone. To get around this constraint, we need a new design that brings data processing and data storage closer together. In this presentation, we will look at how persistent memory will change the way large datasets are stored. We will review the hardware characteristics of 3D XPoint™, a new persistent memory technology with low latency and high capacity. We will also discuss opportunities for further improvement within the HBase framework using persistent memory.

HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment

Apache HBase Improvements and Practices at Xiaomi

Viewers also liked (20)

Apache HBase - Just the Basics

Keynote: The Future of Apache HBase

Keynote: Welcome Message/State of Apache HBase

Apache HBase at Airbnb

Improvements to Apache HBase and Its Applications in Alibaba Search

Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight

Hw09 Practical HBase Getting The Most From Your H Base Install

Chicago Data Summit: Apache HBase: An Introduction

HBase Read High Availability Using Timeline Consistent Region Replicas

Hourglass: a Library for Incremental Processing on Hadoop

HBaseCon 2015: HBase Operations at Xiaomi

Argus Production Monitoring at Salesforce

Apache Mesos at Twitter (Texas LinuxFest 2014)

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

HBaseCon 2015: HBase @ Flipboard

Tales from Taming the Long Tail

Rolling Out Apache HBase for Mobile Offerings at Visa

Breaking the Sound Barrier with Persistent Memory

HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment

Apache HBase Improvements and Practices at Xiaomi

Similar to Keynote: Apache HBase at Yahoo! Scale

Millions of Regions in HBase: Size Matters

HBaseCon 2015: Multitenancy in HBase

Since 2013, Yahoo! has been successfully running multi-tenant HBase clusters. Our tenants run applications ranging from real-time processing (e.g. content personalization, Ad targeting) to operational warehouses (e.g. advertising, content). Tenants are guaranteed an adequate level of resource isolation and security. This is achieved through the use of open source and in-house developed HBase features such as region server groups, group-based replication, and group-based favored nodes. Today, with the increase in adoption and new use cases, we are working towards scaling our HBase clusters to support petabytes of data without compromising on performance and operability. A common tradeoff when scaling a cluster to this size is to increase the size of a region, thus avoiding the problem of having too many regions on a cluster. However, large regions negatively affect the performance and operability of a cluster mainly because region size determines the following: 1. granularity for load distribution, and 2. amount of write amplification due to compaction. Thus we are working towards enabling an HBase cluster to host at least a million regions. In this presentation, we will walk through the key features we have implemented as well as share our experiences working on multi-tenancy and scaling the cluster.

HBaseCon2017 Achieving HBase Multi-Tenancy with RegionServer Groups and Favor...

Achieving HBase Multi-Tenancy with RegionServer Groups and Favored Nodes At Yahoo! HBase has been running as a hosted multi-tenant service since 2013. In a single HBase cluster we have around 30 tenants running various types of workloads (ie batch, near real-time, ad-hoc, etc). Typically such a deployment would cause tenant workloads to negatively affect each other because of resource contention (disk, cpu, network, cache thrashing, etc). Using RegionServer Groups we are able to designate a dedicated subset of RegionServers in a cluster to host only tables of a given tenant (HBASE-6721). Most HBase deployments use HDFS as their distributed filesystem, which in turn does not guarantee that a region’s data is locally available to the hosting regionserver. This poses a problem when providing isolation since the hdfs data blocks may have to be read remotely from a different tenant’s host thus contending for disk or network resources. Favored nodes addresses this problem by providing hints to HDFS on which datanodes data should be stored and only assigns regions to these favored regionservers (HBASE-15531). We will walk through these features explaining our motivation, how they work as well as our experiences running these multi-tenant clusters. These features will be available in Apache HBase 2.0. by Francis Liu and Thiruvel Thirumoolan https://youtu.be/xorEYNyLMbM

Benchmarking Apache Samza: 1.2 million messages per sec per node

Tao Feng

Riding the Stream Processing Wave (Strange loop 2019)

Samarth Shetty

Lessons learned from scaling YARN to 40K machines in a multi tenancy environment

At Microsoft we have 500,000 jobs per day running a special query engine over exabytes of data with above 70% CPU utilization and we have made a big bet on YARN as our Resource Manager. We leverage Federation (YARN-2915) and Mercury (YARN-2877) to scale out to more than 40,000 nodes (spread across clusters) at 3000 allocate/second while achieving <5s response time @95%tile. To get there, we had to overcome several challenges: how do you measure and ensure there are no performance regressions? How do you deal with vast heterogeneous container sizes (from seconds to minutes)? What lessons did we find to get high CPU utilization with Mercury? What issues from HA, JVM, routing policy, throttling and DoS we found running and scaling? Join this session and learn about the challenges and learning from running YARN at humongous scale.

Arc305 how netflix leverages multiple regions to increase availability an i...

Ruslan Meshenberg

Apache Performance Tuning: Scaling Out

Sander Temme

Achieving HBase Multi-Tenancy with RegionServer Groups and Favored Nodes

Bangladesh Network Operators Group

At Yahoo! HBase has been running as a hosted multi-tenant service since 2013. In a single HBase cluster we have around 30 tenants running various types of workloads (ie batch, near real-time, ad-hoc, etc). Typically such a deployment would cause tenant workloads to negatively affect each other because of resource contention (disk, cpu, network, cache thrashing, etc). Using RegionServer Groups we are able to designate a dedicated subset of RegionServers in a cluster to host only tables of a given tenant (HBASE-6721). Most HBase deployments use HDFS as their distributed filesystem, which in turn does not guarantee that a region’s data is locally available to the hosting regionserver. This poses a problem when providing isolation since the hdfs data blocks may have to be read remotely from a different tenant’s host thus contending for disk or network resources. Favored nodes addresses this problem by providing hints to HDFS on which datanodes data should be stored and only assigns regions to these favored regionservers (HBASE-15531). We will walk through these features explaining our motivation, how they work as well as our experiences running these multi-tenant clusters. These features will be available in Apache HBase 2.0.

DNS/DNSSEC by Nurul Islam

MyNOG

Realtime olap architecture in apache kylin 3.0

Shi Shao Feng

HBase Operations and Best Practices

Venu Anuganti

Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...

Flink Forward

Let’s be honest: Running a distributed stateful stream processor that is able to handle terabytes of state and tens of gigabytes of data per second while being highly available and correct (in an exactly-once sense) does not work without any planning, configuration and monitoring. While the Flink developer community tries to make everything as simple as possible, it is still important to be aware of all the requirements and implications In this talk, we will provide some insights into the greatest operations mysteries of Flink from a high-level perspective: - Capacity and resource planning: Understand the theoretical limits. - Memory and CPU configuration: Distribute resources according to your needs. - Setting up High Availability: Planning for failures. - Checkpointing and State Backends: Ensure correctness and fast recovery For each of the listed topics, we will introduce the concepts of Flink and provide some best practices we have learned over the past years supporting Flink users in production.

Hands-on DNSSEC Deployment

Domain Name System (DNS) Fundamentals

WebSniffer

Scylla Summit 2016: Outbrain Case Study - Lowering Latency While Doing 20X IO...

ScyllaDB

Dnscluster @ DevOps Krakow 2013

Slawomir Skowron

Hadoop 3.0 - Revolution or evolution?

Uwe Printz

With Hadoop-3.0.0-alpha2 being released in January 2017, it's time to have a closer look at the features and fixes of Hadoop 3.0. We will have a look at Core Hadoop, HDFS and YARN, and answer the emerging question whether Hadoop 3.0 will be an architectural revolution like Hadoop 2 was with YARN & Co. or will it be more of an evolution adapting to new use cases like IoT, Machine Learning and Deep Learning (TensorFlow)?

Perforce Server: The Next Generation

Perforce

Omid Efficient Transaction Mgmt and Processing for HBaseDataWorks Summit

Similar to Keynote: Apache HBase at Yahoo! Scale (20)

Millions of Regions in HBase: Size Matters

HBaseCon 2015: Multitenancy in HBase

HBaseCon2017 Achieving HBase Multi-Tenancy with RegionServer Groups and Favor...

Benchmarking Apache Samza: 1.2 million messages per sec per node

Riding the Stream Processing Wave (Strange loop 2019)

Lessons learned from scaling YARN to 40K machines in a multi tenancy environment

Arc305 how netflix leverages multiple regions to increase availability an i...

Apache Performance Tuning: Scaling Out

Achieving HBase Multi-Tenancy with RegionServer Groups and Favored Nodes

DNS/DNSSEC by Nurul Islam

Realtime olap architecture in apache kylin 3.0

HBase Operations and Best Practices

Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...

Hands-on DNSSEC Deployment

Domain Name System (DNS) Fundamentals

Scylla Summit 2016: Outbrain Case Study - Lowering Latency While Doing 20X IO...

Dnscluster @ DevOps Krakow 2013

Hadoop 3.0 - Revolution or evolution?

Perforce Server: The Next Generation

Omid Efficient Transaction Mgmt and Processing for HBase

More from HBaseCon

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes

Zhiyong Bai As a high performance and scalable key value database, Zhihu use HBase to provide online data store system along with Mysql and Redis. Zhihu’s platform team had accumulated some experience in technology of container, and this time, based on Kubernetes, we build flexible platform of online HBase system, create multiple logic isolated HBase clusters on the shared physical cluster with fast rapid，and provide customized service for different business needs. Combined with Consul and DNS server, we implement high available access of HBase using client mainly written with Python. This presentation is mainly shared the architecture of online HBase platform in Zhihu and some practical experience in production environment. hbaseconasia2017 hbasecon hbase

hbaseconasia2017: HBase on Beam

Jingcheng Du Apache Beam is an open source and unified programming model for defining batch and streaming jobs that run on many execution engines, HBase on Beam is a connector that allows Beam to use HBase as a bounded data source and target data store for both batch and streaming data sets. With this connector HBase can work with many batch and streaming engines directly, for example Spark, Flink, Google Cloud Dataflow, etc. In this session, I will introduce Apache Beam, and the current implementation of HBase on Beam and the future plan on this. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase Disaster Recovery Solution at Huawei

Ashish Singhi HBase Disaster recovery solution aims to maintain high availability of HBase service in case of disaster of one HBase cluster with very minimal user intervention. This session will introduce the HBase disaster recovery use cases and the various solutions adopted at Huawei like. a) Cluster Read-Write mode b) DDL operations synchronization with standby cluster c) Mutation and bulk loaded data replication d) Further challenges and pending work hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest

Tianying Chang HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程

Mon-Fong Mike Jiang, Kuan-Yu Hubert Fan-Chiang and Tienyu Rebecca Lin 自2011年起，我們就開始使用HBase作為結構化大數據的儲存工具，主要是做為半導體製造設備參數的分析。為了有效進行數據查詢，我們開發Standard Query Language(SQL)的整合介面，最早的方式是(1)自行開發GUI操作介面及(2)透過自行定義SQL語法的方式進行，但是這樣會衍生出很多額外的工作，特別是SQL Parser與對應的HBase API的連結。為了解決此問題，我們解析了Hive QL Parser作為主要的核心，將此部分的原始碼整合進HareDB HBase Client之中，另外，也整合了HBase Coprocessor，可以加速查詢的進行，這個架構我們實際使用在數個半導體製造廠的大數據系統中，也展現了高查詢效率。除此之外，透過整合Kafka來處理串流數據的匯入，同時對於數據分析的呈現也加上Cube建立工具，這些都是實際開發大數據系統時陸續面對的問題與解決方法，我們將分享這一連串的系統開發過程。 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Apache HBase at Netease

Xinxin Fan and Hongxiang Jiang First, we will give a brief introduction about the HBase service at Netease，include the basic cluster info and the key HBase service. And then we will talk same tips about the tuning practices for HBase. Last, we will introduce some improvements at the internal HBase version. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase在Hulu的使用和实践

Qianxi Zhang 1. Hulu是美国最受欢迎的在线视频网站之一，Hulu Beijing是Hulu第二大研发中心。北京大数据基础架构团队负责整个公司的大数据基础架构的研发和运维。 2. HBase在Hulu的概况 3. HBase在Hulu的使用 4. 用户画像系统，存放所有用户的基本信息，用户行为，第三方DMP数据和机器学习结果标签(几十万个Qualifier)，Spark和Spark Streaming读写HBase数据，运行各种机器学习模型，为公司的视频推荐，精准广告和Marketing团队服务 5. HBase在Hulu的优化 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: 基于HBase的企业级大数据平台

Xinyu Zhang, Xueliang Chen and Zheng Fan 基于HBase的大数据平台已经成为中国人寿新一代综合业务处理系统中非常重要的基础性数据平台。目前基于该平台已经整合了上百TB的数据，并将几亿客户的客户、业务、接触数据整合到一个统一的数据模型中，并基于此形成了上千个客户标签。同时，基于该平台为客户、营销员和内部管理人员提供了销售支持、客户服务、运营支持等多类应用。通过APP、网页等形式提供了多种信息的检索和查询，并通过深度学习模型提供了反欺诈等方面的数据应用。 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase at JD.com

Xingbo Peng, Nan Zhang and Bang Wen 1.规模现状 HBase在京东CTO体系中经历了数年的发展，集群规模已经达到3000+台，支持了京东600+业务系统，京东CTO体系的HBase集群，已经经历了多次618和双11的考验。京东CTO体系是HBase的重要用户。 2.应用的业务场景介绍HBase在京东的典型应用的业务，包括监控、风控、推荐、广告等 3.高可用改进介绍我们在HBase集群高可用方面做的一些工作，包括跨机房容灾、多租户-资源分组、集群安全等 4.运维实践主要介绍我们在HBase集群运维上的一些实践，包括：HBase集群监控系统Mummut、报警系统、HBase集群与大数据平台结合、业务运营及数据迁移等 5.未来展望介绍我们正在基于HBase做的及未来要做的一些工作，包括：kylin、phoenix和容器化部署等 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Large scale data near-line loading method and architecture

Shuaifeng Zhou When we do real-time data loading to HBase, we use put/putlist interface. After receiving put request, regionserver will write WAL, write data into memory store, flush memory store to disk-store, then compact files again and again. That precedure occupies too much resource and causing read/write performance decrease. To solve the problem, we provide a kind of near-line loading method and architecture, greatly increase the loading bandwidth, and decrease the influence to read operations. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei

Jieshan Bi and Yanhui Zhong 1. CTBase: A light-weight HBase client for structured data. 1). Schematized table, more friendly for structured data storage. 2). Global secondary index for HBase. 3). HBase Query DSL. JSON based light-weight API. 4) Cluster table. Pre-joining with keys, a better solution for cross-table join queries from HBase. 2. Tagram: Distributed bitmap index implementation with HBase. 1). Distributed bitmap index for accelerating AD-HOC queries with low cardinality columns. 2). Powerful and flexible query API. 3). Tagram offers millisecond-level query latency. 3. CloudTable Service Introduction: HBase on Huawei cloud. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase Practice At XiaoMi

hbaseconasia2017: hbase-2.0.0

HBaseCon2017 Democratizing HBase

As HBase and Hadoop continue to become routine across enterprises, these enterprises inevitably shift priorities from effective deployments to cost-efficient operations. Consolidation of infrastructure, the sum of hardware, software, and system-administrator effort, is the most common strategy to reduce costs. As a company grows, the number of business organizations, development teams, and individuals accessing HBase grows commensurately, creating a not-so-simple requirement: HBase must effectively service many users, each with a variety of use-cases. This is problem is known as multi-tenancy. While multi-tenancy isn’t a new problem, it also isn’t a solved one, in HBase or otherwise. This talk will present a high-level view of the common issues organizations face when multiple users and teams share a single HBase instance and how certain HBase features were designed specifically to mitigate the issues created by the sharing of finite resources.

HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest

HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency.

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

Hundreds of millions of people use Quora to find accurate, informative, and trustworthy answers to their questions. As it so happens, counting things at scale is both an important and a difficult problem to solve. In this talk, we will be talking about Quanta, Quora's counting system built on top of HBase that powers our high-volume near-realtime analytics that serves many applications like ads, content views, and many dashboards. In addition to regular counting, Quanta supports count propagation along the edges of an arbitrary DAG. HBase is the underlying data store for both the counting data and the graph data. We will describe the high-level architecture of Quanta and share our design goals, constraints, and choices that enabled us to build Quanta very quickly on top of our existing infrastructure systems.

HBaseCon2017 Transactions in HBase

In the age of NoSQL, big data storage engines such as HBase have given up ACID semantics of traditional relational databases, in exchange for high scalability and availability. However, it turns out that in practice, many applications require consistency guarantees to protect data from concurrent modification in a massively parallel environment. In the past few years, several transaction engines have been proposed as add-ons to HBase; three different engines, namely Omid, Tephra, and Trafodion were open-sourced in Apache alone. In this talk, we will introduce and compare the different approaches from various perspectives including scalability, efficiency, operability and portability, and make recommendations pertaining to different use cases.

HBaseCon2017 Highly-Available HBase

HBaseCon2017 Apache HBase at Didi

In DiDi Chuxing Company, which is China’s most popular ride-sharing company. we use HBase to serve when we have a bigdata problem. We run three clusters which serve different business needs. We backported the Region Grouping feature back to our internal HBase version so we could isolate the different use cases. We built the Didi HBase Service platform which is popular amongst engineers at our company. It includes a workflow and project management function as well as a user monitoring view. Internally we recommend users use Phoenix to simplify access.even more,we used row timestamp;multidimensional table schema to slove muti dimension query problems C++, Go, Python, and PHP clients get to HBase via thrift2 proxies and QueryServer. We run many important buisness applications out of our HBase cluster such as ETA/GPS/History Order/API metrics monitoring/ and Traffic in the Cloud. If you are interested in any aspects listed above, please come to our talk. We would like to share our experiences with you.

HBaseCon2017 gohbase: Pure Go HBase Client

More from HBaseCon (20)

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes

hbaseconasia2017: HBase on Beam

hbaseconasia2017: HBase Disaster Recovery Solution at Huawei

hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程

hbaseconasia2017: Apache HBase at Netease

hbaseconasia2017: HBase在Hulu的使用和实践

hbaseconasia2017: 基于HBase的企业级大数据平台

hbaseconasia2017: HBase at JD.com

hbaseconasia2017: Large scale data near-line loading method and architecture

hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei

hbaseconasia2017: HBase Practice At XiaoMi

hbaseconasia2017: hbase-2.0.0

HBaseCon2017 Democratizing HBase

HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon2017 Transactions in HBase

HBaseCon2017 Highly-Available HBase

HBaseCon2017 Apache HBase at Didi

HBaseCon2017 gohbase: Pure Go HBase Client

Recently uploaded

Orion Context Broker introduction 20240604

Fermin Galan

Enhancing Research Orchestration Capabilities at ORNL.pdf

Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.

GlobusWorld 2024 Opening Keynote session

Large Language Models and the End of Programming

Matt Welsh

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Mind IT Systems

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Cracking the code review at SpringIO 2024

Paco van Beckhoven

Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production. Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process? In this session we will cover: - The Art of Effective Code Reviews - Streamlining the Review Process - Elevating Reviews with Automated Tools By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.

2024 RoOUG Security model for the cloud.pptx

Georgi Kodinov

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

rickgrimesss22

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

XfilesPro

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j

Graphic Design Crash Course for beginners

e20449

Globus Compute wth IRI Workflows - GlobusWorld 2024

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Google

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-pilot-review/ AI Pilot Review: Key Features ✅Deploy AI expert bots in Any Niche With Just A Click ✅With one keyword, generate complete funnels, websites, landing pages, and more. ✅More than 85 AI features are included in the AI pilot. ✅No setup or configuration; use your voice (like Siri) to do whatever you want. ✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It… ✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again. ✅ZERO Limits On Features Or Usages ✅Use Our AI-powered Traffic To Get Hundreds Of Customers ✅No Complicated Setup: Get Up And Running In 2 Minutes ✅99.99% Up-Time Guaranteed ✅30 Days Money-Back Guarantee ✅ZERO Upfront Cost See My Other Reviews Article: (1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review

OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam

takuyayamamoto1800

Enterprise Resource Planning System in Telangana

NYGGS Automation Suite

Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics. To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/

Understanding Globus Data Transfers with NetSage