SlideShare a Scribd company logo
1 of 28
hosted by
颜禹 Yan Yu
The Application of HBase in New Energy
Vehicle Monitoring System
博尼施科技 Burnish Technology Co. Ltd
hosted by
1. Background
2. Challenges and Decisions
3. System architecture
4. Why HBase
5. Challenges with HBase
6. Data backup in HBase
7. Conclusion
8. Prospect
content
hosted by
Backgroud
 100k running vehicles online
 send 2 packages per minute every vehicle.
 data space
 the origin package size is 1KB.
 parsed package size is about 7KB.
 one vehicle will produce 20mb data per day.
 2TB data were generated per day.
 2.9 billion rows need to write to HBase every day.
 concurrency
 3.3k persistent tps
 100k persistent connections
 3.3MB origin data needs to parse per second
 23.1MB parsed data needs to storage in HBase per second
hosted by
hosted by
Challenges
 Small team
 Limited funds, machines,
 Short deadline
 System integration
 How to handle the huge amount of vehicle data
 Demands are very foggy.
hosted by
Decisions
 Language
 Message queue
 Database
 Develop flow
 Micro service
 Monolithic service
 Deploy and maintain
 Cloud
 Native data center
hosted by
Language
 C/C++
 High performance
 Hard to integrate
 Long development time
 Java
 High performance
 Rich third part packages
 Easy to integrate with big data system, i.e. Hadoop, HBase, spark
 Python
 Sprint development
 Rich third part packages
 Performance issue with multi thread
 Golang
 Easy to write multi thread program
 There is no Golang developer in our team
hosted by
Message
 Redis
 High performance
 High memory requirement
 Hard to scale
 Celery
 More fit for distribution task
 Easy to develop with python
 Redis or rabbitmq as it’s backend
 Kafka
 Write to disk first to ensure the message security
 Support consumer group
 Auto balance
 Enough performance for our system
 Easy to scale
 Rabbitmq
 Classic message queue
 Performance
hosted by
Database
 MySql
 Relational database
 Fit for storage static information
 ORM support
 MongoDB
 Document based
 ORM support
 Hard to maintain and scale
 Hbase
 Column database
 High write performance
 Easy to handle TB
 Easy to scale
 OpenTSDB
 Time series database
 Based on HBase
hosted by
Monolithic service vs micro service
 Monolithic service
 Easy to develop when system is not very complicate
 Acceleration for development
 Build the basic system due the the foggy demands
 Micro service
 Easy to scale in a complicate system
 Rapid iteration
 More developers requirement
hosted by
Develop flow
Dependences on central server.
 Dependences on central server.
 Easy to setup on one server
 Single point failure risk
 Confliction over multi
developers
hosted by
Develop flow
 Dependences on individual docker engine.
 Easy to setup with docker-compose
 No single point risk
 High memory develop machine requirement(starting from 32GB)
hosted by
Deploy and maintain
 Cloud
 Easy setup
 Low cost with small scale
 Fast deployment
 No employees
 Native data center
 Hard setup
 Expensive cost with small scale
 Professional employee to maintain our data center
hosted by
Deploy and maintain
 Deploy system with kubernetes
 Easy to management
 Rapid scale
 Compute and storage split separation
 Deploy basic component with cloud service
 Fast deploy
 Careless
 Easy to get high available service
 No employees
hosted by
How a small team hold a high performance system.
 Individual develop environment with docker and docker-compose.
 Deploy system with kubernetes to reduce the operation cost.
 Develop with pure python code.
 Just build the basic system, another demands delay to second
phase development.
hosted by
The System Architecture
hosted by
The System Architecture
hosted by
System maintain
 Application scale
 Application scale with kubernetes
 Basic component with cloud service
 CI/CD
 CI with jenkins
 CD with jenkins and kubernetes
 Data Backup
 Mysql backup
 Hbase backup
 Message backup
hosted by
Why HBase?
 High write performance
 Quick response for query
 Easy to scale
 SQL support with phoenix
 Aliyun provide HBase SAAS
hosted by
Connect to HBase cluster with python
 Provide native java API
 Connect HBase with thrift
 Happybase provide pythonic API
 SQL support with phoenix
hosted by
challenges with HBase
 Row key design
 Hash prefix + timestamp
 Second index
 Import phoenix support
 Insert index manually
 Table design
 Short column name
 Carefully design the table with demands (i.e. the mileage of every
single vehicle)
 Complex query is very slow.
 Create index
 Export some results to HDFS or MySql (kylin?)
hosted by
Hbase Cluster
hosted by
Data Backup Approach
hosted by
hosted by
Pain point
 Complex query with HBase API still very slow
 Phoenix needs create index to the query speed
 Phoenix query still very slow if there is no index in HBase
 Complex query needs big size of index in HBase
 The queryserver between python and phoenixdb is very weak
hosted by
Conculsion
 Introduced the background of monitoring system
 Our decisions of the system
 Why we choose HBase as our main database
 How we deploy and maintain the system
 Introduced the practice of HBase in the system
hosted by
Prospects
 Rewrite high performance component with golang.
 Split the monolithic system into micro service when the system
becomes complex
 Data analysis
 Fault diagnosis
 Predict the vehicle status
 Data compression
 Opentsdb
 Combine the elasticsearch and Hbase in our application.
hosted by
Thanks

More Related Content

What's hot

HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018  Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataHBaseConAsia2018  Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
Michael Stack
 
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
Redis Labs
 

What's hot (20)

HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at HuaweiHBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
 
HBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and Cloud
HBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and CloudHBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and Cloud
HBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and Cloud
 
HBaseConAsia2018 Track2-4: HTAP DB-System: AsparaDB HBase, Phoenix, and Spark
HBaseConAsia2018 Track2-4: HTAP DB-System: AsparaDB HBase, Phoenix, and SparkHBaseConAsia2018 Track2-4: HTAP DB-System: AsparaDB HBase, Phoenix, and Spark
HBaseConAsia2018 Track2-4: HTAP DB-System: AsparaDB HBase, Phoenix, and Spark
 
HBaseConAsia2018 Track3-5: HBase Practice at Lianjia
HBaseConAsia2018 Track3-5: HBase Practice at LianjiaHBaseConAsia2018 Track3-5: HBase Practice at Lianjia
HBaseConAsia2018 Track3-5: HBase Practice at Lianjia
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
 
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018  Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataHBaseConAsia2018  Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
 
HBaseConAsia2018 Track3-3: HBase at China Life Insurance
HBaseConAsia2018 Track3-3: HBase at China Life InsuranceHBaseConAsia2018 Track3-3: HBase at China Life Insurance
HBaseConAsia2018 Track3-3: HBase at China Life Insurance
 
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
HBaseCon 2012 | Building a Large Search Platform on a Shoestring BudgetHBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
 
HBaseConAsia2018 Keynote1: Apache HBase Project Status
HBaseConAsia2018 Keynote1: Apache HBase Project StatusHBaseConAsia2018 Keynote1: Apache HBase Project Status
HBaseConAsia2018 Keynote1: Apache HBase Project Status
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
 
Scaling HDFS at Xiaomi
Scaling HDFS at XiaomiScaling HDFS at Xiaomi
Scaling HDFS at Xiaomi
 
HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster
HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster
HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster
 
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
 
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsightOptimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
 
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisFloating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
 
HBaseCon 2015: State of HBase Docs and How to Contribute
HBaseCon 2015: State of HBase Docs and How to ContributeHBaseCon 2015: State of HBase Docs and How to Contribute
HBaseCon 2015: State of HBase Docs and How to Contribute
 
Redis TimeSeries
Redis TimeSeries Redis TimeSeries
Redis TimeSeries
 
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
 
What's new with enterprise Redis - Leena Joshi, Redis Labs
What's new with enterprise Redis - Leena Joshi, Redis LabsWhat's new with enterprise Redis - Leena Joshi, Redis Labs
What's new with enterprise Redis - Leena Joshi, Redis Labs
 

Similar to HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Monitoring Systems

Membase Meetup 2010
Membase Meetup 2010Membase Meetup 2010
Membase Meetup 2010
Membase
 
Red Hat Storage Day LA - Persistent Storage for Linux Containers
Red Hat Storage Day LA - Persistent Storage for Linux Containers Red Hat Storage Day LA - Persistent Storage for Linux Containers
Red Hat Storage Day LA - Persistent Storage for Linux Containers
Red_Hat_Storage
 
Building Scalable .NET Web Applications
Building Scalable .NET Web ApplicationsBuilding Scalable .NET Web Applications
Building Scalable .NET Web Applications
Buu Nguyen
 

Similar to HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Monitoring Systems (20)

GlobalsDB: Its significance for Node.js Developers
GlobalsDB: Its significance for Node.js DevelopersGlobalsDB: Its significance for Node.js Developers
GlobalsDB: Its significance for Node.js Developers
 
Membase Meetup 2010
Membase Meetup 2010Membase Meetup 2010
Membase Meetup 2010
 
India Webinar
India WebinarIndia Webinar
India Webinar
 
W23 - Advanced Performance Tactics for WebSphere Performance
W23 - Advanced Performance Tactics for WebSphere PerformanceW23 - Advanced Performance Tactics for WebSphere Performance
W23 - Advanced Performance Tactics for WebSphere Performance
 
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
 
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedInHadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
 
Skalowalna architektura na przykładzie soccerway.com
Skalowalna architektura na przykładzie soccerway.comSkalowalna architektura na przykładzie soccerway.com
Skalowalna architektura na przykładzie soccerway.com
 
VMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a ServiceVMware vFabric Data Director for DB as a Service
VMware vFabric Data Director for DB as a Service
 
Red Hat Storage Day LA - Persistent Storage for Linux Containers
Red Hat Storage Day LA - Persistent Storage for Linux Containers Red Hat Storage Day LA - Persistent Storage for Linux Containers
Red Hat Storage Day LA - Persistent Storage for Linux Containers
 
How the Development Bank of Singapore solves on-prem compute capacity challen...
How the Development Bank of Singapore solves on-prem compute capacity challen...How the Development Bank of Singapore solves on-prem compute capacity challen...
How the Development Bank of Singapore solves on-prem compute capacity challen...
 
How Globe Telecom does Primary Backups via StorReduce to the AWS Cloud
 How Globe Telecom does Primary Backups via StorReduce to the AWS Cloud How Globe Telecom does Primary Backups via StorReduce to the AWS Cloud
How Globe Telecom does Primary Backups via StorReduce to the AWS Cloud
 
Enabling big data & AI workloads on the object store at DBS
Enabling big data & AI workloads on the object store at DBS Enabling big data & AI workloads on the object store at DBS
Enabling big data & AI workloads on the object store at DBS
 
linuxcluster.ppt
linuxcluster.pptlinuxcluster.ppt
linuxcluster.ppt
 
A Deep Dive into SharePoint 2016 architecture and deployment
A Deep Dive into SharePoint 2016 architecture and deploymentA Deep Dive into SharePoint 2016 architecture and deployment
A Deep Dive into SharePoint 2016 architecture and deployment
 
Lunar Way and the Cloud Native "stack"
Lunar Way and the Cloud Native "stack"Lunar Way and the Cloud Native "stack"
Lunar Way and the Cloud Native "stack"
 
Running a Megasite on Microsoft Technologies
Running a Megasite on Microsoft TechnologiesRunning a Megasite on Microsoft Technologies
Running a Megasite on Microsoft Technologies
 
Building Scalable .NET Web Applications
Building Scalable .NET Web ApplicationsBuilding Scalable .NET Web Applications
Building Scalable .NET Web Applications
 
Getting Started with Amazon Redshift
Getting Started with Amazon RedshiftGetting Started with Amazon Redshift
Getting Started with Amazon Redshift
 
How leading financial services organisations are winning with tech
How leading financial services organisations are winning with techHow leading financial services organisations are winning with tech
How leading financial services organisations are winning with tech
 
Chicago Data Summit: Geo-based Content Processing Using HBase
Chicago Data Summit: Geo-based Content Processing Using HBaseChicago Data Summit: Geo-based Content Processing Using HBase
Chicago Data Summit: Geo-based Content Processing Using HBase
 

More from Michael Stack

More from Michael Stack (20)

hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloudhbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
 
hbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinteresthbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinterest
 
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltdhbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
 
hbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didihbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didi
 
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
 
hbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencenthbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencent
 
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
 
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
 
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Componenthbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
 
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibabahbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
 
hbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomihbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomi
 
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Sparkhbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
 
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBasehbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
 
hbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solutionhbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solution
 
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memoryhbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
 
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACLhbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
 
hbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBasehbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBase
 
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
 
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
 
HBaseConAsia2019 Keynote
HBaseConAsia2019 KeynoteHBaseConAsia2019 Keynote
HBaseConAsia2019 Keynote
 

Recently uploaded

一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样
一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样
一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样
AS
 
一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理
一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理
一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理
apekaom
 
原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样
原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样
原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样
AS
 
一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书
一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书
一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书
c6eb683559b3
 
一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样
一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样
一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样
ayvbos
 
一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理
一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理
一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理
AS
 
一比一原版犹他大学毕业证如何办理
一比一原版犹他大学毕业证如何办理一比一原版犹他大学毕业证如何办理
一比一原版犹他大学毕业证如何办理
F
 
如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证
hfkmxufye
 
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
AS
 
一比一原版澳大利亚迪肯大学毕业证如何办理
一比一原版澳大利亚迪肯大学毕业证如何办理一比一原版澳大利亚迪肯大学毕业证如何办理
一比一原版澳大利亚迪肯大学毕业证如何办理
SS
 
Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...
Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...
Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...
mikehavy0
 
一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样
一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样
一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样
ayvbos
 

Recently uploaded (20)

Down bad crying at the gym t shirtsDown bad crying at the gym t shirts
Down bad crying at the gym t shirtsDown bad crying at the gym t shirtsDown bad crying at the gym t shirtsDown bad crying at the gym t shirts
Down bad crying at the gym t shirtsDown bad crying at the gym t shirts
 
一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样
一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样
一比一原版(毕业证书)新西兰怀特克利夫艺术设计学院毕业证原件一模一样
 
[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
 
APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0
APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0
APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0
 
一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理
一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理
一比一原版桑佛德大学毕业证成绩单申请学校Offer快速办理
 
TOP 100 Vulnerabilities Step-by-Step Guide Handbook
TOP 100 Vulnerabilities Step-by-Step Guide HandbookTOP 100 Vulnerabilities Step-by-Step Guide Handbook
TOP 100 Vulnerabilities Step-by-Step Guide Handbook
 
原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样
原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样
原版定制(LBS毕业证书)英国伦敦商学院毕业证原件一模一样
 
一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书
一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书
一比一原版(NYU毕业证书)美国纽约大学毕业证学位证书
 
一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样
一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样
一比一原版(Curtin毕业证书)科廷大学毕业证原件一模一样
 
APNIC Policy Roundup, presented by Sunny Chendi at the 5th ICANN APAC-TWNIC E...
APNIC Policy Roundup, presented by Sunny Chendi at the 5th ICANN APAC-TWNIC E...APNIC Policy Roundup, presented by Sunny Chendi at the 5th ICANN APAC-TWNIC E...
APNIC Policy Roundup, presented by Sunny Chendi at the 5th ICANN APAC-TWNIC E...
 
一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理
一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理
一比一原版(Dundee毕业证书)英国爱丁堡龙比亚大学毕业证如何办理
 
Beyond Inbound: Unlocking the Secrets of API Egress Traffic Management
Beyond Inbound: Unlocking the Secrets of API Egress Traffic ManagementBeyond Inbound: Unlocking the Secrets of API Egress Traffic Management
Beyond Inbound: Unlocking the Secrets of API Egress Traffic Management
 
Abortion Pills In Jeddah+966572737505 & Get cytotec Jeddah
Abortion Pills In Jeddah+966572737505 & Get cytotec JeddahAbortion Pills In Jeddah+966572737505 & Get cytotec Jeddah
Abortion Pills In Jeddah+966572737505 & Get cytotec Jeddah
 
Story Board.pptxrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Story Board.pptxrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrStory Board.pptxrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Story Board.pptxrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
 
一比一原版犹他大学毕业证如何办理
一比一原版犹他大学毕业证如何办理一比一原版犹他大学毕业证如何办理
一比一原版犹他大学毕业证如何办理
 
如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCLA毕业证)加州大学洛杉矶分校毕业证成绩单本科硕士学位证留信学历认证
 
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
 
一比一原版澳大利亚迪肯大学毕业证如何办理
一比一原版澳大利亚迪肯大学毕业证如何办理一比一原版澳大利亚迪肯大学毕业证如何办理
一比一原版澳大利亚迪肯大学毕业证如何办理
 
Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...
Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...
Abortion Clinic in Germiston +27791653574 WhatsApp Abortion Clinic Services i...
 
一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样
一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样
一比一原版(USYD毕业证书)悉尼大学毕业证原件一模一样
 

HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Monitoring Systems

  • 1. hosted by 颜禹 Yan Yu The Application of HBase in New Energy Vehicle Monitoring System 博尼施科技 Burnish Technology Co. Ltd
  • 2. hosted by 1. Background 2. Challenges and Decisions 3. System architecture 4. Why HBase 5. Challenges with HBase 6. Data backup in HBase 7. Conclusion 8. Prospect content
  • 3. hosted by Backgroud  100k running vehicles online  send 2 packages per minute every vehicle.  data space  the origin package size is 1KB.  parsed package size is about 7KB.  one vehicle will produce 20mb data per day.  2TB data were generated per day.  2.9 billion rows need to write to HBase every day.  concurrency  3.3k persistent tps  100k persistent connections  3.3MB origin data needs to parse per second  23.1MB parsed data needs to storage in HBase per second
  • 5. hosted by Challenges  Small team  Limited funds, machines,  Short deadline  System integration  How to handle the huge amount of vehicle data  Demands are very foggy.
  • 6. hosted by Decisions  Language  Message queue  Database  Develop flow  Micro service  Monolithic service  Deploy and maintain  Cloud  Native data center
  • 7. hosted by Language  C/C++  High performance  Hard to integrate  Long development time  Java  High performance  Rich third part packages  Easy to integrate with big data system, i.e. Hadoop, HBase, spark  Python  Sprint development  Rich third part packages  Performance issue with multi thread  Golang  Easy to write multi thread program  There is no Golang developer in our team
  • 8. hosted by Message  Redis  High performance  High memory requirement  Hard to scale  Celery  More fit for distribution task  Easy to develop with python  Redis or rabbitmq as it’s backend  Kafka  Write to disk first to ensure the message security  Support consumer group  Auto balance  Enough performance for our system  Easy to scale  Rabbitmq  Classic message queue  Performance
  • 9. hosted by Database  MySql  Relational database  Fit for storage static information  ORM support  MongoDB  Document based  ORM support  Hard to maintain and scale  Hbase  Column database  High write performance  Easy to handle TB  Easy to scale  OpenTSDB  Time series database  Based on HBase
  • 10. hosted by Monolithic service vs micro service  Monolithic service  Easy to develop when system is not very complicate  Acceleration for development  Build the basic system due the the foggy demands  Micro service  Easy to scale in a complicate system  Rapid iteration  More developers requirement
  • 11. hosted by Develop flow Dependences on central server.  Dependences on central server.  Easy to setup on one server  Single point failure risk  Confliction over multi developers
  • 12. hosted by Develop flow  Dependences on individual docker engine.  Easy to setup with docker-compose  No single point risk  High memory develop machine requirement(starting from 32GB)
  • 13. hosted by Deploy and maintain  Cloud  Easy setup  Low cost with small scale  Fast deployment  No employees  Native data center  Hard setup  Expensive cost with small scale  Professional employee to maintain our data center
  • 14. hosted by Deploy and maintain  Deploy system with kubernetes  Easy to management  Rapid scale  Compute and storage split separation  Deploy basic component with cloud service  Fast deploy  Careless  Easy to get high available service  No employees
  • 15. hosted by How a small team hold a high performance system.  Individual develop environment with docker and docker-compose.  Deploy system with kubernetes to reduce the operation cost.  Develop with pure python code.  Just build the basic system, another demands delay to second phase development.
  • 16. hosted by The System Architecture
  • 17. hosted by The System Architecture
  • 18. hosted by System maintain  Application scale  Application scale with kubernetes  Basic component with cloud service  CI/CD  CI with jenkins  CD with jenkins and kubernetes  Data Backup  Mysql backup  Hbase backup  Message backup
  • 19. hosted by Why HBase?  High write performance  Quick response for query  Easy to scale  SQL support with phoenix  Aliyun provide HBase SAAS
  • 20. hosted by Connect to HBase cluster with python  Provide native java API  Connect HBase with thrift  Happybase provide pythonic API  SQL support with phoenix
  • 21. hosted by challenges with HBase  Row key design  Hash prefix + timestamp  Second index  Import phoenix support  Insert index manually  Table design  Short column name  Carefully design the table with demands (i.e. the mileage of every single vehicle)  Complex query is very slow.  Create index  Export some results to HDFS or MySql (kylin?)
  • 25. hosted by Pain point  Complex query with HBase API still very slow  Phoenix needs create index to the query speed  Phoenix query still very slow if there is no index in HBase  Complex query needs big size of index in HBase  The queryserver between python and phoenixdb is very weak
  • 26. hosted by Conculsion  Introduced the background of monitoring system  Our decisions of the system  Why we choose HBase as our main database  How we deploy and maintain the system  Introduced the practice of HBase in the system
  • 27. hosted by Prospects  Rewrite high performance component with golang.  Split the monolithic system into micro service when the system becomes complex  Data analysis  Fault diagnosis  Predict the vehicle status  Data compression  Opentsdb  Combine the elasticsearch and Hbase in our application.