Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseConAsia2018 Track3-2: HBase at China Telecom


Published on

Chen Ze of China Telecom

Published in: Internet
  • Be the first to comment

HBaseConAsia2018 Track3-2: HBase at China Telecom

  1. 1. hosted by HBase at China Telecom Chen Ze
  2. 2. hosted by Content 01 02 03 Infrastructure & Application HBase usage at China Telecom How do we monitor HBase and optimization Q&A Monitor & Optimization
  3. 3. hosted by Infrastructure & Application HBase usage at China Telecom 01
  4. 4. hosted by Infrastructure Offline computing platform HDFS Hive Impala Storm Elasticsearch Real-time computation platform Telecom Open platform Ad-hoc job scheduling Cluster managerment Monitor & Alert Spark Streaming Spark Sqoop DataX Flume FTP ETL development MapReduce Pig YARN Tachyon Tez Zookeeper HBase SQL MLlibMahout Data exchange Data collecting Kerberos ……
  5. 5. hosted by China Telecom HBase Platform • 322 hosts in a independent HDFS cluster ,32 cores,256GB memory,3.6T*12 disk • 6 HBase clusters with different kinds of application Persistence for Streaming Jobs Online writing/reading Kylin support • 520 TB data ,1 TB/day • HBase1.2.0--CDH 5.12.1
  6. 6. hosted by Persistence for Streaming Jobs • core system • data collecting system
  7. 7. hosted by Data collecting system Collect different kinds of data with different kinds of collecting method Collecting system accounts data DPI/signal/CDR … OSS MSS CRM/VSOP Quasi-real-time data real-time data batch data CRM /VSOP DPI/signal/CDR MSS accounts data OSS Core system
  8. 8. hosted by Data collecting system OSS data collecting use HBase replication to receive data from 31 Provincial branches HBase Provincial branches HBase HBase HBase Shanghai data center replication HDFS
  9. 9. hosted by Core System • core system use HBase to store middle layer data and Aggregation layer data through Spark Streaming
  10. 10. hosted by Online writing/reading application • DPI data service • Location-based data service
  11. 11. hosted by • DPI data service • What is DPI data?
  12. 12. hosted by • DPI data service • How to use DPI data ?
  13. 13. hosted by • Location-based data service Kafka HDFS/Hive Spark cluster Hbase cluster WEB/RestAPI DPI/signal/CDR data Data from Base station Mysql FTP
  14. 14. hosted by • Location-based data service
  15. 15. hosted by Monitor & Optimization How do we monitor HBase and optimization 02
  16. 16. hosted by • HBase Monitor • Tools ganglia-3.6.0 Zabbix-3.2.4 • Ganglia for HBase basic metrics • Zabbix for important items to alert
  17. 17. hosted by • HBase ganglia configuration Edit HBase config file Than we can see many metrics items
  18. 18. hosted by • Important metrics Type Items System status ping Available memory in percent Disk usage HDFS status NameNode port DataNode port JournalNode port HDFS avaliable space HBase status HBase Master Port HBase Region Server Port Zookeeper status ZooKeeper Port ZKFC Port HBase RPC regionserver.TotalCallTime regionserver.ProcessCallTime regionserver.QueueCallTime regionserver.numActiveHandler regionserver.ipc.numCallsInGeneralQueue regionserver.ipc.numOpenConnections regionserver.RegionServer.numCallsInWriteQueue regionserver.RegionServer.numCallsInReadQueue Hbase IO regionserver.Server.Mutate_99th_percentile regionserver.wal.SyncTime_99th_percentile regionserver.server.Get_99th_percentile regionserver.server.ScanTime_99th_percentile
  19. 19. hosted by • Important metrics Type Items HBase region Region num,size BlockCache hit radio JVM GC jvm.JvmMetrics.GcTimeMillis jvm.JvmMetrics.GcCount GC log
  20. 20. hosted by • HBase Debug case • One day, our core data system hanged when trying to connection Hbase cluster…
  21. 21. hosted by HBase Optimization • CMS vs G1 • Read/Write Splitting and some Optimization
  22. 22. hosted by CMS vs G1 CMS to G1 Why?
  23. 23. hosted by Read/Write Splitting Optimization • Use replication • Use Kerberos • Optimization hbase.ipc
  24. 24. hosted by Q&A
  25. 25. hosted by Thanks