Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseConAsia2018 Track1-4: HBase at Didi

107 views

Published on

JingYi Yao of Didi @ HBaseConAsia2018

Published in: Internet
  • Be the first to comment

HBaseConAsia2018 Track1-4: HBase at Didi

  1. 1. hosted by HBaseConAsia2018 August 17,2018 Gehua New Century Hotel Beijing,China
  2. 2. hosted by HBase at DiDi JingYi Yao
  3. 3. hosted by Content 01 02 04 03 HBase Improvements & Applications Phoenix Improvements & Applications GeoMesa Applications Monitor & Operations
  4. 4. hosted by HBase Improvements & Applications 01
  5. 5. hosted by HBase Improvements & Applications ◇ 1.HBase Applications at DiDi ◇ 2.ACL Improvements ◇ 3.Replication Improvements ◇ 4.Reduce User Connection
  6. 6. hosted by HBase Improvements & Applications 1.HBase Applications
  7. 7. hosted by HBase Improvements & Applications 1.HBase Applications
  8. 8. hosted by HBase Improvements & Applications 1.HBase Applications
  9. 9. hosted by HBase Improvements & Applications 1.HBase Applications ◇ Message Queue ◇ GeoGraphic Feature data ◇ Heat Map ◇ User Profile ◇ Log Analysis
  10. 10. hosted by HBase Improvements & Applications 2.Replication Improvements
  11. 11. hosted by HBase Improvements & Applications 2.Replication Improvements ◇ (1).Replication in RS Group ◇ (2).Table Metric ◇ (3).RS Group ratio
  12. 12. hosted by HBase Improvements & Applications 3.Connection ◇ (1).users ◇ (2).Coprocessor: ◇ Regionserver clusterConnection
  13. 13. hosted by HBase Improvements & Applications 4.ACL Improvements
  14. 14. hosted by HBase Improvements & Applications 4.ACL Improvements
  15. 15. hosted by HBase Improvements & Applications 4.ACL Improvements
  16. 16. hosted by HBase Improvements & Applications ◇ reduce zookeeper znode ◇ and watcher 4.ACL Improvements
  17. 17. hosted by HBase Improvements & Applications 5.Others ◇ (1)RPC audit log ◇ (2)RSGroup bug fix ◇ (3)quota ◇ (4)safe drop table ◇ …
  18. 18. hosted by Phoenix Improvements & Applications 02
  19. 19. hosted by Phoenix Improvements & Applications
  20. 20. hosted by Phoenix Improvements & Applications
  21. 21. hosted by Phoenix Improvements & Applications ◇ SLA 99.95% ◇ Secondary indexes ◇ Java-jdbc client P99 35 ms ◇ column default ◇ QueryServer 5.Order History
  22. 22. hosted by Phoenix Improvements & Applications ◇ Secondary indexes : ◇ INDEX Type: Global Index => Covered Index ◇ ASYNC INDEX ◇ SALT_BUCKETS => reverse(ID) & pre split ◇ coprocessor :reduce connection ◇ QueryServer metrics
  23. 23. hosted by Phoenix Improvements & Applications
  24. 24. hosted by Phoenix Improvements & Applications ◇ zk connection: ◇ region num * Index num => 1
  25. 25. hosted by GeoMesa Applications03
  26. 26. hosted by GeoMesa Applications GeoMesa GeoMesa is an open source suite of tools that enables large-scale geospatial querying and analytics on distributed computing systems.
  27. 27. hosted by GeoMesa Applications ◇ 1.Road Detection ◇ 2.Device ID => Trajectory
  28. 28. hosted by GeoMesa Applications
  29. 29. hosted by GeoMesa Applications ◇ Z2 : latitude and longitude ◇ Z3 : latitude, longitude, and time ◇ XZ2 : line strings or polygons ◇ XZ3 : non-Point geometry and has a time attribute ◇ Record/ID ◇ Attribute Index
  30. 30. hosted by GeoMesa Applications ◇ GeoHash: ◇ geohashes have been used to first convert latitude and longitude into a single bit vector
  31. 31. hosted by GeoMesa Applications Z-order curve
  32. 32. hosted by GeoMesa Applications ◇ Weakness ◇ More scan Z-order curve
  33. 33. hosted by GeoMesa Applications ◇ Hilbert filling curve Inedx ◇ Geomesa plan ◇ Coprocessor ◇ GeoMesa Stats Future Experience from Phoenix
  34. 34. hosted by Monitor & Operations04
  35. 35. hosted by Monitor & Operations ◇ (1).HBase ◇ (2).Zookeeper ◇ (3).HDFS ◇ (4).ThriftServer ◇ (5).Queryserver 1.Monitor
  36. 36. hosted by Zookeeper Monitor & Operations
  37. 37. hosted by ◇ echo mntr | nc $zookeeper 2181 ◇ zk_num_alive_connections ◇ zk_open_file_descriptor_count ◇ zk_outstanding_requests ◇ zk_min_latency ◇ zk_followers Monitor & Operations
  38. 38. hosted by ◇ [Put | Get | Scan …]_99th_percentile ◇ Slow Read/Write … ThriftServer QueryServer ◇ print slow prepareStatement ◇ print slow query ◇ ExecuteQueryTime_99th_percentile Monitor & Operations
  39. 39. hosted by ◇ slow disk ◇ edits log => hdfs path HDFS Monitor & Operations
  40. 40. hosted by ◇ DHS(Didi HBase Service): ◇ Visualizing Information: Region hot-spots from Region Metric ◇ Read、Write、StoreSize Monitor & Operations
  41. 41. hosted by Monitor & Operations The highest 10 & The lowest 10
  42. 42. hosted by ◇ Automatic computation:input/Output Size per sec.Store Size.. ◇ Recommended values:Region Count Monitor & Operations
  43. 43. hosted by ◇ (1).read request、write request、Mutate_99th_percentile、Get_99th_percentile、 ScanTime_99th_percentile、receivedBytes、sentBytes ◇ (2).compactionQueueLength、flushQueueLength、splitQueueLength ◇ (3).jvm metrics… ◇ slow request: Rpc p99、gc、compaction queue 、slow log Monitor & Operations HBase Metric
  44. 44. hosted by Monitor & Operations
  45. 45. hosted by Monitor & Operations
  46. 46. hosted by Monitor & Operations yaojingyi@didichuxing.com
  47. 47. hosted by Thanks

×