Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseConAsia2018 Track3-5: HBase Practice at Lianjia

50 views

Published on

GuiXian Zhao of Lianjia

Published in: Internet
  • Be the first to comment

  • Be the first to like this

HBaseConAsia2018 Track3-5: HBase Practice at Lianjia

  1. 1. hosted by Hbase Practice At ke.com Guoxian Zhao
  2. 2. hosted by Content 01 02 04 03 Architecture Evolution Olap Scenario Monitor Scenario Data Mining Scenario 05 Reatime ETL Scenario 06 Hdic Scenario
  3. 3. hosted by Architecture Evolution01
  4. 4. hosted by bigdata architecture for ke Evolution
  5. 5. hosted by Olap Scenario02
  6. 6. hosted by Olap Scenario Requirement List Billion data set subsecond response times High concurrency sql interface
  7. 7. hosted by Olap Scenario kylin
  8. 8. hosted by Olap data architecture map permission Olap Scenario monitor meta scheduler Hbase … hadoop, warehouse + compute … load Alluxio Sparksql Spark cluster presto queryengine adhoc build zk build nginx query query Offline kylin Kylin middleware query Meta manager Cube execute Statistics redis redisMetrics api product
  9. 9. hosted by Data For Kylin  800+cube、16+business  200T storage、160000 million data、1 cube 6000 million  Query/day million, <500ms(95%), <1s (99%)
  10. 10. hosted by Data Product Metrics platform • roll-up drill • multidimensional analysis • self-service configuration report • standard indices
  11. 11. hosted by Performance Optimizations For Hbase SSD-FIRST • Read remote ssds first Client H D D S S D H D D DN1 DN2 DN3 OR
  12. 12. hosted by Performance Optimizations For Hbase others  HDFS short-circuit  Data hedged ,multi thread read  Test stale datanodes, avoid the slow datanode  Shutdown balance  Multiwal, Improve write performance
  13. 13. hosted by Monitor Scenario03
  14. 14. hosted by Monitor senario HAProxy TeleGraf Task metrics Hadoopmetric s TSD TSD HBASE grafana
  15. 15. hosted by Monitor Scenario
  16. 16. hosted by Data Mining Scenario04
  17. 17. hosted by Data Mining Scenario
  18. 18. hosted by RealTime ETL Scenario05
  19. 19. hosted by RealTime ETL Scenario Subtitle Text RealTime DB ETL RealTime log ETL 3 4 HBASE fast write fast read change record bigdata storage
  20. 20. hosted by Hdic Scence06
  21. 21. hosted by Hdic Scenario Subtitle Text 3 4
  22. 22. hosted by Data For Ke
  23. 23. hosted by Thanks

×