Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloudera のサポートエンジニアリング #supennight

2,381 views

Published on

サポートエンジニア Night vol.2 で話した内容です https://techplay.jp/event/651515

Published in: Engineering

Cloudera のサポートエンジニアリング #supennight

  1. 1. 1© Cloudera, Inc. All rights reserved. 1© Cloudera, Inc. All rights reserved. Cloudera (@d1ce_)
  2. 2. 2© Cloudera, Inc. All rights reserved. - (@d1ce_) - Cloudera - Hadoop - Who am I?
  3. 3. 3© Cloudera, Inc. All rights reserved. - Cloudera - Cloudera - - Agenda
  4. 4. 4© Cloudera, Inc. All rights reserved. - Cloudera - Cloudera - - Agenda
  5. 5. 5© Cloudera, Inc. All rights reserved. Cloudera
  6. 6. 6© Cloudera, Inc. All rights reserved. Cloudera
  7. 7. 7© Cloudera, Inc. All rights reserved. - - : The Economist https://www.economist.com/news/briefing/2172163 4-how-it-shaping-up-data-giving-rise-new-economy https://www.economist.com/news/leaders/2172165 6-data-economy-demands-new-approach-antitrust- rules-worlds-most-valuable-resource
  8. 8. 8© Cloudera, Inc. All rights reserved. - 2008 4 - 2017 5 - 1,600 - 28 - - - 3,000 - Cloudera University 45,000
  9. 9. 9© Cloudera, Inc. All rights reserved. - - - Cloudera
  10. 10. 10© Cloudera, Inc. All rights reserved. - - - Cloudera
  11. 11. 11© Cloudera, Inc. All rights reserved. / SE / SE
  12. 12. 12© Cloudera, Inc. All rights reserved. - - / SE - SE
  13. 13. 13© Cloudera, Inc. All rights reserved. - - - -
  14. 14. 15© Cloudera, Inc. All rights reserved. - Cloudera - Cloudera - - Agenda
  15. 15. 16© Cloudera, Inc. All rights reserved. Cloudera
  16. 16. 17© Cloudera, Inc. All rights reserved. Cloudera - 2011 5 - 2018 1 ~300 APJ (Local + Global) (Local + Global) (Local + Global) EMEA US
  17. 17. 18© Cloudera, Inc. All rights reserved. Cloudera APJ (Local + Global) (Local + Global) (Local + Global) EMEA US
  18. 18. 19© Cloudera, Inc. All rights reserved. Cloudera Internal Systems Engineering Support Quality Support Planning & Programs
  19. 19. 20© Cloudera, Inc. All rights reserved. Cloudera Internal Systems Engineering Support Quality Support Planning & Programs
  20. 20. 21© Cloudera, Inc. All rights reserved. Cloudera Internal Systems Engineering Support Quality Support Planning & Programs - - 30 4 Storage Access & Integration Data Processing Enterprise ZooKeeper, HDFS HBase, Accumulo Kudu, Director HCatalog, Hive, Impala, Oozie, Parquet, Avro, Sentry, Navigator Optimizer, Sqoop CDSW, Flume, Kafka, Mahout, Spark, MapReduce, YARN, Pig, Search BDR, CDSW, CM, Hue, Security, Encryption
  21. 21. 22© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs
  22. 22. 23© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs - - - -
  23. 23. 24© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs
  24. 24. 25© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs Internal Systems Engineering - Hadoop - OpenStack -
  25. 25. 26© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs Internal Systems Engineering - Hadoop - EDH Openstack (COTT) - (SSD)
  26. 26. 27© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs Internal Systems Engineering - Hadoop - EDH Openstack (COTT) - (SSD) - - -
  27. 27. 28© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs
  28. 28. 29© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs Support Quality - Support Planning & Programs -
  29. 29. 30© Cloudera, Inc. All rights reserved. Internal Systems Engineering Support Quality Support Planning & Programs
  30. 30. 31© Cloudera, Inc. All rights reserved. - Cloudera - Cloudera - - Agenda
  31. 31. 32© Cloudera, Inc. All rights reserved. - - - - - - -
  32. 32. 33© Cloudera, Inc. All rights reserved. - - It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts. Sherlock Holmes in “A Scandal in Bohemia” by Sir Arthur Conan Doyle via ”Systems Performance” written by Brendan Gregg
  33. 33. 34© Cloudera, Inc. All rights reserved.
  34. 34. 35© Cloudera, Inc. All rights reserved.
  35. 35. 36© Cloudera, Inc. All rights reserved.
  36. 36. 37© Cloudera, Inc. All rights reserved.
  37. 37. 38© Cloudera, Inc. All rights reserved. - Cloudera - - - - OS
  38. 38. 39© Cloudera, Inc. All rights reserved. - SE
  39. 39. 40© Cloudera, Inc. All rights reserved. - - - OS Cluster Validation
  40. 40. 41© Cloudera, Inc. All rights reserved. Cloudera JVM Linux Disk, CPU, Mem JVM Linux Disk, CPU, Mem JVM Linux Disk, CPU, Mem : Kudu: HDFS : HBase : ZooKeeper : Sentry Kerberos/AD/LDAP Tableau SAS YARN MapReduce Spark : Impala : Solr /API : Hive, Pig Docker : CDSW : Hue Ingest : Flume, Kafka S3/ADLS Import/ Export : Sqoop
  41. 41. 42© Cloudera, Inc. All rights reserved. : Kudu: HDFS : HBase : ZooKeeper : Sentry Kerberos/AD/LDAP Tableau SAS YARN MapReduce Spark : Impala : Solr /API : Hive, Pig : CDSW : Hue S3/ADLS JVM Linux Disk, CPU, Mem JVM Linux Disk, CPU, Mem JVM Linux Disk, CPU, Mem Docker Ingest : Flume, Kafka Import/ Export : Sqoop
  42. 42. 43© Cloudera, Inc. All rights reserved. : Kudu: HDFS : HBase : ZooKeeper : Sentry Kerberos/AD/LDAP Tableau SAS YARN MapReduce Spark : Impala : Solr /API : Hive, Pig : CDSW : Hue S3/ADLS JVM Linux Disk, CPU, Mem JVM Linux Disk, CPU, Mem JVM Linux Disk, CPU, Mem Docker Ingest : Flume, Kafka Import/ Export : Sqoop
  43. 43. 44© Cloudera, Inc. All rights reserved. - - - - - - -
  44. 44. 45© Cloudera, Inc. All rights reserved. - UNIX - cat logs | grep “^2018-02-03” | # 2018/2/3 grep “Finished memstore flush of” | cut -d: -f 1,2 | # sed -e ‘s/.$//’ | # 10 uniq -c #
  45. 45. 46© Cloudera, Inc. All rights reserved. - UNIX - cat logs | grep “^2018-02-01” | # 2018/2/1 grep “Finished memstore flush of” | cut -d: -f 1,2 | # sed -e ‘s/.$//’ | # 10 uniq -c #
  46. 46. 47© Cloudera, Inc. All rights reserved. - : Pandas
  47. 47. 50© Cloudera, Inc. All rights reserved. - Cloudera - Cloudera - - Agenda
  48. 48. 51© Cloudera, Inc. All rights reserved. - - - - -
  49. 49. 52© Cloudera, Inc. All rights reserved. - - LVM sar mount, df - - lsblk/blkid/dmsetup -
  50. 50. 53© Cloudera, Inc. All rights reserved. Thank you daisuke@cloudera.com

×