サポートエンジニア Night vol.2 で話した内容です https://techplay.jp/event/651515

1. 1© Cloudera, Inc. All rights reserved. 1© Cloudera, Inc. All rights reserved.
Cloudera
(@d1ce_)

2. 2© Cloudera, Inc. All rights reserved.
- (@d1ce_)
- Cloudera
- Hadoop
-
Who am I?

3. 3© Cloudera, Inc. All rights reserved.
- Cloudera
- Cloudera
-
-
Agenda

4. 4© Cloudera, Inc. All rights reserved.
- Cloudera
- Cloudera
-
-
Agenda

5. 5© Cloudera, Inc. All rights reserved.
Cloudera

6. 6© Cloudera, Inc. All rights reserved.
Cloudera

7. 7© Cloudera, Inc. All rights reserved.
-
-
: The Economist
https://www.economist.com/news/briefing/2172163
4-how-it-shaping-up-data-giving-rise-new-economy
https://www.economist.com/news/leaders/2172165
6-data-economy-demands-new-approach-antitrust-
rules-worlds-most-valuable-resource

8. 8© Cloudera, Inc. All rights reserved.
- 2008 4
- 2017 5
- 1,600
- 28
-
-
- 3,000
- Cloudera University 45,000

9. 9© Cloudera, Inc. All rights reserved.
-
-
-
Cloudera

10. 10© Cloudera, Inc. All rights reserved.
-
-
-
Cloudera

11. 11© Cloudera, Inc. All rights reserved.
/ SE / SE

12. 12© Cloudera, Inc. All rights reserved.
-
- / SE
-
SE

13. 13© Cloudera, Inc. All rights reserved.
-
-
-
-

14. 15© Cloudera, Inc. All rights reserved.
- Cloudera
- Cloudera
-
-
Agenda

15. 16© Cloudera, Inc. All rights reserved.
Cloudera

16. 17© Cloudera, Inc. All rights reserved.
Cloudera
- 2011 5
- 2018 1 ~300
APJ
(Local + Global)
(Local + Global)
(Local + Global)
EMEA
US

17. 18© Cloudera, Inc. All rights reserved.
Cloudera
APJ
(Local + Global)
(Local + Global)
(Local + Global)
EMEA
US

18. 19© Cloudera, Inc. All rights reserved.
Cloudera
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs

19. 20© Cloudera, Inc. All rights reserved.
Cloudera
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs

20. 21© Cloudera, Inc. All rights reserved.
Cloudera
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs
-
- 30 4
Storage
Access &
Integration
Data Processing Enterprise
ZooKeeper, HDFS
HBase, Accumulo
Kudu, Director
HCatalog, Hive,
Impala, Oozie,
Parquet, Avro, Sentry,
Navigator Optimizer,
Sqoop
CDSW, Flume,
Kafka, Mahout,
Spark,
MapReduce,
YARN, Pig, Search
BDR, CDSW, CM,
Hue, Security,
Encryption

21. 22© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs

22. 23© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs
-
-
-
-

23. 24© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs

24. 25© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs
Internal Systems Engineering
- Hadoop
-
OpenStack
-

25. 26© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs
Internal Systems Engineering
- Hadoop
- EDH
Openstack (COTT)
- (SSD)

26. 27© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs
Internal Systems Engineering
- Hadoop
- EDH
Openstack (COTT)
- (SSD)
-
-
-

27. 28© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs

28. 29© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs
Support Quality
-
Support Planning & Programs
-

29. 30© Cloudera, Inc. All rights reserved.
Internal Systems
Engineering
Support
Quality
Support Planning
& Programs

30. 31© Cloudera, Inc. All rights reserved.
- Cloudera
- Cloudera
-
-
Agenda

31. 32© Cloudera, Inc. All rights reserved.
-
-
-
-
-
-
-

32. 33© Cloudera, Inc. All rights reserved.
-
-
It is a capital mistake to theorize
before one has data. Insensibly one
begins to twist facts to suit theories,
instead of theories to suit facts.
Sherlock Holmes in “A Scandal in Bohemia” by Sir Arthur Conan Doyle via
”Systems Performance” written by Brendan Gregg

33. 34© Cloudera, Inc. All rights reserved.

34. 35© Cloudera, Inc. All rights reserved.

35. 36© Cloudera, Inc. All rights reserved.

36. 37© Cloudera, Inc. All rights reserved.

37. 38© Cloudera, Inc. All rights reserved.
- Cloudera
-
-
-
- OS

38. 39© Cloudera, Inc. All rights reserved.
-
SE

39. 40© Cloudera, Inc. All rights reserved.
-
-
- OS
Cluster Validation

40. 41© Cloudera, Inc. All rights reserved.
Cloudera
JVM
Linux
Disk, CPU, Mem
JVM
Linux
Disk, CPU, Mem
JVM
Linux
Disk, CPU, Mem
: Kudu:
HDFS
: HBase :
ZooKeeper
: Sentry Kerberos/AD/LDAP
Tableau
SAS
YARN
MapReduce Spark
: Impala
:
Solr
/API : Hive, Pig
Docker
:
CDSW
: Hue
Ingest :
Flume,
Kafka
S3/ADLS
Import/
Export :
Sqoop

41. 42© Cloudera, Inc. All rights reserved.
: Kudu:
HDFS
: HBase :
ZooKeeper
: Sentry Kerberos/AD/LDAP
Tableau
SAS
YARN
MapReduce Spark
: Impala
:
Solr
/API : Hive, Pig
:
CDSW
: Hue
S3/ADLS
JVM
Linux
Disk, CPU, Mem
JVM
Linux
Disk, CPU, Mem
JVM
Linux
Disk, CPU, Mem
Docker
Ingest :
Flume,
Kafka
Import/
Export :
Sqoop

42. 43© Cloudera, Inc. All rights reserved.
: Kudu:
HDFS
: HBase :
ZooKeeper
: Sentry Kerberos/AD/LDAP
Tableau
SAS
YARN
MapReduce Spark
: Impala
:
Solr
/API : Hive, Pig
:
CDSW
: Hue
S3/ADLS
JVM
Linux
Disk, CPU, Mem
JVM
Linux
Disk, CPU, Mem
JVM
Linux
Disk, CPU, Mem
Docker
Ingest :
Flume,
Kafka
Import/
Export :
Sqoop

43. 44© Cloudera, Inc. All rights reserved.
-
-
-
-
-
-
-

44. 45© Cloudera, Inc. All rights reserved.
- UNIX
- cat logs |
grep “^2018-02-03” | # 2018/2/3
grep “Finished memstore flush of” |
cut -d: -f 1,2 | #
sed -e ‘s/.$//’ | # 10
uniq -c #

45. 46© Cloudera, Inc. All rights reserved.
- UNIX
- cat logs |
grep “^2018-02-01” | # 2018/2/1
grep “Finished memstore flush of” |
cut -d: -f 1,2 | #
sed -e ‘s/.$//’ | # 10
uniq -c #

46. 47© Cloudera, Inc. All rights reserved.
- : Pandas

47. 50© Cloudera, Inc. All rights reserved.
- Cloudera
- Cloudera
-
-
Agenda

48. 51© Cloudera, Inc. All rights reserved.
-
-
-
-
-

49. 52© Cloudera, Inc. All rights reserved.
-
- LVM sar mount, df
-
- lsblk/blkid/dmsetup
-

50. 53© Cloudera, Inc. All rights reserved.
Thank you
daisuke@cloudera.com