The document discusses the BIG DATA department of Recruit Technologies. It describes how the department has used Amazon's Elastic MapReduce (EMR) and Elastic Compute Cloud (EC2) services since 2010 to perform big data analytics on datasets using Hadoop. It provides details on how the department configures and manages EMR clusters on EC2 spot instances to perform tasks like log analysis and recommendation algorithms in a cost-effective manner. Various strategies are discussed around optimizing the use of spot instances and EMR to reduce costs while managing reliability.

1. (C) Recruit Technologies Co.,Ltd. All rights reserved.
EMR

2. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

 29



Apple
Hadoop

3. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
2010
2016
2012
2014
/
/ /
Hadoop
/ /
AWS( EMR)

4. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
2010
2016
2012
2014
/
/ /
Hadoop
/ /
AWS( EMR)

5. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department





6. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA DepartmentEMR

 ( 1 ) ( )


 SQL
 EMR

 Hadoop

 11 100
 EMR

7. (C) Recruit Technologies Co.,Ltd. All rights reserved.

8. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
 EC2

 EC2
30 50% )

 Hadoop

9. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department


 HIGH
 LOW

10. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
client Amazon
EMR
Amazon
EC2
Amazon
EC2
Amazon
EC2
......

11. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department


30
5

12. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

5
15 + 15

13. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

30
5
30

14. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

5
15 + 15

15. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

16. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

17. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

18. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

19. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

20. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

 Mapper, Reducer
 HDFS

21. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
Frequency of being outbid (week)

22. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
Frequency of being outbid (week)
. https://aws.amazon.com/jp/ec2/spot/bid-advisor/ ( 2016-12-09)

23. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

 r3.xlarge
 r3.2xlarge
 r3.4xlarge
 1
 2
 4
/

24. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
インスタンスタイプ
取得するスポットインスタン
スの値段
値段
r3.xlarge $ 0.1 $ 0.1 / 1 = $ 0.1
r3.2xlarge $ 0.3 $ 0.3 / 2 = $ 0.15
r3.4xlarge $ 0.35 $ 0.35 / 4 = $ 0.0875

25. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
 48





10

26. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
 48

1 2 48

27. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department

28. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
 48





10

29. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
 or 4
 r3.xlarge
 r3.2xlarge
 r3.4xlarge
 r3.8xlarge

 e.g. r3.2xlarge
オンデマンド価格 スポットインスタンス価格
$ 0.798
0.798 * 10% = $ 0.0798
0.798 * 20% = $ 0.1596
…
0.798 * 90% = $ 0.7182

30. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
 48





10

31. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department


 WebUI

 JSON