Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EMRでスポットインスタンスの自動入札ツールを作成する

1,499 views

Published on

2016/12/12 BigData-JAWS 勉強会#4での、荻原の講演資料になります

Published in: Technology
  • Login to see the comments

  • Be the first to like this

EMRでスポットインスタンスの自動入札ツールを作成する

  1. 1. (C) Recruit Technologies Co.,Ltd. All rights reserved. EMR
  2. 2. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   29    Apple Hadoop
  3. 3. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department 2010 2016 2012 2014 / / / Hadoop / / AWS( EMR)
  4. 4. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department 2010 2016 2012 2014 / / / Hadoop / / AWS( EMR)
  5. 5. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department    
  6. 6. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA DepartmentEMR   ( 1 ) ( )    SQL  EMR   Hadoop   11 100  EMR
  7. 7. (C) Recruit Technologies Co.,Ltd. All rights reserved.
  8. 8. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  EC2   EC2 30 50% )   Hadoop
  9. 9. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department    HIGH  LOW
  10. 10. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department client Amazon EMR Amazon EC2 Amazon EC2 Amazon EC2 ......
  11. 11. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   30 5
  12. 12. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  5 15 + 15
  13. 13. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  30 5 30
  14. 14. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  5 15 + 15
  15. 15. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  16. 16. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  17. 17. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  18. 18. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  19. 19. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  20. 20. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   Mapper, Reducer  HDFS
  21. 21. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department Frequency of being outbid (week)
  22. 22. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department Frequency of being outbid (week) . https://aws.amazon.com/jp/ec2/spot/bid-advisor/ ( 2016-12-09)
  23. 23. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   r3.xlarge  r3.2xlarge  r3.4xlarge  1  2  4 /
  24. 24. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department インスタンスタイプ 取得するスポットインスタン スの値段 値段 r3.xlarge $ 0.1 $ 0.1 / 1 = $ 0.1 r3.2xlarge $ 0.3 $ 0.3 / 2 = $ 0.15 r3.4xlarge $ 0.35 $ 0.35 / 4 = $ 0.0875
  25. 25. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48      10
  26. 26. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48  1 2 48
  27. 27. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  28. 28. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48      10
  29. 29. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  or 4  r3.xlarge  r3.2xlarge  r3.4xlarge  r3.8xlarge   e.g. r3.2xlarge オンデマンド価格 スポットインスタンス価格 $ 0.798 0.798 * 10% = $ 0.0798 0.798 * 20% = $ 0.1596 … 0.798 * 90% = $ 0.7182
  30. 30. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48      10
  31. 31. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department    WebUI   JSON

×