Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hadoop 5

1,492 views

Published on

Hadoop本輪読会第4回の発表資料です。

Published in: Technology
  • Be the first to comment

Hadoop 5

  1. 1. Hadoop 5 MapReduce
  2. 2. ❖ blog: http://brass.to/blog/ ❖ twitter: @akahigeg ❖ Rails ❖ Mac,Vim ❖ Linux(CentOS),Apache,MySQL ❖ Agile
  3. 3. ❖ X ❖ PHP CakePHP ❖ TDD ❖ 16 Puppet
  4. 4. 5.1 API P127 5.2 P130 5.3 P137 5.4 P140 5.5 P146 5.6 P161 5.7 MapReduce P164
  5. 5. 5.1 API P127 ❖ = XML 5-1 P128 ❖ ‣ ❖ ‣ ‣ final
  6. 6. 5.1 API P129 ❖ ‣ DRY ❖ ‣ ❖
  7. 7. 5.2 P130 ❖ Hadoop ❖ IDE ❖ P131 ‣ -conf P132 ‣ conf ‣ P133
  8. 8. 5.2 P133 ❖ Tool ❖ GenericOptionParser ToolRunner ❖ P135 ❖ P135 ‣ JVM P136 ‣ -conf
  9. 9. 5.3 P137 ❖ map reduce ‣ ‣ OutputCollector ‣ ❖ = ❖ JUnit Mockito TDD ‣ MapReduce MRUnit
  10. 10. 5.3 P137 ❖ MaxTemperatureMapper ‣ Test: 5-4 P137 ‣ mapper ver.1 5-5 P138 ‣ Test: P138 ‣ mapper ver.2 P139 ❖ MaxTemperatureReducer ‣ Test: P139 ‣ reducer 5-6 P140
  11. 11. 5.4 P140 ❖ ‣ map reducer ‣ Tool ❖ JobRunner ‣ MapReduce ‣
  12. 12. 5.4 P142 ❖ ‣ mapper - ❖ 5-8 P143 ❖ mapper 5-9 P144 ❖ TDD
  13. 13. 5.4 P144 ❖ JobRunner ‣ Tool ‣ 5-10 P145 ❖ P145 ‣ HDFS MapReduce ‣ Hadoop ‣
  14. 14. 5.5 P146 ❖ ❖ ‣ JobClient P147 ‣ ID = job_YYYYMMDDHHMM_0000 ‣ ID = ID + _m_000000 ‣ ID = ID + _0000
  15. 15. 5.5 P148 ❖ MapReduce Web UI http://jobtracker-host:50030/ ‣ jobtracker P148 - Hadoop - - - - ❖ P150 - -
  16. 16. 5.5 P152 ❖ ‣ reducer - MapReduce ‣ ‣ ‣
  17. 17. 5.5 P153 ❖ ‣ Web UI ‣ ‣ MapReduce ❖ Web UI ‣ ‣
  18. 18. 5.5 P158 ❖ P158 ‣ P157 ‣ ❖ Hadoop P157 ‣ Chukwa ‣ ‣
  19. 19. 5.5 P159 ❖ ‣ ‣ ‣ ‣ ‣ ‣ IDE
  20. 20. 5.6 P161 ❖ ✓ mapper ✓ reducer ✓ combiner ✓ ✓ ✓
  21. 21. 5.6 P161 ❖ ‣ JobRunner P162 ❖ HPROF P162 ‣ ‣ ‣
  22. 22. 5.7 MapReduce P164 ❖ MapReduce ❖ ‣ UNIX ❖ MapReduce Pig Hive
  23. 23. 5.7 MapReduce P166 ❖ ❖ 2 ❖ ❖ ❖ ‣ JobControl ‣ Hadoop Workflow Scheduler HWS ※
  24. 24. ❖ TDD ‣ ‣ ‣ ❖ ❖ ❖

×