0
Hadoop      5
MapReduce
❖ blog: http://brass.to/blog/
❖ twitter: @akahigeg
❖ Rails

❖ Mac,Vim
❖ Linux(CentOS),Apache,MySQL
❖ Agile
❖   X

❖       PHP            CakePHP

❖ TDD

❖       16    Puppet
5.1      API      P127
5.2               P130
5.3               P137
5.4               P140
5.5               P146
5.6    ...
5.1          API        P127

❖            =
    XML      5-1 P128

❖

    ‣

❖

    ‣

    ‣ final
5.1     API         P129

❖

    ‣         DRY

❖

    ‣

❖
5.2                            P130

❖ Hadoop

❖ IDE

❖                    P131

    ‣ -conf          P132

    ‣         ...
5.2                                          P133

❖ Tool

❖ GenericOptionParser ToolRunner



❖                      P135...
5.3                                 P137

❖ map reduce

    ‣

    ‣             OutputCollector

    ‣

❖         =

❖ JU...
5.3                                             P137

❖ MaxTemperatureMapper
  ‣ Test:                                5-4 ...
5.4                                  P140

❖

    ‣ map reducer

    ‣ Tool

❖            JobRunner

    ‣                ...
5.4                                  P142

❖

    ‣ mapper

      -

❖              5-8 P143

❖ mapper                  5-...
5.4                                           P144

❖            JobRunner

    ‣ Tool

    ‣                             ...
5.5                                         P146

❖

❖

    ‣ JobClient               P147

    ‣        ID = job_YYYYMMDD...
5.5                                                P148

❖ MapReduce Web UI http://jobtracker-host:50030/
    ‣ jobtracker...
5.5                     P152

❖

    ‣ reducer

        -   MapReduce

    ‣

    ‣

    ‣
5.5                   P153

❖

    ‣ Web UI

    ‣

    ‣     MapReduce

❖       Web UI

    ‣

    ‣
5.5                            P158

❖              P158

    ‣                   P157

    ‣

❖ Hadoop         P157

    ...
5.5         P159

❖

    ‣

    ‣

    ‣

    ‣

    ‣

    ‣ IDE
5.6              P161

❖

    ✓ mapper

    ✓ reducer

    ✓ combiner
    ✓

    ✓

    ✓
5.6                          P161

❖

    ‣     JobRunner
              P162

❖ HPROF               P162

    ‣

    ‣

  ...
5.7 MapReduce                             P164

 ❖                       MapReduce


 ❖


     ‣ UNIX

 ❖                 ...
5.7 MapReduce                             P166

 ❖

 ❖ 2

 ❖

 ❖

 ❖

     ‣ JobControl
     ‣ Hadoop Workflow Scheduler HW...
❖ TDD

    ‣

    ‣

    ‣

❖

❖

❖
Hadoop 5
Hadoop 5
Upcoming SlideShare
Loading in...5
×

Hadoop 5

1,285

Published on

Hadoop本輪読会第4回の発表資料です。

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,285
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
32
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide


























  • Transcript of "Hadoop 5"

    1. 1. Hadoop 5 MapReduce
    2. 2. ❖ blog: http://brass.to/blog/ ❖ twitter: @akahigeg ❖ Rails ❖ Mac,Vim ❖ Linux(CentOS),Apache,MySQL ❖ Agile
    3. 3. ❖ X ❖ PHP CakePHP ❖ TDD ❖ 16 Puppet
    4. 4. 5.1 API P127 5.2 P130 5.3 P137 5.4 P140 5.5 P146 5.6 P161 5.7 MapReduce P164
    5. 5. 5.1 API P127 ❖ = XML 5-1 P128 ❖ ‣ ❖ ‣ ‣ final
    6. 6. 5.1 API P129 ❖ ‣ DRY ❖ ‣ ❖
    7. 7. 5.2 P130 ❖ Hadoop ❖ IDE ❖ P131 ‣ -conf P132 ‣ conf ‣ P133
    8. 8. 5.2 P133 ❖ Tool ❖ GenericOptionParser ToolRunner ❖ P135 ❖ P135 ‣ JVM P136 ‣ -conf
    9. 9. 5.3 P137 ❖ map reduce ‣ ‣ OutputCollector ‣ ❖ = ❖ JUnit Mockito TDD ‣ MapReduce MRUnit
    10. 10. 5.3 P137 ❖ MaxTemperatureMapper ‣ Test: 5-4 P137 ‣ mapper ver.1 5-5 P138 ‣ Test: P138 ‣ mapper ver.2 P139 ❖ MaxTemperatureReducer ‣ Test: P139 ‣ reducer 5-6 P140
    11. 11. 5.4 P140 ❖ ‣ map reducer ‣ Tool ❖ JobRunner ‣ MapReduce ‣
    12. 12. 5.4 P142 ❖ ‣ mapper - ❖ 5-8 P143 ❖ mapper 5-9 P144 ❖ TDD
    13. 13. 5.4 P144 ❖ JobRunner ‣ Tool ‣ 5-10 P145 ❖ P145 ‣ HDFS MapReduce ‣ Hadoop ‣
    14. 14. 5.5 P146 ❖ ❖ ‣ JobClient P147 ‣ ID = job_YYYYMMDDHHMM_0000 ‣ ID = ID + _m_000000 ‣ ID = ID + _0000
    15. 15. 5.5 P148 ❖ MapReduce Web UI http://jobtracker-host:50030/ ‣ jobtracker P148 - Hadoop - - - - ❖ P150 - -
    16. 16. 5.5 P152 ❖ ‣ reducer - MapReduce ‣ ‣ ‣
    17. 17. 5.5 P153 ❖ ‣ Web UI ‣ ‣ MapReduce ❖ Web UI ‣ ‣
    18. 18. 5.5 P158 ❖ P158 ‣ P157 ‣ ❖ Hadoop P157 ‣ Chukwa ‣ ‣
    19. 19. 5.5 P159 ❖ ‣ ‣ ‣ ‣ ‣ ‣ IDE
    20. 20. 5.6 P161 ❖ ✓ mapper ✓ reducer ✓ combiner ✓ ✓ ✓
    21. 21. 5.6 P161 ❖ ‣ JobRunner P162 ❖ HPROF P162 ‣ ‣ ‣
    22. 22. 5.7 MapReduce P164 ❖ MapReduce ❖ ‣ UNIX ❖ MapReduce Pig Hive
    23. 23. 5.7 MapReduce P166 ❖ ❖ 2 ❖ ❖ ❖ ‣ JobControl ‣ Hadoop Workflow Scheduler HWS ※
    24. 24. ❖ TDD ‣ ‣ ‣ ❖ ❖ ❖
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×