15 map reduce on azure

1. MapReduce on Azure
UN EXEMPLE CONCRET

2. Se connecter sur notre instance
 ssh pbury@168.63.111.90
 git clone https://gitlab.com/pbury/m2isf_hadoop.git
 cd m2isf_hadoop
 sudo docker-compose –f docker-compose-local.yml build
 sudo docker-compose –f docker-compose-local.yml up
 sudo docker ps
 sudo docker exec –it resourcemanager bash
 Ca y est, on est dans le container !!
 exit
2

3. Sans Hadoop
cat /usr/local/src/ascii_5000.txt |
/usr/local/src/mapper.py | sort -k1,1 |
/usr/local/src/reducer.py
3

4. Hadoop : on prépare le travail
On copie la donnée
hdfs dfs -mkdir /gutemberg
hdfs dfs -put /usr/local/src/ascii_5000.txt /gutemberg
On rentre dans le docker
docker exec –it resourcemanager bash
4

5. On lance Map Reduce
hadoop jar /opt/hadoop-2.7.1/share/hadoop/tools/lib/hadoop-streaming-
2.7.1.jar
-file /usr/local/src/mapper.py
-mapper /usr/local/src/mapper.py
-file /usr/local/src/reducer.py
-reducer /usr/local/src/reducer.py
-input /gutemberg
-output /gutemberg-output5
5

6. Résultats
hdfs dfs -ls /gutemberg-output5/
Found 2 items
-rw-r--r-- 3 root supergroup 0 2019-01-29 07:24 /gutemberg-output5/_SUCCESS
-rw-r--r-- 3 root supergroup 340155 2019-01-29 07:24 /gutemberg-output5/part-00000
6

7. Résultats
hdfs dfs –cat gutemberg-output5/part-00000
7

8. Références
 https://gitlab.com/pbury/m2isf_hadoop
 Créer une clef ssh pour azure :
 https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/virtual-
machines/linux/ssh-from-windows.md
 Powershell : ssh-keygen
8