This is the basic tutorial to help developer or system administrator to build a basic cloud environment with Docker. In this book, I will not use Dockerfile to create a container because I don't know how to use that yet. XDDD In the end of this book, I will summary some problem I haven't solved yet. If there is any mistake, please let me know. GitBook:

2. 1. Introduction
2. Install Docker on Ubuntu 14.04
3. Set Hadoop Environment in a Docker Container
4. Set HBase Environment in a Docker container
5. Export and Import a Docker Image between Nodes in Cluster
6. The Problem I Haven't Solved
7. Possible Problems
Table of Contents

3. This is the basic tutorial to help developer or system administrator to build a basic cloud environment with
Docker.
In this book, I will not use Dockerfile to create a container because I don't know how to use that yet.
XDDD
In the end of this book, I will summary some problem I haven't solved yet.
If there is any mistake, please let me know.
Distribute Cloud Environment with Docker

4. $ sudo apt-get update
$ sudo apt-get install docker.io
Enable tab-completion of Docker commands in BASH
$ source /etc/bash_completion.d/docker.io
First, check your system can deal with https URLs: the /usr/lib/apt/methods/https should exist. If
it doesn't, you need to install the package apt-transport-https
$ apt-get update
$ apt-get install apt-transport-https
Add the Docker repository key to your system keychain.
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D857
Add the Docker repository to your apt source list
$ sudo sh -c “echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker
To verify that everything has worked as exepected:
$ sudo docker run -i -t ubuntu /bin/bash
You should download the latest version of Ubuntu image, and then start bash in a new container.
Show running containers:
Install Docker on Ubuntu 14.04 LTS
Ubuntu-maintained Package Installation
Docker-maintained Package Installation
Note: If you want install the recently version of Docker, you do not need to install
docker.io from Ubuntu
Basic Docker Command-line

5. $ sudo docker ps
Show all images in your local repository:
$ sudo docker images
Run a container from a specific image
$ sudo docker run -i -t <image_id || repository:tag> -bash
Start a existed container:
$ sudo docker start -i <image_id>
Attach a running container:
$ sudo docker attach <container_id>
Exit without shutting down a container:
[Ctrl-p] + [Ctrl-q]
https://docs.docker.com/installation/ubuntulinux/#ubuntu-trusty-1404-lts-64-bit
Reference

6. In a new Docker container, you need to set basic environment first before setting Hadoop.
$ sudo apt-get update
$ sudo apt-get install default-jdk
The default JDK will be installed at /usr/lib/jvm/<java-version>
$ sudo apt-get install git wget vim ssh
$ adduser hduser
Grant a user privileges
$ visudo
Append the hduser you just created below the root and just set privileges specification as root
$ ssh-keygen -t rsa
$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
Set Hadoop Environment in a Docker Container
Update Apt List
Install Java JDK
Install some needed package
Create a user to manage hadoop cluster
Generate SSH authorized key to let socket connection without
password
Set the port of SSH and SSHD

7. Because I use Docker as my distributed tool, so the default ssh port 22 has been listened by host
machine, I need to use another port to communicate between Docker containers in different host
machine. In my example, I will use 2122 port for listening and sending request.
ssh
$ sudo vi /etc/ssh/ssh_config
-> Port 2122
wq!
sshd
$ sudo vi /etc/ssh/sshd_config
-> Port 2122
-> UsePAM no
wq!
I supposed my cloud environment is as following:
VM: 5 nodes (master, master2, slave1, slave2, slave3)
OS: Ubuntu 14.04 LTS
Docker Version: 1.3.1
Hadoop Version: 2.6.0
Download hadoop-2.6.0
$ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
$ tar zxvf hadoop-2.6.0.tar.gz
Set environment path
$ sudo vi /etc/profile
-> export JAVA_HOME=/usr/lib/jvm/<java-version>
-> export HADOOP_HOME=<YOUR_HADOOP_PACKAGE_PATH>
-> export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
-> export PATH=$HADOOP_HOME/bin:$PATH
-> export CLASSPATH=$HADOOP_HOME/lib:$CLASSPATH
:wq!
$ source /etc/profile
All needed will be stored in <HADOOP_HOME>/etc/hadoop
Set Hadoop Environment
Modify Hadoop configuration

8. core-site.xml
<configuration>
<property>
<name>fs.DefaultFS</name>
<value>hdfs://master:9000</value>
<description>The master endpoint in cluster.</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/<HADOOP_HOME>/temp</value>
<description>A base for other temporary directories.</description>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master2:90001</value>
<description>Set secondary namenode to prevent the master node crash.</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/<HADOOP_HOME>/dfs/name</value>
<description>Set the location of name node version.</description>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>file:/<HADOOP_HOME>/dfs/data</value>
<description>Set the location of data node version.</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
<description>The number of replication in cluster.</description>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
</configuration>

9. mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
yarn-stie.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
hadoop-env.sh
$ vi <HADOOP_CONF_DIR>/hadoop-env.sh
-> export JAVA_HOME=/usr/lib/jvm/<java-version>
wq!

10. yarn-env.sh
$ vi <HADOOP_CONF_DIR>/yarn-env.sh
-> export JAVA_HOME=/usr/lib/jvm/<java-version>
wq!
slaves
$ vi <HADOOP_CONF_DIR>/slaves
-> master2
-> slave1
-> slave2
-> slave3

11. After setting Hadoop environment in previous section, you can set hbase environment now.
In the following example, I will use custom zookeeper to manage the resource of my cluster.
HBase version: 0.99.2
Zookeeper version: 3.3.6
$ wget http://ftp.twaren.net/Unix/Web/apache/hbase/hbase-0.99.2/hbase-0.99.2-bin.tar.gz
$ wget http://ftp.twaren.net/Unix/Web/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz
$ tar -zxvf hbase-0.99.2-bin.tar.gz
$ tar -zxvf zookeeper-3.3.6.tar.gz
hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>hdfs://master:60000</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value><ZOOKEEPER_HOME>/data</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
</configuration>
Set HBase Environment in a Docker container
Download HBase and Zookeeper
Set the configuration of HBase

12. hbase-env.sh
$ vi <HBASE_HOME>/conf/hbase-env.sh
-> export HBASE_HOME=<HBASE_HOME>
-> export HADOOP_HOME=<HBASE_HOME>
-> export HBASE_CLASSPATH=$HADOOP_CONF_DIR
-> HBASE_MANAGES_ZK=false
regionservers
$ vi <HBASE_HOME>/conf/slaves
-> master2
-> slave1
-> slave2
-> slave3
As previous setting of hbase-env.sh , you can see I set HBASE_MANAGES_ZK=false to use my custom
zookeeper to manage and monitor the resource of cluster.
zoo.cfg
$ vi <ZOOKEEPER_HOME>/conf/zoo.cfg
-> dataDir=<ZOOKEEPER_HOME>/data
-> clientPort=2181
-> server.1=master:2888:3888
And then add a myid file under <ZOOKEEPER_HOME>/data to tell zookeeper which node is the
zookeeper running on.
for example, as my zoo.cfg set server. 1 = master :2888:3888, it means this zookeeper thread running
on master node binding 2888 port and 3888 port. So I need to tell zookeeper which machine is that
run on.
$ vi myid
-> 1
wq!
$ sudo vi /etc/profile
-> export HBASE_HOME=<HBASE_HOME>
-> export ZOOKEEPER_HOME=<ZOOKEEPER_HOME>
-> export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH
Set the configuration of Zookeeper
Set System Environment

13. After following previous sections to set Hadoop and HBase configurations, we can commit this Docker
image to distribute cloud cluster.
Brief Summary

14. The following figure shows what I plan to distribute my cloud cluster.
If you follow the previous sections, you will get a Docker container with hadoop environment. To use that,
you need to duplicate it to other endpoints you want to distribute to build hadoop cluster.
Save the container as a compressed tar file.
$ sudo docker save <image_repository:tag> > XXXX.tar
After finishing export, I use scp to transport the image to other nodes in cluster.
Export and Import a Docker Image between Nodes in
Cluster
NOTE: My experiment environment is on Windows Azure Virtual
Machine.
Export a Docker container to a image.

15. $ scp -P [port] XXXX.tar [account]@[domain]/<where you want to store>
Now, switch to the master2 to show how to use the image we just transport from master .
$ sudo docker load < XXXX.tar
After loading tar file, we can check the image just importing from local repository.
$ sudo docker images
Now, we can start to use this image to distribute the cluster. In this example, I write a simple shell
script to run Docker image instead of Dockerfile because I haven't knowed how to use Dockerfile
to build a Docker container.
Because --link option is not fit in my situation, I use the basically port mapping to try connect with each
container via internet
$ vi bootstrap.sh
-> sudo docker run -i -t -p 2122:2122 -p 50020:50020 -p 50090:50090 -p 50070:50070 -p 50010:50010 -p
wq!
$ sh bootstrap.sh
In this example, I use master2 as hostname and listen all needed ports from container to endpoint
machine.
Now, we can start to boot the hadoop cluster on. There are some steps before start-dfs.sh .
Each containers in cluster need to do the following statement
$ source /etc/profile
$ sudo vi /etc/hosts
-> put all IP and host name
## for example
Shell script
Distribute with Docker Container

16. 127.0.0.5 master
10.0.0.2 master2
10.0.0.3 slave1
10.0.0.4 slave2
10.0.0.5 slave3
wq!
## restart ssh twice
$ sudo service ssh restart
$ sudo service ssh restart
Master container need to do this
## format hdfs namenode
$ hdfs namenode -format
$ <HADOOP_HOME> sbin/start-dfs.sh
$ <HADOOP_HOME> sbin/start-yarn.sh
## make the root directory of hbase
$ hadoop fs -mkdir /hbase
## start zookeeper
$ zkServer.sh start
## start hbase
$ start-hbase.sh
## TEST
$ jps
If success, you will see the following process running on master

17. Now, we can start using hadoop and hbase to record and analyze data by following this tutorial.
Nevertheless, there are still problems I've met but not solved.
In section 4, after I docker run each containers, I have to modify every /etc/hosts of each containers
to connect each others. But it will happen two problems.
First, it is inconvenient to modify every hosts file if large amounts of endpoint machines.
Second, if reboot containers, the IP of container will be automatically reset by Docker. Suppose you use
HBase as your NoSQL DB, Zookeeper will store old IP and all regionservers can't trace back the
master node.
I've surveyed some method to solve but not implement yet. Look at the following link:
http://jpetazzo.github.io/2013/10/16/configure-docker-bridge-network/
If I solve problems, I will update the book.
I've referenced the before link and try to solve the second problem I met. There are some new problems
happened so that I don't solve it.
I've modify the interface IP and route of container successfully, but I can't ssh into container after do
that. Th following code is what I use, maybe everybody can discuss on this issue in stackoverflow .
http://stackoverflow.com/questions/27937185/assign-static-ip-to-docker-container
pid=$(sudo docker inspect -f '{{.State.Pid}}' <container_name> 2>/dev/null)
sudo rm -rf /var/run/netns/*
sudo ln -s /proc/$pid/ns/net /var/run/netns/$pid
sudo ip link add A type veth peer name B
sudo brctl addif docker0 A
sudo ip link set A up
sudo ip link set B netns $pid
sudo ip netns exec $pid ip link set eth0 down
sudo ip netns exec $pid ip link delete eth0
sudo ip netns exec $pid ip link set dev B name eth0
sudo ip netns exec $pid ip link set eth0 address 12:34:56:78:9a:bc
sudo ip netns exec $pid ip link set eth0 down
sudo ip netns exec $pid ip link set eth0 up
sudo ip netns exec $pid ip addr add 172.17.0.1/16 dev eth0
sudo ip netns exec $pid ip route add default via 172.17.42.1
The Problem I Haven't Solved
Update
2015/01/20

18. ClockOutOfSyncException
Becasue the hadoop cluster time date is not sync, you can use ntpdate asia.pool.ntp.org to sync
date time of each hosts.
ConnectionRefused
Please confirm your IP and hostname in /etc/hosts is correct or not.
continue...
Possible Problems