This report discusses an internship project involving setting up Hadoop clusters using either manual configuration or an on-demand configuration using Docker containers and Ansible for provisioning. The project allows configuring NameNode, DataNodes, TaskTrackers and JobTrackers and provides a live shell to run Linux commands. It uses Ansible for 60% of the configuration and Python CGI for 40%. HDFS and Docker are implemented and the project aims to completely set up Hadoop clusters.

1. A K H I L G O Y A L
1 5 C S U 0 1 6
Summer Intenship
Report

2. About Company
 LinuxWorld Informatics Pvt Ltd is an ISO Certified
company which is dedicated to offering a
comprehensive set of most useful open source and
Commercial training programmes today’s Industry
demands! . It’s main office is in Bangalore.
The Structure of the organization is very simple. It is
fully governed by young and Energetic Technocrats,
dedicated to Open Source technologies and Linux
promotion. The team of faculty is guided by Mr.
Vimal Daga –the chief technicalofficer of Linux
World Informatics Pvt Ltd.

3. About Project
 In our project we have used Hadoop as a framework which is Apache open
source
framework.
My product provides two way to setup hadoop cluster-
Manual Configuration – Provided the node statistics like RAM, CPU and
Hard disk to the client. Cluster will be established as per on click selection
of Name Node, Data Node, Task Tracker and Job Tracker etc.
On demand Configuration – Establishment of cluster using Docker
(Container) and Ansible (DevOps) depending upon the number of
Data Nodes, Task Tracker and Node Manager

4. Features
 In Docker , We can setup Docker Images of type Centos:
latest, Fedora , Ubuntu and we can also manage our own
created docker images (container) by starting, stopping and
Removing our containers.
 We have provided our live shell on which we can run the
Linux Commands.
 We can setup Httpd (Hypertext transfer protocol Daemon
that is web server) server Configuration and NFS Server
Configuration
 In our Project we have used 60% of the Ansible and 40% of
python cgi.

5. NEED

6. Windows 10
Virtual Box 5.2
Redhat Linux 7.3
NFS Docker HDFS HTTPD

7. HDFS Architecture

9. Complete setup

11. Docker