Hadoop installation and Running KMeans Clustering with MapReduce Program on Hadoop

1
Hadoop Installation and Running KMeans Clustering
with MapReduce Program on Hadoop
Introduction
General issue that I will cover in this document are Hadoop installation (in section 1) and running
KMeans Clustering MapReduce program on Hadoop (section 2).
1 Hadoop Installation
I will install Hadoop Single Node Cluster mode in my personal computer using this following
environment .
1. Ubuntu 12.04
2. Java JDK 1.7.0 update 21
3. Hadoop 1.2.1 (stable)
1.1 Prerequisites
Before installing Hadoop, the following point must be done first before installing Hadoop in our
system
1. Sun JDK /Open JDK
I use Sun JDK from oracle instead of Open JDK, the resource package can be downloaded from here :
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
2. Hadoop installer packages
In this report, I will use Hadoop version 1.2.1 (stable). The resource package can be downloaded
from her : http://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/
1.1.1 Configuring Java
In my computer, I have several installed java version, they are Java 7 and java 6. For running Hadoop
program, I need to configure which version I will use. I decide to use the newer version (java version
1.7 update 21) so the following are step by step for configuring latest java version.
1. Configure java
$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/jdk1.6.0_45/bin/java 1 auto mode
1 /usr/lib/jvm/jdk1.6.0_45/bin/java 1 manual mode
* 2 /usr/lib/jvm/jdk1.7.0_21/bin/java 1 manual mode
Press enter to keep the current choice[*], or type selection number: 2

2
2. configure javac
$ sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).
------------------------------------------------------------
0 /usr/lib/jvm/jdk1.6.0_45/bin/javac 1 auto mode
1 /usr/lib/jvm/jdk1.6.0_45/bin/javac 1 manual mode
* 2 /usr/lib/jvm/jdk1.7.0_21/bin/javac 1 manual mode
3. configure javaws
$ sudo update-alternatives --config javaws
There are 2 choices for the alternative javaws (providing /usr/bin/javaws).
------------------------------------------------------------
0 /usr/lib/jvm/jdk1.6.0_45/bin/javaws 1 auto mode
1 /usr/lib/jvm/jdk1.6.0_45/bin/javaws 1 manual mode
* 2 /usr/lib/jvm/jdk1.7.0_21/bin/javaws 1 manual mode
4. check the configuration
To make sure the latest java, javac, and javaws successfully configure, I use this following command
tid@dbubuntu:~$ java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
1.1.2 Hadoop installer
After downloading Hadoop installer package, then we need to extract the installer package into the
desire directory. I downloaded Hadoop installer and locate in the ~/Download directory.
For extracting Hadoop installer package in the local directory, I use the following command
$ tar -xzfv hadoop-1.2.1.tar.gz
1.2 System configuration
In this section I will explain step-by step how to setup and preparing the system for Hadoop single
node cluster in my local compute. The system configuration consists of network configuration for
setup the hosts name, move Hadoop extracted package into desire folder, enabling ssh, and adding
and changing folder permission.
1.2.1 Network configuration
In the hadoop network configuration, all of the machines should have alias instead of IP address. To
configure network aliases, we can edit /etc/hosts on machine that will we use for Hadoop
master and slave. In my case, since I am using single node, the configuration can be done by these
step
1. Open file /etc/hosts as sudoers,

3
$ sudo nano /etc/hosts
2. I add the following lines, inside the curly branch is my IP Address,
127.0.0.1 localhost
164.125.50.127 localhost
Note :
For configuring the local computer into pseudo distributed mode, the following configuration for
/etc/hosts is used
# /etc/hosts (for Hadoop Master and Slave)
192.168.0.1 master
192.168.0.2 slave
1.2.2 User configuration
For security issue, we better to create special user for Hadoop in each machine, however, since I am
working in local computer, I use existing user. The following command is for adding new user and
group
$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser
1.2.3 Configuring SSH
Hadoop requires SSH access to manage its nodes. In this configuration, I also configure SSH access to
localhost for hduser that I already made in the previous section and local existing user.
1. For generating ssh key for hduser, we can use the following command
$ su - hduser
Password:
hduser@dbubuntu:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
/home/hduser/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is:
b8:68:c1:4b:d1:fe:4b:40:2c:1c:b4:37:db:5f:76:ee hduser@dbubuntu
The key's randomart image is:
+--[ RSA 2048]----+
| .o |
| . = |
| = * |
| . * = |
| + = S o . |
| . + + . o o |
| + . o . . |
| . . . . |
| . E |
+-----------------+
hduser@dbubuntu:~$

4
2. Enable ssh access for local machine with newly created key
cat /home/hduser/.ssh/id_rsa.pub >>
/home/hduser/.ssh/authorized_keys
Note :
Since I use local existing user, I do the above step 2 times, also for user tid
1.2.4 Extracting Hadoop installer
I copy the Hadoop Installer package from ~/Download directory into desire folder. In this case I use
/usr/local to locate hadoop installer package.
1. Command for moving extracted Hadoop into
$ cp ~/Downloads/Hadoop-1.2.1 /usr/local/
$ cd /usr/local
2. in order to make hadoop easy to access and handling some update version of Hadoop, I create
symbolic link of hadoop-1.2.1 into hadoop directory
$ ln -s hadoop-1.2.1 hadoop
3. change folder permission of hadoop, so it will accessible by user tid
$ sudo chown -R tid:hadoop hadoop
1.3 Hadoop Configuration
After sucessfully configuring the network, folder, and other configuration, in this part, I will explain
step by step hadoop configuration for each machine. Since I locate my hadoop inside
/usr/local/hadoop so it will be the active directory, and all of the hadoop configurations are
located in conf directory.
1.3.1 conf/masters
Since in this configuration for single node, the content of masters file by default should be like this
Localhost
Note : For multi node cluster we can add the master alias’s name regarding the network setup and
configuration in section 1.2.1
1.3.2 conf/slaves
Same as masters file, the default setup for slaves should be like this
Localhost
Note : for multi node cluster, we can add the master and slave alias(considering network
configuration in section 1.2.1), for example the slaves file might look like this
slave1
slave2
slave3

5
1.3.3 conf/core-site.xml
For core site configuration I specify the location in directory /tmp/Hadoop/app. In this file, I give
the configuration of core site of cluster. Firstly, the file look like this following
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
</configuration>
then I change the configuration to look like this
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop/app</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
1.3.4 Conf/mapred-site.xml
I change the mapred-site.xml , that it look like this following
<configuration>

<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>
The host and port that
the MapReduce job tracker runs
at. If "local", then jobs are run
in-process as a single map
and reduce task.
</description>
</property>
</configuration>

6
1.3.5 hdfs-site.xml
For hdfs-site, for single cluster, I specify the number of replication only 1, if we have several
machine we can add the number of replication.
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can
be specified when the file is created.
The default is used if replication
is not specified in create time.
</description>
</property>
</configuration>
1.3.6 Formatting HDFS via Namenode
Before starting the cluster, we should format the Hadoop File System. It can be formatted once or
more, however formatting the namenode means clear all of the data, so we need to be careful
otherwise we can lose our data
tid@dbubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format
1.4 Running Hadoop
After setting all needed configuration, finally we can start our Hadoop. For running Hadoop daemon,
there are several alternatives,
1.4.1 Starting all daemon at once
For starting all of the service in Hadoop, I use this following command
tid@dbubuntu:/usr/local/hadoop$ bin/start-all.sh
Then, checking whether all of daemons already running using the following command
tid@dbubuntu:/usr/local/hadoop$ jps
1.4.2 Stopping all daemon
For stopping all daemon, we can use the following command
tid@dbubuntu:/usr/local/hadoop$ bin/stop-all.sh

7
2 Hadoop MapReduce Program (KMeans Clustering in Map
Reduce)
In this part I will explain how I can run map reduce program regarding reference from Thomas’s blog
for KMeans Clustering in MapReduce (http://codingwiththomas.blogspot.kr/2011/05/k-means-
clustering-with-mapreduce.html ) with some modification
2.1 Eclipse IDE Setup
There are several ways for setting up the IDE environment so we can easily create MapReduce
Program in Eclipse. For ease development and setup, I am using Eclipse plugin from self build plugin
by creating jar from Hadoop library. The following are step-by-step setting up eclipse IDE
Step 1: copy the pre-build eclipse plugin for Hadoop in directory plugins of Eclipse
Figure 1 Eclipse hadoop plugin inside plugins directory of eclipse
Step 2 : restart Eclipse, then in the perspective part, we will see other perspective in the right
corner, and choose MapReduce Perspective
Figure 2 MapReduce perspective in eclipse IDE
Step 3: Add the server in the map reduce panel.
In my case, because my server is located in the local machine, named as localhost, the detail will
looks like the following

8
Figure 3 Configuration for adding Hadoop Location
After adding the server, in the right side, along with project explorer, we will see the HDFS file
explorer, and right now Eclipse is ready to use for developing MapReduce Application
Figure 4 HDFS Directory explorer
2.2 Source Code Preparation
In this part, I will describe how to prepare the project, package, and class for KMeans Clustering
MapReduce program.
Create new project
1. First, we need to create new MapReduce project, by clicking new project in the upper left
corner of eclipse, and after the following
window pup up, choose MapReduce
project
Figure 5 Choosing MapReduce project

9
2. Fill the project name, and check the reference location of Hadoop. By default, if we are using
eclipse plugin for Hadoop, the folder will be directed to our Hadoop installation folder , then
click “Finish”
Create Package and Class
For KMeans Clustering MapReduce program, based on Thomas’s references, we need to create two
package, one package for clustering model, consists of Model class for Vector, Distance Measure,
and define the ClusterCenter (Vector.java, DistanceMeasurer.java, and ClusterCenter.java) and the
other is package for Main, Mapper, and Reducer Class (KMeans
1. com.clustering.model package Model Class (Vector.java)
Model Class : Vector.java
package com.clustering.model;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.Arrays;
import org.apache.hadoop.io.WritableComparable;
public class Vector implements WritableComparable<Vector>{
private double[] vector;
public Vector(){
super();
}
public Vector(Vector v){
super();
int l= v.vector.length;
this.vector= new double[l];
System.arraycopy(v.vector, 0,this.vector, 0, l);

10
}
public Vector(double x, double y){
super();
this.vector = new double []{x,y};
}
@Override
public void readFields(DataInput in) throws IOException {
// TODO Auto-generated method stub
int size = in.readInt();
vector = new double[size];
for(int i=0;i<size;i++)
vector[i]=in.readDouble();
}
@Override
public void write(DataOutput out) throws IOException {
out.writeInt(vector.length);
for(int i=0;i<vector.length;i++)
out.writeDouble(vector[i]);
}
@Override
public int compareTo(Vector o) {
boolean equals = true;
for (int i=0;i<vector.length;i++){
if (vector[i] != o.vector[i]) {
equals = false;
break;
}
}
if(equals)
return 0;
else
return 1;
}
public double[] getVector(){
return vector;
}
public void setVector(double[]vector){
this.vector=vector;
}
public String toString(){
return "Vector [vector=" + Arrays.toString(vector) + "]";
}
}
2. Distance Measurement Class
Distance Measurement class : DistanceMeasurer.java

11
public class DistanceMeasurer {
public static final double measureDistance(ClusterCenter center,
Vector v) {
double sum = 0;
int length = v.getVector().length;
for (int i = 0; i < length; i++) {
sum += Math.abs(center.getCenter().getVector()[i]
- v.getVector()[i]);
}
return sum;
}
}
3. ClusterCenter
Cluster Center definition : ClusterCenter.java
import java.io.DataInput;
import java.io.DataOutput;
import org.apache.hadoop.io.WritableComparable;
public class ClusterCenter implements WritableComparable<ClusterCenter> {
private Vector center;
public ClusterCenter() {
super();
this.center = null;
}
public ClusterCenter(ClusterCenter center) {
super();
this.center = new Vector(center.center);
}
public ClusterCenter(Vector center) {
super();
this.center = center;
}
public boolean converged(ClusterCenter c) {
return compareTo(c) == 0 ? false : true;
}
@Override
public void write(DataOutput out) throws IOException {
center.write(out);
}
@Override
public void readFields(DataInput in) throws IOException {
this.center = new Vector();
center.readFields(in);
}

12
@Override
public int compareTo(ClusterCenter o) {
return center.compareTo(o.getCenter());
}
/**
* @return the center
*/
public Vector getCenter() {
return center;
}
@Override
public String toString() {
return "ClusterCenter [center=" + center + "]";
}
}
After configuring the class model, the next one is MapReduce Classes, which consist of Mapper,
Reducer, and finally the Main class
1. Mapper class
Mapper class : KMeansMapper.java
package com.clustering.mapreduce;
import java.util.LinkedList;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.mapreduce.Mapper;
import com.clustering.model.ClusterCenter;
import com.clustering.model.DistanceMeasurer;
import com.clustering.model.Vector;
public class KMeansMapper extends
Mapper<ClusterCenter, Vector, ClusterCenter, Vector>{
List<ClusterCenter> centers = new LinkedList<ClusterCenter>();
@Override
protected void setup(Context context) throws IOException,
InterruptedException {
super.setup(context);
Configuration conf = context.getConfiguration();
Path centroids = new Path(conf.get("centroid.path"));
FileSystem fs = FileSystem.get(conf);
SequenceFile.Reader reader = new SequenceFile.Reader(fs,
centroids,
conf);

13
ClusterCenter key = new ClusterCenter();
IntWritable value = new IntWritable();
while (reader.next(key, value)) {
centers.add(new ClusterCenter(key));
}
reader.close();
}
@Override
protected void map(ClusterCenter key, Vector value, Context context)
throws IOException, InterruptedException {
ClusterCenter nearest = null;
double nearestDistance = Double.MAX_VALUE;
for (ClusterCenter c : centers) {
double dist = DistanceMeasurer.measureDistance(c,
value);
if (nearest == null) {
nearest = c;
nearestDistance = dist;
} else {
if (nearestDistance > dist) {
nearest = c;
nearestDistance = dist;
}
}
}
context.write(nearest, value);
}
}
2. Reducer class
Reducer class : KMeansReducer.java
import java.util.LinkedList;
import java.util.List;
import org.apache.hadoop.mapreduce.Reducer;
public class KMeansReducer extends
Reducer<ClusterCenter, Vector, ClusterCenter, Vector>{
public static enum Counter{
CONVERGED
}
List<ClusterCenter> centers = new LinkedList<ClusterCenter>();

14
protected void reduce(ClusterCenter key, Iterable<Vector> values,
Context context) throws IOException,
InterruptedException{
Vector newCenter = new Vector();
List<Vector> vectorList = new LinkedList<Vector>();
int vectorSize = key.getCenter().getVector().length;
newCenter.setVector(new double[vectorSize]);
for(Vector value :values){
vectorList.add(new Vector(value));
for(int i=0;i<value.getVector().length;i++){
newCenter.getVector()[i]+=value.getVector()[i];
}
}
for(int i=0;i<newCenter.getVector().length;i++){
newCenter.getVector()[i] =
newCenter.getVector()[i]/vectorList.size();
}
ClusterCenter center = new ClusterCenter(newCenter);
centers.add(center);
for(Vector vector:vectorList){
context.write(center, vector);
}
if(center.converged(key))
context.getCounter(Counter.CONVERGED).increment(1);
}
protected void cleanup(Context context) throws
IOException,InterruptedException{
super.cleanup(context);
Configuration conf = context.getConfiguration();
Path outPath = new Path(conf.get("centroid.path"));
fs.delete(outPath,true);
final SequenceFile.Writer out = SequenceFile.createWriter(fs,
context.getConfiguration(),
outPath, ClusterCenter.class, IntWritable.class);
final IntWritable value = new IntWritable(0);
for(ClusterCenter center:centers){
out.append(center, value);
}
out.close();
}
}

15
3. Main class
Main class : KMeansClusteringJob.java
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
public class KMeansClusteringJob {
private static final Log LOG =
LogFactory.getLog(KMeansClusteringJob.class);
public static void main(String[] args) throws IOException,
InterruptedException, ClassNotFoundException {
int iteration = 1;
Configuration conf = new Configuration();
conf.set("num.iteration", iteration + "");
Path in = new Path("files/clustering/import/data");
Path center = new
Path("files/clustering/import/center/cen.seq");
conf.set("centroid.path", center.toString());
Path out = new Path("files/clustering/depth_1");
Job job = new Job(conf);
job.setJobName("KMeans Clustering");
job.setMapperClass(KMeansMapper.class);
job.setReducerClass(KMeansReducer.class);
job.setJarByClass(KMeansMapper.class);
SequenceFileInputFormat.addInputPath(job, in);
if (fs.exists(out))
fs.delete(out, true);
if (fs.exists(center))
if (fs.exists(in))
final SequenceFile.Writer centerWriter =
SequenceFile.createWriter(fs,
conf, center, ClusterCenter.class,
IntWritable.class);

16
final IntWritable value = new IntWritable(0);
centerWriter.append(new ClusterCenter(new Vector(1, 1)),
value);
centerWriter.append(new ClusterCenter(new Vector(5, 5)),
value);
centerWriter.close();
final SequenceFile.Writer dataWriter =
SequenceFile.createWriter(fs,
conf, in, ClusterCenter.class,
Vector.class);
dataWriter
.append(new ClusterCenter(new Vector(0,
0)), new Vector(1, 2));
dataWriter.append(new ClusterCenter(new Vector(0, 0)),
new Vector(16, 3));
dataWriter
0)), new Vector(3, 3));
dataWriter
0)), new Vector(2, 2));
dataWriter
0)), new Vector(2, 3));
dataWriter.append(new ClusterCenter(new Vector(0, 0)),
new Vector(25, 1));
dataWriter
0)), new Vector(7, 6));
dataWriter
0)), new Vector(6, 5));
dataWriter.append(new ClusterCenter(new Vector(0, 0)), new
Vector(-1,
-23));
dataWriter.close();
SequenceFileOutputFormat.setOutputPath(job, out);
job.setInputFormatClass(SequenceFileInputFormat.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
job.setOutputKeyClass(ClusterCenter.class);
job.setOutputValueClass(Vector.class);
job.waitForCompletion(true);
long counter = job.getCounters()
.findCounter(KMeansReducer.Counter.CONVERGED).getValue();
iteration++;
while (counter > 0) {
conf = new Configuration();
conf.set("centroid.path", center.toString());
conf.set("num.iteration", iteration + "");
job = new Job(conf);
job.setJobName("KMeans Clustering " + iteration);
job.setMapperClass(KMeansMapper.class);
job.setReducerClass(KMeansReducer.class);
job.setJarByClass(KMeansMapper.class);
Input
vector
K-Center
vector

17
in = new Path("files/clustering/depth_" +
(iteration - 1) + "/");
out = new Path("files/clustering/depth_" +
iteration);
SequenceFileInputFormat.addInputPath(job, in);
if (fs.exists(out))
SequenceFileOutputFormat.setOutputPath(job, out);
job.setInputFormatClass(SequenceFileInputFormat.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
job.setOutputKeyClass(ClusterCenter.class);
job.setOutputValueClass(Vector.class);
job.waitForCompletion(true);
iteration++;
counter = job.getCounters()
.findCounter(KMeansReducer.Counter.CONVERGED).getValue();
}
Path result = new Path("files/clustering/depth_" +
(iteration - 1) + "/");
FileStatus[] stati = fs.listStatus(result);
for (FileStatus status : stati) {
if (!status.isDir() &&
!status.getPath().toString().contains("/_")) {
Path path = status.getPath();
LOG.info("FOUND " + path.toString());
SequenceFile.Reader reader = new
SequenceFile.Reader(fs, path,
conf);
ClusterCenter key = new ClusterCenter();
Vector v = new Vector();
while (reader.next(key, v)) {
LOG.info(key + " / " + v);
}
reader.close();
}
}
}
}

18
Final project listing will look like this
Figure 6 File Listing for KMeansMapReduce Program
2.3 Run the program
Unlike the wordcount program that we have to prepare the input files, in KMeansClustering
program the input is defined inside the KMeansClusteringJob class.
For running KMeansClustering job, since we are already configure the eclipse, we can run the
program natively inside Eclipse, So by pointing out the Main class (KMeansClusteringJob.java) we can
run the project as Hadoop Application
Figure 7 Run Project as hadoop application
The Input (to be defined in KMeansClusteringJob class)
Vector [vector=[16.0, 3.0]]
Vector [vector=[-1.0, -23.0]]
Output from Thomas’s Blog :
ClusterCenter [center=Vector [vector=[13.5, 3.75]]] / Vector [vector=[16.0, 3.0]]
ClusterCenter [center=Vector [vector=[1.4, -2.6]]] / Vector [vector=[1.0, 2.0]]

19
ClusterCenter [center=Vector [vector=[1.4, -2.6]]] / Vector [vector=[-1.0, -23.0]]
Output of my KMeansClusteringJob :
file:/home/tid/eclipse/workspace/MRClustering/files/clustering/depth_3/part-r-00000
14/04/08 15:50:39 INFO mapreduce.KMeansClusteringJob: ClusterCenter [center=Vector
[vector=[13.5, 3.75]]] / Vector [vector=[16.0, 3.0]]
[vector=[1.4, -2.6]]] / Vector [vector=[1.0, 2.0]]
[vector=[1.4, -2.6]]] / Vector [vector=[-1.0, -23.0]]
Complete Log Result of My KMeansClusteringJob
14/04/08 15:50:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
14/04/08 15:50:35 INFO compress.CodecPool: Got brand-new compressor
14/04/08 15:50:35 WARN mapred.JobClient: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same.
14/04/08 15:50:35 WARN mapred.JobClient: No job jar file set. User classes may not be

20
found. See JobConf(Class) or JobConf#setJar(String).
14/04/08 15:50:35 INFO input.FileInputFormat: Total input paths to process : 1
14/04/08 15:50:35 INFO mapred.JobClient: Running job: job_local1343624176_0001
14/04/08 15:50:35 INFO mapred.LocalJobRunner: Waiting for map tasks
14/04/08 15:50:35 INFO mapred.LocalJobRunner: Starting task:
attempt_local1343624176_0001_m_000000_0
14/04/08 15:50:36 INFO util.ProcessTree: setsid exited with exit code 0
14/04/08 15:50:36 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@24bec229
14/04/08 15:50:36 INFO mapred.MapTask: Processing split:
file:/home/tid/eclipse/workspace/MRClustering/files/clustering/import/data:0+558
14/04/08 15:50:36 INFO mapred.MapTask: io.sort.mb = 100
14/04/08 15:50:36 INFO mapred.MapTask: data buffer = 79691776/99614720
14/04/08 15:50:36 INFO mapred.MapTask: record buffer = 262144/327680
14/04/08 15:50:36 INFO compress.CodecPool: Got brand-new decompressor
14/04/08 15:50:36 INFO compress.CodecPool: Got brand-new decompressor
14/04/08 15:50:36 INFO mapred.MapTask: Starting flush of map output
14/04/08 15:50:36 INFO mapred.MapTask: Finished spill 0
14/04/08 15:50:36 INFO mapred.Task: Task:attempt_local1343624176_0001_m_000000_0 is done.
And is in the process of commiting
14/04/08 15:50:36 INFO mapred.LocalJobRunner:
14/04/08 15:50:36 INFO mapred.Task: Task 'attempt_local1343624176_0001_m_000000_0' done.
14/04/08 15:50:36 INFO mapred.LocalJobRunner: Finishing task:
attempt_local1343624176_0001_m_000000_0
14/04/08 15:50:36 INFO mapred.LocalJobRunner: Map task executor complete.
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7b4b3d0e
14/04/08 15:50:36 INFO mapred.Merger: Merging 1 sorted segments
14/04/08 15:50:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of
total size: 380 bytes
14/04/08 15:50:36 INFO mapred.Task: Task:attempt_local1343624176_0001_r_000000_0 is done.
14/04/08 15:50:36 INFO mapred.Task: Task attempt_local1343624176_0001_r_000000_0 is allowed
to commit now
14/04/08 15:50:36 INFO output.FileOutputCommitter: Saved output of task
'attempt_local1343624176_0001_r_000000_0' to files/clustering/depth_1
14/04/08 15:50:36 INFO mapred.LocalJobRunner: reduce > reduce
14/04/08 15:50:36 INFO mapred.Task: Task 'attempt_local1343624176_0001_r_000000_0' done.
14/04/08 15:50:36 INFO mapred.JobClient: map 100% reduce 100%
14/04/08 15:50:36 INFO mapred.JobClient: Job complete: job_local1343624176_0001
14/04/08 15:50:36 INFO mapred.JobClient: Counters: 21
14/04/08 15:50:36 INFO mapred.JobClient: File Output Format Counters
14/04/08 15:50:36 INFO mapred.JobClient: Bytes Written=537
14/04/08 15:50:36 INFO mapred.JobClient: File Input Format Counters
14/04/08 15:50:36 INFO mapred.JobClient: Bytes Read=574
14/04/08 15:50:36 INFO mapred.JobClient: FileSystemCounters
14/04/08 15:50:36 INFO mapred.JobClient: FILE_BYTES_READ=2380
14/04/08 15:50:36 INFO mapred.JobClient: FILE_BYTES_WRITTEN=106876
14/04/08 15:50:36 INFO mapred.JobClient: com.clustering.mapreduce.KMeansReducer$Counter
14/04/08 15:50:36 INFO mapred.JobClient: CONVERGED=2
14/04/08 15:50:36 INFO mapred.JobClient: Map-Reduce Framework
14/04/08 15:50:36 INFO mapred.JobClient: Reduce input groups=2
14/04/08 15:50:36 INFO mapred.JobClient: Map output materialized bytes=384
14/04/08 15:50:36 INFO mapred.JobClient: Combine output records=0
14/04/08 15:50:36 INFO mapred.JobClient: Map input records=9
14/04/08 15:50:36 INFO mapred.JobClient: Reduce shuffle bytes=0
14/04/08 15:50:36 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
14/04/08 15:50:36 INFO mapred.JobClient: Reduce output records=9
14/04/08 15:50:36 INFO mapred.JobClient: Spilled Records=18
14/04/08 15:50:36 INFO mapred.JobClient: Map output bytes=360
14/04/08 15:50:36 INFO mapred.JobClient: Total committed heap usage (bytes)=355991552
14/04/08 15:50:36 INFO mapred.JobClient: CPU time spent (ms)=0
14/04/08 15:50:36 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0

21
14/04/08 15:50:36 INFO mapred.JobClient: SPLIT_RAW_BYTES=139
14/04/08 15:50:36 INFO mapred.JobClient: Map output records=9
14/04/08 15:50:36 INFO mapred.JobClient: Combine input records=0
14/04/08 15:50:36 INFO mapred.JobClient: Reduce input records=9
attempt_local1426850290_0002_m_000000_0
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@26adcd34
file:/home/tid/eclipse/workspace/MRClustering/files/clustering/depth_1/part-r-00000:0+521
attempt_local1426850290_0002_m_000000_0
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3740f768
to commit now
14/04/08 15:50:38 INFO mapred.JobClient: com.clustering.mapreduce.KMeansReducer$Counter
14/04/08 15:50:38 INFO mapred.JobClient: CONVERGED=2

22
attempt_local466621791_0003_m_000000_0
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@373fdd1a
file:/home/tid/eclipse/workspace/MRClustering/files/clustering/depth_2/part-r-00000:0+521
attempt_local466621791_0003_m_000000_0
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@78d3cdb9
to commit now

23
14/04/08 15:50:39 INFO mapreduce.KMeansClusteringJob: FOUND
file:/home/tid/eclipse/workspace/MRClustering/files/clustering/depth_3/part-r-00000
[vector=[1.4, -2.6]]] / Vector [vector=[-1.0, -23.0]]
References
 Hadoop Installation on single node cluster - http://www.michael-noll.com/tutorials/running-
hadoop-on-ubuntu-linux-single-node-cluster/
 KMeansClustering with MapReduce - http://codingwiththomas.blogspot.kr/2011/05/k-
means-clustering-with-mapreduce.html

Hadoop installation and Running KMeans Clustering with MapReduce Program on Hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Hadoop installation and Running KMeans Clustering with MapReduce Program on Hadoop

Similar to Hadoop installation and Running KMeans Clustering with MapReduce Program on Hadoop (20)

Recently uploaded

Recently uploaded (20)

Hadoop installation and Running KMeans Clustering with MapReduce Program on Hadoop