Hands on
Upcoming SlideShare
Loading in...5
×
 

Hands on

on

  • 1,348 views

Hands on

Hands on

Statistics

Views

Total Views
1,348
Views on SlideShare
827
Embed Views
521

Actions

Likes
2
Downloads
51
Comments
0

2 Embeds 521

http://www.nasscom.in 293
http://www.nasscom.org 228

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Hands on Hands on Presentation Transcript

  • Hadoop Hands on Session 1
  • Agenda  Installation  Start Services  Job Tracker and Name Node UI Overview  Interaction with HDFS  Mapper  Reducer  Hadoop Job  Run Map Reduce on HadoopImpetus Proprietary 2
  • Installation  Hadoop installation is as easy as unzipping the binary distribution.  Setup ssh login to localhost- rm -rf .ssh/* ssh-keygen -t rsa > /dev/null ssh-copy-id -i localhost ssh localhostImpetus Confidential 3
  • Start Services  First format the namenode hadoop namenode -format  Now start all 4 services hadoop namenode hadoop datanode hadoop jobtracker hadoop tasktrackerImpetus Confidential 4
  • UI Overview  Name node http://localhost:50070/dfshealth.jsp  Data Node http://localhost:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=% 2F  Job Tracker http://localhost:50030/jobtracker.jsp  Task Tracker http://localhost:50060/tasktracker.jspImpetus Confidential 5
  • Interaction with HDFSImpetus Confidential 6
  • Mapper public static class DemoMapper extends Mapper<LongWritable, Text, Text, IntWritable> { // 11~American President, The (1995)~Comedy|Drama|Romance public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] records = value.toString().split("~"); String[] genres = records[2].split("|"); for (String genre : genres) { context.write(new Text(genre), new IntWritable(1)); } } }Impetus Confidential 7
  • Reducer public static class DemoReducer extends Reducer<Text, IntWritable, Text, IntWritable> { @Override protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int count = 0; for (IntWritable value : values) { count += value.get(); } context.write(key, new IntWritable(count)); } }Impetus Confidential 8
  • Hadoop Job public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf); job.setJarByClass(DemoMapper.class); job.setMapperClass(DemoMapper.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setReducerClass(DemoReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setNumReduceTasks(1); // 1 is default job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); TextInputFormat.addInputPath(job, new Path("/mldata/movies.dat")); TextOutputFormat.setOutputPath(job, new Path("/mldata/moviesout/")); boolean result = job.waitForCompletion(true); System.out.println("Job status: " + result); }Impetus Confidential 9
  • Thank You Q&AImpetus Proprietary 10