Introducción a hadoop

Introducción a Hadoop El bazuca de los datos Iván de Prado Alonso // @ivanprado // @datasalt

Datasalt Foco en el Big Data – Contribución al Open Source – Consultoría & Desarrollo – Formación 2 / 34

BIG“MAC” DATA 3 / 34

Fisonomía de un proyecto Big Data Adquisición Procesamiento Servicio 4 / 34

Tipos de sistemas Big Data● Offline – La latencia no es un problema● Online – La inmediatez de los datos es importante● Mixto – Lo más común Offline Online MapReduce Bases de datos NoSQL Hadoop Motores de búsqueda Distributed RDBMS 5 / 34

“Swiss army knife of the 21st century” Media Guardian Innovation Awardshttp://www.guardian.co.uk/technology/2011/mar/25/media-guardian-innovation-awards-apache-hadoop 6 / 34

Historia● 2004-2006 – Google publica los papers de GFS y MapReduce – Doug Cutting implementa una versión Open Source en Nutch● 2006-2008 – Hadoop se separa de Nutch – Se alcanza la escala web en 2008● 2008-Hasta ahora – Hadoop se populariza y se comienza a explotar comercialmente. Fuente: Hadoop: a brief history. Doug Cutting 7 / 34

Hadoop “The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model” De la página de Hadoop 8 / 34

Sistema de Ficheros Distribuido● Sistema de ficheros distribuido (HDFS) – Bloques grandes: 64 Mb ● Almacenados en el sistema de ficheros del SO – Tolerante a Fallos (replicación) – Formatos habituales: ● Ficheros en formato texto (CSV) ● SequenceFiles – Ristras de pares [clave, valor] 9 / 34

MapReduce● Dos funciones (Map y Reduce) – Map(k, v) : [z,w]* – Reduce(k, v*) : [z, w]*● Ejemplo: contar palabras – Map([documento, null]) -> [palabra, 1]* – Reduce(palabra, 1*) -> [palabra, total]● MapReduce y SQL – SELECT palabra, count(*) GROUP BY palabra● Ejecución distribuida en un cluster con escalabilidad horizontal 10 / 34

El típico Word Count Esto es una linea Esto también Map Reduce reduce(es, {1}) = map(“Esto es una linea”) = es, 1 esto, 1 reduce(esto, {1, 1}) = es, 1 esto, 2 una, 1 reduce(linea, {1}) = linea, 1 linea, 1 map(“Esto también”) = reduce(también, {1}) = esto, 1 también, 1 también, 1 reduce(una, {1}) = una, 1 es, 1 esto, 2 Resultado: linea, 1 también, 1 una, 1 11 / 34

Word Count en Hadoop public class WordCountHadoop extends Configured implements Tool { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable val : values) { sum += val.get(); } ¡Mejor vamos por partes! result.set(sum); context.write(key, result); } } @Override public int run(String[] args) throws Exception { if(args.length != 2) { System.err.println("Usage: wordcount-hadoop <in> <out>"); System.exit(2); } Path output = new Path(args[1]); HadoopUtils.deleteIfExists(FileSystem.get(output.toUri(), conf), output); Job job = new Job(getConf(), "word count hadoop"); job.setJarByClass(WordCountHadoop.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); return 0; } public static void main(String[] args) throws Exception { ToolRunner.run(new SortJobHadoop(), args); } } 12 / 34

Word Count en Hadoop - Mapper public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } 13 / 34

Word Count en Hadoop - Reducer public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } 14 / 34

Word Count en Hadoop – Configuración yejecución if(args.length != 2) { System.err.println("Usage: wordcount-hadoop <in> <out>"); System.exit(2); } Path output = new Path(args[1]); HadoopUtils.deleteIfExists(FileSystem.get(output.toUri(), conf), output); Job job = new Job(getConf(), "word count hadoop"); job.setJarByClass(WordCountHadoop.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); 15 / 34

Ejecución de un Job MapReduce Bloques del fichero de entrada Nodo 1 Nodo 2 Mappers Datos Intermedios Nodo 1 Nodo 2 Reducers Resultado 16 / 34

Serialización ● Writables • Serialización nativa de Hadoop • De muy bajo nivel • Tipos básicos: IntWritable, Text, etc. ● Otras • Thrift, Avro, Protostuff • Compatibilidad hacia atrás. 17 / 34

La curva deaprendizajede Hadoop es alta 18 / 34

Tuple MapReduce● Un MapReduce más simple – Tuplas en lugar de key/value – A nivel de job se define ● Los campos por los que agrupar ● Los campos por los que ordenar – Tuple MapReduce-join 19 / 34

Pangool● Implementación de TupleMap reduce – Desarrollado por Datasalt – OpenSource – Eficiencia equiparable a Hadoop● Objetivo: reemplazar la API de Hadoop● Si quieres aprender Hadoop, empieza por Pangool 20 / 34

Eficiencia de Pangool● Equiparable a Hadoop Ver http://pangool.net/benchmark.html 21 / 34

Pangool – URL resolution● Ejemplo de Join – Muy difícil en Hadoop. Fácil en Pangool.● Problema: – Existen muchos acortadores de URLs y redirecciones – Para analizar datos, suele ser útil reemplazar las URLs por su URL canónica – Supongamos que tenemos ambos datasets ● Un mapa con entradas URL → URL canónica ● Un dataset con URLs (que queremos resolver) y otros campos. – El siguiente job Pangool soluciona el problema de manera escalable. 22 / 34

URL Resolution – Definiendo Schemas static Schema getURLRegisterSchema() { List<Field> urlRegisterFields = new ArrayList<Field>(); urlRegisterFields.add(Field.create("url",Type.STRING)); urlRegisterFields.add(Field.create("timestamp",Type.LONG)); urlRegisterFields.add(Field.create("ip",Type.STRING)); return new Schema("urlRegister", urlRegisterFields); } static Schema getURLMapSchema() { List<Field> urlMapFields = new ArrayList<Field>(); urlMapFields.add(Field.create("url",Type.STRING)); urlMapFields.add(Field.create("canonicalUrl",Type.STRING)); return new Schema("urlMap", urlMapFields); } 23 / 34

URL Resolution – Cargando el fichero aresolver public static class UrlProcessor extends TupleMapper<LongWritable, Text> { private Tuple tuple = new Tuple(getURLRegisterSchema()); @Override public void map(LongWritable key, Text value, TupleMRContext context, Collector collector) throws IOException, InterruptedException { String[] fields = value.toString().split("t"); tuple.set("url", fields[0]); tuple.set("timestamp", Long.parseLong(fields[1])); tuple.set("ip", fields[2]); collector.write(tuple); } } 24 / 34

URL Resolution – Cargando el mapa de URLs public static class UrlMapProcessor extends TupleMapper<LongWritable, Text> { private Tuple tuple = new Tuple(getURLMapSchema()); @Override public void map(LongWritable key, Text value, TupleMRContext context, Collector collector) throws IOException, InterruptedException { String[] fields = value.toString().split("t"); tuple.set("url", fields[0]); tuple.set("canonicalUrl", fields[1]); collector.write(tuple); } } 25 / 34

URL Resolution – Resolución en el reducer public static class Handler extends TupleReducer<Text, NullWritable> { private Text result; @Override public void reduce(ITuple group, Iterable<ITuple> tuples, TupleMRContext context, Collector collector) throws IOException, InterruptedException, TupleMRException { if (result == null) { result = new Text(); } String cannonicalUrl = null; for(ITuple tuple : tuples) { if("urlMap".equals(tuple.getSchema().getName())) { cannonicalUrl = tuple.get("canonicalUrl").toString(); } else { result.set(cannonicalUrl + "t" + tuple.get("timestamp") + "t" + tuple.get("ip")); collector.write(result, NullWritable.get()); } } } } 26 / 34

URL Resolution – Configurando y Lanzandoel job String input1 = args[0]; String input2 = args[1]; String output = args[2]; deleteOutput(output); TupleMRBuilder mr = new TupleMRBuilder(conf,"Pangool Url Resolution"); mr.addIntermediateSchema(getURLMapSchema()); mr.addIntermediateSchema(getURLRegisterSchema()); mr.setGroupByFields("url"); mr.setOrderBy( new OrderBy().add("url", Order.ASC).addSchemaOrder(Order.ASC)); mr.setTupleReducer(new Handler()); mr.setOutput(new Path(output), new HadoopOutputFormat(TextOutputFormat.class), Text.class, NullWritable.class); mr.addInput(new Path(input1), new HadoopInputFormat(TextInputFormat.class), new UrlMapProcessor()); mr.addInput(new Path(input2), new HadoopInputFormat(TextInputFormat.class), new UrlProcessor()); mr.createJob().waitForCompletion(true); 27 / 34

Hadoop vs Pangool URL Resolution URL Resolution en Hadoop en Pangool /** return * Copyright [2012] [Datasalt Systems S.L.] groupUrl.toString() + "," + ip.toString() + "," + timestamp; * } * Licensed under the Apache License, Version 2.0 (the "License"); } * you may not use this file except in compliance with the License. } * You may obtain a copy of the License at * public static class KeyPartitioner implements Partitioner<UrlRegJoinUrlMap, NullWritable> { * http://www.apache.org/licenses/LICENSE-2.0 @Override * public int getPartition(UrlRegJoinUrlMap key, NullWritable value, int * Unless required by applicable law or agreed to in writing, software } numPartitions) { * distributed under the License is distributed on an "AS IS" BASIS, return Math.abs(key.groupUrl.hashCode()) % * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @Override numPartitions; * See the License for the specific language governing permissions and public void readFields(DataInput in) throws IOException { } * limitations under the License. groupUrl.readFields(in); */ sourceId = (int) in.readByte(); @Override package com.datasalt.pangool.examples.urlresolution; if(sourceId == SOURCE_URL_MAP) { public void configure(JobConf arg0) { } import java.io.IOException; cannonicalUrl.readFields(in); } import java.util.ArrayList; } else { import java.util.List; public static class GroupingComparator implements RawComparator<UrlRegJoinUrlMap> { ip.readFields(in); import org.apache.hadoop.fs.Path; timestamp = @Override import org.apache.hadoop.io.LongWritable; in.readLong(); public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) import org.apache.hadoop.io.NullWritable; } { import org.apache.hadoop.io.Text; } try { import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; int offset1 = import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; @Override s1; import org.apache.hadoop.util.ToolRunner; public void write(DataOutput out) throws IOException { int offset2 = groupUrl.write(out); s2; import com.datasalt.pangool.examples.BaseExampleJob; out.writeByte((byte) sourceId); int strSize1 = import com.datasalt.pangool.io.ITuple; if(sourceId == SOURCE_URL_MAP) { WritableComparator.readVInt(b1, offset1);/** import com.datasalt.pangool.io.Schema; int strSize2 = * Copyright [2012] [Datasalt Systems S.L.] import com.datasalt.pangool.io.Schema.Field; cannonicalUrl.write(out); WritableComparator.readVInt(b2, offset2); * import com.datasalt.pangool.io.Schema.Field.Type; } else { offset1 += * Licensed under the Apache License, Version 2.0 (the "License"); import com.datasalt.pangool.io.Tuple; ip.write(out); WritableUtils.decodeVIntSize(b1[offset1]); * you may not use this file except in compliance with the License. import com.datasalt.pangool.tuplemr.TupleMRBuilder; offset2 += * You may obtain a copy of the License at import com.datasalt.pangool.tuplemr.TupleMRException; out.writeLong(timestamp); WritableUtils.decodeVIntSize(b2[offset2]); * import com.datasalt.pangool.tuplemr.TupleMapper; } return * http://www.apache.org/licenses/LICENSE-2.0 import com.datasalt.pangool.tuplemr.TupleReducer; } WritableComparator.compareBytes(b1, offset1, strSize1, b2, offset2, strSize2); * import com.datasalt.pangool.tuplemr.mapred.lib.input.HadoopInputFormat; } catch(IOException e) { * Unless required by applicable law or agreed to in writing, software import com.datasalt.pangool.tuplemr.mapred.lib.output.HadoopOutputFormat; @Override throw new * distributed under the License is distributed on an "AS IS" BASIS, public int hashCode() { RuntimeException(e); * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. /** if(sourceId == SOURCE_URL_MAP) { } * See the License for the specific language governing permissions and * This example shows how to perform reduce-side joins using Pangool. We have one file with URL Registers: ["url", "timestamp", return } * limitations under the License. "ip"] and another file with ((groupUrl.hashCode() * 31) + cannonicalUrl.hashCode()) * 31; */ * canonical URL mapping: ["url", "canonicalUrl"]. We want to obtain the URL Registers file with the url substituted with the } else { @Overridepackage com.datasalt.pangool.benchmark.urlresolution; * canonical one according to the mapping file: ["canonicalUrl", "timestamp", "ip"]. return public int compare(UrlRegJoinUrlMap o1, UrlRegJoinUrlMap o2) { */ ((((groupUrl.hashCode() * 31) + ip.hashCode()) * 31) + (int) timestamp) * 31; return o1.groupUrl.compareTo(o2.groupUrl);import java.io.DataInput; public class UrlResolution extends BaseExampleJob { } }import java.io.DataOutput; } }import java.io.IOException; static Schema getURLRegisterSchema() { List<Field> urlRegisterFields = new ArrayList<Field>(); @Override /**import org.apache.hadoop.conf.Configuration; urlRegisterFields.add(Field.create("url",Type.STRING)); public boolean equals(Object right) { *import org.apache.hadoop.fs.FileSystem; urlRegisterFields.add(Field.create("timestamp",Type.LONG)); if(right instanceof UrlRegJoinUrlMap) { */import org.apache.hadoop.fs.Path; urlRegisterFields.add(Field.create("ip",Type.STRING)); public static class UrlMapClass implements Mapper<LongWritable, Text, UrlRegJoinUrlMap, NullWritable>import org.apache.hadoop.io.LongWritable; return new Schema("urlRegister", urlRegisterFields); UrlRegJoinUrlMap r = (UrlRegJoinUrlMap) right; {import org.apache.hadoop.io.NullWritable; if(sourceId ==import org.apache.hadoop.io.RawComparator; } r.sourceId) { private final UrlRegJoinUrlMap key = new UrlRegJoinUrlMap();import org.apache.hadoop.io.Text;import org.apache.hadoop.io.WritableComparable; static Schema getURLMapSchema() { if(groupUrl.equals(r.groupUrl)) { @Overrideimport org.apache.hadoop.io.WritableComparator; List<Field> urlMapFields = new ArrayList<Field>(); public void configure(JobConf arg0) {import org.apache.hadoop.io.WritableUtils; urlMapFields.add(Field.create("url",Type.STRING)); if(sourceId == SOURCE_URL_MAP) { }import org.apache.hadoop.mapred.FileOutputFormat; urlMapFields.add(Field.create("canonicalUrl",Type.STRING));import org.apache.hadoop.mapred.JobConf; return new Schema("urlMap", urlMapFields); return @Overrideimport org.apache.hadoop.mapred.Mapper; } cannonicalUrl.equals(r.cannonicalUrl); public void close() throws IOException {import org.apache.hadoop.mapred.OutputCollector; }import org.apache.hadoop.mapred.Partitioner; @SuppressWarnings("serial") } else {import org.apache.hadoop.mapred.Reporter; public static class UrlProcessor extends TupleMapper<LongWritable, Text> { @Overrideimport org.apache.hadoop.mapred.TextInputFormat; return ip.equals(r.ip) && timestamp == public void map(LongWritable ignore, Text inValue,import org.apache.hadoop.mapred.lib.MultipleInputs; private Tuple tuple = new Tuple(getURLRegisterSchema()); r.timestamp; OutputCollector<UrlRegJoinUrlMap, NullWritable> context,import org.apache.hadoop.mapreduce.Job; Reporter arg3) throws IOException {import org.apache.hadoop.mapreduce.Reducer; @Override } String[] fields =import org.apache.hadoop.util.GenericOptionsParser; public void map(LongWritable key, Text value, TupleMRContext context, inValue.toString().split("t"); Collector collector) } key.setUrlMap(fields[0], fields[1]);/** throws IOException, InterruptedException { } context.collect(key, NullWritable.get()); * Code for solving the URL Resolution CoGroup Problem in Hadoop Java Map/Red API. } } * <p> String[] fields = return false; } * The URL Resolution CoGroup Problem is: We have one file with URL Registers: {url timestamp ip} and another file with value.toString().split("t"); } * canonical URL mapping: {url cannonicalUrl}. We want to obtain the URL Registers file with the url substituted with tuple.set("url", fields[0]); /** * the canonical one according to the mapping file: {canonicalUrl timestamp ip}. tuple.set("timestamp", public static class Comparator extends WritableComparator { * */ Long.parseLong(fields[1])); public Comparator() { */public class HadoopUrlResolution { tuple.set("ip", fields[2]); public static class UrlRegisterMapClass implements Mapper<LongWritable, Text, UrlRegJoinUrlMap, collector.write(tuple); super(UrlRegJoinUrlMap.class); NullWritable> { public final static int SOURCE_URL_MAP = 0; } } public final static int SOURCE_URL_REGISTER = 1; } private final UrlRegJoinUrlMap key = new UrlRegJoinUrlMap(); public int compare(byte[] b1, int s1, int public static class UrlRegJoinUrlMap implements WritableComparable<UrlRegJoinUrlMap> { @SuppressWarnings("serial") l1, byte[] b2, int s2, int l2) { @Override public static class UrlMapProcessor extends TupleMapper<LongWritable, Text> { try { public void configure(JobConf arg0) { // --- Common fields --- // } private Text groupUrl = new Text(); private Tuple tuple = new Tuple(getURLMapSchema()); int offset1 = s1; private int sourceId; @Override @Override int offset2 = s2; public void close() throws IOException { // --- Url register --- // public void map(LongWritable key, Text value, TupleMRContext context, } private Text ip = new Text(); Collector collector) // Group url private long timestamp; throws IOException, InterruptedException { @Override int strSize1 = WritableComparator.readVInt(b1, offset1); public void map(LongWritable ignore, Text inValue, // --- Url map --- // String[] fields = OutputCollector<UrlRegJoinUrlMap, NullWritable> context, private Text cannonicalUrl = new Text(); value.toString().split("t"); int strSize2 = WritableComparator.readVInt(b2, offset2); Reporter arg3) throws IOException { tuple.set("url", fields[0]); public void setUrlRegister(String groupUrl, String ip, long timestamp) { tuple.set("canonicalUrl", fields[1]); offset1 += WritableUtils.decodeVIntSize(b1[offset1]); String[] fields = this.groupUrl.set(groupUrl); collector.write(tuple); inValue.toString().split("t"); sourceId = SOURCE_URL_REGISTER; } offset2 += WritableUtils.decodeVIntSize(b2[offset2]); key.setUrlRegister(fields[0], fields[2], this.ip.set(ip); } Long.parseLong(fields[1])); this.timestamp = timestamp; int cmp = WritableComparator.compareBytes(b1, offset1, strSize1, b2, offset2, strSize2); context.collect(key, NullWritable.get()); } @SuppressWarnings("serial") } public static class Handler extends TupleReducer<Text, NullWritable> { if(cmp != 0) { } public void setUrlMap(String groupUrl, String cannonicalUrl) { this.groupUrl.set(groupUrl); private Text result; return cmp; public static class Reduce extends Reducer<UrlRegJoinUrlMap, NullWritable, Text, NullWritable> { sourceId = SOURCE_URL_MAP; this.cannonicalUrl.set(cannonicalUrl); @Override } Text result = new Text(); public void reduce(ITuple group, Iterable<ITuple> tuples, TupleMRContext context, Collector collector) offset1 += strSize1; protected void reduce(UrlRegJoinUrlMap key, Iterable<NullWritable> throws IOException, InterruptedException, TupleMRException { values, Context ctx) throws IOException, if (result == null) { offset2 += strSize2; InterruptedException { result = new String cannonicalUrl = null; Text(); // Source id for(@SuppressWarnings("unused") } NullWritable value : values) { String cannonicalUrl = null; int sourceId1 = (int) b1[offset1]; for(ITuple tuple : tuples) { if(key.sourceId == SOURCE_URL_MAP) { int sourceId2 = (int) b2[offset2]; if("urlMap".equals(tuple.getSchema().getName())) { cannonicalUrl = key.cannonicalUrl.toString(); if(sourceId1 != sourceId2) { } else { cannonicalUrl = tuple.get("canonicalUrl").toString(); } else { return sourceId1 > sourceId2 ? 1 : -1; result.set(cannonicalUrl + "t" + key.timestamp + "t" + key.ip); result.set(cannonicalUrl + "t" + tuple.get("timestamp") + "t" + tuple.get("ip")); } ctx.write(result, NullWritable.get()); } collector.write(result, NullWritable.get()); return 0; } } } }; } catch(IOException e) { } } } throw new RuntimeException(e); public final static void main(String[] args) throws IOException, InterruptedException, } ClassNotFoundException { public UrlResolution() { } Configuration conf = new Configuration(); super("UrlResolution: [input_url_mapping] [input_url_regs] [output]"); } String[] otherArgs = new GenericOptionsParser(conf, } args).getRemainingArgs(); static { // register this comparator if(otherArgs.length != 3) { @Override System.err.println("Usage: urlresolution public int run(String[] args) throws Exception { WritableComparator.define(UrlRegJoinUrlMap.class, new Comparator()); <url-map> <url-register> <out>"); if(args.length != 3) { } System.exit(2); failArguments("Wrong number of arguments"); } return -1; @Override JobConf job = new JobConf(conf); } public int compareTo(UrlRegJoinUrlMap o) { FileSystem fS = FileSystem.get(conf); String input1 = args[0]; int cmp = groupUrl.compareTo(o.groupUrl); fS.delete(new Path(otherArgs[2]), true); String input2 = args[1]; if(cmp != 0) { String output = args[2]; return cmp; MultipleInputs.addInputPath(job, new Path(otherArgs[0]), } TextInputFormat.class, UrlMapClass.class); deleteOutput(output); if(sourceId != o.sourceId) { MultipleInputs.addInputPath(job, new Path(otherArgs[1]), return TextInputFormat.class, UrlRegisterMapClass.class); TupleMRBuilder mr = new TupleMRBuilder(conf,"Pangool Url Resolution"); sourceId > o.sourceId ? 1 : -1; mr.addIntermediateSchema(getURLMapSchema()); } job.setJarByClass(HadoopUrlResolution.class); mr.addIntermediateSchema(getURLRegisterSchema()); return 0; mr.setGroupByFields("url"); } job.setPartitionerClass(KeyPartitioner.class); mr.setTupleReducer(new Handler()); job.setOutputValueGroupingComparator(GroupingComparator.class); mr.setOutput(new Path(output), new public String toString() { HadoopOutputFormat(TextOutputFormat.class), Text.class, NullWritable.class); if(sourceId == SOURCE_URL_MAP) { job.setMapOutputKeyClass(UrlRegJoinUrlMap.class); mr.addInput(new Path(input1), new return job.setMapOutputValueClass(NullWritable.class); HadoopInputFormat(TextInputFormat.class), new UrlMapProcessor()); groupUrl.toString() + "," + cannonicalUrl.toString(); mr.addInput(new Path(input2), new } else { job.setOutputKeyClass(Text.class); HadoopInputFormat(TextInputFormat.class), new UrlProcessor()); job.setOutputValueClass(NullWritable.class); mr.createJob().waitForCompletion(true); FileOutputFormat.setOutputPath(job, new Path(otherArgs[2])); return 1; } Job j = new Job(job); j.setReducerClass(Reduce.class); public static void main(String[] args) throws Exception { j.waitForCompletion(true); ToolRunner.run(new UrlResolution(), args); } } } } 28 / 34

Hadoop en el mundo real● Captura de datos – Usando Hadoop – Usando Sqoop para BDs SQL – Con servicios que escriben en el HDFS – Flume – Storm● Aplicación de patrones básicos – Filtrado – Ordenación – Ejecución distribuida – Joins ● Reduce-side, Map-side, en memoria – Producto cruzado – Reconciliado 29 / 34

Hadoop en el mundo real (II)● Encadenamiento de Jobs – Se encadena la salida de un job con la entrada de la de otro, formando un flujo. – Sistemas para flow: ● Oozie● Actualización de bases de datos – Generación/actualización ● de índices Solr ● de BD clave/valor ● de BD SQL 30 / 34

Aclaremos dudas habituales ● Hadoop no es una DB ● Hadoop “aparentemente” sólo procesa datos ● Hadoop no permite “lookups” Hadoop supone un cambio de paradigma que cuesta asimilar 31 / 34

Cambio en la Filosofía ● Reprocesarlo todo siempre. ¡TODO! ● ¿Por qué? • Más tolerante a fallos • Más flexible • Más eficiente. Ej:  Con un HD de 7200 RPM – Random IOPS – 100 – Lectura secuencial – 40 MB/s – Tamaño de registro: 5 Kb  … con que un 1,25% de los registros cambien, es más rápido reescribirlo todo que hacer accesos aleatorios de actualización. – 100 MB, 20.000 registros » Lectura secuencial: 2,5 sg » Lectura aleatoria: 200 sg 32 / 34

Herramientas de alto nivel● Usar Hadoop o Pangool puede no ser apropiado para todos los proyectos● Hay herramientas de mayor nivel que pueden ser muy útiles y más fáciles: – Hive: Análisis de datos con SQL – Pig: Análisis de datos con un lenguaje parecido a SQL – Cascading: framework de programación en Hadoop, que incluye tuplas y gestión automática de flows. – Hadoop streaming: procesamiento de datos sobre Hadoop con cualquier comando/lenguaje de scripting. 33 / 34

GraciasIván de Prado Alonsoivan@datasalt.com@ivanprado

http://www.datasalt.es	555
https://twitter.com	9
http://ristrettonews.com	4
http://webcache.googleusercontent.com	1

Introducción a hadoop

by datasalt on Mar 22, 2012

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 569

Statistics

Introducción a hadoop Presentation Transcript