Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A whirlwind tour of
hadoop
By Eric Marshall
For LOPSA-NJ
an all too brief introduction to the world of big data
Eric Marshall
I work for Airisdata; we’re hiring!
Smallest computer I lost sleep over:
Sinclair-Timex Z81 – 1KB of memory
...
Vocabulary disclaimer
 Just like your favorite swear word, which can act like
many parts of speech and refer to many a th...
My plan of attack
 An intro: the good, the bad and the ugly at 50,000 ft.
 2¢ tour of hadoop’s processing - map reduce
...
Why did this happen?
 Old school –> scale up == larger costlier
monolithic system (or a small cluster there of) i.e.
vert...
The Good
Simple development (when
compared to Message
Passing Interface
programming )
Scale – no shared state,
programmer ...
The Bad
 Not a silver bullet :(
 MapReduce is batch data processing
the time scale is minutes to hours
 MapReduce is ov...
The Ugly?
Welcome to the rest of our talk!
First stop, Map Reduce
Hadoop’s MapReduce
Lisp’s map and reduce
plus the associative property
applied to clusters.
Map()
 Imagine a number of servers with lists of first names –
What is the most popular name?
 Box 1-isabella William av...
Map()
 So we would have:
 Box 1-Isabella William Ava Mia Emma Alexander
 Box 2-Noah Noah Isabella Isabella Emma Emma
 ...
Map()
 So we would have:
 Box 1-(Isabella,1) (William,1) (Ava,1) (Mia,1) (Emma,1)
(Alexander,1)
 Box 2-(Noah,1) (Noah,1...
Shuffle/Sort
 So we would have:
 Box 1-(Alexander,1) (Ava,1) (Emma,1) (Emma,1)
(Emma,1) (Emma,1) (Emma,1)
 Box 2-(Isabe...
Reduce
 So we would have:
 Box 1-(Alexander,1) (Ava,1) (Emma,1) (Emma,1) (Emma,1)
(Emma,1) (Emma,1)
 Box 2-(Isabella,1)...
(This architecture is covered in greater detail in Chapter 4 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
Map/reduce failures
Check the job if:
The job throws an uncaught exception.
The job exits with a nonzero exit code.
The jo...
Instant MR test
 Um, is the system working?
 yarn jar /usr/hdp/2.3.2.0-2950/hadoop-
mapreduce/hadoop-mapreduce-examples....
(HDFS is covered in greater detail in Chapter 3 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
There is a HDFS CLI
 You already know some of the commands:
 hdfs dfs –ls /
 hdfs dfs –du /
 hdfs dfs –rm /
 hdfs dfs...
HDFS failures
 Jobs fail: due to missing
blocks
 Jobs fail: due to moving data
due to down datanodes or
huge ingest
 Wi...
HDFS failures
 Jobs fail: due to missing
blocks
 Jobs fail: due to moving data
due to down datanodes or
huge ingest
 Wi...
The rest of the garden
Distributed Filesystems
- Apache HDFS
outliers:
- Tachyon
- Apache GridGain
- Ignite
- XtreemFS
- C...
10 in 10 minutes!
 Easier Programming: Pig, Spark
 SQL-like tools: Hive, Impala, Hbase
 Data pipefitting: Sqoop, Flume,...
Easier Programming
Pig
What is it: a high level programming language for data
manipulation that abstracts M/R from Yahoo
Why: a few lines of ...
Spark
What is it: computing framework from ampLab, UC Berkeley
Why: high level abstractions and better use of memory
Neat ...
SQL-ish
Hive/HQL
What is it: a data infrastructure and query language from
Facebook
Why: batched SQL queries against HDFS
Neat tri...
Impala
What is it: SQL query engine from Cloudera
Why: fast adhoc queries on subsets of data stored in hadoop
Example:
[im...
HBase
What is it: a non-relational database from Powerset
Why: fast access to large sparse data sets
Example:
hbase(main):...
Data pipefitting
Sqoop
What is it: glue tool for moving data between relational
databases and hadoop
Why: make the cumbersome easier
Exampl...
Flume
What is it: a service for collecting and aggregating logs
Why: because log ingestion is tougher than it seems
Exampl...
Kafka
What is it: message broker from LinkedIn
Why: fast handling of data feeds
Neat trick: no need to worry about missing...
Book keeping
Oozie
What is it: workflow scheduler from Yahoo Banglalore
Why: because cron isn’t perfect
Example:
oozie job -oozie http:...
Zookeeper
What is it: a coordination service from Yahoo
Why: sync info for distributed systems (similar idea behind
DNS or...
Distributed Programming
- Apache MapReduce also MRv2/YARN
- Apache Pig
outliers:
- JAQL
- Apache Spark
- Apache Flink (for...
Distributed Filesystems
- Apache HDFS
outliers:
- Tachyon
- Apache GridGain
- Ignite
- XtreemFS
- Ceph Filesystem
- Red Ha...
NoSQL Databases
:Columnal Data Model
- Apache HBase
outliers:
- Apache Accumulo
- Hypertable
- HP Vertica
:Key Value Data
...
Data Ingestion
:SQL on Hadoop
- Apache Hive
- Apache HCatalog
outliers:
- Cloudera Kudu
- Trafodion
- Apache Drill
- Cloud...
Etc.
Service Programming and
Frameworks
- Apache Zookeeper
- Apache Avro
- Apache Parquet
outliers:
- Apache Thrift
- Apac...
And now a bit of common sense for
sys-admin-ing Hadoop clusters
Avoid
 The usual -
 Don’t let hdfs fill up
 Don’t use all the memory
 Don’t use up all the cpus
 Don’t drop the netwo...
Um, backups?
 Usual suspects plus
 Namenode’s meta data!! (fsimage)
 Hdfs? Well, it would nice but unlikely (if so dist...
Hadoop Management
 Apache Ambari
 Cloudera Manager
Monitoring
 The usual suspects plus…
 JMX support
 Jvm via jstat, jmap etc.
 hdfs
 Mapred
 conf/hadoop-metrics.prope...
User management
 Hdfs quotas
 Access controls
 Internal and
 external
 MR schedulers
 Fifo, Fair, Capacity
 Kerbero...
Configuration
 /etc/hadoop/conf
 Lots of knobs!
 !Ojo! –
 Lots of overrides
 Get the basic system solid before securi...
Want more?
(Disclaimer: I receive nothing from O’Reilly. Not even a Christmas card…)
Fin
 Thanks for listening
 Slides:
http://www.slideshare.net/ericwilliammarshall/hadoop-
for-sysadmins
 Any questions?
What’s in a name?
 Doug Cutting seems to have been inspired by his
family. Lucene is his wife’s middle name, and her
mate...
What to do?
Combinations of the usual stuff:
 Numerical Summarizations
 Filtering
 Altering Data Organization
 Joining...
federation
(Image from Chapter 2 of Eric Sammer’s Hadoop Operations by O’Reilly)
Hadoop for sysadmins
Hadoop for sysadmins
Upcoming SlideShare
Loading in …5
×

Hadoop for sysadmins

440 views

Published on

A whirlwind tour of hadoop - an all too brief introduction to the world of big data

Published in: Technology
  • Be the first to comment

Hadoop for sysadmins

  1. 1. A whirlwind tour of hadoop By Eric Marshall For LOPSA-NJ an all too brief introduction to the world of big data
  2. 2. Eric Marshall I work for Airisdata; we’re hiring! Smallest computer I lost sleep over: Sinclair-Timex Z81 – 1KB of memory Largest computer I lost sleep over: SGI Altix 4700 – 1 TB of memory
  3. 3. Vocabulary disclaimer  Just like your favorite swear word, which can act like many parts of speech and refer to many a thing; hadoop vocabulary has the same problem  Casually, people refer to hadoop as storage, processing, a programming model(s), clustered machines. The same problem exists for other terms in the lexicon, so ask me when I make less sense than usual.
  4. 4. My plan of attack  An intro: the good, the bad and the ugly at 50,000 ft.  2¢ tour of hadoop’s processing - map reduce  2¢ tour of hadoop’s storage – hdfs  A blitz tour of the rest of the hadoop ecosystem
  5. 5. Why did this happen?  Old school –> scale up == larger costlier monolithic system (or a small cluster there of) i.e. vertical scaling  Different approach – all road lead to scale out  Assume failures  Smart software, cheap hardware  Don’t move data; bring processing to data
  6. 6. The Good Simple development (when compared to Message Passing Interface programming ) Scale – no shared state, programmer don’t need to know the topology, easy to add hardware Automatic parallelization and distribution of tasks Fault tolerance Works with commodity hardware Open source!
  7. 7. The Bad  Not a silver bullet :(  MapReduce is batch data processing the time scale is minutes to hours  MapReduce is overly simplify/abstracted – you are stuck with the M/R model and it is hard to work smarter  MapReduce is low level compared to high-level languages like SQL  Not all work decomposes well into parallelized M/R  Open source :)
  8. 8. The Ugly? Welcome to the rest of our talk! First stop, Map Reduce
  9. 9. Hadoop’s MapReduce Lisp’s map and reduce plus the associative property applied to clusters.
  10. 10. Map()  Imagine a number of servers with lists of first names – What is the most popular name?  Box 1-isabella William ava mia Emma Alexander  Box 2-Noah NOAH Isabella Isabella emma Emma  Box 3-emma Emma Liam liam mason Isabella Map() would apply a function to each element independent of order. For example, capitalize each word (MapReduce is covered in greater detail in Chapter 2 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
  11. 11. Map()  So we would have:  Box 1-Isabella William Ava Mia Emma Alexander  Box 2-Noah Noah Isabella Isabella Emma Emma  Box 3-Emma Emma Liam Liam Mason Isabella Map() could be apply function to make pairs For example, Isabella becomes (Isabella, 1)
  12. 12. Map()  So we would have:  Box 1-(Isabella,1) (William,1) (Ava,1) (Mia,1) (Emma,1) (Alexander,1)  Box 2-(Noah,1) (Noah,1) (Isabella,1) (Isabella,1) (Emma,1) (Emma,1)  Box 3-(Emma,1) (Emma,1) (Liam,1) (Liam,1) (Mason,1) (Isabella,1) Now we are almost ready for the reduce, but first the sort and shuffle
  13. 13. Shuffle/Sort  So we would have:  Box 1-(Alexander,1) (Ava,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1)  Box 2-(Isabella,1) (Isabella,1) (Isabella,1) (Isabella,1)  Box 3-(Liam,1) (Liam,1) (Mason,1) (Mia,1) (Noah,1) (Noah,1) (William,1) Now for the reduce, our function would sum all the of the 1s, and return name and count
  14. 14. Reduce  So we would have:  Box 1-(Alexander,1) (Ava,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1)  Box 2-(Isabella,1) (Isabella,1) (Isabella,1) (Isabella,1)  Box 3-(Liam,1) (Liam,1) (Mason,1) (Mia,1) (Noah,1) (Noah,1) (William,1) Now for the reduce, our function would sum all the of the 1s, and return name and count  Box 1-(Alexander,1) (Ava,1) (Emma,5)  Box 2-(Isabella,4)  Box 3-(Liam,2) (Mason,1) (Mia,1) (Noah,2) (William,1) (https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html for similar coded in java )
  15. 15. (This architecture is covered in greater detail in Chapter 4 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
  16. 16. Map/reduce failures Check the job if: The job throws an uncaught exception. The job exits with a nonzero exit code. The job fails to report progress to the tasktracker for a configurable amount of time. (i.e. hung, stuck, slow) Check the node if: the same node keeps killing jobs…check the node Check the Job tracker/RM if: jobs are lost or stuck and then they all fail 
  17. 17. Instant MR test  Um, is the system working?  yarn jar /usr/hdp/2.3.2.0-2950/hadoop- mapreduce/hadoop-mapreduce-examples.jar pi 10 100 (your jar most likely will be somewhere else)
  18. 18. (HDFS is covered in greater detail in Chapter 3 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
  19. 19. There is a HDFS CLI  You already know some of the commands:  hdfs dfs –ls /  hdfs dfs –du /  hdfs dfs –rm /  hdfs dfs –cat /  There are other modes than dfs: dfsadmin, namenode, datanode, fsck, zkfc, balancer, etc.
  20. 20. HDFS failures  Jobs fail: due to missing blocks  Jobs fail: due to moving data due to down datanodes or huge ingest  Without NN HA – single point of failure for everything  Regular file system mayhem that you already know and love  plus the usual perms issues
  21. 21. HDFS failures  Jobs fail: due to missing blocks  Jobs fail: due to moving data due to down datanodes or huge ingest  Without NN HA – single point of failure for everything  Regular file system mayhem that you already know and love  plus the usual perms issues
  22. 22. The rest of the garden Distributed Filesystems - Apache HDFS outliers: - Tachyon - Apache GridGain - Ignite - XtreemFS - Ceph Filesystem - Red Hat GlusterFS - Quantcast File System QFS - Lustre Security outliers: - Apache Sentry - Apache Knox Gateway - Apache Ranger Distributed Programming - Apache MapReduce also MRv2/YARN - Apache Pig outliers: - JAQL - Apache Spark - Apache Flink (formerly Stratosphere) - Netflix PigPen - AMPLab SIMR - Facebook Corona - Apache Twill - Damballa Parkour - Apache Hama - Datasalt Pangool - Apache Tez - Apache Llama - Apache DataFu - Pydoop - Kangaroo - TinkerPop - Pachyderm MapReduce NewSQL Databases outliers: - TokuDB - HandlerSocket - Akiban Server - Drizzle - Haeinsa - SenseiDB - Sky - BayesDB - InfluxDB NoSQL Databases :Columnal Data Model - Apache HBase outliers: - Apache Accumulo - Hypertable - HP Vertica :Key Value Data Model - Apache Cassandra - Riak - Redis - Linkedin Volemort :Document Data Model outliers: - MongoDB - RethinkDB - ArangoDB - CouchDB :Stream Data Model outliers: - EventStore :Key-Value Data Model outliers: - Redis DataBase - Linkedin Voldemort - RocksDB - OpenTSDB :Graph Data Model outliers: - Neo4j - ArangoDB - TitanDB - OrientDB - Intel GraphBuilder - Giraph - Pegasus - Apache Spark Scheduling - Apache Oozie outliers: - Linkedin Azkaban - Spotify Luigi - Apache Falco
  23. 23. 10 in 10 minutes!  Easier Programming: Pig, Spark  SQL-like tools: Hive, Impala, Hbase  Data pipefitting: Sqoop, Flume, Kafka  Book keeping: Oozie, Zookeeper
  24. 24. Easier Programming
  25. 25. Pig What is it: a high level programming language for data manipulation that abstracts M/R from Yahoo Why: a few lines of code to munge data Example: filtered_words = FILTER words BY word MATCHES 'w+'; word_groups = GROUP filtered_words BY word; word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS count, group AS word; (Pig is covered in greater detail in Alan Gate’s Programming Pig by O’Reilly And in Chapter 16 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
  26. 26. Spark What is it: computing framework from ampLab, UC Berkeley Why: high level abstractions and better use of memory Neat trick: in-memory RDDs Example: scala> val linesWithSpark = textFile.filter(line => line.contains("Spark")) Or, in python: >>> linesWithSpark = textFile.filter(lambda line: "Spark" in line) (Spark is covered in greater detail by Matei Zaharia et al. in Learning Spark by O’Reilly Also of note is Advanced Analytics with Spark – it shows Spark’s capabilities well but moves way too quick to be truly useful. It is covered in Chapter 19 of Tom White’s Hadoop – The Definitive Guide by O’Reilly – lastest ed. Only)
  27. 27. SQL-ish
  28. 28. Hive/HQL What is it: a data infrastructure and query language from Facebook Why: batched SQL queries against HDFS Neat trick: stores metadata so you don’t have to Example: hive> LOAD DATA INPATH ‘/user/work/input/BX-BooksCorrected.csv’ OVERWRITE INTO TABLE BXDataSet; hive> select yearofpublication, count(booktitle) from bxdataset group by yearofpublication; (Hive is covered in greater detail by Jason Ruthergenlen et al. in Programming HIve by O’Reilly. Instant Apache Hive Essentials How-To by Darren Lee by Packt was useful to me as tutorial. It is also covered in Chapter 17 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
  29. 29. Impala What is it: SQL query engine from Cloudera Why: fast adhoc queries on subsets of data stored in hadoop Example: [impala-host:21000] > select count(*) from customer_address; (nada, let me know if you hit pay dirt)
  30. 30. HBase What is it: a non-relational database from Powerset Why: fast access to large sparse data sets Example: hbase(main):001:0> create 'test', 'cf' 0 row(s) in 0.4170 seconds  Hbase::Table – test hbase(main):003:0> put 'test', 'row1', 'cf:a', 'value1' 0 row(s) in 0.0850 seconds hbase(main):006:0> scan 'test' ROW COLUMN+CELL row1 column=cf:a, timestamp=1421762485768, value=value1 (HBase is covered in Chapter 20 of Tom White’s Hadoop – The Definitive Guide by O’Reilly And in covered in greater detail in Lars George’s HBase – The Definitive Guide by O’Reilly)
  31. 31. Data pipefitting
  32. 32. Sqoop What is it: glue tool for moving data between relational databases and hadoop Why: make the cumbersome easier Example: sqoop list-databases --connect jdbc:mysql://mysql/employees –username joe -- password myPassword (HBase is covered in greater detail in Chapter 16 of Tom White’s Hadoop – The Definitive Guide by O’Reilly There is also a cookbook that covered a few worthy gotchas: Apache Sqoop Cookbook Kathleen Ting by O’Reilly)
  33. 33. Flume What is it: a service for collecting and aggregating logs Why: because log ingestion is tougher than it seems Example: # Define a memory channel on agent called memory-channel. agent.channels.memory-channel.type = memory # Define a source on agent and connect to channel memory-channel. agent.sources.tail-source.type = exec agent.sources.tail-source.command = tail -F /var/log/system.log agent.sources.tail-source.channels = memory-channel # Define a sink that outputs to logger. agent.sinks.log-sink.channel = memory-channel agent.sinks.log-sink.type = logger # Define a sink that outputs to hdfs. agent.sinks.hdfs-sink.channel = memory-channel agent.sinks.hdfs-sink.type = hdfs agent.sinks.hdfs-sink.hdfs.path = hdfs://localhost:54310/tmp/system.log/ agent.sinks.hdfs-sink.hdfs.fileType = DataStream # Finally, activate. agent.channels = memory-channel agent.sources = tail-source agent.sinks = log-sink hdfs-sink (I haven’t read much on Flume; if you find something clever let me know!)
  34. 34. Kafka What is it: message broker from LinkedIn Why: fast handling of data feeds Neat trick: no need to worry about missing data or double processing data Example: > bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test This is a message This is another message > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from- beginning This is a message This is another message (I disliked the one book I read but I found the online docs very readable! http://kafka.apache.org/ Also check out the design docs http://kafka.apache.org/documentation.html#design )
  35. 35. Book keeping
  36. 36. Oozie What is it: workflow scheduler from Yahoo Banglalore Why: because cron isn’t perfect Example: oozie job -oozie http://localhost:8080/oozie -config examples/apps/map- reduce/job.properties -run (Oozie is covered in greater detail in Islam & Srinivasan’s Apache Oozie: The Workflow Scheduler by O’Reilly)
  37. 37. Zookeeper What is it: a coordination service from Yahoo Why: sync info for distributed systems (similar idea behind DNS or LDAP) Example: [zkshell: 14] set /zk_test junk cZxid = 5 ctime = Fri Jun 05 13:57:06 PDT 2009 mZxid = 6 mtime = Fri Jun 05 14:01:52 PDT 2009 pZxid = 5 [zkshell: 15] get /zk_test junk cZxid = 5 ctime = Fri Jun 05 13:57:06 PDT 2009 mZxid = 6 mtime = Fri Jun 05 14:01:52 PDT 2009 pZxid = 5 (Zookeeper is covered in greater detail in Zookeeper: Distributed Process Cooridination by O’Reilly And in Chapter 21 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
  38. 38. Distributed Programming - Apache MapReduce also MRv2/YARN - Apache Pig outliers: - JAQL - Apache Spark - Apache Flink (formerly Stratosphere) - Netflix PigPen - AMPLab SIMR - Facebook Corona - Apache Twill - Damballa Parkour - Apache Hama - Datasalt Pangool - Apache Tez - Apache Llama - Apache DataFu - Pydoop - Kangaroo - TinkerPop - Pachyderm MapReduce
  39. 39. Distributed Filesystems - Apache HDFS outliers: - Tachyon - Apache GridGain - Ignite - XtreemFS - Ceph Filesystem - Red Hat GlusterFS - Quantcast File System QFS - Lustre
  40. 40. NoSQL Databases :Columnal Data Model - Apache HBase outliers: - Apache Accumulo - Hypertable - HP Vertica :Key Value Data Model - Apache Cassandra - Riak - Redis - Linkedin Volemort :Document Data Model outliers: - MongoDB - RethinkDB - ArangoDB - CouchDB :Stream Data Model outliers: - EventStore :Key-Value Data Model outliers: - Redis DataBase - Linkedin Voldemort - RocksDB - OpenTSDB :Graph Data Model outliers: - Neo4j - ArangoDB - TitanDB - OrientDB - Intel GraphBuilder - Giraph - Pegasus - Apache Spark NewSQL Databases outliers: - TokuDB - HandlerSocket - Akiban Server - Drizzle - Haeinsa - SenseiDB - Sky - BayesDB - InfluxDB
  41. 41. Data Ingestion :SQL on Hadoop - Apache Hive - Apache HCatalog outliers: - Cloudera Kudu - Trafodion - Apache Drill - Cloudera Impala - Facebook Presto - Datasalt Splout SQL - Apache Spark - Apache Tajo - Apache Phoenix - Apache MRQL - Kylin Data Ingestion - Apache Flume - Apache Sqoop outliers: - Facebook Scribe - Apache Chukwa - Apache Storm - Apache Kafka - Netflix Suro - Apache Samza - Cloudera Morphline - HIHO - Apache NiFi
  42. 42. Etc. Service Programming and Frameworks - Apache Zookeeper - Apache Avro - Apache Parquet outliers: - Apache Thrift - Apache Curator - Apache Karaf - Twitter Elephant Bird - Linkedin Norbert Scheduling - Apache Oozie outliers: - Linkedin Azkaban - Spotify Luigi - Apache Falcon - Schedoscope Security outliers: - Apache Sentry - Apache Knox Gateway - Apache Ranger System Deployment and Management outliers: - Apache Ambari - Cloudera Manager - Cloudera HUE - Apache Whirr - Apache Mesos - Myriad - Marathon - Brooklyn - Hortonworks HOYA - Apache Helix - Apache Bigtop - Buildoop - Deploop
  43. 43. And now a bit of common sense for sys-admin-ing Hadoop clusters
  44. 44. Avoid  The usual -  Don’t let hdfs fill up  Don’t use all the memory  Don’t use up all the cpus  Don’t drop the network  <insert fav disaster>  Resource Exhaustion by users  Hardware Failure (drives are the king of this domain)
  45. 45. Um, backups?  Usual suspects plus  Namenode’s meta data!! (fsimage)  Hdfs? Well, it would nice but unlikely (if so distcp)  Snapshots
  46. 46. Hadoop Management  Apache Ambari  Cloudera Manager
  47. 47. Monitoring  The usual suspects plus…  JMX support  Jvm via jstat, jmap etc.  hdfs  Mapred  conf/hadoop-metrics.properties  http://namenode:50070/  http://namenode:50070/jmx
  48. 48. User management  Hdfs quotas  Access controls  Internal and  external  MR schedulers  Fifo, Fair, Capacity  Kerberos can be used as well
  49. 49. Configuration  /etc/hadoop/conf  Lots of knobs!  !Ojo! –  Lots of overrides  Get the basic system solid before security and performance  Watch the units – some are in megabytes but some are in bytes!  Have canary jobs  Ensure same configs are everywhere (including uniform dns/host)
  50. 50. Want more? (Disclaimer: I receive nothing from O’Reilly. Not even a Christmas card…)
  51. 51. Fin  Thanks for listening  Slides: http://www.slideshare.net/ericwilliammarshall/hadoop- for-sysadmins  Any questions?
  52. 52. What’s in a name?  Doug Cutting seems to have been inspired by his family. Lucene is his wife’s middle name, and her maternal grandmother’s first name. His son, as a toddler, used Nutch as the all-purpose word for meal and later named a yellow stuffed elephant Hadoop. Doug said he “was looking for a name that wasn’t already a web domain and wasn’t trademarked, so I tried various words that were in my life but not used by anybody else. Kids are pretty good at making up words.”
  53. 53. What to do? Combinations of the usual stuff:  Numerical Summarizations  Filtering  Altering Data Organization  Joining Data  I/O
  54. 54. federation (Image from Chapter 2 of Eric Sammer’s Hadoop Operations by O’Reilly)

×