This document provides an overview of big data ecosystems, including common log formats, compression techniques, data collection methods, distributed storage options like HDFS and S3, distributed processing frameworks like Hadoop MapReduce and Storm, workflow managers, real-time storage options, and other related topics. It describes technologies like Kafka, HBase, Cassandra, Pig, Hive, Oozie, and Azkaban; compares advantages and disadvantages of HDFS, S3, HBase and other storage systems; and provides references for further information.

1. BigData Eco System overview
●Log Formats
●Compression
●Collecting Data
●Distributed Storage
●Distributed Processing
●Workflow Management
●Realtime Read Write Storage
●Others

2. Log Formats
● Google Protocol Buffers
● Thrift
Typed Data (and int is an int etc)
Named Flexible Schema
Backwards compatibility

3. Compression
Default (slowish)
● GZIP (splittable)
● Bzip2 (slowest)
● 7zip (only for long archival)
Fastest
● Snappy (not splittable)
● LZO (not splittable)
●

4. Compression LZO/Snappy
● Saves data
● CPU/RAM impact is negligible
● Less bytes in disk makes for faster reads and
writes

5. Collecting Data
File collection
● BigStreams
Pub Subscribe collection
● Kafka + Kafka-Collector
● Scribe
● Flume

6. Data Pipeline
● Reliable
● Auto healing
● Cache data for N days
● Automatic
● Fast
● Distributed

7. Distributed Storage
● HDFS (master slaves)
● S3 (bucket, key → blob, no master)
● GridFS ?
● NFS (not for high writes or big data)

8. Hadoop Distributed File System
Advantages
● Simple (more or less)
● Works with every day hardware (cheap to scale)
● Proven scalability to petabytes
● Lends itself to efficient distributed batch processing
Disadvantages
● Single Point of Failure (HA is a work in progress)
● All meta-data must fit in master's RAM
● No Random Read/Writes

9. S3
Advantages
● Cheap
● distributed
● Good for data archival
Disadvantages
● Data is stored externally
● Does not lend itself to batch processing of large
volumes of data

10. Distributed Processing
● Hadoop MapReduce
● GridGain
● Storm
● Akka Actors

11. Hadoop MapReduce
● Used for distributed serial batch processing
● Works with HDFS
● Simple concept but complex APIs
● Lots of higher level APIs for querying (Pig/Hive)
● Not for random indexed reads
● Not for small data i.e. < 10 gigs

12. GridGain
● Fast In-Memory queries
● Not attached to any specific datastorage
● API is java/script based

13. Storm
● In-Memory
● Distributed
● Stream based aggregation/processing
● Supports sending partially aggregated data to
backends like Hbase/Cassandra

14. Akka Actors
● Concurrent processing constructs based on the
erlang actor model
● Latest versions support distributed RPC
communication via Netty or ZeroMQ.
● Used for building distributed fast processing
systems.

15. M/R High level languages
SQL
• Hive
Imperative
• Pig
Lisp
• Cascalog
R
• Hive JDBC Connection

16. Apache Pig
Advantages
● Simple and programmable
● UDFS and Loader/Store APIs are simple
● Spill to disk to avoid OOM
Disadvantages
● Low level
● Schema-less
●

17. Hive
Advantages
● SQL interface
● Server mode
Fast
Disadvantages
● Complex UDF Load/Store (SERDE) API
● Does not spill to disk like pig to avoid OOM

18. Workflow Management
● Glue
● Oozie
● Azkaban
● Bash

19. Glue
● Workflows for devops.
● No XML.
● Polygot language approach supports Groovy,
Scala,Ruby(JRuby), Python(Jython), Clojure,
JavaScript.
● Data driven and cronbased worfklows
● Separate configuration from workflows

20. Oozie
● XML
● UI for build workflows using blocks, (still have to
program the components)
● Buy another pair of glasses

21. Azkaban
● Based on Flows
– There consist of binaries described by a job text file
● Concentrates on generic scheduling and retries
in a traditional sense.
● Flow UI

22. Bash
● Don't do workflows in bash
● Know your bash for simple adhoc searches and
processing
● Again do not do workflows in bash

23. Realtime Read Write Storage
● Hbase and Accumulo
● Cassandra

24. Hbase and Accumulo
● Both are based on the BigTable paper from
Google.
● Column based storage
● Integrates with HDFS
● Tables act as distributed indexes
● Region Servers are single points of failure
● Aimed at faster reads than writes

25. Cassandra
● Based on the Dynamo papers from amazon
● No single point of failure
● Aimed at faster writes than reads
● Default eventual consistency with configurable
durability options (at the cost of writing speed)
● Column Counters

26. Others
● Lucene (api for building fast indexes)
● Solr and Elastic search
– Built on top of lucene
– Distributed indexes
– Fast query times
● Mongo DB (document db)
● Redis (fast in-memory db)
– Lots of basic constructs, easy to build bloom filters
– Great for realtime

27. References
http://cassandra.apache.org/
http://hbase.apache.org/
http://hbase.apache.org/
http://hbase.apache.org/
http://hadoop.apache.org/
http://hive.apache.org/
http://pig.apache.org/
http://storm-project.net/
http://lucene.apache.org/core/
http://lucene.apache.org/solr/
http://www.elasticsearch.org/
https://github.com/nathanmarz/cascalog
http://akka.io/
http://www.datastax.com/dev/blog/big-analytics-with-r-cassandra-and-hive
http://gerritjvv.github.io/glue/
https://code.google.com/p/bigstreams/
https://github.com/facebook/scribe
http://flume.apache.org/
http://redis.io/
http://www.mongodb.org/
http://netty.io/
http://zeromq.org/
http://kafka.apache.org/
http://www.gridgain.com/
http://docs.mongodb.org/manual/core/gridfs/
http://aws.amazon.com/s3/