1. TiE SV Big Data Panel
Oct 13, 2011

3. What did Google do?
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

4. What did Google do?
Store files
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

5. What did Google do?
Process
data
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

6. What did Google do?
Ingest data
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

7. What did Google do?
Store records & tables
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

8. What did Google do?
High level domain specific
language
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

9. What did Google do?
Chain together complex workloads
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

10. What did Google do?
Schedule them
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

11. What did Google do?
Columnar format + metadata
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

12. What did Google do?
End user queries
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

13. What did Google do?
Coordinate within
system
Dremel
Evenflow Evenflow Dremel
MySQL Sawzall
Bigtable
Gateway MapReduce / GFS
Chubby
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

14. The pattern repeated
HiPal
Databee Databee Hive
Hive
HBase
Scribe
Zookeeper
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

15. The pattern repeated
Oozie Oozie Hive
Pig & Hive
Data HBase
Highway
Zookeeper
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

16. The pattern repeated
Azkaban Azkaban
Sqoop Pig
Voldemort
Kafka
Zookeeper
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

17. The pattern repeated
Cloudera’s Distribution Including Apache Hadoop
Hue Hue
Oozie Oozie Hive
Sqoop Hive / Pig
HBase
Flume
Zookeeper
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

18. Project summary
Topic Project(s)
File storage HDFS
Record storage Hbase, Hypertabe, Accumulo
Metadata storage Hive, Hcatalog
Batch data processing MapReduce
Streaming data processing S4, Storm
Graph processing Giraph, X-Rime
Query language Hive
Dataflow language Pig
Database integration Sqoop
Event data collection Flume, Scribe
Test & assembly Bigtop
Distributed lock Zookeeper
Web access Hue
Workflow Oozie, Azkaban
File format Avro, RCFile, Protocol Buffers, Sequence File
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

21. POSSIBLE
with
BIG DATA
anything
is

23. Celebrate Next
Saturday