This document discusses big data and different approaches to processing large datasets. It introduces how internet-enabled devices generate huge amounts of data daily. Storing and processing 10 terabytes of data on Amazon EC2 is then used as an example, showing how using multiple machines in parallel can process the data much faster than a single machine. Common approaches using Hadoop, MongoDB, and Spark are overviewed, along with examples of how each can be used for storage and computation. A demo of setting up an Amazon cluster and processing Wikipedia data with Hadoop and Spark is proposed. Issues around big data in Vietnam are also briefly covered.

1. Big Data in Action
Ngon Pham, Lana Engineer

2. Introduction
●
●
●
●
●
Introduction
Problem
Approach
Demo
Big Data in Vietnam

3. Introduction
● Internet-enabled devices
○ Tons of data generated every second
● Hardware becomes much cheaper
○ We can now store and process much more data

4. Problem
● How to process 10TB, how long and how
much?
○ Assume
■ Amazon EC2
■ HDD read at 50MB/s
■ Computation time is less than I/O time

5. Problem
● 1 machine, 1 core, 1 HDD
○ Time: 55.56 hours
○ Amazon Cost: $0.12 x 55.56 = $6.67
● 10 machines, 40 cores, 40 HDD
○ Time: 1.39 hours
○ Amazon Cost: $0.48 x 10 x 1.39 = $6.67
⇒ The same cost but 40x faster

6. Question
● How to divide data/process between
machines?
● How to make each process read data inside
the machine directly instead of another?
● How to replicate data, restore the process if
there is failure?
● Lots of task management questions...

7. Approach
● Hadoop
● MongoDB
● Spark

8. Hadoop Approach
● Storage
○ HDFS

9. Hadoop Approach
● Computation
○ MapReduce

10. MongoDB Approach
● Storage
○ Document

11. MongoDB Approach
● Computation
○ SQL
○ Aggregation
○ MapReduce

12. Spark Approach
● Storage
○ Resilient distributed
dataset (RDD)
○ Persistent backed by
HDFS / HBase...

13. Spark Approach
● Computation
○ Mixed
○ In-memory
computing

14. Demo
● Hadoop
○ Run script to create Amazon cluster
○ Play with Hadoop / HDFS / Spark
○ Process Wikipedia data
● MongoDB
○ Collect data from different sources and analyze

15. Big Data in Vietnam

16. Big Data in Vietnam
● Why is MongoDB popular?
○ Lots of PHP developers prefer
○ Simple to setup and use
○ Similar to MySQL

17. Big Data in Vietnam
● Hadoop is used by a few big local online
companies & international startups
○ Analyze tons of data
○ Create new competitive advantage
⇒ But there is a big shortage of skilled engineers

18. Q&A
Q&A