This document discusses the process of collecting, storing, processing and analyzing big data. It covers the key concepts and technologies for collecting data using tools like Apache Sqoop and Kafka, storing data using clusters, file systems, NoSQL databases and concepts like sharding and replication. It also discusses processing data using parallel and distributed processing with Hadoop, and analyzing data using Apache Phoenix which provides a SQL interface to query HBase databases.

1. Collecting, Storing and
Analyzing Big Data
trieunt@fpt.com.vn
tantrieuf31@gmail.com
Big Data Process Development

2. Agenda
Collecting → Storing → Processing → Analyzing → Learning → Reacting
Data engineering process: 3 tasks
1. Collecting
a. Concepts
b. Technology
2. Storing
a. Big Data Storage Concepts
b. Big Data Storage Technology
3. Processing
a. Big Data Processing Concepts
b. Big Data Processing Technology
Data Science/Machine Learning process: 3 tasks
4) Analyzing → 5) Learning → 5) Reacting

3. Big Data Analytics Lifecycle
Collecting
Storing
Processing

4. (Collecting) → Storing → Processing → Analyzing
→ Learning → Reacting

5. Collecting

6. Collecting tools
Batch collecting: Apache Sqoop ( from DBMS to Apache Hadoop)
Real-time collecting: RFX-tracking (from stream data to Apache Kafka)

7. Collecting → (Storing) → Processing → Analyzing
→ Learning → Reacting

8. Storing Concepts
Clusters
File Systems and Distributed File Systems
NoSQL
Sharding
Replication
Sharding and Replication
CAP Theorem
ACID
BASE

9. Clusters

10. NoSQL

11. Sharding

13. Replication (Master-Slave)

14. Replication (Peer-to-Peer)

15. CAP Theorem

17. Collecting → Storing → (Processing) → Analyzing
→ Learning → Reacting

18. Processing concepts
Parallel Data Processing
Distributed Data Processing
Hadoop
Processing Workloads
Cluster
Processing in Batch Mode
Processing in Realtime Mode

19. Parallel Data Processing

20. Distributed Data Processing

21. Hadoop
Hadoop is a versatile framework that provides both processing and
storage capabilities

22. Batch processing (offline processing)

23. Transactional processing

24. Cluster

25. Map and Reduce Tasks

26. Processing in Realtime Mode

27. Tools

28. When standard relational database
(Oracle,MySQL, ...) is not good enough
the “analytic system” MySQL database from a startup, tracking all actions in
mobile games: iOS, Android, ...

29. 3 common problems in Big Data System
1. Size: the volume of the datasets is a critical factor.
2. Complexity: the structure, behaviour and permutations of the datasets is
a critical factor.
3. Technologies: the tools and techniques which are used to process a
sizable or complex dataset is a critical factor.

30. What is Apache Phoenix ?
Apache Phoenix is a SQL skin over HBase.
It means scaling Phoenix just like scale-up and
scale-out the Hbase

31. Phoenix
SQL Engine

32. Interesting features of Apache Phoenix
● Embedded JDBC driver implements the majority of java.sql interfaces, including
the metadata APIs.
● Allows columns to be modeled as a multi-part row key or key/value cells.
● Full query support with predicate push down and optimal scan key formation.
● DDL support: CREATE TABLE, DROP TABLE, and ALTER TABLE for
adding/removing columns.
● Versioned schema repository. Snapshot queries use the schema that was in
place when data was written.
● DML support: UPSERT VALUES for row-by-row insertion, UPSERT SELECT for
mass data transfer between the same or different tables, and DELETE for
deleting rows.
● Limited transaction support through client-side batching.
● Single table only - no joins yet and secondary indexes are a work in progress.
● Follows ANSI SQL standards whenever possible
● Requires HBase v 0.94.2 or above
● 100% Java

34. the Phoenix table schema

35. Setting JDBC Phoenix Driver

36. Phoenix and SQL tool in Eclipse 4

37. Phoenix vs Hive
(running over HDFS and HBase)
http://phoenix.apache.org/performance.html
Performance: Phoenix vs Hive

38. Readings
1. https://medium.baqend.com/real-time-stream-processors-a-survey-and-d
ecision-guidance-6d248f692056#.s00ac9xtu
2. https://medium.baqend.com/nosql-databases-a-survey-and-decision-guid
ance-ea7823a822d#.pn63unwx6
3. https://www.infoq.com/articles/apache-kafka
4. https://docs.google.com/document/d/1ZtEhLw3lrQSeNWVEJkKLLy8B8t9zA
0MGRqmCOV3_hsA/edit?usp=sharing