The document provides a comprehensive overview of big data concepts, principles, and practical use cases, highlighting the importance of big data in various applications. It outlines the architectural design using lambda architecture, details the data engineering process, and includes a case study on a chat bot with a video recommendation engine. Additionally, it offers guidance for students on learning big data and resources for further education.

1. Concepts, use cases
and principles to build
big data systems
http://www.bigdatavietnam.org
https://www.facebook.com/bigdatavn Compiled by Nguyễn Tấn Triều

2. Key Contents
1. Introduction to the key Big Data concepts
○ The Origins of Big Data
○ What is Big Data ?
○ Why is Big Data So Important ?
○ How Is Big Data Used In Practice ?
2. Introduction to the key principles of Big Data Systems
○ How to design Data Pipeline in 6 steps
○ Using Lambda Architecture for big data processing
3. Practical case study
○ Chat bot with Video Recommendation Engine
4. FAQ for student

3. Introduction to the
key Big Data
concepts
○ The Origins of Big Data
○ What is Big Data ?
○ Why is Big Data so
important ?
○ How Is Big Data used in
practice ?

4. The Origins of Big Data

5. https://www.kdnuggets.com/2017/02/origins-big-data.html

6. What is Big Data ?

7. What is Big Data ?

8. What is Big Data ?

9. Why is Big Data So Important ?

10. Why is Big Data So Important ?

12. Source: https://internetofthingsagenda.techtarget.com/definition/Internet-of-Things-IoT
How Is Big Data Used In Practice ?

13. How Is Big Data Used In Practice ?

14. Why is Big Data So Important ?

16. How Is Big Data Used In Practice ?
Device Analytics
Which device is most
popular used ?

17. How Is Big Data Used In Practice ?
Time-series Analytics
The peak hours of system

18. How Is Big Data Used In Practice ?
GeoLocation Heatmap Analytics

19. Introduction to the
key principles of
Big Data Systems
○ How to design Data
Pipeline in 6 steps
○ Using Lambda
Architecture for big
data processing

20. How to design Data Pipeline Systems
Collecting → Storing → Processing → Analyzing → Learning → Visualizing
Data engineering process: 3 tasks
1. Collecting
a. Concepts
b. Technology
2. Storing
a. Big Data Storage Concepts
b. Big Data Storage Technology
3. Processing
a. Big Data Processing Concepts
b. Big Data Processing Technology
Data Science/Machine Learning process: 3 tasks
4) Analyzing → 5) Learning → 5) Visualizing

21. Data Engineer Tasks Data Analyst Tasks
Big Data Analytics Lifecycle
Collecting
Storing
Processing
Analyzing
Learning
Visualizing

22. (Collecting) → Storing → Processing → Analyzing
→ Learning → Reacting

23. Collecting

24. Collecting tools
Batch collecting: Apache Sqoop ( from DBMS to Apache Hadoop)
Real-time collecting: Log Collector with Apache Kafka

25. Collecting → (Storing) → Processing → Analyzing
→ Learning → Reacting

26. Storing Concepts
● Clusters
● Scale-Up vs Scale-Out
● File Systems and Distributed File Systems
● NoSQL
● Sharding
● Replication
● Sharding and Replication
● CAP Theorem

27. Clusters

28. Scale-Up vs Scale-Out

29. Database in Big Data

30. NoSQL

31. NoSQL

32. Sharding

34. Replication (Master-Slave)

35. Replication (Peer-to-Peer)

36. CAP Theorem

37. Collecting → Storing → (Processing) → Analyzing
→ Learning → Reacting

38. Processing concepts
● Parallel Data Processing
● Distributed Data Processing
● Hadoop
● Processing Workloads
● Cluster
● Processing in Batch Mode
● Processing in Realtime Mode

39. Parallel Data Processing

40. Distributed Data Processing

41. Hadoop
Hadoop is a versatile framework that provides both processing and
storage capabilities

42. Batch processing (offline processing)

43. Transactional processing

44. Cluster

45. Map and Reduce Tasks

46. Processing in Realtime Mode

47. When standard relational database
(Oracle,MySQL, ...) is not good enough
the “analytic system” MySQL database from a startup, tracking all actions in
mobile games: iOS, Android, ...

48. 3 common problems in Big Data System
1. Size: the volume of the datasets is a critical factor.
2. Complexity: the structure, behaviour and permutations of the datasets is
a critical factor.
3. Technologies: the tools and techniques which are used to process a
sizable or complex dataset is a critical factor.

49. Key ideas of Lambda Architecture in Big Data System

50. Practical case
study Chat bot with Video
Recommendation Engine

51. Problem
● A company want to develop a chat bot for
news recommendation
● They want to classify data into standard
categories (26 categories) for
user-friendly query
● The engineering team have develop a
data pipeline for system

52. Solution Diagram
Big Data
is here
Author @tantrieuf31

53. Problem: Topic Classification for News

54. Solution Diagram

55. FAQ for students
How to learn Big Data ?
Job Opportunity
Ref resources

56. How to learn Big Data ?
1. Have lots of passion, curiosity with data
2. Knowledge about data structure, statistics and basic maths
3. Love to solve complex problems with data-driven mindset
4. Database knowledge: when to use NoSQL vs RDBMS
5. Knowledge about distributed computing
6. Linux / Open Source Tools
7. Programming language: Python / Java / SQL / JavaScript
8. English skills

57. Big Data Job Market is really hot
https://www.class-central.com/subject/big-data

58. Some good books for self-learning
● http://sachvui.com/ebook/du-lieu-lon-big-data.281.html
● https://drive.google.com/open?id=0B3dHGVpTXDOhQXJCR01PVkpQMGM
● https://drive.google.com/file/d/1rPvfio6EkaUvGtgfQoq9p9Fa2ljOMIn1/view?usp=sharing
● https://drive.google.com/open?id=0B3dHGVpTXDOhVTBKX09NUnlLcm8

59. Free MOOC
https://www.class-central.com/subject/big-data