The document discusses big data architecture and its characteristics, focusing on its generation, storage, and processing through various systems and applications. It presents an example of an EKG scenario where extensive data accumulation occurs with numerous users, illustrating the scale and requirements for big data management. It also outlines how to build a big data server farm architecture using cloud services like AWS, emphasizing the need for a robust and scalable infrastructure.

1. Big Data Architecture
Jordan Chung
2015

2. Agenda
• Big-Data data characteristic
• Big-Data whole process pictures
• Big-Data scenario (EKG example)
• How to build Big-Data server farm architecture

3. Jordan’s Profile
• Linkedin: https://www.linkedin.com/profile/view?id=5911091
• Facebook: https://www.facebook.com/profile.php?id=548778295
• FB club: https://www.facebook.com/SoftwareABC
Skype: jordan_chung, Line: jordan_chung, Wechat: chung_jordan
-- Business Development
-- Product development life cycle management
-- Product Management
-- Project Management
-- Solution Architect
-- Recruiting
-- Quality Assurance
-- Cloud solution
-- Golf platform http://www.pgggame.com
-- Pre-sales
-- eMarketing
-- eBook, Video, Audio, Karaok digital content solution.

4. Big-Data Data Characteristic
• Automatically generated by machine (or sensor)
• Typically an entirely new source of data
• Not designed to be friendly ( or not readable)
• May not have much values (價值)

5. Big-Data Source
Users
Systems
Applications (Desktop / Mobile)
Sensors
Large and growing files
Source Generator

6. Big-Data Process Picture (I)
Source Generation Web service Storage Reporting
traffic

7. Big-Data Process Picture (II)
Source Generation Storage ReportingWeb service
More traffic

8. Big-Data Process Picture (III)
Source Generation Storage ReportingWeb service
More traffic

9. Big-Data Process Picture (IV)
Source Generation Storage ReportingWeb service
More Data
需要以TB /PB來幾算
More traffic

10. Big-Data Process Picture (V)
Source Generation Storage ReportingWeb service
More Data
需要以TB /PB來幾算
More traffic

11. Big-Data Process Picture (VI)
Source Generation Storage Reporting
1. 分析的Power
2. 分析的時間
Web service
More DataMore traffic
需要以TB /PB來幾算

12. Big-Data Process Picture (VII)
Source Generation Storage Reporting
1. 分析的Computing Power
2. 分析的時間
需要以TB /PB來幾算 分析的方法
Web service
** 紅色的項目 是在Big data 中會出現且不太好解決的問題
More DataMore traffic Analytics
Model

13. One Tool?
One tool to rule Big
Data Service out

14. Big-Data Scenario - EKG
• User wear EKG (心電圖) device and post the message to
server every second
• Data format can be data text or number
• One hour will generate EKG data around 1 MB
• If having 10,000 user this EKG service,
then will roughly accumulate around 90 TB one year.
Which means 10,000 connection every seconds

15. 情境示意圖
Sensor
RF/BLE http
(text) (text)
SQL
NoSQL DB
(text)
Server Farm
Client site

16. How to build Big-Data
Server farm

17. ELB
EC2 (Web)
Big-Data Server Architecture with
AWS Cloud
S3
EMR
EC2(Web)
MySQL (A/A/A)
定期
EC2(NoSQL)
ELB
EC2 (API/Web)
For APP
For Web
Sensor
RF/BLE
http
APP
(text)
http
Reporting
Take a bit time
Real Time Can be Batch
SQS/Kinese
EC2 ( Consumer Web)
EC2 (Cluster)
Mgr server
Auto -Scale
Auto -Scale

18. Build server farm architecture with Machines
FW (Active – Standby)
HA: 30 seconds
A S
…
VIP
internet
Web server
Switch
LBS
Switch
DB server
Cache Servers
VIP
Management server
Deployment server
IPS
VNX 5200 (Horizontal storage solution)
Switch
Switch
Storage
Can have single value
但機率很小很小
Switch

19. 有機會合作， 將會了解Super Value