Upcoming SlideShare
×

# 13 09-28 hadoop-in_taiwan_2013_opening

1,027 views
976 views

Published on

Published in: Travel, Technology
3 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

Views
Total views
1,027
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
36
0
Likes
3
Embeds 0
No embeds

No notes for slide

### 13 09-28 hadoop-in_taiwan_2013_opening

1. 1. 2013-09-28 Hadoop in Taiwan 2013 Three New Trends of Big Data 即時‧安全‧易用 王耀聰 / 國家高速網路與計算中心 Jazz Yao-Tsung Wang / NCHC <jazz@nchc.narl.org.tw>
2. 2. 2013-09-28 Hadoop in Taiwan 2013 2 教師節快樂！謝謝各位蒞臨！教師節快樂！謝謝各位蒞臨！ 感謝主辦單位 與贊助廠商 祝台下的老師們教師節快樂！ Happy Teacher's Day !!
3. 3. 2013-09-28 Hadoop in Taiwan 2013 3 3 Vs of Big Data3 Vs of Big Data 3 巨量資料的挑戰在於如何管理「數量」、「增加率」與「多樣性」 Volume 資料數量 (amount of data) Velocity 資料增加率 (speed of data in/out) Variety 資料多樣性 (data types, sources) Batch ( 批次作業 ) Realtime ( 即時資料 ) TB EB Unstructured 非結構化資料 Semi-structured 半結構化資料 Structured 結構化資料 PB 參考來源： [1] Laney, Douglas. "3D Data Management: Controlling Data Volume, Velocity and Variety" (6 February 2001) [2] Gartner Says Solving 'Big Data' Challenge Involves More Than Just Managing Volumes of Data, June 2011
4. 4. 2013-09-28 Hadoop in Taiwan 2013 4 Life of Big DataLife of Big Data ：蒐、存、取、析、用：蒐、存、取、析、用
5. 5. 2013-09-28 Hadoop in Taiwan 2013 5 Big Data is the Answer - What was the Question?Big Data is the Answer - What was the Question? 參考來源： Big Data is the Answer - What was the Question? http://www.saama.com/blog/bid/76211/Big-Data-is-the-Answer-What-was-the-Question
6. 6. 2013-09-28 Hadoop in Taiwan 2013 6 Big Data at Rest – MapReduce FrameworkBig Data at Rest – MapReduce Framework 6 Volume VelocityVariety TB EB PB Realtime Batch Structured Unstructured M apReduce Fram ework PetabyteFileSystem HadoopHadoop HPCCHPCC 存、取、析
7. 7. 2013-09-28 Hadoop in Taiwan 2013 7 Big Data in Motion –Big Data in Motion – In-Memory ProcessingIn-Memory Processing 、、 Predictive AnalyticsPredictive Analytics Volume VelocityVariety TB EB PB Realtime Batch Structured Unstructured HBase / DrillHBase / Drill Impala / SparkImpala / Spark 取、析、用
8. 8. 2013-09-28 Hadoop in Taiwan 2013 8 Big Data in Motion –Big Data in Motion – Streaming Data Collection / Data CleaningStreaming Data Collection / Data Cleaning 8 Volume VelocityVariety TB EB PB Realtime Batch Structured Unstructured Message QueueMessage Queue ( AMQP , RabbitMQ )( AMQP , RabbitMQ ) Storm / KafkaStorm / Kafka 蒐、存 ( 前處理 )
9. 9. 2013-09-28 Hadoop in Taiwan 2013 9 NoHadoop ?! Not Only Hadoop !!NoHadoop ?! Not Only Hadoop !! Source: Lambda Architecture, 8. March 2013 http://www.ymc.ch/en/lambda-architecture-part-1 HBase Storm ElephantDB Or Voldemort Hadoop
10. 10. 2013-09-28 Hadoop in Taiwan 2013 10 Next Step : Big Data SecurityNext Step : Big Data Security 當我們緊密相連 ..... 世界政經：歐盟想分 Tweeter 找出經濟、政治的脈動 國家安全：美國 PRISM 計劃 ( 網軍 ! 終極警探 4.0 ) 組織如何因應 APT ? Big Data 平台本身的安全性 ? 有太多安全的問題等待解決！ Source: Gartner (March 2011), 'Big Data' Is Only the Beginning of Extreme Information Management, 7 April 2011, http://www.gartner.com/id=1622715 權限管控 品質管控 數量管控
11. 11. 2013-09-28 Hadoop in Taiwan 2013 11 To Find the Value of Big Data We need Data Scientist Team ! 電機 資訊 數學數學 統計統計 商商 做決策 資 料 科 學 家 分 析 軟 體 重點在找到價值 Value
12. 12. 2013-09-28 Hadoop in Taiwan 2013 12 議程安排 ( 上午場次 ) 即時‧安全‧易用
13. 13. 2013-09-28 Hadoop in Taiwan 2013 13 議程安排 ( 下午場次 )