Big Data Taiwan 2014 Track1-3: Big Data, Big Challenge — Splunk 幫你解決 Big Data...Etu Solution
講者:SYSTEX 數據加值應用發展部產品經理 | 陶靖霖
議題簡介:認清現實吧! Big Data 是個熱門詞彙、熱門議題,但是問題的核心仍然圍繞在資料處理的流程、架構與技術,要踏入 Big Data 的領域,使用者會遭遇哪些挑戰? Splunk 被譽為「全球最佳的 Big Data Company」,究竟在資料處理的流程中擁有什麼獨特的技術優勢,能夠幫助使用者克服這些挑戰?又有哪些成功幫助使用者從資料中萃取出價值的應用案例?歡迎來認識 Splunk 以及全球 Big Data 成功案例。
update:2016/06/28
Example & homework: https://github.com/Phate334/MapReduceExample
簡報已更新並重新整理
This slide show you how to deploy Hadoop cluster with Cloudera and set up your own development environment to easy to test your MapReduce app.
這份簡報分成三個部分:
1.架設hadoop叢集,今年縮短了這部分的介紹,只放上一些參考的連結。
2.準備一個可以在本機端測試Mapreduce的開發環境,而不需要每次都將程式送到叢集中。
3.三個簡單的例子,用來介紹Mapreduce框架。
Big Data Taiwan 2014 Track1-3: Big Data, Big Challenge — Splunk 幫你解決 Big Data...Etu Solution
講者:SYSTEX 數據加值應用發展部產品經理 | 陶靖霖
議題簡介:認清現實吧! Big Data 是個熱門詞彙、熱門議題,但是問題的核心仍然圍繞在資料處理的流程、架構與技術,要踏入 Big Data 的領域,使用者會遭遇哪些挑戰? Splunk 被譽為「全球最佳的 Big Data Company」,究竟在資料處理的流程中擁有什麼獨特的技術優勢,能夠幫助使用者克服這些挑戰?又有哪些成功幫助使用者從資料中萃取出價值的應用案例?歡迎來認識 Splunk 以及全球 Big Data 成功案例。
update:2016/06/28
Example & homework: https://github.com/Phate334/MapReduceExample
簡報已更新並重新整理
This slide show you how to deploy Hadoop cluster with Cloudera and set up your own development environment to easy to test your MapReduce app.
這份簡報分成三個部分:
1.架設hadoop叢集,今年縮短了這部分的介紹,只放上一些參考的連結。
2.準備一個可以在本機端測試Mapreduce的開發環境,而不需要每次都將程式送到叢集中。
3.三個簡單的例子,用來介紹Mapreduce框架。
該投影片介紹了利用CDH(Cloudera’s Cloudera Distribution Including Apache Hadoop)來架設Hadoop叢集時所需要準備的環境,其中包括硬體規格、系統環境與軟體版本。
未來蒐集的相關資源會放在Diggo如下的兩個library中:
https://www.diigo.com/user/phate334/cloudera
https://www.diigo.com/user/phate334/hadoop
8. 巨量資料的現況 …
Current Status of Big Data …..
'' Big data is like teenage sex:
everyone talks about it,
nobody really knows how to do it,
everyone thinks everyone else is doing it,
so everyone claims they are doing it .. ''
– Dan Ariely, Professor at Duke University
and Professor at Center for Advanced Hindsight
11. 快速部署 Hadoop 環境
Hadoop 的部署模型現況
2014/04/11 OSDC.TW
Jazz Wang
<jazzwang@etusolution.com>
12. 12
Spectrum of Hadoop Deployment 2013
On-premise full custom : 完全自建的 Hadoop 叢集,高度客制化,例子:Yahoo!
Hadoop Appliance:購買廠商提供的 Hadoop 叢集硬體與軟體
Hadoop Hosting:類似主機代管(Hosting)概念,依需求租賃實體 Hadoop 節點
Hadoop as a Service:由雲服務廠商提供的虛擬化 Hadoop 運算環境(Ex. EMR)
Source: Accenture Technology Labs , “Four Options for Enterprise IT when Deploying the Big Data System Hadoop”, June 24, 2013
http://www.accenture.com/us-en/Pages/insight-hadoop-deployment-comparison.aspx
Hadoop Deployment Comparison Study -- Price-performance comparison between a bare-metal Hadoop cluster and Hadoop-as-a-Service
http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Technology-Labs-Where-To-Deploy-Your-Hadoop-CLusters-Executive-Summary.pdf
18. 18
Spectrum of Hadoop Deployment 2013
On-premise full custom : 完全自建的 Hadoop 叢集,高度客制化,例子:Yahoo!
Hadoop Appliance:購買廠商提供的 Hadoop 叢集硬體與軟體
Hadoop Hosting:類似主機代管(Hosting)概念,依需求租賃實體 Hadoop 節點
Hadoop as a Service:由雲服務廠商提供的虛擬化 Hadoop 運算環境(Ex. EMR)
Source: Accenture Technology Labs , “Four Options for Enterprise IT when Deploying the Big Data System Hadoop”, June 24, 2013
http://www.accenture.com/us-en/Pages/insight-hadoop-deployment-comparison.aspx
Hadoop Deployment Comparison Study -- Price-performance comparison between a bare-metal Hadoop cluster and Hadoop-as-a-Service
http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Technology-Labs-Where-To-Deploy-Your-Hadoop-CLusters-Executive-Summary.pdf
21. 21
Spectrum of Hadoop Deployment 2013
On-premise full custom : 完全自建的 Hadoop 叢集,高度客制化,例子:Yahoo!
Hadoop Appliance:購買廠商提供的 Hadoop 叢集硬體與軟體
Hadoop Hosting:類似主機代管(Hosting)概念,依需求租賃實體 Hadoop 節點
Hadoop as a Service:由雲服務廠商提供的虛擬化 Hadoop 運算環境(Ex. EMR)
Source: Accenture Technology Labs , “Four Options for Enterprise IT when Deploying the Big Data System Hadoop”, June 24, 2013
http://www.accenture.com/us-en/Pages/insight-hadoop-deployment-comparison.aspx
Hadoop Deployment Comparison Study -- Price-performance comparison between a bare-metal Hadoop cluster and Hadoop-as-a-Service
http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Technology-Labs-Where-To-Deploy-Your-Hadoop-CLusters-Executive-Summary.pdf
31. Spectrum of Hadoop Deployment 2013
On-premise full custom : 完全自建的 Hadoop 叢集,高度客制化,例子:Yahoo!
Hadoop Appliance:購買廠商提供的 Hadoop 叢集硬體與軟體
Hadoop Hosting:類似主機代管(Hosting)概念,依需求租賃實體 Hadoop 節點
Hadoop as a Service:由雲服務廠商提供的虛擬化 Hadoop 運算環境(Ex. EMR)
Source: Accenture Technology Labs , “Four Options for Enterprise IT when Deploying the Big Data System Hadoop”, June 24, 2013
http://www.accenture.com/us-en/Pages/insight-hadoop-deployment-comparison.aspx
Hadoop Deployment Comparison Study -- Price-performance comparison between a bare-metal Hadoop cluster and Hadoop-as-a-Service
http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Technology-Labs-Where-To-Deploy-Your-Hadoop-CLusters-Executive-Summary.pdf
34. 34
「堅」跟「快」不是唯一的考量
Source: Accenture Technology Labs , “Four Options for Enterprise IT when Deploying the Big Data System Hadoop”, June 24, 2013
http://www.accenture.com/us-en/Pages/insight-hadoop-deployment-comparison.aspx
Where to deploy your Hadoop clusters? Executive Summary
http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Technology-Labs-Where-To-Deploy-Your-Hadoop-CLusters-Executive-Summary.pdf
ROIReturn on investment
投資回報率