Download as PDF, PPTX
Uploaded byYa-hui Lin
雲端與Big data
The document discusses cloud computing and big data. It begins by watching introductory videos on these topics. Cloud computing is defined as utilizing computer resources over the internet. Key characteristics include on-demand self-service, network access, resource pooling, rapid elasticity, and measured service. The document discusses how cloud computing represents a new business era for information technology, with platforms as a service (PaaS) being a new application platform. Public clouds offer a pay-as-you-go utility model for IT resources.
雲端與Big data
- 1. 雲端 與 BigData Kun-Ta Chuang (莊坤達), Ph.D. Assistant Professor National Cheng Kung University
- 2. Preliminaries ∗ Before goinginto the discussion, we see videos talking about the future 2
- 3. What is CloudComputing? ∗ Okay, we still watch a video before starting the discussion about ‘Cloud Computing’ 3
- 4. What is BigData? ∗ Sure. We also start by watching a video! 4
- 5. ∗ Cloud Computingand Big Data are the definite consequence of the internet age! ∗ We start the discussion from ‘Cloud Computing’ 5
- 6. Introduction to CloudComputing ∗ What is Cloud Computing? ∗ We have different perspectives from different sides ∗ According to wikipedia, "Cloud computing is Internet-based ("Cloud") development and use of computer technology. "
- 7. The NIST CloudDefinition Framework Hybrid Clouds Deployment Private Community Models Public Cloud Cloud Cloud Service Software as a Platform as a Infrastructure as a Models Service (SaaS) Service (PaaS) Service (IaaS) On Demand Self-Service Essential Broad Network Access Rapid Elasticity Characteristics Resource Pooling Measured Service Massive Scale Resilient Computing Common Homogeneity Geographic Distribution Characteristics Virtualization Service Orientation Low Cost Software Advanced Security 7
- 8. What is CloudComputing? ∗ A new business opportunity? ∗ Is it far beyond distributed/grid/cluster computing? ∗ Or, just a new term? ∗ Is it a new Holy Grail? I don’t understand what we would do ∗ Web 3.0, new web-scale problem? differently in the light of cloud computing ∗ Social, Location, Mobile other than changing the wording of some of our ads Oracle’s CEO Larry Ellison
- 9. New philosophy? What wedo in the past
- 10. In the CloudEra
- 11. We don’t needto work here
- 12. The Rise ofa New Era in IT Cloud Platform as a Service Web Application Servers PC / Client-Server Unix Services Mainframe COBOL Each new era in computing brings a new application platform: for the Cloud era it is “PaaS”
- 13.
- 14. Where can weget money? From Gartner (March, 2009)
- 15. It is anew Era, but Is it a new business model? ∗ Let’s turn to review the history of the IC industry ∗ Do you think why Fabless Design Houses are so strong in the past 10+ years?
- 16. Systems Design Manufacturing Saber SysStudio VMM HW/SW Magellan SysVerilog Formality DC Ultra Test VIP IC VCS NTB Virtual Compiler Platform Star RCXT Connect. DesignWare IP Analog IP (Phys) CHIP Power Hercules CATS Sigma C Proteus SiVL PrimeTime FE TCAD NanoSim PrimeYield HSIM BE TCAD HSPICE DFM Manuf. TCAD Yield Test Libraries Mgmt Chips
- 17. Today: Global ICMarket Systems $1.26T Front-End Manufacturing EDA $21.9B Computers Communications Masks* $4.0 B Consumer $3.3B Industrial Lithography/Mask Making CMP equipment Military… Ion Implanters Deposition Embedded SW $2.5B Etching and Cleaning Silicon Other Wafers $11.4B Back-End Manufacturing IP $1.4B $6.6B Assembly Equipment Assembly Inspect. Semiconductors $269.9B Dicing Bonding Micros, DSP Packaging Memory Int. Assembly Sys ASIC, ASSP Chips Total Test Analog Discrete Foundry Wafers $20.9B 2008 Data (*2006) Source: VLSI Research, Gartner, IC Insights, SEMI, Information Network, Synopsys Estimates
- 18.
- 19.
- 20. Cloud -- NotJust a New Term? ∗ Is ‘Cloud Computing’ far beyond distributed/grid/cluster computing? ∗ Is it also mature? ∗ 鑑古知今
- 21. Do we haveTSMC and Synopsys in the Cloud IT industry? ∗ Amazon AWS Marketplace 21
- 22. Look back ∗ Wehave TSMC and Synopsys, but we still need ASML, National Instruments
- 23. ∗ VMWARE 23
- 24. Cloud Hierarchy ∗ IaaS ∗ Infrastrature ∗ PaaS ∗ Platform ∗ SaaS ∗ Software
- 25. Technology Hierarchy User Level 應用 Social Computing, Enterprise, ISV,… User-Level 程式語言 Middleware Web 2.0 介面, Mashups, Workflows, … 控制 Qos Neqotiation, Ddmission Control, Core Middleware Pricing, SLA Management, Metering… 虛擬化 VM, VM management and Deployment System Level 25
- 26. Deployment models Public cloud Community cloud Hybrid cloud Private cloud We talk about: Public Cloud - A cloud is available in pay-as- you-go to the general public 26
- 27. Utility Computing --Pay as you go ∗ Hours purchased via cloud ∗ Cloud computing offers computing can be economic benefits of distributed non-uniformly elasticity and in time transference of risk Utility Computing – the service being sold in public cloud Cloud Services = SaaS + Utility Computing
- 28. The spirit of‘Pay as you go’ ∗ No longer require the Large Capital ∗ Don’t concerned about Over-Provisioning or Under- Provisioning for prediction ∗ 選課系統 ∗ Startup companies ∗ Companies with large batch-oriented tasks can be finish quickly ∗ More elasticity of resources
- 29. Example(Provision for peakload) 最高峰 :500servers 最低峰 :100servers 雲端需要24*300=7200(小時*伺服器) 傳統模式下需要500*24=12000(小時*伺服器)雲 端可以節省約1.7倍的cost!!!
- 30. Example(Under-provision) Active user –People use the site regularly Defector – People abandon the sites Suppose 10% of active user become defector who receive poor service due to under-provision
- 31. Cloud can help ∗The appearance of infinite computing resource is available to overcome load surges ∗ The elimination of an up-front commitment by cloud users ∗ The ability to pay for use of computing resources on a short term ∗ Remember: 要喝牛奶,你不必買頭牛 31
- 32. Famous new Companies ∗ 30,000,000 users ∗ Based on Amazon AWS ∗ Django web framework ∗ PostgreSQL database ∗ Memory cache by Redis ∗ Merged by Facebook Quoted from http://instagram-engineering.tumblr.com/post/13649370142/what- powers-instagram-hundreds-of-instances-dozens-of
- 33. Famous new Companies ∗Also based on Amazon AWS
- 34. Cloud Cost ∗ 在矽谷每個月租serverx元， 頻寬x元 在台灣每個月租server 0.5~1x元，頻寬30~40x元!! --- 翟本喬 ∗ 在美國租伺服器,每台每月169~229美元，可是流量超 出我的預期…最後我的信用卡額度每個月3萬美金(約 90萬台幣)才夠用 --- 陳士駿 ∗ 在台灣會更慘，每個月90萬美金(2700萬台幣) 34
- 35. Price ∗ Is Cloud-Servicereally cheaper?? ∗ Depend on your age/finance situations, you rent or buy houses
- 36. General Obstacles andOpportunities in Clouds
- 37. Top 10 Obstaclesand Opportunities for Cloud Computing
- 38. Top 10 Obstaclesand Opportunities for Cloud Computing ∗ 1.Availability/Business Continuity ∗ Q: User/Organization worry about whether utility computing services will have adequate availability or company may even go out of business ∗ A:Multiple and different cloud computing providers
- 39. Top 10 Obstaclesand Opportunities for Cloud Computing ∗ 2.Data Lock-In ∗ Q:The Storage API for cloud computing are still essentially proprietary, cannot easily extract by customers ∗ A: Standardize APIs ;Compatible SW to enable Surge of Hybird of Cloud Computing
- 40. Top 10 Obstaclesand Opportunities for Cloud Computing ∗ 3.Data Confidentiality/Auditability ∗ Q: Cloud user face security threats both from outsides and insides the cloud Outside : any third-party , cloud vender Inside : cloud user ∗ A: cloud user : virtualization ∗ cloud vender : user-level encryption ∗ any third-party : firewall
- 41. Top 10 Obstaclesand Opportunities for Cloud Computing ∗ 4.Data Transfer Bottlenecks ∗ Q : The cost of data transfer is high and transfer rate ∗ is slow because data is in surprising size ∗ A: ship disks
- 42. Top 10 Obstaclesand Opportunities for Cloud Computing ∗ 7.Bugs in large scale distributed systems ∗ Q:Bugs can’t appear in smaller configuration ,but appear in production data center ∗ A:Use distributed VMs
- 43. Top 10 Obstaclesand Opportunities for Cloud Computing ∗ 10.Software Licensing ∗ Q : Cloud provisions pay more money ∗ A : Open source or pay-for-use license ∗ Why open source?? Cost issues in startup teams
- 44.
- 45. Talking about ‘BigData’ 45
- 46. New Data Source ∗The number of smart phone will exceed 1 billion in 2014, as expected
- 47. ∗ The numberof app download is more than 10 billion Quoted from http://android- developers.blogspot.com/search/label/Android%20Market
- 48. Web-Scale Problems It is BIG DATA! ∗ Characteristics: ∗ Definitely data-intensive ∗ May also be processing intensive ∗ Examples: ∗ Crawling, indexing, searching, mining the Web ∗ Social Network ∗ Web 3.0 applications
- 49. ∗ In 2007 the average was 5,000 tweets per day ∗ In 2008 that had grown to 300,000 ∗ In 2009 tweets per day averaged 2.5 million ∗ In 2010 that number was 35 million tweets per day ∗ In the month of March 2011 alone, 140 million tweets are being sent on average per day. http://www.marketinggum.com/twitter-statistics-2011-updated-stats/ 49
- 50. ∗ Twitter isthe top 8 website Quoted from http://www.alexa.com/topsites 50
- 51. Web-Scale Problems It is BIG DATA! http://archive.org/index.php ∗ Wayback Machine has 2 PB + 20 TB/month (2006) ∗ Google processes 20 PB a day (2008) ∗ “all words ever spoken by human beings” ~ 5 EB ∗ NOAA has ~1 PB climate data (2007) ∗ CERN’s LHC will generate 15 PB a year (2008) 640K ought to be enough for anybody. 51
- 52. Quoted from “Nosqlbig data Hadoop with microsoft” 52
- 53. What is thescale of BigData? ∗ We can capture the scale of 300GB, since we have a hard disk more than the size nowaday 53
- 54. What is thescale of BigData? Quoted from “Nosql big data Hadoop with microsoft” 54
- 55. What is thescale of BigData? 55
- 56. Quoted from “bigdata the next frontier for 56 innovation competition and productivity”
- 57. Quoted from “bigdata the next frontier for 57 innovation competition and productivity”
- 58. For Big DataAnalytics ∗ They cannot be solved by a set of machines ∗ Many machines? ∗ Distributed/grid/cluster computing? ∗ We need huge machines! ∗ Less-communication between computers ∗ Less-synchronization systems
- 60. Big Data Initiativein US 60
- 61. Big Data isthe trend Open Its Power! 61
- 62.
- 63.
- 64.
- 65.
- 66. Third-party Cloud Services ∗Play as a web-services to provide Relation Database functionalities ∗ Solve (2) Data Lock-In Issues
- 67.
- 68.
- 69. Traditional Database modelis no longer workable! 69
- 70.
- 71. They are thefuture ∗ We have data and Computing Everywhere! ∗ New terms: M2M, Internet of Things ∗ The IT industry is growing but changing ∗ Software and Idea are more valuable than Hardware and Labor ∗ Small/Diverse/Open-Source Software is more beneficial 71
- 72. They are thefuture ∗ Cross-discipline will be the best way to evolve with the trend ∗ Good to touch Data-Driven Sciences ∗ Data Mining ∗ Since Software is the king, welcome to join us ∗ 9:00~12:00 Thursday ∗ 4204@CSIE Building ∗ Many Talks about software or big data processing from experts in software industries such as Google, Yahoo!, Synopsys, Trend Micro 72
- 73. Q&A ∗ Taiwan Ready? ∗ Our Network environment? ∗ Our Software environment? ∗ Our Creation? ∗ No Matter you like it or not, the surge is coming ∗ Thinking Big for the new Opportunities! 73