Your SlideShare is downloading. ×
0
@natishalom
About GigaSpaces
The Reality of Big Data…2.7 ZB0.5 Petabytes66%Global Digital DataTwo years’ tweetsPlan to use Big Data/Cloud43%think that ...
Large ISV Case Study• Application– Call Center surveillance• Background– Previously – voice data• Goal for a new system– M...
Ever Growing DataDeeper CorrelationTight Performance
A Classic Case for..
A Typical Big Data System
CostBusinessImpact
Big Datain the Cloud
Big Data in the Cloud- 3 Reasons• Skills– Do you really need/want this all in-house?• Huge amounts of external data– Does ...
• Auto start VMs• Install and configureapp components• Monitor• Repair• (Auto) Scale• Burst…
Running Bare-Metal forhigh I/O workloads, Publiccloud for sporadicworkloads
• ConsistentManagement• Automation Throughthe Entire Stack
Reducing the Complexity17My RecipesWrap all your system elements into easy-to-userecipes, providing you with consistent, a...
Reducing the Infrastructure CostConsistent ManagementAbstractionTypical Big Data SystemCreates an abstraction between your...
What about-Performance?-Deterministic Latency?
Bare Metal vs. Virtualization BenchmarkSource: Petestrenna8.84%14.36%24.46%2.41X10.84XDisk I/OCPU and MemoryNetwork I/ODis...
Bare Metal vs. Virtualization BenchmarkSource: NTT DOCOMO
3Xmore compute resourcesfor the same workload!Non Deterministic Latency
Bare Metal OpenStack Support
“We took this single image, picked it upfrom public cloud into a Rackspace-powered private cloud and saw a4X increased eff...
Automation FrameworksConfiguration Centric APP Centric (PaaS)
Built-in Support for Big Data StacksReal Time Relational DBClustersNoSQL Clusters HadoopStorm MySQL MongoDB Hadoop (Hive,P...
Moving from Existing Data Center toOpenStack?Consistent ManagementNon Virtualized Data Center OpenStack CloudCloud Driver
Storm on OpenStack
BigData Services Catalogue on OpenStack (HP)
Large ISV Case Study• Application– Call Center surveillance system• Background– Previously – voice data• Goal for a new sy...
Additional Benefits• True Cloud Economics• One product -> anyCustomer Environment• Increased Agility
http://www.cloudifysource.orghttp://github.com/CloudifySource
Additional References• Bare Metal Cloud/PaaS• OpenStack Baremetal Project• Big Data in the Cloud• Big Data in the Cloud us...
Big Data on OpenStack
Big Data on OpenStack
Big Data on OpenStack
Big Data on OpenStack
Big Data on OpenStack
Upcoming SlideShare
Loading in...5
×

Big Data on OpenStack

1,471

Published on

The massive computing and storage resources that are needed to support big data applications make cloud environments an ideal fit. In this session, you'll learn how to build your big data "database on-demand" using MongoDB, Cassandra, Solr, MySQL, or any other big data solution, as well as manage your big data application using a new open source framework called “Cloudify.” All this, on top of the OpenStack cloud.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,471
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
77
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Reasons why people would be concerned of moving data into the cloud- I suppose one thing you could mention "against" having data in the cloud is the fear of losing control of your data (high cost of transfer, lock in etc)Talk about private/public cloud
  • GigaSpaces Big Data Survey:http://www.gigaspaces.com/sites/default/files/product/BigDataSurvey_Report.pdfForbes on Big Data & Cloud http://www.forbes.com/sites/forrester/2012/08/15/big-data-meets-cloud/38% of all companies from our survey are planning a BI SaaS project before the end of 2013. Many of those respondents (27%) plan to complement their existing BI solutions and a smaller number (11%) actually plan to fully replace their existing BI with a cloud solutionForrester:This year we will hit a volume of 2.7 zettabytes of global digital data ~20% of all tweets include a link that needs to be opened to understand its context.[ii] All tweets from the past two years take 0.5 petabytes to store; it simply doesn’t make sense for every company interested in social media to start storing the same big data in-house.http://www.globaltelecomsbusiness.com/article/3133566/Big-data-becomes-priority-as-executives-tackle-complexity-of-business-analytics.htmlCompanies are most interested in getting access to data in real time (54%), accessing data from multiple devices (51%) and accessing data from remote/flexible locations (44%). Yet, getting access to data in real time emerges as the biggest challenge for companies (52%) along with speed of data delivery (50%); 
• 43% think that data analytics could be improved in their organisation if data analytics was part of cloud services delivered with third-party expertise. 
   
  • Big data requires a spectrum of advanced technologies, skills, and investments. Do you really need/want this all in-house?Big data includes huge amounts of external data. Does it make sense to move and manage all this data behind your firewall?Big data needs a lot of data services. Focus on the value of your differentiated data analysis instead of big data management.http://www.forbes.com/sites/forrester/2012/08/15/big-data-meets-cloud/
  • Consistent Management: Making the deployment, installation, scaling, fail-over looks the same through the entire stack
  • References:http://serverfault.com/questions/261974/how-much-overhead-does-x86-x64-virtualization-havehttp://petersenna.com/en/projects/81-performance-overhead-and-comparative-performance-of-4-virtualization-solutions
  • References:http://serverfault.com/questions/261974/how-much-overhead-does-x86-x64-virtualization-havehttp://petersenna.com/en/projects/81-performance-overhead-and-comparative-performance-of-4-virtualization-solutions
  • http://gigaom.com/2013/04/16/top-5-lessons-learned-at-openstack-summit/
  • Transcript of "Big Data on OpenStack"

    1. 1. @natishalom
    2. 2. About GigaSpaces
    3. 3. The Reality of Big Data…2.7 ZB0.5 Petabytes66%Global Digital DataTwo years’ tweetsPlan to use Big Data/Cloud43%think that theirorganization’s data analytics could beimproved if data analytics was part ofcloud services
    4. 4. Large ISV Case Study• Application– Call Center surveillance• Background– Previously – voice data• Goal for a new system– Monitor data & voice– Multiple data sources– Advanced correlations
    5. 5. Ever Growing DataDeeper CorrelationTight Performance
    6. 6. A Classic Case for..
    7. 7. A Typical Big Data System
    8. 8. CostBusinessImpact
    9. 9. Big Datain the Cloud
    10. 10. Big Data in the Cloud- 3 Reasons• Skills– Do you really need/want this all in-house?• Huge amounts of external data– Does it make sense to move andmanage all this data behind yourfirewall?• Focus on the value of your data– Instead of big data managementHolger Kisker
    11. 11. • Auto start VMs• Install and configureapp components• Monitor• Repair• (Auto) Scale• Burst…
    12. 12. Running Bare-Metal forhigh I/O workloads, Publiccloud for sporadicworkloads
    13. 13. • ConsistentManagement• Automation Throughthe Entire Stack
    14. 14. Reducing the Complexity17My RecipesWrap all your system elements into easy-to-userecipes, providing you with consistent, automatedmanagement of your Big DataConsistent ManagementTypical Big Data SystemScaleMonitorUpdateDeployOne manager easily &consistently handles allsystem functions.
    15. 15. Reducing the Infrastructure CostConsistent ManagementAbstractionTypical Big Data SystemCreates an abstraction between your Big Data systemrecipe/blueprint and the target environment. This meansyou can take the same blueprint and simply point it atdifferent environments without making any changes toyour application.Testing ProductionDevelopmentClientEnvironmentScaleMonitorUpdateDeploy
    16. 16. What about-Performance?-Deterministic Latency?
    17. 17. Bare Metal vs. Virtualization BenchmarkSource: Petestrenna8.84%14.36%24.46%2.41X10.84XDisk I/OCPU and MemoryNetwork I/ODisk LatencyMicro-operations
    18. 18. Bare Metal vs. Virtualization BenchmarkSource: NTT DOCOMO
    19. 19. 3Xmore compute resourcesfor the same workload!Non Deterministic Latency
    20. 20. Bare Metal OpenStack Support
    21. 21. “We took this single image, picked it upfrom public cloud into a Rackspace-powered private cloud and saw a4X increased efficiency running thatworkload.”Jim O’Neill CIO at HubSpot
    22. 22. Automation FrameworksConfiguration Centric APP Centric (PaaS)
    23. 23. Built-in Support for Big Data StacksReal Time Relational DBClustersNoSQL Clusters HadoopStorm MySQL MongoDB Hadoop (Hive,Pig,..)GigaSpaces XAP Postgress Cassandra ZooKeeperCouchbaseElasticSearch
    24. 24. Moving from Existing Data Center toOpenStack?Consistent ManagementNon Virtualized Data Center OpenStack CloudCloud Driver
    25. 25. Storm on OpenStack
    26. 26. BigData Services Catalogue on OpenStack (HP)
    27. 27. Large ISV Case Study• Application– Call Center surveillance system• Background– Previously – voice data• Goal for a new systemMonitor data & voiceMultiple data sourcesAdvanced correlations MissionAccomplished
    28. 28. Additional Benefits• True Cloud Economics• One product -> anyCustomer Environment• Increased Agility
    29. 29. http://www.cloudifysource.orghttp://github.com/CloudifySource
    30. 30. Additional References• Bare Metal Cloud/PaaS• OpenStack Baremetal Project• Big Data in the Cloud• Big Data in the Cloud using Cloudify• Putting Hadoop On Any Cloud (A video presentation)• In Memory Computing (Data Grid) for Big Data• Using the Cloudify Player as an Open Source Framework forBuilding Your Own Cloud Application Marketplace on OpenStack• Going native: The move to bare-metal cloud services• New bare metal cloud offerings emerging• How much overhead does x86/x64 virtualization have?• Amazon EC2 versus Bare Metal and KVM? The inside story on whatyou thought you knew about EC2
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×