Scaling and Managing Big Data Apps in the Cloud
 

Like this? Share it with your network

Share

Scaling and Managing Big Data Apps in the Cloud

on

  • 1,759 views

Title: Scaling and Managing Big Data Apps on Public Clouds ...

Title: Scaling and Managing Big Data Apps on Public Clouds

Abstract: The massive computing and storage resources that are needed to support big data applications make on-demand, elastic cloud environments an ideal fit. However, managing your big data app on the cloud is no walk in the park - configuration, orchestration, H/A, auto-scaling are all quite complex when it comes to choosing the right cloud for you, whether it’s public, private or a hybrid cloud - which is where Cloudify and Eucalyptus come together. In this session, you'll learn how to deploy, manage, monitor and scale your big data apps on the open source Eucalyptus cloud platform using Cloudify, as well as easily test drive your apps locally and then migrate the workload to Amazon Web Services EC2.

Statistics

Views

Total Views
1,759
Views on SlideShare
1,501
Embed Views
258

Actions

Likes
5
Downloads
48
Comments
0

4 Embeds 258

http://cloudifysource.tumblr.com 235
https://twitter.com 21
http://www.kred.com 1
http://webcache.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Reasons why people would be concerned of moving data into the cloud- I suppose one thing you could mention "against" having data in the cloud is the fear of losing control of your data (high cost of transfer, lock in etc)Talk about private/public cloud
  • GigaSpaces Big Data Survey:http://www.gigaspaces.com/sites/default/files/product/BigDataSurvey_Report.pdfForbes on Big Data & Cloud http://www.forbes.com/sites/forrester/2012/08/15/big-data-meets-cloud/38% of all companies from our survey are planning a BI SaaS project before the end of 2013. Many of those respondents (27%) plan to complement their existing BI solutions and a smaller number (11%) actually plan to fully replace their existing BI with a cloud solutionForrester:This year we will hit a volume of 2.7 zettabytes of global digital data ~20% of all tweets include a link that needs to be opened to understand its context.[ii] All tweets from the past two years take 0.5 petabytes to store; it simply doesn’t make sense for every company interested in social media to start storing the same big data in-house.http://www.globaltelecomsbusiness.com/article/3133566/Big-data-becomes-priority-as-executives-tackle-complexity-of-business-analytics.htmlCompanies are most interested in getting access to data in real time (54%), accessing data from multiple devices (51%) and accessing data from remote/flexible locations (44%). Yet, getting access to data in real time emerges as the biggest challenge for companies (52%) along with speed of data delivery (50%); 
• 43% think that data analytics could be improved in their organisation if data analytics was part of cloud services delivered with third-party expertise. 
   
  • Big data requires a spectrum of advanced technologies, skills, and investments. Do you really need/want this all in-house?Big data includes huge amounts of external data. Does it make sense to move and manage all this data behind your firewall?Big data needs a lot of data services. Focus on the value of your differentiated data analysis instead of big data management.http://www.forbes.com/sites/forrester/2012/08/15/big-data-meets-cloud/
  • Consistent Management: Making the deployment, installation, scaling, fail-over looks the same through the entire stack
  • Any app - All clouds

Scaling and Managing Big Data Apps in the Cloud Presentation Transcript

  • 1. @natishalom
  • 2. About GigaSpaces 2
  • 3. The Reality of Big Data.. 2.7 ZB Global Digital Data0.5 Petabytes 43% think that data analytics could be improved in their Two years tweets organization if data analytics was part of cloud services 66%Plan to use Big Data/Cloud 4
  • 4. Large ISV Case Study• Application – Call Center surveillance• Background – Previously – voice data• Goal for a new system – Monitor data & voice – Multiple data sources – Advanced correlations
  • 5. Ever Growing DataDeeper CorrelationTight Performance
  • 6. A Classic Case for...
  • 7. A Typical Big Data System…
  • 8. BusinessCost Impact
  • 9. Big Datain the Cloud
  • 10. Big Data in the Cloud - 3 Reasons • Skills – Do you really need/want this all in- house? • Huge amounts of external dataHolger Kisker – Does it make sense to move and manage all this data behind your firewall? • Focus on the value of your data – Instead of big data management
  • 11. • Auto start VMs• Install and configure app components• Monitor• Repair• (Auto) Scale• Burst…
  • 12. Use Eucalyptus for privatedata , AWS for sporadicworkloads ..
  • 13. • Consistent Management• Automation Through the Entire Stack
  • 14. Consistent Management Recipes consistent description for running any app:  What middleware services to run  Dependencies between services  How to install services  Where application and service binaries are  When to spawn or terminate instances  How to monitor each of the services 16
  • 15. Choosing the Right Cloud for the Job compute { template "SMALL_LINUX" }SMALL_LINUX : template{ SMALL_LINUX : template imageId linuxImageId imageId "us-east-1/ami-76f0061f“ remoteDirectory "/home/user/gs-files" remoteDirectory "/home/ec2-user/gs-files“ machineMemoryMB 1600 machineMemoryMB 1600 hardwareId “m1.medium” hardwareId "m1.small" locationId “us-west-1” locationId "us-east-1" localDirectory "upload“ localDirectory "upload" keyFile “myEucaKeyFile.pem” keyFile "myKeyFile.pem" username "user" options ([ options ([ "securityGroups" : ["default"]as "securityGroups" : ["default"] as String[],String[], "keyPair" : "myKeyFile" "keyPair" : keyPair ]) ]) overrides (["jclouds.ec2.ami-query":"", overrides ([“endpoint” : "jclouds.ec2.cc-ami-query":""])“http://communitycloud.eucalyptus.com”]) privileged true privileged true }} 18
  • 16. Automation across the stack 1 Upload your recipe. 2 Cloudify creates VM’s & installs agents 3 Agents install and manage your app 4 Cloudify automate the scaling
  • 17. Big Data On Demand with CloudifyRelational DB Clusters NoSQL Clusters HadoopMySQL MongoDB Hadoop (Hive, Pig,..)Postgress Cassandra Storm Couchbase ZooKeeper ElasticSearch
  • 18. Demo Time: Storm Cluster ® Copyright 2011 Gigaspaces Ltd. All Rights 22 Reserved
  • 19. Large ISV Case Study• Application – Call Center surveillance system• Background – Previously – voice data• Goal for a new system Monitor data & voice Multiple data sources Advanced correlations Mission Accomplished
  • 20. Additional Benefits • True Cloud Economics • One product -> Any Customer Environment • Increased Agility
  • 21. http://www.cloudifysource.orghttp://github.com/CloudifySource