• Like

Big Data in the Cloud

  • 1,024 views
Uploaded on

Adopting Hadoop to manage your Big Data is an important step, but not the end-solution to your Big Data challenges. Here are some of the additional considerations you must face: …

Adopting Hadoop to manage your Big Data is an important step, but not the end-solution to your Big Data challenges. Here are some of the additional considerations you must face:
Choosing the right cloud for the job: The massive computing and storage resources that are needed to support Big Data applications make cloud environments an ideal fit, and more than ever, there is a growing number of choices of cloud infrastructure types and providers. Given the diverse options, and the dynamic environments involved, it becomes ever more important to maintain the flexibility for all your IT needs.
Big Data is a complex beast: It involves many and different moving parts, in large clusters, and is continually growing and evolving. Managing such an environment manually is not a viable option. The question is, how can you achieve automation of all this complexity?
The world beyond Hadoop: Big Data is not just Hadoop – there is a whole rapidly growing ecosystem to contend with, including NoSQL, data processing, analytics tools… As well as your own application services. How can you manage deployment, configuration, scaling and failover of all the different pieces, in a consistent way?
In this session, you’ll learn how to deploy and manage your Hadoop cluster on any Cloud, as well as manage the rest of your big data application stack using a new open source framework called Cloudify.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,024
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
43
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • GigaSpaces Big Data Survey:http://www.gigaspaces.com/sites/default/files/product/BigDataSurvey_Report.pdfForbes on Big Data & Cloud http://www.forbes.com/sites/forrester/2012/08/15/big-data-meets-cloud/38% of all companies from our survey are planning a BI SaaS project before the end of 2013. Many of those respondents (27%) plan to complement their existing BI solutions and a smaller number (11%) actually plan to fully replace their existing BI with a cloud solutionForrester:This year we will hit a volume of 2.7 zettabytes of global digital data ~20% of all tweets include a link that needs to be opened to understand its context.[ii] All tweets from the past two years take 0.5 petabytes to store; it simply doesn’t make sense for every company interested in social media to start storing the same big data in-house.http://www.globaltelecomsbusiness.com/article/3133566/Big-data-becomes-priority-as-executives-tackle-complexity-of-business-analytics.htmlCompanies are most interested in getting access to data in real time (54%), accessing data from multiple devices (51%) and accessing data from remote/flexible locations (44%). Yet, getting access to data in real time emerges as the biggest challenge for companies (52%) along with speed of data delivery (50%); 
• 43% think that data analytics could be improved in their organisation if data analytics was part of cloud services delivered with third-party expertise. 
   
  • Big data requires a spectrum of advanced technologies, skills, and investments. Do you really need/want this all in-house?Big data includes huge amounts of external data. Does it make sense to move and manage all this data behind your firewall?Big data needs a lot of data services. Focus on the value of your differentiated data analysis instead of big data management.http://www.forbes.com/sites/forrester/2012/08/15/big-data-meets-cloud/
  • Consistent Management: Making the deployment, installation, scaling, fail-over looks the same through the entire stack
  • Any app - All clouds

Transcript

  • 1. @natishalom
  • 2. About GigaSpaces 2
  • 3. Big Data In The Cloud 2.7 ZB Global Digital Data 0.5Petabytes 43% think that data analytics could be improved in their Two years tweets organization if data analytics was part of cloud services 66% Will or plan to use Big Data in the cloud 3
  • 4. Large ISV Case Study• Application – Call Center surveillance• Background – Previously – voice data• Goal for a new system – Monitor data & voice – Multiple data sources – Advanced correlations
  • 5. Ever Growing DataDeeper CorrelationTight Performance
  • 6. A Classic Case for..
  • 7. A Typical Big Data System…
  • 8. BusinessCost Impact
  • 9. Big Datain the Cloud
  • 10. Big Data in the Cloud- 3 Reasons • Skills – Do you really need/want this all in- house? • Huge amounts of external data.Holger Kisker – Does it make sense to move and manage all this data behind your firewall? • Focus on the value of your data – Instead of big data management.
  • 11. • Auto start VMs• Install and configure app components• Monitor• Repair• (Auto) Scale• Burst…
  • 12. Running Bare-Metal forhigh I/O workloads, Publiccloud for sporadicworkloads ..
  • 13. • Consistent Management• Automation Through the Entire Stack
  • 14. http://code.zynga.com/2012/02/the-evolution-of-zcloud/
  • 15. http://code.mixpanel.com/2010/11/08/amazon-vs-rackspace/
  • 16. Realization: What You Really Care about Is App Portability
  • 17. Consistent Management Recipes consistent description for running any app:  What middleware services to run  Dependencies between services  How to install services  Where application and service binaries are  When to spawn or terminate instances  How to monitor each of the services. ® Copyright 2012 GigaSpaces Ltd. All Rights 25 Reserved
  • 18. Choosing the Right Cloud for the Job compute { template "SMALL_LINUX" }SMALL_LINUX : template{ SMALL_LINUX : template imageId "1234" imageId "us-east-1/ami-76f0061f“ machineMemoryMB 3200 remoteDirectory "/home/ec2-user/gs-files“ hardwareId "103" machineMemoryMB 1600 remoteDirectory "/root/gs-files" hardwareId "m1.small" localDirectory "upload" locationId "us-east-1" keyFile "gigaPGHP.pem" localDirectory "upload" options ([ keyFile "myKeyFile.pem" "openstack.securityGroup" : "default", "openstack.keyPair" : "gigaPGHP" options ([ ]) "securityGroups" : ["default"]as privileged true String[],} "keyPair" : "myKeyFile" ]) overrides (["jclouds.ec2.ami-query":"", "jclouds.ec2.cc-ami-query":""]) privileged true } ® Copyright 2012 GigaSpaces Ltd. All Rights 27 Reserved
  • 19. Automation across the stack 1 Upload your recipe. 2 Cloudify creates VM’s & installs agents 3 Agents install and manage your app 4 Cloudify automate the scaling
  • 20. Demo Time..
  • 21. Try a simple demo yourselflaunch.cloudifysource.org/d
  • 22. Large ISV Case Study• Application – Call Center surveillance system• Background – Previously – voice data• Goal for a new system Monitor data & voice Multiple data sources Advanced correlations Mission Accomplished
  • 23. Additional Benefits • True Cloud Economics • One product -> any Customer Environment • Increased Agility
  • 24. http://www.cloudifysource.orghttp://github.com/CloudifySource