Big Data is growing rapidly in terms of volume, variety, and velocity. The cloud is well-suited to handle Big Data challenges by providing elastic and scalable infrastructure, which optimizes resources and reduces costs compared to traditional IT. In the cloud, users can collect, store, analyze and share large amounts of data without upfront investment, and scale easily as needs change. Real-world examples show how companies in industries like banking, retail, and advertising are using the cloud's Big Data services to gain insights from large datasets.

1. BIG Data on AWS
Paul Duffy

2. Characteristics of
Big Data
How the Cloud Is
Big Data’s Best Friend
Big Data on the Cloud
In the Real World

3. Characteristics of Big Data

4. The cost of data generation is falling rapidly
Dramatic increase in volume, velocity and
variety of data

5. BIG DATA
A collection of tools, techniques and technologies that
allow you to work productively with data at any scale.

6. Big Data is Getting Bigger
2.7 Zetabytes in 2012
Over 90% will be
unstructured
Data spread across a wide
array of silos

7. Features driven by MapReduce

8. Variable data structures and sources
Computer Generated Human Generated
• Application server logs • Twitter “Fire Hose” 50m
(web sites, games) tweets/day 1,400%
• Sensor data (weather, growth per year
water, smart grids) • Blogs/Reviews/Emails/P
• Images/videos (traffic, ictures
security cameras) • Social Graphs:
Facebook, Linked-in,
Contacts

9. The Role of Data
is Changing

10. Traditional analytics required a
fixed data model,
based on pre-known questions
Big Data promotes data exploration and
experimentation which leads to innovation

11. Collection & Computation Collaboration
Generation
storage & analytics & sharing

12. Lower costs,
faster throughput
Collection & Computation Collaboration
Generation
storage & analytics & sharing
Increased pressure on traditional IT and tools

13. Require tools designed for data
collection and computation at
any volume, velocity or format.

14. Software
• Designed for distribution
• Easy programming models
• Flexible language choice
• Platform for abstraction and ecosystem
• Good example: Hadoop

15. Infrastructure
• Designed for distribution
• Easy programming models
• Flexible language choice
• Platform for abstraction and ecosystem
• Good example: Cloud computing

16. Software
Infrastructure

17. How the Cloud Is
Big Data’s Best Friend

18. How do we define the cloud?
By Benefits!

19. No Cap Ex
Pay Per
Elasticity
Use
Cloud
Fast Time to Market Focus on core
competency

20. Why is the Cloud
Big Data’s Best Friend?

21. We know we want collect, store, organize, analyze and
share it.
But we have limited resources.

22. The Cloud Optimizes
Precious IT Resources
i.e. Skilled People

23. “Over the next decade, the number of files or containers that
encapsulate the information in the digital universe will grow by
75x.
While the pool of IT staff available to manage them will grow
only slightly. At 1.5x”
- 2011 IDC Digital Universe Study

24. Deploying a Hadoop cluster is hard

25. Cloud computing
30% 70%
The Old Managing All of the
IT World Using Big Data
“Undifferentiated Heavy Lifting”

26. Cloud computing
30% 70%
The Old Managing All of the
IT World Using Big Data
“Undifferentiated Heavy Lifting”
Cloud-Based Configuring
Infrastructure Analyzing and Using Big Data
Cloud Assets
70% 30%

27. Managed
Reusability
Services
Scale Innovation

28. Managed
Reusability
Services
Scale Innovation

29. Managed
Reusability
Services
Scale Innovation

30. Managed
Reusability
Services
Scale Innovation

31. Managed
Reusability
Services
Scale Innovation

32. The Cloud Optimizes
Capacity Resources

33. Elastic Compute Capacity
On and Off Fast Growth
Variable peaks Predictable peaks

34. Elastic Compute Capacity
WASTE
On and Off Fast Growth
Variable peaks Predictable peaks
CUSTOMER DISSATISFACTION

35. Elastic Compute Capacity
Capacity Traditional
IT capacity
Elastic cloud capacity
Time
Your IT needs

36. Elastic Compute Capacity
On and Off Fast Growth
Variable peaks Predictable peaks

37. The Cloud Empowers Users
to Balance Cost and Time

38. 1 instance for 500 hours
=
500 instances for 1 hour
I like this!
I scale

39. The Cloud
Reduces Cost
For Experimentation

40. The Cloud
Enables Collection and Storage
of Big Data

41. Storage Costs are Declining

42. Simple Storage Service
1 Trillion
1000,000
750,000
500,000
250,000
0,000
750k+ peak transactions per second

43. Global Accessibility
Region
US-WEST (N. California) EU-WEST (Ireland)
GOV CLOUD ASIA PAC (Tokyo)
US-EAST (Virginia)
US-WEST (Oregon)
ASIA PAC
(Singapore)
SOUTH AMERICA (Sao Paulo)

44. Amazon DynamoDB
Managed NoSQL database service
Unlimited size
Unlimited scale
Flexible key/value store
Consistent, low latencies (single digit milliseconds, SSD)
Robust, durable data storage
Integrated analytics with Elastic MapReduce

45. Amazon Elastic MapReduce
On-demand, managed analytics platform
Powered by Hadoop
Integrated with Spot instances to lower costs
Vibrant ecosystem of tools
Elastic clusters
Flexible programming model (Java, Python, Ruby etc)

46. Big Data on the Cloud
In the Real World

47. Big Data Verticals
Social
Media/Advertisi Financial
Oil & Gas Retail Life Sciences Security Network/Gamin
ng Services
g
User
Anti-virus
Targeted Monte Carlo Demographics
Recommend
Advertising Simulations
Seismic Genome Fraud
Usage analysis
Analysis Analysis Detection
Image and
Transactions
Video Risk Analysis
Processing Analysis Image In-game
Recognition metrics

48. Visualizations

49. Bank – Monte Carlo Simulations
“The AWS platform was a good fit for its
unlimited and flexible computational power to
23 Hours to our risk-simulation process requirements.
With AWS, we now have the power to decide
20 Minutes how fast we want to obtain simulation
results, and, more importantly, we have the
ability to run simulations not possible before
due to the large amount of infrastructure
required.” – Castillo, Director, Bankinter

50. Recommendations
The Taste Test http://www.etsy.com/tastetest

51. Recommendations
Gift Ideas for Facebook Friends
etsy.com/gifts

53. Click Stream Analysis
User recently
purchased a
sports movie and Targeted Ad
is searching for (1.7 Million per day)
video games

54. Characteristics of
Big Data
How the Cloud Is
Big Data’s Best Friend
Big Data on the Cloud
In the Real World

55. Thank you…