Your SlideShare is downloading. ×
0
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Journey Through the AWS Cloud; Application Services

4,401

Published on

The fourth in our series of webinars, 'Journey Through the AWS Cloud'. This complimentary presentation discusses the use of services offered by AWS that alleviate the need for you to install and …

The fourth in our series of webinars, 'Journey Through the AWS Cloud'. This complimentary presentation discusses the use of services offered by AWS that alleviate the need for you to install and manage software on EC2 instances. We introduce the key services customers employ to keep them focused on developing their applications, whilst AWS takes care of running the scalable and reliable building blocks upon which they are built.

Published in: Technology
1 Comment
4 Likes
Statistics
Notes
  • Recording can be found here: http://www.youtube.com/watch?v=jaTbZBXMMqE
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
4,401
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
134
Comments
1
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Journey through the Cloud: Application Services Ryan Shuttleworth – Technical Evangelist @ryanAWS
  • 2. Journey through the cloudCommon use cases & stepping stones into the AWS cloud Learning from customer journeys Best practices to bootstrap your projects
  • 3. Application Services Build upon services built for the cloud Address common pain points in application architectures Reduce operational management of software componentsFocus on application function, not undifferentiated heavy lifting
  • 4. AgendaWhy AWS for application servicesServices overviewDive into select servicesWhere to go next
  • 5. Why AWS for application services?
  • 6. Application ServicesServices that wrap software you’d commonly install and manage yourself
  • 7. Application ServicesDatabases AnalyticsMiddleware Frameworks
  • 8. Application ServicesScalability Reliability Availability
  • 9. Application ServicesScalability Reliability Availability Do you focus on Does core Is middlewarescaling frameworks component ‘glue’ difficult to rather than reliability affect make as optimizing your your application available as you code? track record? need it?
  • 10. Application Services Operational Management
  • 11. Application Services Operational Management Do you spend more time managing application services than you do building and managing assets core to your business?
  • 12. 30% 70% On-Premise Your Managing All of theInfrastructure Business “Undifferentiated Heavy Lifting”
  • 13. 30% 70% On-Premise Your Managing All of theInfrastructure Business “Undifferentiated Heavy Lifting” AWS More Time to Focus on Configuring Your Cloud-Based Your Business Cloud AssetsInfrastructure 70% 30%
  • 14. Infrastructure & Compute ScalingApplication Services Security CDN Backup DNS Database Building blocks for applications Storage Load BalancingDesigned and built for the cloud Workflow MonitoringAvailable at end of a web service call Networking Messaging
  • 15. Storage & Archive AWS is used in a variety of ways… By managing time-consuming Brought on massive DynamoDB database administration tasks, capacity in an operationally short RDS allows SEGA to focus on period of time to handle demand business critical applications peaks during SuperBowlSaved months of development & Relies on Simple Workflow toarchitecture time and focused on orchestrate complex, heterogeneousapplication development instead scientific workflows by using Cloud Search
  • 16. Business & technical drivers You might be able to:
  • 17. Business & technical drivers You might be able to: Reduce costsRunning software on EC2 can bemore expensive than consuming functionality as a service
  • 18. Business & technical drivers You might be able to: Reduce costs Improve reliabilityRunning software on EC2 can be Services built with inherent multi-more expensive than consuming AZ functionality improve functionality as a service application reliability
  • 19. Business & technical drivers You might be able to: Reduce costs Improve reliabilityRunning software on EC2 can be Services built with inherent multi-more expensive than consuming AZ functionality improve functionality as a service application reliability Scale easilyServices scale as you grow and need them, without large investments in infrastructure
  • 20. Business & technical drivers You might be able to: Reduce costs Improve reliabilityRunning software on EC2 can be Services built with inherent multi-more expensive than consuming AZ functionality improve functionality as a service application reliability Scale easily Re-focus energiesServices scale as you grow and Spend less time doing need them, without large undifferentiated heavy lifting and investments in infrastructure more time on your business
  • 21. Q Instance Availability Zone Decoupled Reliableapplications on messageEC2 instances queue on instance Region
  • 22. Q Q Q Instance Instance InstanceAvailability Zone Availability Zone Availability Zone Multiple Technical instances implementation for reliability Region
  • 23. Q Q Q Instance Instance Instance Availability Zone Availability Zone Availability Zone Remove software Regionrunning on EC2
  • 24. Simplified operations Availability Zone Availability Zone Availability ZoneReduced costs Amazon Simple Queue Service (SQS) Region Replace with a regional service
  • 25. Queuing on AWS Setup & manage instances Install & configure queuing middlewareSet up persistent message storeImplement clustering across AZs for HA Implement queue monitoring systems Implement queuing in applications
  • 26. Queuing on AWS Amazon SQS Setup & manage instances Create a queue Install & configure queuing HTTP PUT to place messages middleware HTTP GET (long polling) to pullSet up persistent message store messagesImplement clustering across AZs for HA Implement queue monitoring systems Implement queuing in applications
  • 27. AWS application services
  • 28. Databases Middleware Frameworks Analytics Relational ElastiCache Elastic Database CloudSearch MapReduce Service Simple Simple Queuing Email SimpleDB Service Service Simple Notification CloudWatch DynamoDB Service Simple Workflow
  • 29. Databases Middleware Frameworks Analytics Relational ElastiCache Elastic Database CloudSearch MapReduce Service Simple Simple Queuing Email SimpleDB Service Service Simple Notification CloudWatch DynamoDB Service Simple Workflow
  • 30. Amazon Relational Database Service
  • 31. Source: Forrester Security planning License training Backup, recovery load and unload Script automationPerformance Installation, upgrade,and tuning patching, migration
  • 32. Migration Backup and Schema design recovery Patching Query construction Configuration Software upgradesFrequent server upgrades Storage upgrades Query optimization Hardware crash
  • 33. Migration Focus on Backup and these Schema design recovery things Patching Query construction Configuration Software upgradesFrequent server upgrades Storage upgrades Query optimization Hardware crash
  • 34. Migration Backup and Schema design recovery Patching Instead of Query construction these Configuration Software upgradesFrequent server upgrades Storage upgrades Query optimization Hardware crash
  • 35. Near zero administration Painless patching, automatic upgrades Cloudwatch monitoring, metric alarms One click. High Availability.
  • 36. One click.High availability with Multi-AZ Automated deployment across multiple AZs Synchronous replication from master to replica Automatic fail-over; replica promoted to master Test fail-over
  • 37. Push-button scale, high performance Scale storage from 5Gb to 1Tb of storage Scale instance from small to 4XL (better I/O) Add Read Replicas with asynchronous replication Add ElastiCache for performance
  • 38. Relational Database Service Database-as-a-Service No need to install or manage database instances Scalable and fault tolerant configurations Feature Details Platform support Create MySQL, SQL Server and Oracle RDBMS Preconfigured Get started instantly with sensible default settings Automated patching Keep your database platform up to date automatically Backups Automatic backups and point in time recovery and full DB backups Backups Volumes can be snapshotted for point in time restore Failover Automated failover to slave hosts in event of a failure Replication Easily create read-replicas of your data and seamlessly replicate data across availability zones
  • 39. RDBMS on AWS Setup & manage instances Install & configure database platform Configure backupsImplement master-slave for HA Implement read-replicas for performanceManage maintenance updates
  • 40. RDBMS on AWS RDS Setup & manage instances Create RDS instance Install & configure database Select Multi-AZ platform Choose backup period Configure backups Choose maintenance windowsImplement master-slave for HA Implement read-replicas for performanceManage maintenance updates
  • 41. Amazon Relational Database Service (Amazon RDS) databases stores forum threads, site content, and project configuration data.High availability Multi-AZ database deployment to handle live game metadata and user-generated content.Enterprise-grade fault tolerance for protecting customer data.By managing time-consuming database administration tasks, Amazon RDS allows SEGA to focus on business critical applications.
  • 42. Amazon DynamoDB
  • 43. Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Scalability
  • 44. Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Reality: performance degrades with scale Scalability
  • 45. Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Hardware provisioning Data sharding Data caching Reality: performance Cluster management degrades with scale Fault management Scalability
  • 46. Dial it up DynamoDB Provisioned read/write performance per table Predictable high performance scaled via console or API
  • 47. Low provisioned throughput Table Partition SSD RegionIllustrative diagram only
  • 48. Low provisioned throughput Replica Table Replica Partition Partition Partition SSD SSD SSD Availability Zone Availability Zone Availability Zone RegionIllustrative diagram only
  • 49. Low provisioned throughput Table Partition SSD RegionIllustrative diagram only
  • 50. Increased provisioned throughput Table Table Table Table Table Partition Partition Partition Partition Partition SS SS SS SS SS D D D D D Table Table Table Table Table Partition Partition Partition Partition Partition SS SS SS SS SS D D D D D RegionIllustrative diagram only
  • 51. High provisioned throughput Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition RegionIllustrative diagram only
  • 52. DynamoDB Feature DetailsProvisioned throughput NoSQL Provisioned Dial up or down provisioned read/writedatabase throughput capacityFast, predictable performance Predictable Average single digit millisecond latenciesFully distributed, fault tolerant performance from SSD backed infrastructurearchitecture Strong consistency Be sure you are reading the most up to date values Fault tolerant Data replicated across availability zones Monitoring Integrated to Cloud Watch Secure Integrates with AWS Identity and Access Management (IAM) Elastic Integrates with Elastic MapReduce for MapReduce complex analytics on large datasets
  • 53. NoSQL on AWSPerform capacity planning and determine nodes required Define PIOPS EBS volumes & instance types Setup & manage instancesInstall NoSQL database software Configure replica sets Implement auto-scaling Define tables Define key strategy
  • 54. NoSQL on AWS DynamoDBPerform capacity planning and Define tables determine nodes required Define key strategy Define PIOPS EBS volumes & Choose performance level instance types Setup & manage instancesInstall NoSQL database software Configure replica sets Implement auto-scaling Define tables Define key strategy
  • 55. “AWS gave us the flexibility to bring a massive amount of capacity online in a short period of DynamoDB: time and allowed us to do so in an operationally over 500,000 writes per straightforward way. second AWS is now Shazam’s cloud provider of choice,” Amazon EMR:more than 1 million writes Jason Titus, per second CTO
  • 56. Amazon Simple Workflow
  • 57. Simple Workflow 1 2 Task A 3 Task B (Auto-scaling)Flow framework for simplification ofcross system task coordination Task CLong running transaction state and taskdistribution
  • 58. A typicalbusinessworkflow…Multiple steps Update Update € Ship SendMultiple decision Inventory $ or €? account order Emailpoints New Order Update $ account
  • 59. A typicalbusinessworkflow… System A System C System System EMultiple steps Update Update € D Ship SendMultiple decision Inventory $ or €? account order EmailpointsHeterogeneous New Ordersystems Update $ account System B
  • 60. A typicalbusiness Process logic Middlewareworkflow…State managed in System A System C System D System Eend systems Update Update € Ship Send Inventory $ or €? account order EmailProcess logicembedded in New OrderapplicationsComplex queuing,message ordering,de-duplication anddependencies State Update $Scaling and failover accountof tasks troublesome System B
  • 61. Implemented in Workflow metadata & stateSimpleWorkflow… DeciderImplement a‘Decider’ with simple, Update Update € Ship Send Inventory $ or €? accountlinear decision logic order EmailProcesses/tasks in a New Orderworkflow become‘Workers’All state & metadatais handled by highly Workeravailable and durable Update $AWS ‘Workflow’ account
  • 62. Amazon SimpleWorkflow Service (SWF)
  • 63. DeciderGet Decision Task Return Result Amazon Simple Workflow Service (SWF)
  • 64. Decider Get Decision Task Return Result Amazon Simple Workflow Service (SWF)Get Activity Task Return Result Worker
  • 65. Workflows on AWS Setup & manage instances Install & configure workflow middlewareArchitect for high availability Implement workflows in proprietary/complicated languages Implement process audit
  • 66. Workflows on AWS Simple Workflow Setup & manage instances Implement ‘decider’ in language Install & configure workflow of choice middleware Implement task ‘workers’ toArchitect for high availability consume work Implement workflows in proprietary/complicated languages Implement process audit
  • 67. Amazon CloudSearch
  • 68. select *from articleswhere contentlike ‘%cloud%’
  • 69. 1 million hits a day? 5 million items?
  • 70. Finding things is trickier than first appears
  • 71. +
  • 72. Anatomy of a search Results & Ranking Tokenization FieldedFacets
  • 73. Endpointsdoc-mydomain.us-east-1.cloudsearch.amazonaws.com search-mydomain.us-east-1. cloudsearch.amazonaws.com
  • 74. Testing Query String FacetsResults
  • 75. Integration Domain http://search-toilets-in-oz-m6gko2xujhti47jv5w62eldyoy.us- east-1.cloudsearch.amazonaws.com/2011-02-01/search ?q=True+Blue &return-fields= address1 %2Caddressnote %2Cf__name %2Clastupdatedate %2Clatitude %2Clongitude Query String %2Cnotes %2Cpostcode %2Ctoiletdetails_id %2Ctoileturl Search Options %2Ctext_relevance
  • 76. Integration Search Results { "rank":"-text_relevance", "match-expr":"(label True Blue)", "hits":{"found":1, "start":0, "hit":[{"id":"toiletdetails_csv_41", "data":{"address1":["51-59 Samwell Street"], "addressnote":["3 male and 3 female toilets"], "f__name":["Croydon True Blue Visitor"], "lastupdatedate":["2010-03-04Z"], "latitude":["-18.204249"], "longitude":["142.244534"], "notes":[], "postcode":["4871"], "text_relevance":["309"], "toiletdetails_id":["40”] }] }, "info”{ "rid":"ccd66a5219f938d295e4326391ce31e1359fb7bc306564", "time-ms":3, "cpu-time-ms":0} }
  • 77. Scalable
  • 78. Time: 1800hRequests <80% CPU Small Instance Partition 1 Copy 1 Time Elastic Search
  • 79. Time: 2000hRequests >80% CPU Small Instance Partition 1 Copy 2 Small Instance Small Instance Partition 1 Partition 1 Copy 1 Copy 1 Time Elastic Search
  • 80. Time: 2200hRequests >80% CPU Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  • 81. Time: 0000hRequests Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 <30% CPU Copy 2 Copy 2 Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  • 82. Cost savingsRequests Traditional required capacity Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  • 83. Cost savingsRequests Traditional required capacity Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Elastic Capacity Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  • 84. Cost savingsRequests Traditional required capacity Small Instance Savings Partition 1 Savings Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Elastic Capacity Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  • 85. Cloud Search Search Instance Partition 1 Copy 1 Traffic Search Instance Partition 1 Request Copy 2 volume &complexity Search Instance Partition 1 Copy n
  • 86. Data Document quantity & sizeCloudSearch Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Copy 1 Copy 1 Copy 1
  • 87. Data Document quantity & size Cloud Search Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Copy 1 Copy 1 Copy 1 Traffic Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Request Copy 2 Copy 2 Copy 2 volume &complexity Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Copy n Copy n Copy n
  • 88. Search on AWSPerform capacity planning on size of indexes Setup & manage required number of instances Install & configure search software such as Solr across clusterManage cluster partitioning and size over time Implement monitoring
  • 89. Search on AWS Cloud SearchPerform capacity planning on size Create search domain in console of indexes Upload documents Setup & manage required Retrieve results number of instances Install & configure search software such as Solr across clusterManage cluster partitioning and size over time Implement monitoring
  • 90. Amazon Elastic MapReduce
  • 91. 1 instance for 100 hours =100 instances for 1 hour
  • 92. Small instance = $8
  • 93. Elastic MapReduce Feature Details Scalable Use as many or as few compute instances running Hadoop as you want. Modify the number of instances while your job flow is runningIntegrated with Works seamlessly with S3 as origin other services and output. Integrates with DynamoDB Managed, elastic Hadoop clusterComprehensive Supports languages such as Hive and Pig for defining analytics, and allows Integrates with S3 & DynamoDB complex definitions in Cascading, Leverage Hive & Pig analytics scripts Java, Ruby, Perl, Python, PHP, R, or Integrates with instance types such as spot C++ Cost effective Works with Spot instance types Monitoring Monitor job flows from with the management console
  • 94. But what is it?
  • 95. A frameworkSplits data into piecesLets processing occur Gathers the results
  • 96. Very large dataset (e.g TBs)
  • 97. Lots of instances of ‘X’Very large dataset (e.g TBs)
  • 98. Lots of instances of ‘X’Very large dataset (e.g TBs) Split the log into many small pieces
  • 99. Process in an Lots of EMR cluster instances of ‘X’Very large dataset (e.g TBs) Split the log into many small pieces
  • 100. Process in an Lots of EMR cluster instances of ‘X’Very large dataset (e.g TBs) Split the Aggregate log into the results many small from all pieces the nodes
  • 101. Process in an Lots of EMR cluster instances of ‘X’Very large dataset Aggregate (e.g TBs) Split the Aggregate view of ‘X’ log into the results many small from all pieces the nodes
  • 102. Very large dataset Aggregate (e.g TBs) Insight in a fraction of the time view of ‘X’
  • 103. 1 instance for 100 hours =100 instances for 1 hour
  • 104. Small instance = $8
  • 105. 1 instance for 1,000 hours =1,000 instances for 1 hour
  • 106. Small instance = $80
  • 107. S3Input data
  • 108. S3 Input dataCode Elastic MapReduce
  • 109. S3 Input dataCode Elastic Name MapReduce node
  • 110. S3 Input dataCode Elastic Name MapReduce node Elastic cluster
  • 111. S3 Input dataCode Elastic Name MapReduce node HDFS Elastic cluster
  • 112. S3 Input dataCode Elastic Name MapReduce node Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  • 113. S3 Input dataCode Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  • 114. S3Input data Output S3 + SimpleDB
  • 115. Hadoop on AWSConfigure & manage EC2 instances Setup Namenode Setup Datanodes Create HDFS file system Setup extras (Hive, Pig etc) Prepare data Execute job
  • 116. Hadoop on AWS Elastic MapReduceConfigure & manage EC2 instances Specify desired cluster size Setup Namenode Execute job Setup Datanodes Create HDFS file system Setup extras (Hive, Pig etc) Prepare data Execute job
  • 117. Features powered by Amazon Elastic MapReduce: People Who Viewed this Also Viewed Review highlights Auto complete as you type on search Search spelling suggestions Top searches Ads200 Elastic MapReduce jobs per day Processing 3TB of data
  • 118. Elastic MapReduce "With Amazon Elastic MapReduce, there was no upfront investment in hardware, no hardware procurement delay, and no need to hire additional operations staff. Because of the flexibility of the platform, our first new online advertising campaign experienced a 500% increase in return on ad spend from a similar campaign a year before.” Mark Taylor, Program Director
  • 119. Where to go next
  • 120. http://aws.typepad.comhttp://aws.amazon.com/whitepapers
  • 121. Powerof60.com$100 of creditsThat’s enough to run anAmazon Elastic ComputeCloud (Amazon EC2)Standard Small Instancefor approximately 600hoursOr ~600 EC2 StandardSmall Instances for 1 hour
  • 122. Summary
  • 123. AWS is more than compute & storageDatabases, middleware and complex frameworks as servicesRemove the operational burden from running common software servicesApplication services gives access scalable, sophisticated software without the overheads
  • 124. aws.amazon.com

×