Aws for the Retail Industry, Webinar, September 2012

4,107 views

Published on

In this webinar, Ryan Shuttleworth, Technology Evangelist, Amazon Web Services and Adam Bidwell, eCommerce Manager, Kurt Geiger will discuss how retailers are using AWS to carry out a number of business critical functions. This complementary webinar will discuss in detail Kurt Geiger's experience of using AWS to run their Magento eCommerce engine and Ryan Shuttleworth will cover an overview of AWS along with a number case studies of how retail customers are implementing AWS to run their business.

Published in: Technology

Aws for the Retail Industry, Webinar, September 2012

  1. 1. AWS for the Retail Industry Ryan Shuttleworth – Technical Evangelist @ryanAWSAdam Bidwell – eCommerce Manager, Kurt Geiger
  2. 2. Agenda Amazon Web Services Background Utility computing & Elasticity AWS & Retail Security & compliance Highly available customer facing systems Core platforms Customer analytics Kurt Geiger – Customer Story
  3. 3. Your feedback is importantTell us:What’s good, what’s notWhat you want to see at these eventsWhat you want AWS to deliver foryou
  4. 4. background
  5. 5. Consumer Business Seller IT Infrastructure Business Business Tens of millions of Sell on Amazon Cloud computing active customer websites infrastructure for accounts hosting web-scale Use Amazon solutions technology for your own retail website Eight countries: Hundreds of US, UK, Germany, Leverage Amazon’s thousands of Japan, France, Canada, massive fulfillment registered customers China, Italy center network in over 190 countries
  6. 6. About Amazon Web How did Amazon… ServicesDeep experience in building and operating global web scale systems ? …get into cloud computing?
  7. 7. Over 10 years in the makingEnablement of sellers on AmazonInternal need for scalable deployment environmentEarly forays proved developers were hungry for more
  8. 8. AWS MissionEnable businesses and developers to use web services* to build scalable, sophisticated applications. *What people now call “the cloud”
  9. 9. Each day AWS adds the equivalent server capacity to power Amazon when it was a global, $2.76B enterprise (circa 2000)
  10. 10. Objects in S3 1 Trillion 1000.000 750.000 500.000 250.000 0.000 750k+ peak transactions per second
  11. 11. Utility computing
  12. 12. Utility computing On demand Pay as you go Uniform Available
  13. 13. Utility computing On demand Pay as you go Uniform Available
  14. 14. Utility computing
  15. 15. Utility computing On demand Pay as you go Compute Scaling Security CDN Backup DNS Database Storage Load Balancing Workflow Monitoring Networking Uniform Messaging Available
  16. 16. On a global footprint Region US-WEST (N. California) EU-WEST (Ireland) GOV CLOUD ASIA PAC (Tokyo) US-EAST (Virginia)US-WEST (Oregon) ASIA PAC (Singapore) SOUTH AMERICA (Sao Paulo)
  17. 17. On a global footprint Availability Zone
  18. 18. On a global footprint Edge Locations London(2) Seattle South Bend New York (2) Amsterdam Newark Stockholm Dublin Palo Alto TokyoSan Jose Frankfurt(2) Paris(2) Ashburn(2) Milan Osaka Los Angeles (2) Jacksonville Dallas(2) Hong Kong St.Louis Miami Singapore(2) Sydney Sao Paulo
  19. 19. Elasticity
  20. 20. Elastic capacity Traditional IT capacityCapacity Time Your IT needs
  21. 21. Elastic capacity On and Off Fast Growth Variable peaks Predictable peaks
  22. 22. Elastic capacity WASTE On and Off Fast Growth Variable peaks Predictable peaks CUSTOMER DISSATISFACTION
  23. 23. Elastic capacityCapacity Traditional IT capacity Elastic cloud capacity Time Your IT needs
  24. 24. Elastic capacity On and Off Fast Growth Variable peaks Predictable peaks
  25. 25. 503 Service Temporarily UnavailableThe server is temporarily unable to serviceyour request due to maintenance downtime or capacity problems. Please try again later.
  26. 26. 503 Service Temporarily UnavailableThe server is temporarily unable to serviceyour request due to maintenance downtime or capacity problems. Please try again later.
  27. 27. From one instance…
  28. 28. …to thousands
  29. 29. And back again…
  30. 30. 40 servers to 5000 in 3 days EC2 scaled to peak of 5000 instances Number of EC2 Instances “Techcrunched” Launch of Facebook modification Steady state of ~40 instances 4/12/2008 4/13/2008 4/14/2008 4/15/2008 4/16/2008 4/17/2008 4/18/2008 4/19/2008 4/20/2008
  31. 31. Security you can rely upon
  32. 32. Shared responsibility Foundation Services Amazon Compute Storage Database Networking Availability Zones AWS Global Edge Locations Infrastructure Regions
  33. 33. Shared responsibilitySarbanes-Oxley (SOX) FISMA A&AsISO 27001 Certification Multiple NIST Low Approvals to Operate (ATO) NIST Moderate, GSA issued ATOPayment Card Industry Data Security FedRAMP Standard (PCI DSS) Level 1 Compliant DIACAP MAC III Sensitive IATOSAS70(SOC 1) Type II Audit Customers have deployed various compliant applications such as HIPAA (healthcare) Foundation Services Amazon Compute Storage Database Networking Availability Zones AWS Global Edge Locations Infrastructure Regions
  34. 34. Shared responsibility Customer Data Platform, Applications, Identity & Access Management You Operating System, Network & Firewall Configuration Client-side Data Encryption & Data Server-side Encryption Network Traffic Protection Integrity Authentication (File System and/or Data) (Encryption/Integrity/Identity) Foundation Services Amazon Compute Storage Database Networking Availability Zones AWS Global Edge Locations Infrastructure Regions
  35. 35. AWS and Retail
  36. 36. 1Customer facing infrastructure
  37. 37. Rule 1: Service all web requestsa) Make sure requests get to your ‘front door’ DNS Application Data
  38. 38. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’Request DNS Application Data
  39. 39. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’Request DNS Application Data
  40. 40. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’Request DNS Application Data Clients can’t resolve …then this is you? irrelevant
  41. 41. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’ Request DNS Application Data Feature Details Global Supported from AWS global edge locations for fast and reliable domain name resolution “100% Scalable Automatically scales based upon query volumes Available” Route53 Latency based routing Supports resolution of endpoints based upon latency, enabling multi- SLA region application delivery Integrated Integrates with other AWS services allowing Route 53 to front loadhttp://aws.amazon.com/route53/sla balancers, S3 and EC2 Secure Integrates with IAM giving fine grained control over DNS record access
  42. 42. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arriveRequest DNS Application Data Route53
  43. 43. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arriveRequest DNS Application Data Region Availability Zone Elastic load balancing Route53 Availability Zone Multi-availability zone Multi-region Availability Zone Elastic Load Balancer Availability Zone Region
  44. 44. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arrive c) Have the data to form a responseRequest DNS Application Data Region Availability Zone Route53 Availability Zone Availability Zone Elastic Load Balancer Availability Zone Region
  45. 45. Rule 1: Service all web requests a) Make sure requests get to your ‘front door’ b) Make sure you open the door when they arrive c) Have the data to form a response Request DNS Application Data RegionMulti-AZ RDS Availability Zone(Master-slave) Route53 Availability ZoneInter-regionreplication Availability ZoneRead-replicas Elastic Load Balancer Availability Zone Region
  46. 46. Rule 2: Service requests as fast as possible
  47. 47. Rule 2: Service requests as fast as possiblea) Choose the fastest route Request Route53 Region Region B A
  48. 48. Rule 2: Service requests as fast as possiblea) Choose the fastest route Request Route53 16ms 92ms Region Region B A
  49. 49. Rule 2: Service requests as fast as possiblea) Choose the fastest route Request Route53 16ms 92ms Region Region B A
  50. 50. Rule 2: Service requests as fast as possiblea) Choose the fastest route Request Route53 Region A DNS entry 16ms Region Region B A
  51. 51. Rule 2: Service requests as fast as possible a) Choose the fastest route b) Offload your application serversCloudFront 3 Served from S3World-wide content distribution network /images/*Easily distribute content to end users with lowlatency, high data transfer speeds, and nocommitments. London 2 Served from EC2 *.php Paris 1 Single CNAME NY www.mysite.com
  52. 52. Rule 2: Service requests as fast as possiblea) Choose the fastest routeb) Offload your application servers Without CloudFront EC2 webservers/app servers loaded by user requests
  53. 53. Rule 2: Service requests as fast as possiblea) Choose the fastest routeb) Offload your application servers With CloudFront Load of user requests pushed into CloudFront, EC2 cluster can scale down Offload Scale Down
  54. 54. Rule 2: Service requests as fast as possible a) Choose the fastest route b) Offload your application serversNo CDN CDN for CDN for Static Static & Content Dynamic Content Offload Scale Down Response TimeResponse Time Response Time Server Load Server Server Load Load
  55. 55. Rule 3: Handle requests at any scalea) Scale up Vertical Scaling From $0.02/hr Scale up with Elastic Compute Cloud (EC2) Basic unit of compute capacity Range of CPU, memory & local disk options 14 Instance types available, from micro through cluster compute to SSD backed
  56. 56. Rule 3: Handle requests at any scalea) Scale upb) Scale out as-create-auto-scaling-group MyGroupTriggerauto-scaling --launch-configuration MyConfigpolicy --availability-zones eu-west-1a --min-size 4 --max-size 200 Auto-scaling Automatic re-sizing of compute clusters based upon demand
  57. 57. Rule 3: Handle requests at any scalea) Scale upb) Scale out Manually By Schedule Send an API call or use CLI to Scale up/down based on date and timelaunch/terminate instances – Only need to specify capacity change (+/-) By Policy Auto-RebalanceScale in response to changing conditions, Instances are automatically based on user configured real-time launched/terminated to ensure the monitoring and alerts application is balanced across multiple Azs
  58. 58. Rule 3: Handle requests at any scalea) Scale upb) Scale out Manually By Schedule Preemptive manual scaling of Send an API call or use CLI to Regular scaling up and down of Scale up/down based on date and timelaunch/terminate instances – Only need capacity instances e.g. before a marketing event add(+/-) to specify capacity change 10 more e.g. scale from 0 to 2 to process SQS instances messages every night or double capacity on a Friday night By Policy Auto-RebalanceScale in response to changing conditions, Instances are automatically Dynamic scale based upon based on user configured real-time Maintain capacity across launched/terminated to ensure the monitoringmetrics custom and alerts application is balancedzones multiple availability across e.g. SQS queue depth, Average CPU load, e.g. Instance availability maintained in Azs ELB latency event of AZ becoming unavailable
  59. 59. Rule 3: Handle requests at any scale a) Scale up b) Scale out c) Dial it up Elastic Block Store DynamoDBProvisioned IOPS up to 1000 per EBS Provisioned read/write performance per volume table Predictable performance for Predictable high performance scaled via demanding workloads such as console or API databases
  60. 60. Rule 4: Simplify architecture with services Relational Database ServiceUse RDS for databases Database-as-a-Service No need to install or manage database instances Scalable and fault tolerant configurations DynamoDB Use DynamoDB for Provisioned throughput NoSQL database high performance key- Fast, predictable performance value DB Fully distributed, fault tolerant architecture
  61. 61. Rule 4: Simplify architecture with services Amazon SQS Reliable messageProcessing results Reliable, highly scalable, queue service queuing without for storing messages as they travel Amazon SQS between instances additional software 1 Processing task/processing trigger 2Push inter-process Simple Workflow Task Aworkflows into the Reliably coordinate processing steps Task B 3 across applicationscloud with SWF (Auto-scaling) Integrate AWS and non-AWS resources Manage distributed state in complex systems Task C
  62. 62. Rule 4: Simplify architecture with services Document Server Cloud SearchDon’t install search Elastic search engine based uponsoftware, use Amazon A9 search engine Fully managed service withCloudSearch Search sophisticated feature set Server Scales automatically Results Elastic MapReduce Elastic Hadoop cluster Process large volumes Integrates with S3 & DynamoDB of data cost effectively Leverage Hive & Pig analytics scripts with EMR Integrates with instance types such as spot
  63. 63. “Amazon CloudSearch is a game-changingproduct that has allowed us to deliver powerfulnew search capabilities. Our customers can now find what they are looking for faster and more easily than ever before… ….We saved many months of re-architecture and development time by going with Amazon CloudSearch” Don MacAskill CEO & Chief Geek SmugMug
  64. 64. 10 Million records44 GB collectionmore than 2,000 operationsper secondOrder volumes increasesubstantially during theholidays necessitatingelasticity
  65. 65. 2Core platforms
  66. 66. Certification of SAP BusinessObjects business intelligence solutions and SAP Rapid Deployment Solutions (RDS) on Linux & Windows Server 2008 R2Certification of SAP Business All-in-One on Linux & Windows Server 2008 R2Certified database engines for production SAP deployments: MaxDB, DB2, MS SQL Server 2008 R2
  67. 67. Non production Backup, archive and Production systems recovery systems(dev, test, staging) (databases, AMIs) (Analytics, branch etc) http://aws.amazon.com/sap/
  68. 68. Relational Database Service Database-as-a-Service No need to install or manage database instances Scalable and fault tolerant configurations Feature Details Platform support Create MySQL, SQL Server and Oracle RDBMS Preconfigured Get started instantly with sensible default settingsAutomated patching Keep your database platform up to date automatically Backups Automatic backups and point in time recovery and full DB backups Backups Volumes can be snapshotted for point in time restore Failover Automated failover to slave hosts in event of a failure Replication Easily create read-replicas of your data and seamlessly replicate data across availability zones
  69. 69. Pilot light architecture Disaster recovery in AWSBuild resources around replicated dataset Keep ‘pilot light’ on by replicating core databasesBuild AWS resources around dataset and leave in stopped state
  70. 70. Pilot light architecture Disaster recovery in AWSBuild resources around Scale resources in AWS in replicated dataset response to a DR event Keep ‘pilot light’ on by replicating core Start up pool of resources in AWS when databases events dictateBuild AWS resources around dataset and Match current production capacity through leave in stopped state auto-scaling polcies
  71. 71. Pilot light architecture Disaster recovery in AWSBuild resources around Scale resources in AWS in replicated dataset response to a DR event Keep ‘pilot light’ on by replicating core Start up pool of resources in AWS when databases events dictateBuild AWS resources around dataset and Match current production capacity through leave in stopped state auto-scaling policies Switch-over to system in AWS
  72. 72. 3Customer analytics
  73. 73. Big DataWe can collect more
  74. 74. Big Data There is more
  75. 75. Big DataAnd data has gravity…
  76. 76. Storage Big Data Compute Data has gravityApp Data App http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
  77. 77. Storage Big Data Compute …and inertia at volume… Data http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
  78. 78. Storage Big Data Compute …easier to move applications to the data Data http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/
  79. 79. Lorem ipsum dolor sitStorage Big Data Compute met, consectetur Bring compute capacity to the data dipiscing elit. Etiam Lorem ipsum dolor uis ligula neque, eget amet, consecte enenatis sem. Personal adipiscing elit. EtiaSuspendisse non eros quis ligula neque, eg ulla, at placerat nibh.Cras id lectus mattis est Very large dataset venenatis se Suspendisse non er llamcorper blandit.seeks strong & nulla, at placerat nibhProin ut nisi vitae enim ulputate tempor. consistent compute for Cras id lectus mattisPhasellus id commodo est ullamcorper ros. Mauris necshort term relationship, blandit. Proin ut nisi ignissim turpis. Nunc vitae enim vulputate possibly longer. GSOH a tempor. Phasellus id Cras id lectus mattis plus aws.amazon.com commodo eros. Mauris nec dignissim est ullamcorper turpis. Nunc
  80. 80. Cloud has the power to process
  81. 81. Storage Big Data Compute From one instance…
  82. 82. Storage Big Data Compute …to thousands
  83. 83. Storage Big Data Compute and back again…
  84. 84. The revolution
  85. 85. have data
  86. 86. have datacan store
  87. 87. have datacan store can analyse
  88. 88. economically
  89. 89. fast
  90. 90. Who is your customer really? What do people really like?What is happening socially with your products? How do people really use your products?
  91. 91. 96
  92. 92. Lesson 1: don’t leave your Amazon account logged in at homeLesson 2: use the data you have to drive proactive processes
  93. 93. 1 instance for 100 hours =100 instances for 1 hour
  94. 94. Small instance = $8
  95. 95. Amazon Elastic MapReduce
  96. 96. Elastic MapReduceManaged, elastic Hadoop clusterIntegrates with S3 & DynamoDBLeverage Hive & Pig analytics scriptsIntegrates with instance types such as spot Feature Details Scalable Use as many or as few compute instances running Hadoop as you want. Modify the number of instances while your job flow is running Integrated with other services Works seamlessly with S3 as origin and output. Integrates with DynamoDB Comprehensive Supports languages such as Hive and Pig for defining analytics, and allows complex definitions in Cascading, Java, Ruby, Perl, Python, PHP, R, or C++ Cost effective Works with Spot instance types Monitoring Monitor job flows from with the management console
  97. 97. But what is it?
  98. 98. A frameworkSplits data into piecesLets processing occur Gathers the results
  99. 99. S3 + DynamoDB Input dataCode Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  100. 100. Very large click log (e.g TBs)
  101. 101. Lots of actions by John SmithVery large click log (e.g TBs)
  102. 102. Lots of actions by John SmithVery large click log (e.g TBs) Split the log into many small pieces
  103. 103. Process in an Lots of actions EMR cluster by John SmithVery large click log (e.g TBs) Split the log into many small pieces
  104. 104. Process in an Lots of actions EMR cluster by John SmithVery large click log (e.g TBs) Split the Aggregate log into the results many small from all pieces the nodes
  105. 105. Process in an Lots of actions EMR cluster by John SmithVery large What click log John (e.g TBs) Split the Aggregate log into the results Smith many small from all did pieces the nodes
  106. 106. Very large What click log John (e.g TBs) Insight in a fraction of the time Smith did
  107. 107. 1 instance for 100 hours =100 instances for 1 hour
  108. 108. Small instance = $8
  109. 109. 1 instance for 1,000 hours =1,000 instances for 1 hour
  110. 110. Small instance = $80
  111. 111. Features powered by Amazon Elastic MapReduce: People Who Viewed this Also Viewed Review highlights Auto complete as you type on search Search spelling suggestions Top searches Ads200 Elastic MapReduce jobs per day Processing 3TB of data
  112. 112. “With AWS, our developers can now do things they couldn’t before……Our systems team can focus their energies on other challenges.” Dave Marin Search and data-mining engineer
  113. 113. Elastic MapReduceWeb log analysis and recommendation algorithms
  114. 114. Adam BidwelleCommerce Manager
  115. 115. Overview of Kurt GeigerKurt Geiger are responsible for the operation of threeretail websites:• Kurtgeiger.com• Shoeaholics.com• Ninewest.co.ukIn total serving upwards of a half-million page views aday.
  116. 116. Our interest in Amazon, is to host:• Frontend systems - three Magento installations which the stores are built on.• Administration systems – backend tasks, such as product enrichment and reporting.• Testing – load-testing systems, and other ‘sandpit’ tasks• Research/Development – one-off installations for investigation purposes.
  117. 117. Challenges faced by Kurt Geiger:• Rapidly changing business needs – fast pace makes it difficult to predict long-term requirements• Marketing activity – drives large traffic spikes
  118. 118. Why Amazon?• Unique model – we’ve used several cloud providers Amazon offer a wide range of network/server infrastructure and services.• Self-service – 24/7 help yourself approach, suits us to take what we need when we need it
  119. 119. Future• Larger capacity architectures• More API based “pop-up” systems on demand• Reserved instances - further cost savings
  120. 120. Benefits• Hourly billing – the cost adapts with our current set up, no tie-in• Large capacity – Whether capacity will be there is not a consideration, it just is• Trusted provider – architecture still requires planning for good reliability, but AWS has robust infrastructure to build on
  121. 121. aws.amazon.com/free

×