AWS Summit Benelux 2013 - Media and Online Advertising on AWS

1,051 views
896 views

Published on

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,051
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
47
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

AWS Summit Benelux 2013 - Media and Online Advertising on AWS

  1. 1. Media and Online Advertising on AWS Jan Borch – AWS Solutions Architect
  2. 2. Media Application on AWS Music streaming
  3. 3. Media Application on AWS Video streaming
  4. 4. Media Application on AWS Digital publishing
  5. 5. Media Application on AWS Media sharing
  6. 6. Martijn Bakker Chief Engineer, WeTransfer
  7. 7. Martijn Bakker, Chief Engineer, WeTransfer AWS Summit Benelux 2013
  8. 8. WeTransfer • Send big files anywhere • Up to 2GB per transfer • Files are stored for 7 days • Beautiful backgrounds (50/50 split between ads & art) 2
  9. 9. WeTransfer • WeTransfer Plus: € 120 per year • Up to 5GB per transfer • Store up to 50GB permanently • Your own backgrounds (no ads)
  10. 10. WeTransfer • 1.8 Million transfers per day • Downloads: 25 Gigabit per second • 1.5 Million requests per hour (site + API) • Over a Petabyte of storage used on S3 (Peak measurements - July 2013)
  11. 11. Company • 15 wonderful, dedicated people • Founded & based in Amsterdam • Originated from Oy Communications
  12. 12. Origins • Purely a functional tool • Design company needs to send big files to clients • FTP & friends “technical”, “confusing”
  13. 13. Early development • OyTransfer + advertising = WeTransfer • Goals: • beautiful • easy to use • secure
  14. 14. Growth • Double the transfers every 3 months • Previous hoster: • could not match growth tempo adequately • hardware-based platform; adds maintenance & development costs
  15. 15. Growth • Our 3 necessities: • Storage • Support • Scalability
  16. 16. Growth • AWS: on-demand, available right away • Initial migration: • development time: 1 month • using S3 through EC2 instances
  17. 17. The new WeTransfer • Built for & with AWS: uses RDS, EC2, S3, CloudWatch, DynamoDB, Route53, ElastiCache • Ruby + HTML5 + JavaScript (frontend) • Backend tailored around S3 • Launched January of this year
  18. 18. WeTransfer and S3 • Virtually unlimited storage capacity • Redundancy: always available • Fast, and cheap compared to similar offers • Dramatically less costs
  19. 19. WeTransfer and S3 • Uses the multipart upload mechanism (where possible) • Resumable uploads • Uploads go directly to S3 thanks to CORS support • Worker instances to process uploaded content
  20. 20. WeTransfer and S3 • Secure upload / download, and encryption • Regionalized: storage facilities all over the world to ensure proper speeds to end users • No maintenance
  21. 21. So why S3? • Fast & flexible • Almost no time spent on maintenance • Virtually limitless capacity at the tips of your fingers
  22. 22. https://wetransfer.com/jobs
  23. 23. Online Advertising on AWS YOUR AD HERE
  24. 24. Common challenge for Advertising Platforms ...
  25. 25. ... device and media fragmentation ... TXT
  26. 26. ... scaled to millions of users
  27. 27. 503 Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
  28. 28. 503 Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
  29. 29. Maintain availability from one server…
  30. 30. …to thousands
  31. 31. Let's take a journey ...
  32. 32. Let's take a journey ... Store
  33. 33. AWS S3 Storage for the Internet
  34. 34. STATIC FILES REPOSITORY AMAZON S3 MEDIA AD SERVED TO USER
  35. 35. Let's take a journey ... Store
  36. 36. Let's take a journey ... Transform
  37. 37. AWS Elastic Transcoder Video transcoding in the cloud
  38. 38. VIDEO FILES REPOSITORY AMAZON S3 MEDIA AD SERVED TO USER AMAZON ELASTIC TRANSCODER
  39. 39. Let's take a journey ... Transform
  40. 40. Let's take a journey ... Deliver
  41. 41. AWS CloudFront Web service for content delivery
  42. 42. Dallas(2) St.Louis Miami JacksonvilleLos Angeles (2) Palo Alto Seattle Ashburn(2) Newark New York (2) Dublin London(2) Amsterdam Stockholm Frankfurt(2) Paris(2) Singapore(2) Hong Kong Tokyo Sao Paulo South Bend San Jose Osaka Milan Sydney Reach a global audienceReach a global audience
  43. 43. CONTENT DELIVERY NETWORK AMAZON CLOUDFRONT IMPRESSION LOGS
  44. 44. Simple HLS video streaming architecture In-house content publication server Source Video Assets in S3 S3
  45. 45. Simple HLS video streaming architecture In-house content publication server Source Video Assets in S3 Video transcoded into HLS S3Elastic Transcoder
  46. 46. Simple HLS video streaming architecture In-house content publication server Source Video Assets in S3 Video transcoded into HLS Edge Delivery using CloudFront Stockholm NY CloudFront S3Elastic Transcoder
  47. 47. AWS CLI  aws s3 cp video.avi s3://mybucket/video  aws elastictranscoder create-job --pipeline-id 1379510897399-mxjrif --input '{"Key":"video/video.avi"}' --outputs '[{"Key":"sample","PresetId":"1234-123", ...}]'
  48. 48. Let's take a journey ... Deliver
  49. 49. Let's take a journey ... Match
  50. 50. AWS EC2 Resizable compute capacity in the cloud
  51. 51. cc2.8xlarge Virtual core: 32 - 2 x Intel Xeon Memory: 60,5 GiB I/O performance: 10 Gbit Virtual core: 1 Memory: 1.7 GiB I/O performance: Moderate m1.small cr1.8xlarge Virtual core: 32 - 2 x Intel Xeon Memory: 240 GiB I/O performance: 10 Gbit SSD Instance store: 240 GB cr1.8xlarge Virtual core: 16 Memory: 60.5 GiB I/O performance: 10 Gbit SSD Instance store: 2 x 1TB cr1.8xlarge Virtual core: 16 Memory: 117 GiB I/O performance: 10 Gbit Instance store: 24 x 2TB EC2 instance types
  52. 52. Amazon Route 53 Highly available and scalable Domain Name System Extremely reliable and cost effective Feature Details Global Supported from AWS global edge locations for fast and reliable domain name resolution Scalable Automatically scales based upon query volumes Latency based routing Supports resolution of endpoints based upon latency, enabling multi-region application delivery Integrated Integrates with other AWS services allowing Route 53 to front load balancers, S3 and EC2 Reach a global audience
  53. 53. Link to Ad Resource AMAZON EC2 + AUTOSCALING Ad Servers AMAZON ELB
  54. 54. AWS DYNAMODB fast & fully managed NoSQL database service
  55. 55. AMAZON DYNAMODB PROFILES DATABASE
  56. 56. ad-id advertiser max-price imps to deliver imps delivered 1 AAA 100 50000 1200 2 BBB 150 30000 2500 user-id attribute1 attribute2 attribute3 attribute4 A XXX XXX XXX XXX B YYY YYY YYY YYY not many rows so many rows frequent update (near realtime) batch manner update Ads Profiles(user-cookie) Very general table structure
  57. 57. Let's take a journey ... Match
  58. 58. Let's take a journey ... Capture
  59. 59. Click-through Servers AMAZON EC2 + AUTOSCALING
  60. 60. AWS OPSWORKS INTEGRATED APPLICATION MANAGEMENT
  61. 61. Stack
  62. 62. LayerStack
  63. 63. InstancesLayerStack
  64. 64. ScaleInstancesLayerStack
  65. 65. Agent on each EC2 instance OpsWorkstalks with The heart of the service
  66. 66. Instance lifecycle and configuration hooks
  67. 67. Cookbooks script "install_composer" do interpreter "bash" user "root" cwd "#{node[:deploy][:myphotoapp][:deploy_to]}/ current" code <<-EOH curl -s https://getcomposer.org/installer | php php composer.phar install EOH end Amazon S3 Git repository
  68. 68. Let's take a journey ... Capture
  69. 69. Let's take a journey ... Report
  70. 70. CLICK-THROUGH LOG FILES AMAZON S3
  71. 71. AMAZON ELASTIC MAP REDUCE CLICK-THROUGH LOG FILES
  72. 72. Data Growth GB TB PB Data Growth
  73. 73. Data Growth Data Growth Server Logs Click Analysis Impression logs
  74. 74. Sampling Big Data Time to process Inflexible Complexities of Big Data
  75. 75. Sampling Big Data Inflexible Complexities of Big Data Elastic Map Reduce & Redshift
  76. 76. Sampling Big Data Complexities of Big Data “Queryable” Elastic Map Reduce & Redshift
  77. 77. Big Data Complexities of Big Data “Queryable”All Data Elastic Map Reduce & Redshift
  78. 78. Data Insight Turning Data into Information
  79. 79. Data Insight Elastic MapReduce Turning Data into information Redshift
  80. 80. AWS Elastic Map Reduce Process vast amounts of data using Hadoop
  81. 81. AWS Redshift Fast, fully managed, petabyte-scale data warehouse service
  82. 82. Let's take a journey ... Store Transform Deliver Match Capture Report
  83. 83. Amazon Web Services
  84. 84. Garry Turkington, CTO, Improve Digital
  85. 85. PRESENTATION TYPE IMPROVE DIGI Amazon Web Services at Improve Digital IMPROVE DIGITAL 26 September 2013 Garry Turkington CTO g.turkington@improvedigital.com @garryimprove
  86. 86. • Cloud-based Real Time Advertising Technology • Focus on the premium publisher / media owner • Integrations with thousands of Demand Partners • Decision driven by Real Time Data • Offices in UK, NL, DE & ES • +100 Premium Publishers IMPROVE DIGITAL ABOUT US
  87. 87. Use AWS in conjunction with dedicated physical infrastructure 2 sides to the story • Front end: serving of ads to end-users • Back-end: Data processing and dev/test Use of AWS
  88. 88. • Fleet of ad servers running mostly on EC2 • Ad serving process is computationally expensive and has strict time constraints • Need ability to spin up additional instances based on demand: horizontally scalable system • Place ad servers in different regions to reduce serving latency; big benefit of EC2 over physical kit • Grow fleets in different regions separately Serving ads
  89. 89. • S3/Glacier used for policy-driven data retention • S3 is the starting point for AWS and on-premises data processing jobs • S3 used as a shared storage space between distributed components • VPC used to integrate AWS and on-premises flows and systems • Automated deployment of dev/test software into VPC EC2 has been great Backend systems
  90. 90. As a startup it was almost a no-brainer • Didn't want/need overhead of own physical infrastructure • Pricing model with hugely reduced (or zero) up-front cost is an easy sell • Coordinating the ability to quickly grow the ad server fleet is *hard* with a physical data centre • As a more mature company the above still apply • In addition our needs have also matured from the lower level "give us servers and storage" Why do we use AWS?
  91. 91. Lessons learned: It works!  Service integration is often ridiculously easy: pull S3 data into EMR, set up auto-scaling etc  Geographic data locality -- helps with compliance  Automatic cost reductions does wonders for corporate acceptance  Continuous evolution of the services means that they suddenly can be a great fit
  92. 92. Lessons learned: It works in its own way  Need to understand exactly what each service offers  Need to design for fault tolerance; instances can fail at high scale  Had to work hard to get our network to integrate with VPC  Can't save you from yourself; poor design is poor design  Would still love to see another region in EU
  93. 93. • Growth means more of all the above • Want to re-evaluate services that weren't a great fit for us in the past (RDS, DynamoDB) • Believe we can use data processing services (Elastic MapReduce in particular) alongside on-premises systems • Looking to Cloud Formation/ Elastic Beanstalk and Opsworks to extend automation much further The future

×