Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Customer Presentation - Mediafed


Published on

Alan Williamson, Co-Founder of Mediafed presents at the AWS Start-Up Event - London about their use of Amazon S3 and EC2.

Published in: Technology, Sports
  • Be the first to comment

  • Be the first to like this

AWS Customer Presentation - Mediafed

  1. 1. Amazon AWS & Mediafed Alan Williamson Director, MediaFed ltd Co-Founder, aw2.0 ltd the cloud experts
  2. 2. MediaFed Ltd <ul><li>RSS Experts </li></ul><ul><li>Monetize feeds </li></ul><ul><li>Track feeds </li></ul><ul><li>Value Added feeds </li></ul><ul><li>Feed Compliance </li></ul><ul><li>Focused on Premium publishers </li></ul>
  3. 3. MediaFed’s work flow MediaFed Publishers MediaFed Services Millions of RSS Clients
  4. 4. Traditional Setup <ul><li>Fixed number of ‘real’ strokeable servers </li></ul><ul><li>Professionally managed servers </li></ul><ul><li>Costing £xxx’s per month just for one server </li></ul>
  5. 5. Then things got interesting Started to grow FAST! /
  6. 6. Where did MediaFed need to scale? <ul><li>Front end Web Servers </li></ul><ul><ul><li>Servicing the requests for feeds, article views, and click thru’s </li></ul></ul><ul><li>RSS Ad Servers </li></ul><ul><ul><li>Processing the ad insertion’s and click-thru’s </li></ul></ul><ul><li>Feed Crawlers </li></ul><ul><ul><li>Retrieving the feed content, cleaning and conversion </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Storage of feed data, and statistical analysis, running to TB’s of data </li></ul></ul>
  7. 7. <ul><li>The current hosting company was asked for a quote </li></ul><ul><li>It came as quite a shock to us </li></ul>
  8. 8. <ul><li>We needed servers FAST </li></ul><ul><li>Time Quoted 10 working days! </li></ul><ul><li>Need a modern approach </li></ul>
  10. 10. Which Cloud Provider … <ul><li>… had experience with a number of cloud providers </li></ul><ul><li>… each one with their strengths and weaknesses </li></ul><ul><li>… different price/performance points </li></ul><ul><li>But … </li></ul>
  11. 11. It’s “Hug Amazon” day!
  12. 12. Amazon’s EC2 HOT features <ul><li>Instance start up parameter passing </li></ul><ul><li>Instance snap shot </li></ul><ul><li>VERY FAST instance startup </li></ul><ul><li>Large ‘ free ’ disk space with each instance </li></ul>
  13. 13. Some EC2 problems … <ul><li>No VLAN support </li></ul><ul><li>No ability to turn off public IP address </li></ul><ul><li>No reverse DNS </li></ul>
  14. 14. .. and instances do go down! But “ssssh” you didn’t hear it from me!
  15. 15. Our Amazon Family Usage <ul><li>Amazon EC2 usage </li></ul><ul><ul><li>Runs the main infrastructure; the bread’n’butter </li></ul></ul><ul><li>SimpleDB </li></ul><ul><ul><li>Used for something quite special </li></ul></ul><ul><li>S3 </li></ul><ul><ul><li>Long term storage of log files and DB snapshots </li></ul></ul>
  16. 16. Usage of EC2 Instances <ul><li>The majority of all our services are Java based </li></ul><ul><ul><li>High performance custom built code; no J2EE! </li></ul></ul><ul><li>Load balancing provided by nginx </li></ul><ul><li>Database provided by MySQL 5 </li></ul>
  17. 17. Provisioning of instances <ul><li>Manage our deployment as a single ‘stack’ </li></ul><ul><ul><li>/opt/mediafed/ </li></ul></ul><ul><li>All applications, scripts, binaries in there </li></ul><ul><li>Require no external, pre-installed software </li></ul><ul><li>Complete stack is held within SVN </li></ul><ul><ul><li>First thing that happens when an instance boots up is to update itself with the latest from SVN </li></ul></ul>
  18. 18. Keep it Simple <ul><li>No multiple instances to worry about </li></ul><ul><li>Any instance can do any job in the system we require </li></ul><ul><li>Horrendously transportable! </li></ul><ul><li>Instances are ready to work as soon as they are powered up </li></ul>
  19. 19. Our goals … <ul><li>Over reaching goal was to make it an SSH free zone </li></ul><ul><li>Misbehaving instances would simply be shut off and a new one powered up </li></ul><ul><li>We viewed the “instance” as a component that could be used; not as a server that it really was </li></ul>
  20. 20. Starting up … <ul><li>From the time we issue the command to start up to it actually doing REAL work </li></ul><ul><ul><li>Usually 3 to 4 minutes!!!! </li></ul></ul>
  21. 21. EC2 #1 Killer Feature <ul><li>Parameter passing into EC2 instances </li></ul><ul><li>We pass in a configuration string on startup that specifies what it will do: </li></ul><ul><ul><li>ec2-xxxx -d “TYPE=WEB,ROLE=RSSPOOL3” </li></ul></ul><ul><li>Instance runs a ‘boot script’ that then runs up the necessary service we require </li></ul>
  22. 22. But others services in the system need to know about the new instances ?????
  23. 23. SimpleDB! <ul><li>We use SimpleDB as a sophisticated internal ‘DNS’ or service directory </li></ul><ul><li>When a service is started, it registers its role with SimpleDB including all IP, Port and other information </li></ul><ul><li>Other services keep an eye on changes in this configuration and automatically update their own configuration files accordingly </li></ul>
  24. 24. Amazon SimpleDB as DNS <ul><li>Allow us to add in new web servers into the farm for load balancing without doing a thing except start it up </li></ul><ul><li>Add in additional MySQL slaves/masters without touching anything </li></ul><ul><li>And of course, we can remove and shutdown an instance as easily </li></ul><ul><li>Big advantage, we can use the same SimpleDB technique for other cloud providers too! </li></ul>
  25. 25. Amazon is cheap! <ul><li>Doubling up instances for redundancy is very cheap to do </li></ul><ul><li>Mediafed is running an on-demand system for a fraction of the cost it would have cost to run a traditional server room </li></ul><ul><li>Turn off servers at low points in the week, no cost, no hassle! </li></ul>
  26. 26. No more hardware hassles! <ul><li>The previous hosting company for Mediafed DID NOT recover from a hard disk failure </li></ul><ul><li>Fortunately Mediafed had a back up </li></ul>
  27. 27. Some cloud advice …
  28. 28. Not a cloud in sight ! Forget the marketing hype - Cloud Providers do go down
  29. 29. … and no two providers are the same NO CLOUD STANDARD
  30. 30. So can you move the whole house?
  31. 31. Don’t buy into the hype <ul><li>All the usual ‘out sourcing’ rules apply </li></ul><ul><ul><li>Always look at the worse case scenario </li></ul></ul><ul><ul><li>Plan for the worse case scenario </li></ul></ul><ul><ul><li>Assume the worse case scenario will happen </li></ul></ul><ul><ul><li>Practice the worse case scenario </li></ul></ul>
  32. 32. Remember … you can’t just nip down and reboot the server
  33. 33. But take a moment and … <ul><li>Think of your servers as single use devices instead of the typical “LAMP” </li></ul><ul><li>Managing up to 10 servers via SSH isn’t a problem </li></ul><ul><ul><li>is doing the same for 10, 20, 50+ servers really practical? </li></ul></ul><ul><li>Hold off on that purchase order to Dell and talk to your software guys to see if you can re-jig current things to work within the ‘cloud platform’, or Linux-OnDemand </li></ul>
  34. 34. [email_address] UK +44 1556 568 103 / US +1 650 331 9758 the cloud experts – putting it to work