AWS Customer Presentation - Mediafed


Published on

Alan Williamson, Co-Founder of Mediafed presents at AWS Start-Up Event - London.

Published in: Technology, Sports
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AWS Customer Presentation - Mediafed

  1. Amazon AWS & Mediafed Alan Williamson Director, MediaFed ltd Co-Founder, aw2.0 ltd the cloud experts
  2. MediaFed Ltd <ul><li>RSS Experts </li></ul><ul><li>Monetize feeds </li></ul><ul><li>Track feeds </li></ul><ul><li>Value Added feeds </li></ul><ul><li>Feed Compliance </li></ul><ul><li>Focused on Premium publishers </li></ul>
  3. MediaFed’s work flow MediaFed Publishers MediaFed Services Millions of RSS Clients
  4. Traditional Setup <ul><li>Fixed number of ‘real’ strokeable servers </li></ul><ul><li>Professionally managed servers </li></ul><ul><li>Costing £xxx’s per month just for one server </li></ul>
  5. Then things got interesting Started to grow FAST! /
  6. Where did MediaFed need to scale? <ul><li>Front end Web Servers </li></ul><ul><ul><li>Servicing the requests for feeds, article views, and click thru’s </li></ul></ul><ul><li>RSS Ad Servers </li></ul><ul><ul><li>Processing the ad insertion’s and click-thru’s </li></ul></ul><ul><li>Feed Crawlers </li></ul><ul><ul><li>Retrieving the feed content, cleaning and conversion </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Storage of feed data, and statistical analysis, running to TB’s of data </li></ul></ul>
  7. <ul><li>The current hosting company was asked for a quote </li></ul><ul><li>It came as quite a shock to us </li></ul>
  8. <ul><li>We needed servers FAST </li></ul><ul><li>Time Quoted 10 working days! </li></ul><ul><li>Need a modern approach </li></ul>
  10. Which Cloud Provider … <ul><li>… had experience with a number of cloud providers </li></ul><ul><li>… each one with their strengths and weaknesses </li></ul><ul><li>… different price/performance points </li></ul><ul><li>But … </li></ul>
  11. It’s “Hug Amazon” day!
  12. Amazon’s EC2 HOT features <ul><li>Instance start up parameter passing </li></ul><ul><li>Instance snap shot </li></ul><ul><li>VERY FAST instance startup </li></ul><ul><li>Large ‘ free ’ disk space with each instance </li></ul>
  13. Some EC2 problems … <ul><li>No VLAN support </li></ul><ul><li>No ability to turn off public IP address </li></ul><ul><li>No reverse DNS </li></ul>
  14. .. and instances do go down! But “ssssh” you didn’t hear it from me!
  15. Our Amazon Family Usage <ul><li>Amazon EC2 usage </li></ul><ul><ul><li>Runs the main infrastructure; the bread’n’butter </li></ul></ul><ul><li>SimpleDB </li></ul><ul><ul><li>Used for something quite special </li></ul></ul><ul><li>S3 </li></ul><ul><ul><li>Long term storage of log files and DB snapshots </li></ul></ul>
  16. Usage of EC2 Instances <ul><li>The majority of all our services are Java based </li></ul><ul><ul><li>High performance custom built code; no J2EE! </li></ul></ul><ul><li>Load balancing provided by nginx </li></ul><ul><li>Database provided by MySQL 5 </li></ul>
  17. Provisioning of instances <ul><li>Manage our deployment as a single ‘stack’ </li></ul><ul><ul><li>/opt/mediafed/ </li></ul></ul><ul><li>All applications, scripts, binaries in there </li></ul><ul><li>Require no external, pre-installed software </li></ul><ul><li>Complete stack is held within SVN </li></ul><ul><ul><li>First thing that happens when an instance boots up is to update itself with the latest from SVN </li></ul></ul>
  18. Keep it Simple <ul><li>No multiple instances to worry about </li></ul><ul><li>Any instance can do any job in the system we require </li></ul><ul><li>Horrendously transportable! </li></ul><ul><li>Instances are ready to work as soon as they are powered up </li></ul>
  19. Our goals … <ul><li>Over reaching goal was to make it an SSH free zone </li></ul><ul><li>Misbehaving instances would simply be shut off and a new one powered up </li></ul><ul><li>We viewed the “instance” as a component that could be used; not as a server that it really was </li></ul>
  20. Starting up … <ul><li>From the time we issue the command to start up to it actually doing REAL work </li></ul><ul><ul><li>Usually 3 to 4 minutes!!!! </li></ul></ul>
  21. EC2 #1 Killer Feature <ul><li>Parameter passing into EC2 instances </li></ul><ul><li>We pass in a configuration string on startup that specifies what it will do: </li></ul><ul><ul><li>ec2-xxxx -d “TYPE=WEB,ROLE=RSSPOOL3” </li></ul></ul><ul><li>Instance runs a ‘boot script’ that then runs up the necessary service we require </li></ul>
  22. But others services in the system need to know about the new instances ?????
  23. SimpleDB! <ul><li>We use SimpleDB as a sophisticated internal ‘DNS’ or service directory </li></ul><ul><li>When a service is started, it registers its role with SimpleDB including all IP, Port and other information </li></ul><ul><li>Other services keep an eye on changes in this configuration and automatically update their own configuration files accordingly </li></ul>
  24. Amazon SimpleDB as DNS <ul><li>Allow us to add in new web servers into the farm for load balancing without doing a thing except start it up </li></ul><ul><li>Add in additional MySQL slaves/masters without touching anything </li></ul><ul><li>And of course, we can remove and shutdown an instance as easily </li></ul><ul><li>Big advantage, we can use the same SimpleDB technique for other cloud providers too! </li></ul>
  25. Amazon is cheap! <ul><li>Doubling up instances for redundancy is very cheap to do </li></ul><ul><li>Mediafed is running an on-demand system for a fraction of the cost it would have cost to run a traditional server room </li></ul><ul><li>Turn off servers at low points in the week, no cost, no hassle! </li></ul>
  26. No more hardware hassles! <ul><li>The previous hosting company for Mediafed DID NOT recover from a hard disk failure </li></ul><ul><li>Fortunately Mediafed had a back up </li></ul>
  27. Some cloud advice …
  28. Not a cloud in sight ! Forget the marketing hype - Cloud Providers do go down
  29. … and no two providers are the same NO CLOUD STANDARD
  30. So can you move the whole house?
  31. Don’t buy into the hype <ul><li>All the usual ‘out sourcing’ rules apply </li></ul><ul><ul><li>Always look at the worse case scenario </li></ul></ul><ul><ul><li>Plan for the worse case scenario </li></ul></ul><ul><ul><li>Assume the worse case scenario will happen </li></ul></ul><ul><ul><li>Practice the worse case scenario </li></ul></ul>
  32. Remember … you can’t just nip down and reboot the server
  33. But take a moment and … <ul><li>Think of your servers as single use devices instead of the typical “LAMP” </li></ul><ul><li>Managing up to 10 servers via SSH isn’t a problem </li></ul><ul><ul><li>is doing the same for 10, 20, 50+ servers really practical? </li></ul></ul><ul><li>Hold off on that purchase order to Dell and talk to your software guys to see if you can re-jig current things to work within the ‘cloud platform’, or Linux-OnDemand </li></ul>
  34. [email_address] UK +44 1556 568 103 / US +1 650 331 9758 the cloud experts – putting it to work