Cloud based Projects at Belfast eScience Centre


Published on

A presentation by Terry Harmer of the Belfast eScience Centre at the Repositories and the Cloud meeting organised by Eduserv and JISC in London on Feb 23 2010.

Published in: Education, Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Cloud based Projects at Belfast eScience Centre

  1. 1. Cloud based Projects atBelfast e-Science CentreAn Overview<br />Terry Harmer<br />London<br />1<br />February 2010<br /><br />
  2. 2. What do I do?<br />Technical Directorof Belfast e-Science<br />Develop project ideas for digital economy applications<br />Form consortia to bid for funding <br /> … usually write the project funding proposals<br /> … funding from EPSRC, INI, TSB and private companies<br />Lead Technical architect for projects<br />Project Manager<br /> … also do software development<br />These projects are (and increasingly so) based around utility infrastructure consisting of owned and multiple utility vendors.<br />London <br />February 2010<br />2<br />
  3. 3. Talk Outline<br />Talk objective<br />BeSC? <br />..some history of BeSC applications<br />Evolution of our infrastructure<br />2 Examples of utility-centric deployed applications<br />Issues<br />London<br />February 2010<br />3<br />
  4. 4. Objective<br />To present some large-scale projects that are in field deployment with established user groups<br />Dynamic and utility cloud focused<br />Why this approach and what advantages has this approach given us.<br />Issues, advantages, problems, pitfalls, <br />Headline<br />Technical<br />London <br />February 2010<br />4<br />
  5. 5. Belfast e-Science Centre?<br />Belfast e-Science was established in 2002 with funding from EPSRC and the DTI under the UK e-Science programme.<br />Funded since then by TSB, EPSRC, INI, MoD, QinetiQ<br />Currently one of four EPSRC Platform Award funded e-Science Centres in the UK.<br />BeSC is entirely self funding (and has been since 2002)<br />We have the attitude and tend to operate like a small R&D company<br />Don’t really use resources within a University infrastructure<br />Have close connections with many companies but less with host Uni.<br />Mainly deal with commercial users and organisations.<br />Have a tight budget and (perhaps too) big ambitions. <br />London <br />February 2010<br />5<br />
  6. 6. A bit of BeSC Context<br />As somewhat of an accidental decision, BeSC focused on commercial/industrial applications<br />Some of the accident was a result of the initial DTI Centre funding emphasising commercial applications and Tony Hey’sdarwinian view of e-science programme.<br />The industrial/commercial focus grew from the challenges within the application areas which we felt offered something new and distinct to the e-Science community.<br />No one else was focusing there so it made us unique<br />There are real and significant challenges<br />London<br />February 2010<br />6<br />
  7. 7. Why Commercial/Industrial?<br />Media domain <br />Speed : user driven<br />Security : video is treated as money.<br />large data sizes : larger than LHC for example<br /><ul><li>Financial domain
  8. 8. Speed : 100,000+ share trades per second
  9. 9. Security : company business
  10. 10. heavily regulated – where how data is moved
  11. 11. On-demand infrastructure/resources
  12. 12. Hosting/utility management are necessary parts of a dynamic digital economy and technology still required. </li></ul>Applications<br />Digital media –BBC, QinetiQ<br />Financial Services –First Derivatives, ??<br />Military Applications –UK MoD, QinietQ<br />Technology<br />Resources, Auto-deployment, on-demand resources<br />Management of owned and 3rd party clouds<br />Autonomic management, SLAs and scaling<br />London<br />February 2010<br />7<br />
  13. 13. Experimental Environment (2003)<br />London<br />February 2010<br />8<br />BBCNI<br />
  14. 14. Experimental Environment (Spring 2004)<br />London <br />February 2010<br />9<br />
  15. 15. Experimental Environment (Autumn 2004)<br />London <br />February 2010<br />10<br />
  16. 16. Experimental Environment (Autumn 2005)<br />London <br />February 2010<br />11<br />
  17. 17. Deployment Environment 2010<br />London <br />February 2010<br />12<br />
  18. 18. 2 Examples<br />Financial Services with FD<br />On-demand media with BBC/QinetiQ/BT<br />London <br />February 2010<br />13<br />
  19. 19. First Derivatives plc<br />Provider of software to banks and financial services companies.<br />Have software in 9 of the top 10 banks.<br />Develop auto-trading software<br />Provide financial services, consulting, technology outsourcing, design etc<br />London<br />February 2010<br />14<br />
  20. 20. Simple Architecture<br />London<br />February 2010<br />15<br />
  21. 21. Solution 1: 2004-2005<br />London<br />February 2010<br />16<br />
  22. 22. Current Cloud Solution (2006- )<br />London<br />February 2010<br />17<br />
  23. 23. Digital Media (2002-)<br />Working in the evolving on-demand media environment<br />Partners: BBC/QinetiQ/BT….completed late 2009<br />Started pre- iPlayer and YouTube!<br />Concern early was on better resource utilisation in and expensive and highly dynamic environment.<br />Early model of pooled resources<br />Most recently in on-demand media infrastructures<br />Project PRISM with BBC/QinetiQ/BT<br />Supporting game console to Phone to set-top box access.<br />Much of our work now is on military media infrastructures.<br />London <br />February 2010<br />18<br />
  24. 24. A circuit based infrastructure<br />London<br />February 2010<br />19<br />
  25. 25. A Media SOA (this Slide dates from 2003!)<br />London<br />20<br />February 2010<br />
  26. 26. Mobile Non-geographic services (slide from 2005!)<br />London<br />21<br />February 2010<br />
  27. 27. A Dynamic Utility Cloud<br />London<br />February 2010<br />22<br />
  28. 28. Work flow<br />Work with BBC is winding down<br />Expertise is moving to military applications.<br />Managing currently around 1+ petabytes of media content<br />Has managed close to 2petabytes in the last 4 years.<br />London <br />February 2010<br />23<br />
  29. 29. Infrastructure Summary<br />Dynamic collections of services<br />Managing real user groups<br />Service scale to established SLAs<br />We attempt to keep our deployed infrastructure low<br />Our infrastructure is a mix of owned and utility infrastructure<br />Buying capacity and storage on demand is our norm.<br />increasingly the utility part is the majority for processing and user interfaces<br />Owned infrastructure is a secure repository.<br />London <br />February 2010<br />24<br />
  30. 30. Advantages (headline)<br />Develop an infrastructure that suits the application we are deploying.<br />The cost of ownership is pretty low.<br />As an R&D organisation we can punch above our small size and relatively small budget.<br />Experiment with great flexibility running parallel shared infrastructures.<br />Reach out to real user groups <br />….. Unconstrained by (often entirely justified) corporate/academic infrastructure procedures.<br />You own what you need to own for as long as you need to own it and it can be configured exactly for your needs.<br />London <br />February 2010<br />25<br />
  31. 31. Issues (general)<br />Utility resource market is immature<br />We treat providers as a commodity market place.<br />The offerings can be difficult to compare<br />No standard unit of compute/storage<br />Prices will be dependant on the user usage pattern<br />What you get and what you can buy varies widely<br />Some attempts at customer lock-in to providers<br />Multi-provider clouds can be (relatively) expensive<br />Need to think carefully about what is stored, where it is stored, how long it is stored, who has access<br />We have put a lot of work into automated policy based content management….because we do not have the people to manage this.<br />Based around SAML and XACML<br />London <br />February 2010<br />26<br />
  32. 32. Issues (general)<br />Provider APIs and features constantly changing.<br />No standard API<br />New services and providers appearing. <br />APIs not very well documented<br />Weak SLAs from providers<br />Currently we build our infrastructure assuming there is no SLA.<br />London <br />February 2010<br />27<br />
  33. 33. Issues (technical)<br />Machine performance unpredictable. <br />CPU features especially unpredictable and can make a big difference to compute-heavy tasks <br />e.g. we are heavy video transcoding users.<br />individual instances can be (surprisingly) unreliable (hosts DO crash)<br />Bandwidth unpredictable and can be costly<br />Required to manage OS images<br />proliferation of images; <br />using anything but vendor images requires trust in creator. <br />nobody has a trust framework - you have to trust that user<br />creating own images (or using other peoples) means more machines to keep up to date!<br />London <br />February 2010<br />28<br />
  34. 34. Issues (security)<br />low latency to other consumers' boxes decreases attacker cost and time to perform timing attacks<br />nefarious, rich attackers can get on your box and slow you down or potentially compromise key generation<br />See<br />DDoS on cloud providers can be very damaging to everyone in it; <br />Larger providers just increase the cost of the attack but reward is also high<br /> (see )<br /><ul><li>No (meaningful) security QoS</li></ul>post-attack analysis challenging - in most clouds you cannot inspect a disk to see logs without starting up machine<br /><ul><li>Potential data security issues: who has access to physical boxes? </li></ul>e.g. Amazon recommends all data on disks be encrypted<br />London <br />February 2010<br />29<br />
  35. 35. Issues (banal)<br />Area not well understood<br />What was the inventory tag of the machine?<br />Why are you not using our in-house IS cluster?<br />London <br />February 2010<br />30<br />