Building a platform from open source


Published on

Building a platform from open source

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Building a platform from open source

  1. 1. Building  a  pla+orm  from  open  source   Dustin Whittle
  2. 2. Building  a  pla+orm  from  open  source  
  3. 3. Who  am  I?     •  A  hacker  on  the  web  for  the  last  decade   •  Worked  with  symfony  since  open  source   •  Developed  ysymfony  at  Yahoo!   •  Consultant  helping  startups  scale  web  apps   •  A  speaker  at  conferences  around  the  world     @dusBnwhiDle   hDp://  
  4. 4. When  I  started  developing…  
  5. 5. It  was  really  simple  
  6. 6. 1  shared  host  /  simple  PHP  scripts  
  7. 7. Then  I  started  working  on  lots  of   consulBng  projects  
  8. 8. I  realized  I  needed  a  beDer  set  of  tools  
  9. 9. I  started  exploring  web  frameworks  
  10. 10. I realized reading and using other peoples code made me a better, more productive developer
  11. 11. I started talking about how frameworks like symfony and YUI improve quality and productivity at conferences
  12. 12. Then I went to work at Yahoo to help build a frontend platform based on an open source framework
  13. 13. I realized web application stacks are very similar Lots of the same problems and solutions
  14. 14. Users   Load  Balancers   Frontend   PHP   APC,  PEAR,  PECL   Linux     symfony  /  YUI     Apache   Custom  Modules   Backend   MySQL/Memcache   Web  Services  /  YQL   Ad  API   User  API  
  15. 15. Why  a  frontend  pla+orm?   Build  applicaBons  to  requirements    Do  exactly  what  you  need:  no  more,  no   less     Choose  open  source  or  build  your  own   Everyone  uses  a  framework   If  you  use  open  source,  support/maintenance  is   free   Understand  that  frameworks  add  a  lot  of   overhead  (features  have  a  cost)  
  16. 16. Needs  from  a  frontend  pla+orm   Fit  server  environments  (Linux/PHP5/Apache)   Development  Cycle  –  How  easy  to  develop,  test,  and   deploy?   Clean  separaBon  between  data,  logic,  and  display   Independent  model  layer  to  fit  service  oriented   architecture   Extensible  and  pluggable   InternaBonalizaBon  and  localizaBon  support   Detailed  documentaBon  and  acBve  community  of   support   Open  source  and  ability  to  contribute  back  
  17. 17. Why  a  framework  at  all?   Another  so]ware  layer  (ysymfony,  yphp,   yapache)   Factors  out  common  paDerns   Code  Layout   ConfiguraBon   URL  RouBng   AuthenBcaBon  /  Security   Form  ValidaBon  /  RepopulaBon   InternaBonalizaBon  /  LocalizaBon   Encourages  good  design   AbstracBon  >  Consistency  >  Maintainability    
  18. 18. The  choice  to  adopt  symfony?   Philosophy   Full-­‐stack  framework  for  building  complex  web   applicaBons   Adopt  best  ideas  from  anywhere,  using  exisBng   code  if  available  (Mojavi,  Prado,  Rails,  Django)    Design   Clean  separaBon  between  Model,  View,  and   Controller   Controller  using  modules  and  acBons   Views  using  templates  in  straight  PHP  with  helpers   Easy  to  reuse  view  modules  to  compose  a  page   Layouts,  Components,  ParBals,  Slots  
  19. 19. The  choice  to  use  symfony   Configurability  /  Flexibility   Features  we  do  not  want  are  easily  disabled   Use  of  factories  for  easy  customizaBon   DocumentaBon  /  Support  Community   The  DefiniBve  Guide  to  symfony   Askeet,  Jobeet,  Cookbooks,  Advents   AcBve  community  with  wiki,  mailing  lists,   forums,  irc  channel    
  20. 20. A  look  at  Yahoo!  Answers   hDp://   Yahoo!  Answers  is  the  largest  collecBon  of   human  knowledge  on  the  Web  with  more   than  135  million  users  and  515  million   answers  worldwide  (Yahoo!  Internal  Data,   March  2008).   Yahoo!  Answers  is  the  2nd  ranked  educaBon   &  reference  site  on  the  web  (comScore)   Available  in  26  markets  and  12  languages  
  21. 21. Yahoo!  Answers  at  the  beginning   Started  as  a  small  development  team  on   PHP4  from  a  fork  of  Yahoo!  Taiwan   Knowledge+   Launched  December  2005  by  December   2006  there  were  60  million  users  and  65   million  answers   The  code  base  eventually  became  difficult  to   maintain  and  iterate  new  features   Large  distributed  development  teams  
  22. 22. JavaScript  Framework   UBliBes  -­‐  YAHOO,  Dom,  Event,  AnimaBon,  Browser   History  Manager,  ConnecBon  Manager,  Cookie,   DataSource,  Drag  and  Drop,  Element,  Get,   ImageLoader,  JSON,  Resize,  Selector,  Loader   Controls  /  Widgets  -­‐  AutoComplete,  BuDon,   Calendar,  Charts,  Color  Picker,  DataTable,   ImageCropper,  Rich  Text  Editor,  Slider,  Uploader   Container  (Module,  Overlay,  Panel,  ToolBp,  Dialog),   Layout  Manager,  Menu,  TabView,  TreeView   Debug  –  Logger,  Profiler,  Test   Y!  Developer  Network  –  YUI  Javascript  
  23. 23. CSS  FoundaBon   Reset  -­‐  Neutralizes  browser  CSS  styles   Base  -­‐  Applies  consistent  style  foundaBon   Fonts  -­‐  FoundaBon  for  typography  and  font-­‐sizing   Grids  -­‐  Thousands  of  wireframe  layouts   User  Interface  Design  PaDerns  Library   Proven  soluBons  to  common  interfaces   hDp://   Grade  Browser  Support  /  Progressive  Enhancement   Y!  Developer  Network  –  YUI  CSS  
  24. 24. More  than  275  funcBonal  examples   hDp://   YSlow  +  Performance  Rules   hDp://   YUI  Blog   hDp://   Mailing  List  @  Yahoo!  Groups   hDp://­‐ javascript/     Y!  Developer  Network  –  DocumentaBon  
  25. 25. The  big  picture   A  complete  pla+orm  for  building  web   applicaBons  from  frameworks   PHP  Framework   JavaScript  Framework   CSS  Framework   UI  Design  PaDerns  +  Best  PracBces   Development  Tools  (logger,  profiler,  debugger,   docs)   Unit  +  FuncBonal  TesBng  Frameworks  (LIME  /   YUI  Test)   Deployment  Tools  
  26. 26. What  does  Yahoo!  change?   Minor  changes  to  fit  our  environment   Most  changes  are  easily  implemented  via  factories/plugins    Dropped  the  ORM  and  pushed  down  the  stack  (SOA)   Added  a  parallel  API  Dispatcher  (ysfAPIClientPlugin)   Added  dimensions  to  configuraBons   (ysfDimensionsPlugin)   Integrated  R3  translaBon/template  management   (ysfR3Plugin)   R3  -­‐  hDp://     Created  a  build  and  deployment  soluBon  (ysfBuildPlugin)   Uses  internal  tools  for  packaging/deployment   Integrate  support  for  Y!  User  Interface  libraries   (ysfYUIPlugin)  
  27. 27. Propel  or  Doctrine  or  ???   No  ORM  for  large  projects   Doctrine  for  medium  sized  projects   Service  Oriented  Architecture   Pla+orms/Data  as  services   Thin  Controller/Fat  Model   Use  PHP  as  the  frontend  glue   No  heavy  li]ing  in  PHP  =  Push  down  the  stack   Java/C++/Erlang  +  JSON/XML  
  28. 28. Localizing  with  dimensions   Cascading  ConfiguraBon  based  on  YAML   Framework  -­‐>  Project  -­‐>  ApplicaBon  -­‐>  Module   Extending  the  cascade  to  be  based  on   dimensions   Dimensions  can  be  anything  (and  can  be  chained   together)   Data  Center  +  Environment  for  customizing   configuraBons   Culture  for  localizing  user  interface  +  data   Theme  for  customizing  look  and  feel   User  info  (is  user  on  corporate  intranet?)   Caching  
  29. 29. Development  Tools  
  30. 30. A  build  and  deployment  system   Aggregate  and  minify  stylesheets  and  javascripts   Rewrite  templates,  css,  js  for  CDN  (Akamai,  S3,  …)   Generate  translaBons  for  configuraBons  +  templates   Generate  configuraBon  cache   Aggregate  core  classes  +  remove  debug  statements   Run  lint,  unit,  funcBonal  tests   Package  applicaBons  as  .tgz   Deployment  via  packages  
  31. 31. Performance   Hello  world  benchmarks  are  not  useful   You  do  not  use  a  framework  to  write  hello  world   die(‘hello  world’);   Performance  is  relaBve  to  the  features  you  use   ORM   I18N   Output  Escaping   Forms/ValidaBon     Ability  to  scale  !=  performance  
  32. 32. What  does  it  mean  to  scale?   A  system  whose  performance  improves  a]er   adding  hardware,  proporBonally  to  the   capacity  added,  is  said  to  be  a  scalable   system.   High  Availability  +  Scalability  +  Performance   Bigger  dataset,  more  traffic,  maintainable   Not  about  performance   PHP  is  slow,  but  it  is  very  rarely  your  boDleneck   Languages  do  not  scale,  architectures  do  
  33. 33. Scaling  –  The  basics  in  PHP    PHP  is  rarely  the  boDleneck  (even  though  it  can  be  slow)   “Most  performance  comes  not  from  the  language,  but  from  applicaBon   design”  –  Rasmus    Share  Nothing  Architecture   Independent,  self-­‐sufficient,  no  single  point  of  contenBon   No  local  storage  =  No  PHP  Sessions   Use  a  database   Use  a  small  signed  cookie  (beDer)   Important  data  in  database   Individual  expiraBon  on  session  objects   Small  data  items   Use  an  in-­‐memory  cache   Memcache    Premature  opBmizaBon  is  the  root  of  all  evil.   “  vs.  ‘,  echo  vs.  print      Improve  performance  by  queuing  jobs  that  can  take  a  long  Bme  and   caching  data  that  is  expensive  to  query  
  34. 34. Once  an  app  has  a  well  designed   architecture,  developers  can  focus  on   performance  specific  to  the  app  
  35. 35. Use  Xdebug/Webgrind  and  XHProf  to   debug  and  profile  applicaBon   performance  
  36. 36. hDp://  
  37. 37. Web  applicaBon  performance  isn’t  just   about  the  backend,  a  lot  of  open  source   tools  can  help  improve  the  frontend.  
  38. 38. As  the  framework  matured  and  gained   recogniBon  more  companies  started   choosing  symfony    
  39. 39. At  Yahoo  scaling  applicaBons  was  easy:   infrastructure/process  already  exists  
  40. 40. As  I  work  on  my  own  startup  I  realized  I   need  to  create  the  same  scalable  and   fault  tolerant  environment,  except   without  invesBng  millions  of  dollars.  
  41. 41. A  quality  pla+orm  is  built  from  best   pracBces  
  42. 42. OperaBonal  Best  PracBces   •  Global  data  centers  for  performance  and   fault  tolerance   •  Automated  provisioning  of  networks,   servers,  storage,  etc.   •  Automated  builds  +  deployments   •  Monitoring  +  ReporBng  (performance/ costs)   •  Centralized  Logging  +  Management  
  43. 43. Development  Best  PracBces   •  Clean  /  Decoupled  /  Documented  code   •  Stand  on  the  shoulders  of  giants   •  Leverage  open  source  communiBes     •  Source  Control  +  Project  Management   •  Unit  +  FuncBonal  TesBng   •  ConBnuous  IntegraBon   •  Either  you  are  always  ready  to  deploy  or  you   are  never  ready  to  deploy  
  44. 44. Do  it  yourself  for  cheap   Open  source  so]ware  =  Free   •  Linux   •  Apache   •  PHP   •  MySQL  /  Mongo   •  Memcache  /  Gearman   •  symfony  /  Doctrine  /  Twig  /  Swi]  /   PHPUnit  
  45. 45. Today  it  is  possible  for  anyone  to  scale   a  web  pla+orm  just  like  a  big  company.  
  46. 46. Welcome  to  the  cloud.  
  47. 47. It’s  so  cheap  its  free!       Who  doesn’t  love  free?  
  48. 48. hDp://  
  49. 49. A  load  balanced  cluster  with  a   relaBonal  database  and  storage   configured  how  ever  you  want  in   mulBple  data  centers  around  the  world.  
  50. 50. For  free.  And  then  for  relaBvely  cheap.  
  51. 51. Its  not  fair!  Only  for  new  customers.  
  52. 52. A  way  to  monitor  and  manage  your   cloud  
  53. 53. A  free  global  DNS  cloud  service  
  54. 54. A  free  source  control  +  issue  tracking   service  that  enables  public  or  private   collaboraBon  
  55. 55. A  complete  pla+orm  for  developing,   tesBng,  and  deploying  scalable  web   applicaBons.  
  56. 56. Free  cloud  infrastructure:    Amazon  +  Zerigo  +  Landscape     Free  open  source  code:    Symfony  +  Zend  +  YUI  +  AlloyUI     Free  development  tools:    GitHub  +  Xdebug  +  PHPUnit     Built  in  best  pracBces  +  support  
  57. 57. Many  lessons  have  been  learned  and  a   beDer  pla+orm  has  evolved.  
  58. 58. Symfony2  and  YUI3  
  59. 59. BeDer  Components:    lighter,  faster,  cleaner     BeDer  Development  Model:    ConBnuous  IntegraBon    BeDer  tesBng/documentaBon     Stronger  CommuniBes:    Github  enables  contribuBons    Galleries  for  plugins/bundles  
  60. 60. Symfony   Symfony  2  is  a  set  of  cohesive  yet  decoupled   components   Easier  to  use  a  component  to  solve  a  single   problem   Easier  to  build  micro  frameworks  that  solve   specific  problems   Teams  generally  prefer  soluBons  that  are   specific  to  their  exact  problem   Selling  the  full  stack  can  be  difficult  when  a  team   only  wants  a  few  components
  61. 61. Symfony2  stands  on  the  shoulders  of   giants  and  doesn’t  reinvent  the  wheel.  
  62. 62. We  even  use  Zend  Framework  
  63. 63. Symfony2  is  so  good  it  is  only  alpha  and   there  are  already  major  sites  live  in   producBon.  
  64. 64. Take  a  look  at  the  Symfony2  web   profiler  
  65. 65. Take  a  look  at  the  Symfony  sandbox!  
  66. 66.­‐sandbox  
  67. 67. QuesBons?  
  68. 68. Hiring!   If  you  are  a  great  developer  looking  to  make   your  next  move,  let  me  introduce  you  to:     Plancast   TripIt   Awesm   OpenSky   Gogobot  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.