Your SlideShare is downloading. ×
0
Building a platform from open source
SYMFONY-PROJECT.COM | DEVELOPER.YAHOO.COM
Overview
Why symfony?
symfony vs. ysymfony
Social Search: Delicious and Answers
International: ShopGenie.co.uk, FoxyTunes,...
Who am I?
Working with symfony since open source
  symfony Core Team Member
Responsible for the development and support of...
EXAMPLES | TUTORIALS | CODE SAMPLES

DEVELOPER.YAHOO.COM
FREEBSD | LINUX | APACHE | PHP | MYSQL | BUGZILLA | HADOOP | SYMFONY

    YAHOO! IS POWERED BY OPEN
      SOURCE TECHNOLOG...
W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL

YAHOO! EMBRACES OPEN
    STANDARDS
YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES

      YAHOO! GIVES BACK TO OPEN
               SOUR...
YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING
  | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATIO...
Conferences | Hack Days | HackU | Tech Talks | YDN Theater

 YAHOO! ENGAGES COMMUNITIES WITH
OPEN HACK EVENTS AROUND THE W...
Users	
  

                                         Load	
  Balancers	
  

                                               ...
Why a frontend platform?
Rasmus says “frameworks are not well suited for
  Yahoo!”
  Build applications to requirements
  ...
Yahoo! needs from a frontend platform
Fit existing environment (RHEL/PHP5/Apache)
Development Cycle – How easy to develop,...
Why a framework at all?
Another software layer (ysymfony, yphp, yapache)
Factors out common patterns
  Code Layout
  Cong...
The choice to adopt symfony?
Philosophy
  Full-stack framework for building complex web
    applications
  Adopt best idea...
The choice to use symfony
Congurability / Flexibility
   Features we do not want are easily disabled
   Use of factories ...
Why ysymfony for Yahoo! teams?
Eliminate common patterns by adding a layer on PHP
   Code layout/structure (MVC)
   Congu...
How Yahoo! uses symfony
Full Stack
   Yahoo! Bookmarks
   Yahoo! Answers
   Delicious
   Yahoo! Widgets
   Yahoo! Applicat...
A look at Yahoo! Answers
http://answers.yahoo.com
Yahoo! Answers is the largest collection of human
  knowledge on the Web...
Yahoo! Answers at the beginning
Started as a small development team on PHP4 from a
  fork of Yahoo! Taiwan Knowledge+
Laun...
The big picture
A complete platform for building web applications
  from frameworks
  PHP Framework
  JavaScript Framework...
What does Yahoo! change?
Minor changes to t our environment
   Most changes are easily implemented via factories/plugins
...
Propel or Doctrine or ???
No ORM for large projects
Doctrine for medium sized projects
Service Oriented Architecture
  Pla...
Localizing with dimensions
Cascading Conguration based on YAML
  Framework -> Project -> Application -> Module
Extending ...
A build and deployment system
Aggregate and minify stylesheets and javascripts
Rewrite templates, css, js for CDN (Akamai,...
Debug Toolbars
Hello World Performance
Hello world benchmarks are generally not useful
   You don’t use a framework to write hello world
...
What does it mean to scale?
A system whose performance improves after adding
  hardware, proportionally to the capacity ad...
Scaling – The basics in PHP
PHP is rarely the bottleneck (even though it can be slow)
   “Most performance comes not from ...
Scaling Databases – The basics
Drop the ORM
   It’s a choice of convenience over performance
Master/Slave Replication
   F...
Improving latency with Caching
Always use PHP opcode cache (APC, Xcache, etc)
   Use for routing and i18n cache
Memcache (...
Tweaking Performance
Do not use .htaccess (move to real apache cong)
Set a minimal include path
Increase realpath_cache_s...
Symfony (v2)
Symfony 2 is a set of cohesive yet decoupled
  components
  This makes it much easier to use single component...
Do it yourself for cheap
Open source software = Free
   Apache
   PHP
   MySQL
   Memcache / Perlbal / MogileFS / Squid / ...
What is Yahoo! Developer Network?
 The Yahoo! Developer Network offers open source tools
   and open data APIs to make it ...
Y! Developer Network – YUI Javascript
 JavaScript Framework
    Utilities - YAHOO, Dom, Event, Animation, Browser History
...
Y! Developer Network – YUI CSS
 CSS Foundation
    Reset - Neutralizes browser CSS styles
    Base - Applies consistent st...
Y! Developer Network – Documentation
 More than 275 functional examples
    http://developer.yahoo.com/yui/examples/
 YSlo...
A Query Language for the Web




      SELECT	
  	
  *	
  FROM	
  INTERNET	
  
Before YQL
 Thousands of web services that provide valuable data
 Require developers to read documentation and form
   URL...
Y! Open Stack – YQL
 SQL-Like Language
    Synonymous with Data access
    Familiar to developers
    Expressive enough to...
developer.yahoo.com/yql/console/
YQL – Open Tables
   Twitter                            Flickr
   Google                             Upcoming
   Facebook ...
YQL – Open Tables
YQL – Open Tables
YQL – Open Tables
YQL - Examples

 select * from social.connections
 select * from delicious.feeds.popular
 select * from ickr.photos.inter...
YQL – API End Points
 OAuth Endpoint
 http://query.yahooapis.com/v1/yql?q=...

 Public Endpoint
 http://query.yahooapis.co...
Open Source Communities
http://www.symfony-project.org/jobeet/1_2/
Documentation
http://www.symfony-project.org/doc

Practical symfony - Jobeet
The Denitive Guide to symfony
The symfony Re...
Books worth reading
QUESTIONS?
(I will be around, so please say hello and ask questions)
EXAMPLES | TUTORIALS | CODE SAMPLES
DEVELOPER.YAHOO.COM
WANT TO JOIN YAHOO?
  WE ARE HIRING!
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
OSDC.tw - Building a platform from open source
Upcoming SlideShare
Loading in...5
×

OSDC.tw - Building a platform from open source

1,850

Published on

Open Source Developer Conference Taiwan- Building a platform from open source

Presented by Dustin Whittle

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,850
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
34
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Transcript of "OSDC.tw - Building a platform from open source"

  1. 1. Building a platform from open source
  2. 2. SYMFONY-PROJECT.COM | DEVELOPER.YAHOO.COM
  3. 3. Overview Why symfony? symfony vs. ysymfony Social Search: Delicious and Answers International: ShopGenie.co.uk, FoxyTunes, Digu Yahoo! Open Strategy Developer Tools YUI, Design Patterns, Tutorials Data & Social APIs YQL: Yahoo! Query Language
  4. 4. Who am I? Working with symfony since open source symfony Core Team Member Responsible for the development and support of ysymfony at Yahoo! Y! Answers, Delicious, Y! Widgets, Y! Application Platform Developer advocacy for Yahoo! Developer Network Contact dustin@yahoo-inc.com @dustinwhittle
  5. 5. EXAMPLES | TUTORIALS | CODE SAMPLES DEVELOPER.YAHOO.COM
  6. 6. FREEBSD | LINUX | APACHE | PHP | MYSQL | BUGZILLA | HADOOP | SYMFONY YAHOO! IS POWERED BY OPEN SOURCE TECHNOLOGIES
  7. 7. W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL YAHOO! EMBRACES OPEN STANDARDS
  8. 8. YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES YAHOO! GIVES BACK TO OPEN SOURCE
  9. 9. YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER | MUSIC | ANSWERS | SHOPPING | FINANCE | TRAVEL YAHOO! SHARES ITS DATA THROUGH OPEN APIS AND WEB SERVICES
  10. 10. Conferences | Hack Days | HackU | Tech Talks | YDN Theater YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS AROUND THE WORLD
  11. 11. Users   Load  Balancers   Frontend   Apache   PHP   ysymfony  /  YUI   Custom  Modules   APC,  PEAR,  PECL,  Custom  Extensions   Backend   MySQL/Memcache   YQL  /  Web  Services   Ad  API   User  API   Linux  
  12. 12. Why a frontend platform? Rasmus says “frameworks are not well suited for Yahoo!” Build applications to requirements Do exactly what you need: no more, no less Understand that frameworks add a lot of overhead Choosing functional components is a better t Despite choosing open source or building your own Everyone uses a framework If you use open source, use only the pieces you need
  13. 13. Yahoo! needs from a frontend platform Fit existing environment (RHEL/PHP5/Apache) Development Cycle – How easy to develop, test, and deploy? Clean separation between data, logic, and display (MVC) Independent model layer to t service oriented architecture Extensible and pluggable Internationalization and localization support Detailed documentation and active community of support Open source and ability to contribute back
  14. 14. Why a framework at all? Another software layer (ysymfony, yphp, yapache) Factors out common patterns Code Layout Conguration URL Routing Authentication / Security Form Validation / Repopulation Internationalization / Localization Encourages good design Abstraction > Consistency > Maintainability
  15. 15. The choice to adopt symfony? Philosophy Full-stack framework for building complex web applications Adopt best ideas from anywhere, using existing code if available (Mojavi, Prado, Rails, Django) Design Clean separation between Model, View, and Controller Controller using modules and actions Views using templates in straight PHP with helpers Easy to reuse view modules to compose a page Layouts, Components, Partials, Slots
  16. 16. The choice to use symfony Congurability / Flexibility Features we do not want are easily disabled Use of factories for easy customization Documentation / Support Community The Denitive Guide to symfony Askeet, Jobeet, Cookbooks, Advents Active community with wiki, mailing lists, forums, irc channel
  17. 17. Why ysymfony for Yahoo! teams? Eliminate common patterns by adding a layer on PHP Code layout/structure (MVC) Conguration Internationalization ysymfony is just a toolkit Learn one set of tools Shift between multiple projects Consistency Long term maintainability through platform
  18. 18. How Yahoo! uses symfony Full Stack Yahoo! Bookmarks Yahoo! Answers Delicious Yahoo! Widgets Yahoo! Application Platform ShopGenie.co.uk, Foxytunes, Digu Internal Dashboards + Reporting Tools Components Dependency Injection Forms Caching (APC/Memcache) Routing
  19. 19. A look at Yahoo! Answers http://answers.yahoo.com Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008). Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore) Available in 26 markets and 12 languages
  20. 20. Yahoo! Answers at the beginning Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+ Launched December 2005 by December 2006 there were 60 million users and 65 million answers The code base eventually became difcult to maintain and iterate new features Large distributed development teams (US / UK)
  21. 21. The big picture A complete platform for building web applications from frameworks PHP Framework JavaScript Framework CSS Framework UI Design Patterns + Best Practices Development Tools (logger, proler, debugger, docs) Unit + Functional Testing Frameworks (LIME / YUI Test) Deployment Tools (rsync deployment system)
  22. 22. What does Yahoo! change? Minor changes to t our environment Most changes are easily implemented via factories/plugins Dropped the ORM and pushed down the stack (SOA) Added a parallel API Dispatcher (ysfAPIClientPlugin) Added dimensions to congurations (ysfDimensionsPlugin) Integrated R3 translation/template management (ysfR3Plugin) R3 - http://developer.yahoo.com/r3/ Created a build and deployment solution (ysfBuildPlugin) Uses internal tools for packaging/deployment Integrate support for Y! User Interface libraries (ysfYUIPlugin)
  23. 23. Propel or Doctrine or ??? No ORM for large projects Doctrine for medium sized projects Service Oriented Architecture Platforms/Backends as services (reusable to all) Thin Controller/Fat Model (where model == services) Use PHP as the frontend glue No heavy lifting in PHP = Push down the stack Java/C++/Erlang + JSON/XML
  24. 24. Localizing with dimensions Cascading Conguration based on YAML Framework -> Project -> Application -> Module Extending the cascade to be based on dimensions Dimensions can be anything (and can be chained together) Data Center + Environment for customizing congurations Culture for localizing user interface + data Theme for customizing look and feel User info (is user on corporate intranet?) Caching
  25. 25. A build and deployment system Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, …) Generate translations for congurations + templates Generate conguration cache Aggregate core classes + remove debug statements Run lint, unit, functional tests Package applications as .tgz Deployment via packages Integrated with Yahoo! internal tools
  26. 26. Debug Toolbars
  27. 27. Hello World Performance Hello world benchmarks are generally not useful You don’t use a framework to write hello world die(‘hello world’); Performance is relative to the features you use ORM I18N Output Escaping Forms/Validation Ability to scale != performance
  28. 28. What does it mean to scale? A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system. High Availability + Scalability + Performance Bigger dataset, more trafc, maintainable Not about performance PHP is slow, but it is very rarely your bottleneck Languages do not scale, architectures do Planning to grow and planning to fail Capacity Planning Business Continuity Planning
  29. 29. Scaling – The basics in PHP PHP is rarely the bottleneck (even though it can be slow) “Most performance comes not from the language, but from application design” – Rasmus Share Nothing Architecture Independent, self-sufcient, no single point of contention No local storage = No PHP Sessions Use a database (works for distributed) Use a small signed cookie (ideal) Important data in database Individual expiration on session objects Small data items Use a distributed cache Memcache Forget about small efciencies Premature optimization is the root of all evil. “ vs. ‘, echo vs. print
  30. 30. Scaling Databases – The basics Drop the ORM It’s a choice of convenience over performance Master/Slave Replication First steps Helps with reads, writes are still bottleneck Partitioning Segmenting data Sharding (horizontal partitioning) Segmenting data onto different physical machines Make problems smaller, easier to grow
  31. 31. Improving latency with Caching Always use PHP opcode cache (APC, Xcache, etc) Use for routing and i18n cache Memcache (distributed cache) Use for view cache Distributed invalidation can be a pain sfViewCacheManager makes this easy! Be intelligent about cache_keys (uri, user, state) There is a ne line to caching At what point do you spend more time managing the cache, than reading from it?
  32. 32. Tweaking Performance Do not use .htaccess (move to real apache cong) Set a minimal include path Increase realpath_cache_size + realpath_cache_ttl Use apc.stat=0 Don’t use features you do not need Disable in settings.yml / factories.yml Use core_compile to aggregate classes to reduce le i/o Remove debug statements + optimize le lookups sfOptimizerPlugin / project:optimize Use @routeName and use caching for factories
  33. 33. Symfony (v2) Symfony 2 is a set of cohesive yet decoupled components This makes it much easier to use single component to solve a single problem Which makes it easier to build micro frameworks that solve very specic problems Yahoo! Teams generally prefer solutions that are specic to their exact problem Selling the full stack can be difcult when a team only wants a few components Symfony 2 is the right direction, even if it breaks backwards compatability symfony-reloaded.org
  34. 34. Do it yourself for cheap Open source software = Free Apache PHP MySQL Memcache / Perlbal / MogileFS / Squid / Gearman symfony / Doctrine / Propel / Swift Nagios / Cacti Amazon Shared Infrastructure = Cheap EC2 Cloud S3 Storage + Cloudfront SimpleDB
  35. 35. What is Yahoo! Developer Network? The Yahoo! Developer Network offers open source tools and open data APIs to make it easy for developers to build applications and mashups. 50+ APIs / Web Services Developer Dashboard to create/manage Oauth applications Tutorials + Code Samples on using our apis Complete API Documentation Yahoo! User Interface libraries ASTRA Flash Components Design Patterns Library Evangelism: Conferences / Theater / Blogs / Events
  36. 36. Y! Developer Network – YUI Javascript JavaScript Framework Utilities - YAHOO, Dom, Event, Animation, Browser History Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView Debug – Logger, Proler, Test
  37. 37. Y! Developer Network – YUI CSS CSS Foundation Reset - Neutralizes browser CSS styles Base - Applies consistent style foundation Fonts - Foundation for typography and font-sizing Grids - Thousands of wireframe layouts User Interface Design Patterns Library Proven solutions to common interfaces http://developer.yahoo.com/ypatterns/ Grade Browser Support / Progressive Enhancement
  38. 38. Y! Developer Network – Documentation More than 275 functional examples http://developer.yahoo.com/yui/examples/ YSlow + Performance Rules http://developer.yahoo.com/performance YUI Blog http://yuiblog.com/ Mailing List @ Yahoo! Groups http://tech.groups.yahoo.com/group/ydn-javascript/
  39. 39. A Query Language for the Web SELECT    *  FROM  INTERNET  
  40. 40. Before YQL Thousands of web services that provide valuable data Require developers to read documentation and form URLs/queries. Data is isolated and can not be combined Needs combining, tweaking, shaping even after it gets to the developer.
  41. 41. Y! Open Stack – YQL SQL-Like Language Synonymous with Data access Familiar to developers Expressive enough to get the right data Self Describing - show, desc table Allows you to query, lter and join data across Web Services.
  42. 42. developer.yahoo.com/yql/console/
  43. 43. YQL – Open Tables Twitter Flickr Google Upcoming Facebook Delicious Friendfeed Dopplr Wesabe Github Whitepages New York Times Zillow Shopping Search …any web service can be Weather as easy as SQL Available on github - http://github.com/spullara/yql-tables/
  44. 44. YQL – Open Tables
  45. 45. YQL – Open Tables
  46. 46. YQL – Open Tables
  47. 47. YQL - Examples select * from social.connections select * from delicious.feeds.popular select * from ickr.photos.interestingness select * from friendfeed.status select * from github.checkins
  48. 48. YQL – API End Points OAuth Endpoint http://query.yahooapis.com/v1/yql?q=... Public Endpoint http://query.yahooapis.com/v1/public/yql?q= YQL Console http://developer.yahoo.com/yql/console
  49. 49. Open Source Communities
  50. 50. http://www.symfony-project.org/jobeet/1_2/
  51. 51. Documentation http://www.symfony-project.org/doc Practical symfony - Jobeet The Denitive Guide to symfony The symfony Reference Guide symfony Forms in Action Doctrine ORM for PHP Cookbooks + Cheat sheets + Wiki API Documentation
  52. 52. Books worth reading
  53. 53. QUESTIONS? (I will be around, so please say hello and ask questions)
  54. 54. EXAMPLES | TUTORIALS | CODE SAMPLES DEVELOPER.YAHOO.COM
  55. 55. WANT TO JOIN YAHOO? WE ARE HIRING!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×