• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Building A Platform From Open Source At Yahoo
 

Building A Platform From Open Source At Yahoo

on

  • 10,376 views

Building a platform from symfony at Yahoo! ...

Building a platform from symfony at Yahoo!

Presented at symfonyLive 2010 in Paris, France by Dustin Whittle.

Join us for a case study on using open source tools to build a platform for enterprise web applications with symfony. The focus of this session will be on how Yahoo! has built web applications that scale with symfony. Find out what worked and what didn't when building scalable web applications with the symfony framework.

* Why symfony?
* symfony vs ysymfony
* Social Search: Delicious and Answers
* YOS: Developer Tool & Application Platform
* Internal Tools: Customer Care + Dashboards
* The Platform + Components
* Yahoo! symfony Plugins
* Developer Tools - YUI3, YQL, Design Patterns, etc

Statistics

Views

Total Views
10,376
Views on SlideShare
8,653
Embed Views
1,723

Actions

Likes
27
Downloads
311
Comments
1

25 Embeds 1,723

http://imsardine.wordpress.com 429
http://symfony.com 318
http://www.symfonylab.com 291
http://www.symfony-project.org 161
http://swik.net 126
http://developer.yahoo.net 103
http://www.techgig.com 102
http://developer.yahoo.com 70
http://www.phphatesme.com 53
http://www.slideshare.net 24
http://coderwall.com 13
http://raulbarraltamayo.wordpress.com 10
https://developer.yahoo.com 4
http://webcache.googleusercontent.com 4
http://translate.googleusercontent.com 2
http://static.slidesharecdn.com 2
http://www.useseo.com 2
http://www.phpframeworks.com 2
http://127.0.0.1:8795 1
http://www.thewebhatesme.com 1
http://localhost:3000 1
http://feeds2.feedburner.com 1
https://ioio.nu 1
http://www.hanrss.com 1
http://timesjobs.techgig.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • wow nice one.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Building A Platform From Open Source At Yahoo Building A Platform From Open Source At Yahoo Presentation Transcript

    • Building a platform from open source
      Dustin Whittle
    • Overview
      Why symfony?
      symfony vs. ysymfony
      What does scaling really mean?
      Social Search: Delicious and Answers
      International: ShopGenie.co.uk, FoxyTunes, Wretch.cc
      Yahoo! Open Strategy
      What is the Yahoo! Open Stack?
      Developer Tools
      YUI, Design Patterns, Tutorials
      Data & Social APIs
      YQL: Yahoo! Query Language
      Profiles, Connections, Updates, …
      Geo, Flickr, Delicious, Upcoming
      YOS SDK for PHP
    • Who am I?
      Working with symfony since open source
      symfony Core Team Member
      Responsible for the development and support of ysymfony at Yahoo!
      Worked with Y! Answers, Delicious, Y! Widgets, Y! Bookmarks, Yahoo! Application Platform
      Developer advocacy for Yahoo! Developer Network
      Consultant
      Commercial symfony support + training (USA)
    • EXAMPLES | TUTORIALS | CODE SAMPLES
      developer.yahoo.com
    • Freebsd | linux | apache | php | mysql | BUGZILLA | hadoop | symfony
      Yahoo! Is powered by open source technologies
    • W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL
      YAHOO! embraces open standards
    • YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES
      YAHOO! gives back to open source
    • YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER | MUSIC | ANSWERS | SHOPPING | FINANCE | TRAVEL
      YAHOO! SHARES ITS DATA THROUGH OPEN APIs AND WEB SERVICES
    • Conferences | Hack Days | HackU | Tech Talks | YDN Theater
      YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS around the world
    • Users
      LoadBalancers
      Frontend
      Backend
      Linux
      ysymfony/ YUI
      Apache
      Custom Modules
      PHPAPC, PEAR, PECL, Custom Extensions
      UserAPI
      MySQL/Memcache
      Web Services
      Ad API
    • Why a frontend platform?
      Rasmus says “frameworks are not well suited for Yahoo!”
      Build applications to requirements
      Do exactly what you need: no more, no less
      Understand that frameworks add a lot of overhead
      Choosing functional components is a better fit
      Despite choosing open source or building your own
      Everyone uses a framework
      If you use open source, use only the pieces you need
    • Y! needs from a frontend platform
      Fit existing environment (RHEL/PHP5/Apache)
      Development Cycle – How easy to develop, test, and deploy?
      Clean separation between data, logic, and display (MVC)
      Independent model layer to fit service oriented architecture
      Extensible and pluggable
      Internationalization and localization support
      Detailed documentation and active community of support
      Open source and ability to contribute back
    • Why a framework at all?
      Another software layer (ysymfony, yphp, yapache)
      Factors out common patterns
      Code Layout
      Configuration
      URL Routing
      Authentication / Security
      Form Validation / Repopulation
      Internationalization / Localization
      Encourages good design
      Abstraction > Consistency > Maintainability
    • The choice to adopt symfony?
      Philosophy
      Full-stack framework for building complex web applications
      Adopt best ideas from anywhere, using existing code if available (Mojavi, Prado, Rails, Django)
      Design
      Clean separation between Model, View, and Controller
      Controller using modules and actions
      Views using templates in straight PHP with helpers
      Easy to reuse view modules to compose a page
      Layouts, Components, Partials, Slots
    • The choice to use symfony
      Configurability / Flexibility
      Features we do not want are easily disabled
      Use of factories for easy customization
      Documentation / Support Community
      The Definitive Guide to symfony
      Askeet, Jobeet, Cookbooks, Advents
      Active community with wiki, mailing lists, forums, irc channel
    • Why ysymfony for Yahoo! teams?
      Eliminate common patterns by adding a layer on PHP
      Code layout/structure (MVC)
      Configuration
      Internationalization
      ysymfony is just a toolkit
      Learn one set of tools
      Shift between multiple projects
      Consistency
      Long term maintainability through platform
    • How Yahoo! uses symfony
      Full Stack
      Yahoo! Bookmarks
      Yahoo! Answers
      Delicious
      Yahoo! Widgets
      Yahoo! Application Platform
      ShopGenie.co.uk, Foxytunes, Wretch.cc
      Internal Dashboards + Reporting Tools
      Components
      Dependency Injection
      Forms
      Caching (APC/Memcache)
      Routing
    • symfony MVC framework
    • A look at Yahoo! Answers
      http://answers.yahoo.com
      Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008).
      Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore)
      Available in 26 markets and 12 languages
    • Yahoo! Answers at the beginning
      Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+
      Launched December 2005 by December 2006 there were 60 million users and 65 million answers
      The code base eventually became difficult to maintain and iterate new features
      Large distributed development teams (US / UK)
    • The big picture
      A complete platform for building web applications from frameworks
      PHP Framework
      JavaScript Framework
      CSS Framework
      UI Design Patterns + Best Practices
      Development Tools (logger, profiler, debugger, docs)
      Unit + Functional Testing Frameworks (LIME / YUI Test)
      Deployment Tools (rsync deployment system)
    • What does Yahoo! change?
      Minor changes to fit our environment
      Most changes are easily implemented via factories/plugins
      Dropped the ORM and pushed down the stack (SOA)
      Added a parallel API Dispatcher (ysfAPIClientPlugin)
      Added dimensions to configurations (ysfDimensionsPlugin)
      Integrated R3 translation/template management (ysfR3Plugin)
      R3 - http://developer.yahoo.com/r3/
      Created a build and deployment solution (ysfBuildPlugin)
      Uses internal tools for packaging/deployment
      Integrate support for Y! User Interface libraries (ysfYUIPlugin)
    • Propel or Doctrine or ???
      No ORM for large projects
      Doctrine for medium sized projects
      Service Oriented Architecture
      Platforms/Backendsas services (reusable to all)
      Thin Controller/Fat Model (where model == services)
      Use PHP as the frontend glue
      No heavy lifting in PHP = Push down the stack
      Java/C++/Erlang + JSON/XML
    • Localizing with dimensions
      Cascading Configuration based on YAML
      Framework -> Project -> Application -> Module
      Extending the cascade to be based on dimensions
      Dimensions can be anything (and can be chained together)
      Data Center + Environment for customizing configurations
      Culture for localizing user interface + data
      Theme for customizing look and feel
      User info (is user on corporate intranet?)
      Caching
    • symfony I18n + L10n
      Translation + Template Management
      Integrated with symfony i18n + view layers
      <r3:trans>hello</r3:trans> vs __(‘hello’)
      Dictionary Management (YALA)
      SQLite/XLIFF Backend
      Added additional tasks
      Importing/Exporting XLIFF dictionary
      Generating sample translations for projects
      Find missing strings
    • A build and deployment system
      Aggregate and minify stylesheets and javascripts
      Rewrite templates, css, js for CDN (Akamai, S3, …)
      Generate translations for configurations + templates
      Generate configuration cache
      Aggregate core classes + remove debug statements
      Run lint, unit, functional tests
      Package applications as .tgz
      Deployment via packages
      Integrated with Yahoo! internal tools
    • Hello World Performance
      Hello world benchmarks are generally not a useful measurement
      You don’t use a framework to write hello world
      die(‘hello world’);
      Performance is relative to the features you use
      ORM
      I18N
      Output Escaping
      Forms/Validation
      Ability to scale != performance
    • What does it mean to scale?
      A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.
      High Availability + Scalability + Performance
      Bigger dataset, more traffic, maintainable
      Not about performance
      PHP is slow, but it is very rarely your bottleneck
      Languages do not scale, architectures do
      Planning to grow and planning to fail
      Capacity Planning
      Business Continuity Planning
    • Scaling – Planning
      Planning hardware purchases and hosting options to have as much as you need without breaking your wallet
      Partitioning and distributing databases to support large datasets and simultaneous transactions
      Offline transactions and queuing
      Monitoring your applications to find and clear bottlenecks
      Providing service APIs and using services from other providers to increase your site's reach and capabilities
      Think Minimal, Plan to grow, Plan to fail
    • Scaling – The basics in PHP
      PHP is rarely the bottleneck (even though it can be slow)
      “Most performance comes not from the language, but from application design” – Rasmus
      Share Nothing Architecture
      Independent, self-sufficient, no single point of contention
      No local storage = No PHP Sessions
      Use a database (works for distributed)
      Use a small signed cookie (ideal)
      Important data in database
      Individual expiration on session objects
      Small data items
      Use a distributed cache
      Memcache
      Forget about small efficiencies
      Premature optimization is the root of all evil.
      “ vs. ‘, echo vs. print
    • Scaling Databases – The basics
      Drop the ORM
      It’s a choice of convenience over performance
      Master/Slave Replication
      First steps
      Helps with reads, writes are still bottleneck
      Partitioning
      Segmenting data
      Sharding (horizontal partitioning)
      Segmenting data onto different physical machines
      Make problems smaller, easier to grow
    • Improving latency with Caching
      Always use PHP opcode cache (APC, Xcache, etc)
      Use for routing and i18n cache
      Memcache (distributed cache)
      Use for view cache
      Distributed invalidation can be a pain
      sfViewCacheManager makes this easy!
      Be intelligent about cache_keys (uri, user, state)
      There is a fine line to caching
      At what point do you spend more time managing the cache, than reading from it?
    • Tweaking Performance
      Do not use .htaccess (move to real apache config)
      Set a minimal include path
      Increase realpath_cache_size + realpath_cache_ttl
      Use apc.stat=0
      Don’t use features you do not need
      Disable in settings.yml/ factories.yml
      Use core_compile to aggregate classes to reduce file i/o
      Remove debug statements + optimize file lookups
      sfOptimizerPlugin / project:optimize
      Use @routeName and use caching for factories
    • Symfony (v2)
      Symfony 2 is a set of cohesive yet decoupled components
      This makes it much easier to use single component to solve a single problem
      Which makes it easier to build micro frameworks that solve very specific problems
      Yahoo! Teams generally prefer solutions that are specific to their exact problem
      Selling the full stack can be difficult when a team only wants a few components
      Symfony 2 is the right direction, even if it breaks backwards compatability
    • Do it yourself for cheap
      Open source software = Free
      Apache
      PHP
      MySQL
      Memcache / Perlbal / MogileFS / Squid / Gearman
      symfony / Doctrine / Propel / Swift
      Nagios / Cacti
      Amazon Shared Infrastructure = Cheap
      EC2 Cloud
      S3 Storage + Cloudfront
      SimpleDB
    • Y! OS – The Open Stack
    • What Yahoo! offer developers
      • Yahoo! Developer Network
      • Developer Tools (YUI, etc)
      • Social APIs
      • Profiles
      • Connections
      • Updates
      • Data APIs
      • Flickr, Delicious, Search, etc
      • Yahoo! Query Language
      • Yahoo! Application Platform
      • OpenSocial
    • developer.yahoo.com
    • What is Yahoo! Developer Network?
      The Yahoo! Developer Network offers open source tools and open data APIs to make it easy for developers to build applications and mashups.
      50+ APIs / Web Services
      Developer Dashboard to create/manage Oauth applications
      Tutorials + Code Samples on using our apis
      Complete API Documentation
      Yahoo! User Interface libraries
      ASTRA Flash Components
      Design Patterns Library
      Evangelism: Conferences / Theater / Blogs / Events
    • JavaScript Framework
      Utilities - YAHOO, Dom, Event, Animation, Browser History Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader
      Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader
      Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView
      Debug – Logger, Profiler, Test
      Y! Developer Network – YUI Javascript
    • CSS Foundation
      Reset - Neutralizes browser CSS styles
      Base - Applies consistent style foundation
      Fonts - Foundation for typography and font-sizing
      Grids - Thousands of wireframe layouts
      User Interface Design Patterns Library
      Proven solutions to common interfaces
      http://developer.yahoo.com/ypatterns/
      Grade Browser Support / Progressive Enhancement
      Y! Developer Network – YUI CSS
    • More than 275 functional examples
      http://developer.yahoo.com/yui/examples/
      YSlow + Performance Rules
      http://developer.yahoo.com/performance
      YUI Blog
      http://yuiblog.com/
      Mailing List @ Yahoo! Groups
      http://tech.groups.yahoo.com/group/ydn-javascript/
      Y! Developer Network – Documentation
    • A Query Language for the Web
      SELECT * FROM INTERNET
    • Thousands of web services that provide valuable data
      Require developers to read documentation and form URLs/queries.
      Data is isolated and can not be combined
      Needs combining, tweaking, shaping even after it gets to the developer.
      Before YQL
    • SQL-Like Language
      Synonymous with Data access
      Familiar to developers
      Expressive enough to get the right data
      Self Describing - show, desc table
      Allows you to query, filter and join data across Web Services.
      Y! Open Stack – YQL
    • YQL – Open Tables
      Twitter
      Google
      Facebook
      Friendfeed
      Wesabe
      Whitepages
      Zillow
      Search
      Weather
      Flickr
      Upcoming
      Delicious
      Dopplr
      Github
      New York Times
      Shopping
      …any web service can be as easy as SQL
      Available on github - http://github.com/spullara/yql-tables/
    • YQL – Open Tables
    • YQL – Open Tables
    • YQL – Open Tables
    • select * from social.connections
      select * from delicious.feeds.popular
      select * from flickr.photos.interestingness
      select * from friendfeed.status
      select * from github.checkins
      YQL - Examples
    • OAuth Endpoint
      http://query.yahooapis.com/v1/yql?q=...
      Public Endpoint
      http://query.yahooapis.com/v1/public/yql?q=
      YQL Console
      http://developer.yahoo.com/yql/console
      YQL – API End Points
    • Books worth reading
    • (I will be around for the whole event, so please say hello and ask questions)
      Questions?
    • enjoy the rest ofsymfonylive 2010
      EXAMPLES | TUTORIALS | CODE SAMPLES
      developer.yahoo.com
    • Want to join Yahoo?We are hiring!