Building A Platform From Open Source At Yahoo
Upcoming SlideShare
Loading in...5
×
 

Building A Platform From Open Source At Yahoo

on

  • 10,502 views

Building a platform from symfony at Yahoo! ...

Building a platform from symfony at Yahoo!

Presented at symfonyLive 2010 in Paris, France by Dustin Whittle.

Join us for a case study on using open source tools to build a platform for enterprise web applications with symfony. The focus of this session will be on how Yahoo! has built web applications that scale with symfony. Find out what worked and what didn't when building scalable web applications with the symfony framework.

* Why symfony?
* symfony vs ysymfony
* Social Search: Delicious and Answers
* YOS: Developer Tool & Application Platform
* Internal Tools: Customer Care + Dashboards
* The Platform + Components
* Yahoo! symfony Plugins
* Developer Tools - YUI3, YQL, Design Patterns, etc

Statistics

Views

Total Views
10,502
Views on SlideShare
8,773
Embed Views
1,729

Actions

Likes
27
Downloads
311
Comments
1

26 Embeds 1,729

http://imsardine.wordpress.com 431
http://symfony.com 320
http://www.symfonylab.com 292
http://www.symfony-project.org 161
http://swik.net 126
http://developer.yahoo.net 103
http://www.techgig.com 102
http://developer.yahoo.com 70
http://www.phphatesme.com 53
http://www.slideshare.net 24
http://coderwall.com 13
http://raulbarraltamayo.wordpress.com 10
https://developer.yahoo.com 4
http://webcache.googleusercontent.com 4
http://www.phpframeworks.com 2
http://translate.googleusercontent.com 2
http://static.slidesharecdn.com 2
http://www.useseo.com 2
http://timesjobs.techgig.com 1
http://www.thewebhatesme.com 1
http://feeds2.feedburner.com 1
http://localhost:3000 1
http://127.0.0.1:8795 1
https://ioio.nu 1
http://www.hanrss.com 1
https://imsardine.wordpress.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • wow nice one.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building A Platform From Open Source At Yahoo Building A Platform From Open Source At Yahoo Presentation Transcript

  • Building a platform from open source
    Dustin Whittle
  • Overview
    Why symfony?
    symfony vs. ysymfony
    What does scaling really mean?
    Social Search: Delicious and Answers
    International: ShopGenie.co.uk, FoxyTunes, Wretch.cc
    Yahoo! Open Strategy
    What is the Yahoo! Open Stack?
    Developer Tools
    YUI, Design Patterns, Tutorials
    Data & Social APIs
    YQL: Yahoo! Query Language
    Profiles, Connections, Updates, …
    Geo, Flickr, Delicious, Upcoming
    YOS SDK for PHP
  • Who am I?
    Working with symfony since open source
    symfony Core Team Member
    Responsible for the development and support of ysymfony at Yahoo!
    Worked with Y! Answers, Delicious, Y! Widgets, Y! Bookmarks, Yahoo! Application Platform
    Developer advocacy for Yahoo! Developer Network
    Consultant
    Commercial symfony support + training (USA)
  • EXAMPLES | TUTORIALS | CODE SAMPLES
    developer.yahoo.com
  • Freebsd | linux | apache | php | mysql | BUGZILLA | hadoop | symfony
    Yahoo! Is powered by open source technologies
  • W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL
    YAHOO! embraces open standards
  • YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES
    YAHOO! gives back to open source
  • YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER | MUSIC | ANSWERS | SHOPPING | FINANCE | TRAVEL
    YAHOO! SHARES ITS DATA THROUGH OPEN APIs AND WEB SERVICES
  • Conferences | Hack Days | HackU | Tech Talks | YDN Theater
    YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS around the world
  • Users
    LoadBalancers
    Frontend
    Backend
    Linux
    ysymfony/ YUI
    Apache
    Custom Modules
    PHPAPC, PEAR, PECL, Custom Extensions
    UserAPI
    MySQL/Memcache
    Web Services
    Ad API
  • Why a frontend platform?
    Rasmus says “frameworks are not well suited for Yahoo!”
    Build applications to requirements
    Do exactly what you need: no more, no less
    Understand that frameworks add a lot of overhead
    Choosing functional components is a better fit
    Despite choosing open source or building your own
    Everyone uses a framework
    If you use open source, use only the pieces you need
  • Y! needs from a frontend platform
    Fit existing environment (RHEL/PHP5/Apache)
    Development Cycle – How easy to develop, test, and deploy?
    Clean separation between data, logic, and display (MVC)
    Independent model layer to fit service oriented architecture
    Extensible and pluggable
    Internationalization and localization support
    Detailed documentation and active community of support
    Open source and ability to contribute back
  • Why a framework at all?
    Another software layer (ysymfony, yphp, yapache)
    Factors out common patterns
    Code Layout
    Configuration
    URL Routing
    Authentication / Security
    Form Validation / Repopulation
    Internationalization / Localization
    Encourages good design
    Abstraction > Consistency > Maintainability
  • The choice to adopt symfony?
    Philosophy
    Full-stack framework for building complex web applications
    Adopt best ideas from anywhere, using existing code if available (Mojavi, Prado, Rails, Django)
    Design
    Clean separation between Model, View, and Controller
    Controller using modules and actions
    Views using templates in straight PHP with helpers
    Easy to reuse view modules to compose a page
    Layouts, Components, Partials, Slots
  • The choice to use symfony
    Configurability / Flexibility
    Features we do not want are easily disabled
    Use of factories for easy customization
    Documentation / Support Community
    The Definitive Guide to symfony
    Askeet, Jobeet, Cookbooks, Advents
    Active community with wiki, mailing lists, forums, irc channel
  • Why ysymfony for Yahoo! teams?
    Eliminate common patterns by adding a layer on PHP
    Code layout/structure (MVC)
    Configuration
    Internationalization
    ysymfony is just a toolkit
    Learn one set of tools
    Shift between multiple projects
    Consistency
    Long term maintainability through platform
  • How Yahoo! uses symfony
    Full Stack
    Yahoo! Bookmarks
    Yahoo! Answers
    Delicious
    Yahoo! Widgets
    Yahoo! Application Platform
    ShopGenie.co.uk, Foxytunes, Wretch.cc
    Internal Dashboards + Reporting Tools
    Components
    Dependency Injection
    Forms
    Caching (APC/Memcache)
    Routing
  • symfony MVC framework
  • A look at Yahoo! Answers
    http://answers.yahoo.com
    Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008).
    Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore)
    Available in 26 markets and 12 languages
  • Yahoo! Answers at the beginning
    Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+
    Launched December 2005 by December 2006 there were 60 million users and 65 million answers
    The code base eventually became difficult to maintain and iterate new features
    Large distributed development teams (US / UK)
  • The big picture
    A complete platform for building web applications from frameworks
    PHP Framework
    JavaScript Framework
    CSS Framework
    UI Design Patterns + Best Practices
    Development Tools (logger, profiler, debugger, docs)
    Unit + Functional Testing Frameworks (LIME / YUI Test)
    Deployment Tools (rsync deployment system)
  • What does Yahoo! change?
    Minor changes to fit our environment
    Most changes are easily implemented via factories/plugins
    Dropped the ORM and pushed down the stack (SOA)
    Added a parallel API Dispatcher (ysfAPIClientPlugin)
    Added dimensions to configurations (ysfDimensionsPlugin)
    Integrated R3 translation/template management (ysfR3Plugin)
    R3 - http://developer.yahoo.com/r3/
    Created a build and deployment solution (ysfBuildPlugin)
    Uses internal tools for packaging/deployment
    Integrate support for Y! User Interface libraries (ysfYUIPlugin)
  • Propel or Doctrine or ???
    No ORM for large projects
    Doctrine for medium sized projects
    Service Oriented Architecture
    Platforms/Backendsas services (reusable to all)
    Thin Controller/Fat Model (where model == services)
    Use PHP as the frontend glue
    No heavy lifting in PHP = Push down the stack
    Java/C++/Erlang + JSON/XML
  • Localizing with dimensions
    Cascading Configuration based on YAML
    Framework -> Project -> Application -> Module
    Extending the cascade to be based on dimensions
    Dimensions can be anything (and can be chained together)
    Data Center + Environment for customizing configurations
    Culture for localizing user interface + data
    Theme for customizing look and feel
    User info (is user on corporate intranet?)
    Caching
  • symfony I18n + L10n
    Translation + Template Management
    Integrated with symfony i18n + view layers
    <r3:trans>hello</r3:trans> vs __(‘hello’)
    Dictionary Management (YALA)
    SQLite/XLIFF Backend
    Added additional tasks
    Importing/Exporting XLIFF dictionary
    Generating sample translations for projects
    Find missing strings
  • A build and deployment system
    Aggregate and minify stylesheets and javascripts
    Rewrite templates, css, js for CDN (Akamai, S3, …)
    Generate translations for configurations + templates
    Generate configuration cache
    Aggregate core classes + remove debug statements
    Run lint, unit, functional tests
    Package applications as .tgz
    Deployment via packages
    Integrated with Yahoo! internal tools
  • Hello World Performance
    Hello world benchmarks are generally not a useful measurement
    You don’t use a framework to write hello world
    die(‘hello world’);
    Performance is relative to the features you use
    ORM
    I18N
    Output Escaping
    Forms/Validation
    Ability to scale != performance
  • What does it mean to scale?
    A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.
    High Availability + Scalability + Performance
    Bigger dataset, more traffic, maintainable
    Not about performance
    PHP is slow, but it is very rarely your bottleneck
    Languages do not scale, architectures do
    Planning to grow and planning to fail
    Capacity Planning
    Business Continuity Planning
  • Scaling – Planning
    Planning hardware purchases and hosting options to have as much as you need without breaking your wallet
    Partitioning and distributing databases to support large datasets and simultaneous transactions
    Offline transactions and queuing
    Monitoring your applications to find and clear bottlenecks
    Providing service APIs and using services from other providers to increase your site's reach and capabilities
    Think Minimal, Plan to grow, Plan to fail
  • Scaling – The basics in PHP
    PHP is rarely the bottleneck (even though it can be slow)
    “Most performance comes not from the language, but from application design” – Rasmus
    Share Nothing Architecture
    Independent, self-sufficient, no single point of contention
    No local storage = No PHP Sessions
    Use a database (works for distributed)
    Use a small signed cookie (ideal)
    Important data in database
    Individual expiration on session objects
    Small data items
    Use a distributed cache
    Memcache
    Forget about small efficiencies
    Premature optimization is the root of all evil.
    “ vs. ‘, echo vs. print
  • Scaling Databases – The basics
    Drop the ORM
    It’s a choice of convenience over performance
    Master/Slave Replication
    First steps
    Helps with reads, writes are still bottleneck
    Partitioning
    Segmenting data
    Sharding (horizontal partitioning)
    Segmenting data onto different physical machines
    Make problems smaller, easier to grow
  • Improving latency with Caching
    Always use PHP opcode cache (APC, Xcache, etc)
    Use for routing and i18n cache
    Memcache (distributed cache)
    Use for view cache
    Distributed invalidation can be a pain
    sfViewCacheManager makes this easy!
    Be intelligent about cache_keys (uri, user, state)
    There is a fine line to caching
    At what point do you spend more time managing the cache, than reading from it?
  • Tweaking Performance
    Do not use .htaccess (move to real apache config)
    Set a minimal include path
    Increase realpath_cache_size + realpath_cache_ttl
    Use apc.stat=0
    Don’t use features you do not need
    Disable in settings.yml/ factories.yml
    Use core_compile to aggregate classes to reduce file i/o
    Remove debug statements + optimize file lookups
    sfOptimizerPlugin / project:optimize
    Use @routeName and use caching for factories
  • Symfony (v2)
    Symfony 2 is a set of cohesive yet decoupled components
    This makes it much easier to use single component to solve a single problem
    Which makes it easier to build micro frameworks that solve very specific problems
    Yahoo! Teams generally prefer solutions that are specific to their exact problem
    Selling the full stack can be difficult when a team only wants a few components
    Symfony 2 is the right direction, even if it breaks backwards compatability
  • Do it yourself for cheap
    Open source software = Free
    Apache
    PHP
    MySQL
    Memcache / Perlbal / MogileFS / Squid / Gearman
    symfony / Doctrine / Propel / Swift
    Nagios / Cacti
    Amazon Shared Infrastructure = Cheap
    EC2 Cloud
    S3 Storage + Cloudfront
    SimpleDB
  • Y! OS – The Open Stack
  • What Yahoo! offer developers
    • Yahoo! Developer Network
    • Developer Tools (YUI, etc)
    • Social APIs
    • Profiles
    • Connections
    • Updates
    • Data APIs
    • Flickr, Delicious, Search, etc
    • Yahoo! Query Language
    • Yahoo! Application Platform
    • OpenSocial
  • developer.yahoo.com
  • What is Yahoo! Developer Network?
    The Yahoo! Developer Network offers open source tools and open data APIs to make it easy for developers to build applications and mashups.
    50+ APIs / Web Services
    Developer Dashboard to create/manage Oauth applications
    Tutorials + Code Samples on using our apis
    Complete API Documentation
    Yahoo! User Interface libraries
    ASTRA Flash Components
    Design Patterns Library
    Evangelism: Conferences / Theater / Blogs / Events
  • JavaScript Framework
    Utilities - YAHOO, Dom, Event, Animation, Browser History Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader
    Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader
    Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView
    Debug – Logger, Profiler, Test
    Y! Developer Network – YUI Javascript
  • CSS Foundation
    Reset - Neutralizes browser CSS styles
    Base - Applies consistent style foundation
    Fonts - Foundation for typography and font-sizing
    Grids - Thousands of wireframe layouts
    User Interface Design Patterns Library
    Proven solutions to common interfaces
    http://developer.yahoo.com/ypatterns/
    Grade Browser Support / Progressive Enhancement
    Y! Developer Network – YUI CSS
  • More than 275 functional examples
    http://developer.yahoo.com/yui/examples/
    YSlow + Performance Rules
    http://developer.yahoo.com/performance
    YUI Blog
    http://yuiblog.com/
    Mailing List @ Yahoo! Groups
    http://tech.groups.yahoo.com/group/ydn-javascript/
    Y! Developer Network – Documentation
  • A Query Language for the Web
    SELECT * FROM INTERNET
  • Thousands of web services that provide valuable data
    Require developers to read documentation and form URLs/queries.
    Data is isolated and can not be combined
    Needs combining, tweaking, shaping even after it gets to the developer.
    Before YQL
  • SQL-Like Language
    Synonymous with Data access
    Familiar to developers
    Expressive enough to get the right data
    Self Describing - show, desc table
    Allows you to query, filter and join data across Web Services.
    Y! Open Stack – YQL
  • YQL – Open Tables
    Twitter
    Google
    Facebook
    Friendfeed
    Wesabe
    Whitepages
    Zillow
    Search
    Weather
    Flickr
    Upcoming
    Delicious
    Dopplr
    Github
    New York Times
    Shopping
    …any web service can be as easy as SQL
    Available on github - http://github.com/spullara/yql-tables/
  • YQL – Open Tables
  • YQL – Open Tables
  • YQL – Open Tables
  • select * from social.connections
    select * from delicious.feeds.popular
    select * from flickr.photos.interestingness
    select * from friendfeed.status
    select * from github.checkins
    YQL - Examples
  • OAuth Endpoint
    http://query.yahooapis.com/v1/yql?q=...
    Public Endpoint
    http://query.yahooapis.com/v1/public/yql?q=
    YQL Console
    http://developer.yahoo.com/yql/console
    YQL – API End Points
  • Books worth reading
  • (I will be around for the whole event, so please say hello and ask questions)
    Questions?
  • enjoy the rest ofsymfonylive 2010
    EXAMPLES | TUTORIALS | CODE SAMPLES
    developer.yahoo.com
  • Want to join Yahoo?We are hiring!