Building a platform from open source

Building a pla+orm from open source Dustin Whittle

Building a pla+orm from open source

Who am I? •  A hacker on the web for the last decade •  Worked with symfony since open source •  Developed ysymfony at Yahoo! •  Consultant helping startups scale web apps •  A speaker at conferences around the world dusBn.whiDle@gmail.com @dusBnwhiDle hDp://dusBnwhiDle.com/

When I started developing…

It was really simple

1 shared host / simple PHP scripts

Then I started working on lots of consulBng projects

I realized I needed a beDer set of tools

I started exploring web frameworks

I realized reading and using other peoples code made me a better, more productive developer

I started talking about how frameworks like symfony and YUI improve quality and productivity at conferences

Then I went to work at Yahoo to help build a frontend platform based on an open source framework

I realized web application stacks are very similar Lots of the same problems and solutions

Users Load Balancers Frontend Apache PHP symfony / YUI Custom Modules APC, PEAR, PECL Backend MySQL/Memcache Web Services / YQL Ad API User API Linux

Why a frontend pla+orm? Build applicaBons to requirements Do exactly what you need: no more, no less Choose open source or build your own Everyone uses a framework If you use open source, support/maintenance is free Understand that frameworks add a lot of overhead (features have a cost)

Needs from a frontend pla+orm Fit server environments (Linux/PHP5/Apache) Development Cycle – How easy to develop, test, and deploy? Clean separaBon between data, logic, and display Independent model layer to ﬁt service oriented architecture Extensible and pluggable InternaBonalizaBon and localizaBon support Detailed documentaBon and acBve community of support Open source and ability to contribute back

Why a framework at all? Another so]ware layer (ysymfony, yphp, yapache) Factors out common paDerns Code Layout ConﬁguraBon URL RouBng AuthenBcaBon / Security Form ValidaBon / RepopulaBon InternaBonalizaBon / LocalizaBon Encourages good design AbstracBon > Consistency > Maintainability

The choice to adopt symfony? Philosophy Full-‐stack framework for building complex web applicaBons Adopt best ideas from anywhere, using exisBng code if available (Mojavi, Prado, Rails, Django) Design Clean separaBon between Model, View, and Controller Controller using modules and acBons Views using templates in straight PHP with helpers Easy to reuse view modules to compose a page Layouts, Components, ParBals, Slots

The choice to use symfony Conﬁgurability / Flexibility Features we do not want are easily disabled Use of factories for easy customizaBon DocumentaBon / Support Community The DeﬁniBve Guide to symfony Askeet, Jobeet, Cookbooks, Advents AcBve community with wiki, mailing lists, forums, irc channel

A look at Yahoo! Answers hDp://answers.yahoo.com Yahoo! Answers is the largest collecBon of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008). Yahoo! Answers is the 2nd ranked educaBon & reference site on the web (comScore) Available in 26 markets and 12 languages

Yahoo! Answers at the beginning Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+ Launched December 2005 by December 2006 there were 60 million users and 65 million answers The code base eventually became diﬃcult to maintain and iterate new features Large distributed development teams

Y! Developer Network – YUI Javascript JavaScript Framework UBliBes -‐ YAHOO, Dom, Event, AnimaBon, Browser History Manager, ConnecBon Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader Controls / Widgets -‐ AutoComplete, BuDon, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader Container (Module, Overlay, Panel, ToolBp, Dialog), Layout Manager, Menu, TabView, TreeView Debug – Logger, Proﬁler, Test

Y! Developer Network – YUI CSS CSS FoundaBon Reset -‐ Neutralizes browser CSS styles Base -‐ Applies consistent style foundaBon Fonts -‐ FoundaBon for typography and font-‐sizing Grids -‐ Thousands of wireframe layouts User Interface Design PaDerns Library Proven soluBons to common interfaces hDp://developer.yahoo.com/ypaDerns/ Grade Browser Support / Progressive Enhancement

Y! Developer Network – DocumentaBon More than 275 funcBonal examples hDp://developer.yahoo.com/yui/examples/ YSlow + Performance Rules hDp://developer.yahoo.com/performance YUI Blog hDp://yuiblog.com/ Mailing List @ Yahoo! Groups hDp://tech.groups.yahoo.com/group/ydn-‐ javascript/

The big picture A complete pla+orm for building web applicaBons from frameworks PHP Framework JavaScript Framework CSS Framework UI Design PaDerns + Best PracBces Development Tools (logger, proﬁler, debugger, docs) Unit + FuncBonal TesBng Frameworks (LIME / YUI Test) Deployment Tools

What does Yahoo! change? Minor changes to ﬁt our environment Most changes are easily implemented via factories/plugins Dropped the ORM and pushed down the stack (SOA) Added a parallel API Dispatcher (ysfAPIClientPlugin) Added dimensions to conﬁguraBons (ysfDimensionsPlugin) Integrated R3 translaBon/template management (ysfR3Plugin) R3 -‐ hDp://developer.yahoo.com/r3/ Created a build and deployment soluBon (ysfBuildPlugin) Uses internal tools for packaging/deployment Integrate support for Y! User Interface libraries (ysfYUIPlugin)

Propel or Doctrine or ??? No ORM for large projects Doctrine for medium sized projects Service Oriented Architecture Pla+orms/Data as services Thin Controller/Fat Model Use PHP as the frontend glue No heavy li]ing in PHP = Push down the stack Java/C++/Erlang + JSON/XML

Localizing with dimensions Cascading ConﬁguraBon based on YAML Framework -‐> Project -‐> ApplicaBon -‐> Module Extending the cascade to be based on dimensions Dimensions can be anything (and can be chained together) Data Center + Environment for customizing conﬁguraBons Culture for localizing user interface + data Theme for customizing look and feel User info (is user on corporate intranet?) Caching

Development Tools

A build and deployment system Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, …) Generate translaBons for conﬁguraBons + templates Generate conﬁguraBon cache Aggregate core classes + remove debug statements Run lint, unit, funcBonal tests Package applicaBons as .tgz Deployment via packages

Performance Hello world benchmarks are not useful You do not use a framework to write hello world die(‘hello world’); Performance is relaBve to the features you use ORM I18N Output Escaping Forms/ValidaBon Ability to scale != performance

What does it mean to scale? A system whose performance improves a]er adding hardware, proporBonally to the capacity added, is said to be a scalable system. High Availability + Scalability + Performance Bigger dataset, more traﬃc, maintainable Not about performance PHP is slow, but it is very rarely your boDleneck Languages do not scale, architectures do

Scaling – The basics in PHP PHP is rarely the boDleneck (even though it can be slow) “Most performance comes not from the language, but from applicaBon design” – Rasmus Share Nothing Architecture Independent, self-‐suﬃcient, no single point of contenBon No local storage = No PHP Sessions Use a database Use a small signed cookie (beDer) Important data in database Individual expiraBon on session objects Small data items Use an in-‐memory cache Memcache Premature opBmizaBon is the root of all evil. “ vs. ‘, echo vs. print Improve performance by queuing jobs that can take a long Bme and caching data that is expensive to query

Once an app has a well designed architecture, developers can focus on performance speciﬁc to the app

Use Xdebug/Webgrind and XHProf to debug and proﬁle applicaBon performance

hDp://xdebug.org/docs/proﬁler

Web applicaBon performance isn’t just about the backend, a lot of open source tools can help improve the frontend.

As the framework matured and gained recogniBon more companies started choosing symfony

At Yahoo scaling applicaBons was easy: infrastructure/process already exists

As I work on my own startup I realized I need to create the same scalable and fault tolerant environment, except without invesBng millions of dollars.

A quality pla+orm is built from best pracBces

OperaBonal Best PracBces •  Global data centers for performance and fault tolerance •  Automated provisioning of networks, servers, storage, etc. •  Automated builds + deployments •  Monitoring + ReporBng (performance/ costs) •  Centralized Logging + Management

Development Best PracBces •  Clean / Decoupled / Documented code •  Stand on the shoulders of giants •  Leverage open source communiBes •  Source Control + Project Management •  Unit + FuncBonal TesBng •  ConBnuous IntegraBon •  Either you are always ready to deploy or you are never ready to deploy

Do it yourself for cheap Open source so]ware = Free •  Linux •  Apache •  PHP •  MySQL / Mongo •  Memcache / Gearman •  symfony / Doctrine / Twig / Swi] / PHPUnit

Today it is possible for anyone to scale a web pla+orm just like a big company.

Welcome to the cloud.

It’s so cheap its free! Who doesn’t love free?

hDp://aws.amazon.com/free

A load balanced cluster with a relaBonal database and storage conﬁgured how ever you want in mulBple data centers around the world.

For free. And then for relaBvely cheap.

Its not fair! Only for new customers.

A way to monitor and manage your cloud

A free global DNS cloud service

A free source control + issue tracking service that enables public or private collaboraBon

A complete pla+orm for developing, tesBng, and deploying scalable web applicaBons.

Free cloud infrastructure: Amazon + Zerigo + Landscape Free open source code: Symfony + Zend + YUI + AlloyUI Free development tools: GitHub + Xdebug + PHPUnit Built in best pracBces + support

Many lessons have been learned and a beDer pla+orm has evolved.

Symfony2 and YUI3

BeDer Components: lighter, faster, cleaner BeDer Development Model: ConBnuous IntegraBon BeDer tesBng/documentaBon Stronger CommuniBes: Github enables contribuBons Galleries for plugins/bundles

Symfony Symfony 2 is a set of cohesive yet decoupled components Easier to use a component to solve a single problem Easier to build micro frameworks that solve specific problems Teams generally prefer soluBons that are specific to their exact problem Selling the full stack can be difficult when a team only wants a few components symfony-reloaded.org

Symfony2 stands on the shoulders of giants and doesn’t reinvent the wheel.

We even use Zend Framework

Symfony2 is so good it is only alpha and there are already major sites live in producBon.

Take a look at the Symfony2 web proﬁler

Take a look at the Symfony sandbox!

github.com/symfony/symfony-‐sandbox

QuesBons?

Hiring! If you are a great developer looking to make your next move, let me introduce you to: Plancast TripIt Awesm OpenSky Gogobot

http://www.berejeb.com	43
http://coderwall.com	12
http://www.devetdesign.com	6
http://localhost:3000	1

Building a platform from open source

by Dustin Whittle on Nov 04, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 62

Statistics

Building a platform from open source — Presentation Transcript