SlideShare a Scribd company logo
1 of 41
Download to read offline
Flickr and PHP
Cal Henderson
What’s Flickr
• Photo sharing
• Open APIs
Logical Architecture
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
Physical Architecture
Static Servers Database Servers Node Servers
Web Servers
Users
Where is PHP?
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
Other than PHP?
• Smarty for templating
• PEAR for XML and Email parsing
• Perl for controlling…
• ImageMagick, for image processing
• MySQL (4.0 / InnoDb)
• Java, for the node service
• Apache 2, Redhat, etc. etc.
Big Application?
• One programmer, one designer, etc.
• ~60,000 lines of PHP code
• ~60,000 lines of templates
• ~70 custom smarty functions/modifiers
• ~25,000 DB transactions/second at peak
• ~1000 pages per second at peak
Thinking outside the web app
• Services
– Atom/RSS/RDF Feeds
– APIs
• SOAP
• XML-RPC
• REST
• PEAR::XML::Tree
More cool stuff
• Email interface
– Postfix
– PHP
– PEAR::Mail::mimeDecode
• FTP
• Uploading API
• Authentication API
• Unicode
Even more stuff
• Real time application
• Cool flash apps
• Blogging
– Blogger API (1 & 2)
– Metaweblog API
– Atom
– LiveJournal
APIs are simple!
• Modeled on XML-RPC (Sort of)
• Method calls with XML responses
• SOAP, XML-RPC and REST are just transports
• PHP endpoints mean we can use the same application
logic as the website
XML isn’t simple :(
• PHP 4 doesn’t have good a XML parser
• Expat is cool though (PEAR::XML::Parser)
• Why doesn’t PEAR have XPath?
– Because PEAR is stupid!
– PHP 4 sucks!
I love XPath
if ($tree->root->name == 'methodResponse'){
if (($tree->root->children[0]->name == 'params')
&& ($tree->root->children[0]->children[0]->name == 'param')
&& ($tree->root->children[0]->children[0]->children[0]->name == 'value')
&& ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array')
&& ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){
$rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0];
}
if ($tree->root->children[0]->name == 'fault'){
$fault = $tree->root->children[0];
return $fault;
}
}
$nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()');
if (count($nodes)){
$rsp = array_pop($nodes);
}else{
list($fault) = $tree->select_nodes('/methodResponse/fault');
return $fault;
}
Creating API methods
• Stateless method-call APIs are easy to extend
• Adding a method requires no knowledge of the transport
• Adding a method once makes it available to all the
interfaces
• Self documenting
Red Hot Unicode Action
• UTF-8 pages
• CJKV support
• It’s really cool
Unicode for all
• It’s really easy
– Don’t need PHP support
– Don’t need MySQL support
– Just need the right headers
– UTF-8 is 7-bit transparent
– (Just don’t mess with high characters)
• Don’t use HtmlEntities()!
• But bear in mind…
• JavaScript has patchy Unicode support
• People using your APIs might be stupid
Scaling the beast
• Why PHP is great
• MySQL scaling
• Search scaling
• Horizontal scaling
Why PHP is great
• Stateless
– We can bounce people around servers
– Everything is stored in the database
– Even the smarty cache
– “Shared nothing”
– (so long as we avoid PHP sessions)
MySQL Scaling
• Our database server started to slow
• Load of 200
• Replication!
MySQL Replication
• But it only gives you more SELECT’s
• Else you need to partition vertically
• Re-architecting sucks :(
Looking at usage
• Snapshot of db1.flickr.com
– SELECT’s 44,220,588
– INSERT’s 1,349,234
– UPDATE’s 1,755,503
– DELETE’s 318,439
– 13 SELECT’s per I/U/D
Replication is really cool
• A bunch of slave servers handle all the SELECT’s
• A single master just handles I/U/D’s
• It can scale horizontally, at least for a while.
Searching
• A simple text search
• We were using RLIKE
• Then switched to LIKE
• Then disabled it all together
FULLTEXT Indexes
• MySQL saves the day!
• But they’re only supported my MyISAM tables
• We use InnoDb, because it’s a lot faster
• We’re doomed
But wait!
• Partial replication saves the day
• Replicate the portion of the database we want to search.
• But change the table types on the slave to MyISAM
• It can keep up because it’s only handling I/U/D’s on a
couple of tables
• And we can reduce the I/U/D’s with a little bit of vertical
partitioning
JOIN’s are slow
• Normalised data is for sissies
• Keep multiple copies of data around
• Makes searching faster
• Have to ensure consistency in the application logic
Our current setup
Slave Farm
DB1
Master I/U/D’s
SELECT’s
Search Slave
Farm
Search
SELECT’s
DB3
Main Search
slave
DB2
Main Slave
Horizontal scaling
• At the core of our design
• Just add hardware!
• Inexpensive
• Not exponential
• Avoid redesigns
Talking to the Node Service
• Everyone speaks XML (badly)
• Just TCP/IP - fsockopen()
• We’re issuing commands, not requesting data, so we
don’t bother to parse the response
– Just substring search for state=“ok”
• Don’t rely on it!
RSS / Atom / RDF
• Different formats
• All quite bad
• We’re generating a lot of different feeds
• Abstract the difference away using templates
• No good way to do private feeds. Why is nobody working
on this? (WSSE maybe?)
Receiving email
• Want users to be able to email photos to Flickr
• Just get postfix to pipe each mail to a PHP script
• Parse the mail and find any photos
• Cellular phone companies hate you
• Lots of mailers are retarded
– Photos as text/plain attachments :/
Upload via FTP
• PHP isn’t so great at being a daemon
• Leaks memory like a sieve
• No threads
• Java to the rescue
• Java just acts as an FTPd and passes all uploaded files
to PHP for processing
• (This isn’t actually public)
• Bricolage does this I think. Maybe Zope?
Blogs
• Why does everyone loves blogs so much?
• Only a few APIs really
– Blogger
– Metaweblog
– Blogger2
– Movable Type
– Atom
– Live Journal
It’s all broken
• Lots of blog software has broken interfaces
• It’s a support nightmare
• Manila is tricky
• But it all works, more or less
• Abstracted in the application logic
• We just call blogs_post_message();
Back to those APIs
• We opened up the Flickr APIs a few weeks ago
• Programmers mainly build tools for other programmers
• We have Perl, python, PHP, ActionScript, XMLHTTP and
.NET interface libraries
• But also a few actual applications
Flickr Rainbow
Tag Wallpaper
So what next?
• Much more scaling
• PHP 5?
• MySQL 5?
• Taking over the world
Flickr and PHP
Cal Henderson
Any Questions?

More Related Content

What's hot

PostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databasePostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databaseBarry Jones
 
JSON REST API for WordPress
JSON REST API for WordPressJSON REST API for WordPress
JSON REST API for WordPressTaylor Lovett
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettextNgoc Dao
 
Reviewing RESTful Web Apps
Reviewing RESTful Web AppsReviewing RESTful Web Apps
Reviewing RESTful Web AppsTakuto Wada
 
Go from a PHP Perspective
Go from a PHP PerspectiveGo from a PHP Perspective
Go from a PHP PerspectiveBarry Jones
 
The JSON REST API for WordPress
The JSON REST API for WordPressThe JSON REST API for WordPress
The JSON REST API for WordPressTaylor Lovett
 
Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Manish Pandit
 
PLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfPLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfAlfresco Software
 
OrientDB & Lucene
OrientDB & LuceneOrientDB & Lucene
OrientDB & Lucenewolf4ood
 
Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Taylor Lovett
 
WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013Curtiss Grymala
 
Best Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseBest Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseTaylor Lovett
 
Best Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsBest Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsTaylor Lovett
 
Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Wen-Tien Chang
 
WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013Curtiss Grymala
 
Software Development with Open Source
Software Development with Open SourceSoftware Development with Open Source
Software Development with Open SourceOpusVL
 
You Got React.js in My PHP
You Got React.js in My PHPYou Got React.js in My PHP
You Got React.js in My PHPTaylor Lovett
 

What's hot (19)

Apache Jackrabbit
Apache JackrabbitApache Jackrabbit
Apache Jackrabbit
 
PostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databasePostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty database
 
JSON REST API for WordPress
JSON REST API for WordPressJSON REST API for WordPress
JSON REST API for WordPress
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
 
Reviewing RESTful Web Apps
Reviewing RESTful Web AppsReviewing RESTful Web Apps
Reviewing RESTful Web Apps
 
Go from a PHP Perspective
Go from a PHP PerspectiveGo from a PHP Perspective
Go from a PHP Perspective
 
The JSON REST API for WordPress
The JSON REST API for WordPressThe JSON REST API for WordPress
The JSON REST API for WordPress
 
Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2
 
PLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfPLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring Surf
 
OrientDB & Lucene
OrientDB & LuceneOrientDB & Lucene
OrientDB & Lucene
 
Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch
 
WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013
 
Best Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseBest Practices for WordPress in Enterprise
Best Practices for WordPress in Enterprise
 
Best Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsBest Practices for Building WordPress Applications
Best Practices for Building WordPress Applications
 
Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3
 
WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache Solr
 
Software Development with Open Source
Software Development with Open SourceSoftware Development with Open Source
Software Development with Open Source
 
You Got React.js in My PHP
You Got React.js in My PHPYou Got React.js in My PHP
You Got React.js in My PHP
 

Viewers also liked

Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and ApproachesScalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approachesadunne
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson
 
Php internal architecture
Php internal architecturePhp internal architecture
Php internal architectureElizabeth Smith
 
Building Big on the Web
Building Big on the WebBuilding Big on the Web
Building Big on the WebCal Henderson
 
Fifty Rock Solid Templates
Fifty Rock Solid TemplatesFifty Rock Solid Templates
Fifty Rock Solid TemplatesAbhishek Shah
 
Head Over Heals - My Love Song
Head Over Heals - My Love SongHead Over Heals - My Love Song
Head Over Heals - My Love SongAbhishek Shah
 
Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Jennifer Bell
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLJoe Stump
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Aaron Saray
 
6 3 tier architecture php
6 3 tier architecture php6 3 tier architecture php
6 3 tier architecture phpcefour
 
Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture AppDynamics
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tigerElizabeth Smith
 
Benefits of Open Government Data
Benefits of Open Government DataBenefits of Open Government Data
Benefits of Open Government DataJennifer Bell
 
Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Fwdays
 
Domain Driven Design using Laravel
Domain Driven Design using LaravelDomain Driven Design using Laravel
Domain Driven Design using Laravelwajrcs
 

Viewers also liked (20)

Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and ApproachesScalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 
Php internal architecture
Php internal architecturePhp internal architecture
Php internal architecture
 
Building Big on the Web
Building Big on the WebBuilding Big on the Web
Building Big on the Web
 
Fifty Rock Solid Templates
Fifty Rock Solid TemplatesFifty Rock Solid Templates
Fifty Rock Solid Templates
 
Head Over Heals - My Love Song
Head Over Heals - My Love SongHead Over Heals - My Love Song
Head Over Heals - My Love Song
 
Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)
 
Nature
NatureNature
Nature
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQL
 
Going Places
Going PlacesGoing Places
Going Places
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
 
6 3 tier architecture php
6 3 tier architecture php6 3 tier architecture php
6 3 tier architecture php
 
Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture
 
Thassos - Greece
Thassos - GreeceThassos - Greece
Thassos - Greece
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
 
Benefits of Open Government Data
Benefits of Open Government DataBenefits of Open Government Data
Benefits of Open Government Data
 
Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"
 
Aids Out Life In
Aids Out Life InAids Out Life In
Aids Out Life In
 
Slab
SlabSlab
Slab
 
Domain Driven Design using Laravel
Domain Driven Design using LaravelDomain Driven Design using Laravel
Domain Driven Design using Laravel
 

Similar to flickr's architecture & php

Flickr Php
Flickr PhpFlickr Php
Flickr Phproyans
 
Flickr Architecture Presentation
Flickr Architecture PresentationFlickr Architecture Presentation
Flickr Architecture Presentationweb25
 
Flickr and PHP - Cal Henderson
Flickr and PHP - Cal HendersonFlickr and PHP - Cal Henderson
Flickr and PHP - Cal Hendersonkangaro10a
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Servicesroyans
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Servicesroyans
 
hibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfhibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfPatiento Del Mar
 
New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1Stefan Schmidt
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesBrett Meyer
 
Running MongoDB in the Cloud
Running MongoDB in the CloudRunning MongoDB in the Cloud
Running MongoDB in the CloudTony Tam
 
Hard Coding as a design approach
Hard Coding as a design approachHard Coding as a design approach
Hard Coding as a design approachOren Eini
 
Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Aaron Blythe
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?DATAVERSITY
 
REST APIs in Laravel 101
REST APIs in Laravel 101REST APIs in Laravel 101
REST APIs in Laravel 101Samantha Geitz
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
Big Data Developers Moscow Meetup 1  - sql on hadoopBig Data Developers Moscow Meetup 1  - sql on hadoop
Big Data Developers Moscow Meetup 1 - sql on hadoopbddmoscow
 
Modeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeModeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeDavid Boike
 
Rails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineRails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineDavid Keener
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlySarah Guido
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerIke Ellis
 
Oracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionOracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionJeff Smith
 

Similar to flickr's architecture & php (20)

Flickr Php
Flickr PhpFlickr Php
Flickr Php
 
Flickr Architecture Presentation
Flickr Architecture PresentationFlickr Architecture Presentation
Flickr Architecture Presentation
 
Flickr and PHP - Cal Henderson
Flickr and PHP - Cal HendersonFlickr and PHP - Cal Henderson
Flickr and PHP - Cal Henderson
 
Top ten-list
Top ten-listTop ten-list
Top ten-list
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
 
hibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfhibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdf
 
New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
 
Running MongoDB in the Cloud
Running MongoDB in the CloudRunning MongoDB in the Cloud
Running MongoDB in the Cloud
 
Hard Coding as a design approach
Hard Coding as a design approachHard Coding as a design approach
Hard Coding as a design approach
 
Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
 
REST APIs in Laravel 101
REST APIs in Laravel 101REST APIs in Laravel 101
REST APIs in Laravel 101
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
Big Data Developers Moscow Meetup 1  - sql on hadoopBig Data Developers Moscow Meetup 1  - sql on hadoop
Big Data Developers Moscow Meetup 1 - sql on hadoop
 
Modeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeModeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught Me
 
Rails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineRails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search Engine
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute Beginner
 
Oracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionOracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG Edition
 

More from coolpics

J9gumn6ysd33f
J9gumn6ysd33fJ9gumn6ysd33f
J9gumn6ysd33fcoolpics
 
Funny Planes
Funny PlanesFunny Planes
Funny Planescoolpics
 
Funny Bus Banners
Funny Bus BannersFunny Bus Banners
Funny Bus Bannerscoolpics
 
awesome pictures of transistors copulating
awesome pictures of transistors copulating awesome pictures of transistors copulating
awesome pictures of transistors copulating coolpics
 
Superb pictures of the earth from space
Superb pictures of the earth from spaceSuperb pictures of the earth from space
Superb pictures of the earth from spacecoolpics
 
Cool Pictures
Cool PicturesCool Pictures
Cool Picturescoolpics
 
Funny pictures
Funny picturesFunny pictures
Funny picturescoolpics
 
Cool art that will mess with your head...don't miss
Cool art that will mess with your head...don't missCool art that will mess with your head...don't miss
Cool art that will mess with your head...don't misscoolpics
 
The scariest car crash ever
The scariest car crash everThe scariest car crash ever
The scariest car crash evercoolpics
 
Great snaps from the African continent
Great snaps from the African continentGreat snaps from the African continent
Great snaps from the African continentcoolpics
 
What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?coolpics
 

More from coolpics (15)

J9gumn6ysd33f
J9gumn6ysd33fJ9gumn6ysd33f
J9gumn6ysd33f
 
Funny Planes
Funny PlanesFunny Planes
Funny Planes
 
Funny Bus Banners
Funny Bus BannersFunny Bus Banners
Funny Bus Banners
 
cool pics
cool picscool pics
cool pics
 
coolpics
coolpicscoolpics
coolpics
 
coolpics
coolpicscoolpics
coolpics
 
cool pics
cool picscool pics
cool pics
 
awesome pictures of transistors copulating
awesome pictures of transistors copulating awesome pictures of transistors copulating
awesome pictures of transistors copulating
 
Superb pictures of the earth from space
Superb pictures of the earth from spaceSuperb pictures of the earth from space
Superb pictures of the earth from space
 
Cool Pictures
Cool PicturesCool Pictures
Cool Pictures
 
Funny pictures
Funny picturesFunny pictures
Funny pictures
 
Cool art that will mess with your head...don't miss
Cool art that will mess with your head...don't missCool art that will mess with your head...don't miss
Cool art that will mess with your head...don't miss
 
The scariest car crash ever
The scariest car crash everThe scariest car crash ever
The scariest car crash ever
 
Great snaps from the African continent
Great snaps from the African continentGreat snaps from the African continent
Great snaps from the African continent
 
What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?
 

Recently uploaded

The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoUXDXConf
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsUXDXConf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024TopCSSGallery
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...CzechDreamin
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomCzechDreamin
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfFIDO Alliance
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyUXDXConf
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfFIDO Alliance
 

Recently uploaded (20)

The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 

flickr's architecture & php

  • 1. Flickr and PHP Cal Henderson
  • 2. What’s Flickr • Photo sharing • Open APIs
  • 3. Logical Architecture Page Logic Application Logic DatabasePhoto Storage API EndpointsTemplates 3rd Party Apps Flickr Apps Node Service Flickr.comEmail Users
  • 4. Physical Architecture Static Servers Database Servers Node Servers Web Servers Users
  • 5. Where is PHP? Page Logic Application Logic DatabasePhoto Storage API EndpointsTemplates 3rd Party Apps Flickr Apps Node Service Flickr.comEmail Users
  • 6. Other than PHP? • Smarty for templating • PEAR for XML and Email parsing • Perl for controlling… • ImageMagick, for image processing • MySQL (4.0 / InnoDb) • Java, for the node service • Apache 2, Redhat, etc. etc.
  • 7. Big Application? • One programmer, one designer, etc. • ~60,000 lines of PHP code • ~60,000 lines of templates • ~70 custom smarty functions/modifiers • ~25,000 DB transactions/second at peak • ~1000 pages per second at peak
  • 8. Thinking outside the web app • Services – Atom/RSS/RDF Feeds – APIs • SOAP • XML-RPC • REST • PEAR::XML::Tree
  • 9. More cool stuff • Email interface – Postfix – PHP – PEAR::Mail::mimeDecode • FTP • Uploading API • Authentication API • Unicode
  • 10. Even more stuff • Real time application • Cool flash apps • Blogging – Blogger API (1 & 2) – Metaweblog API – Atom – LiveJournal
  • 11. APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can use the same application logic as the website
  • 12. XML isn’t simple :( • PHP 4 doesn’t have good a XML parser • Expat is cool though (PEAR::XML::Parser) • Why doesn’t PEAR have XPath? – Because PEAR is stupid! – PHP 4 sucks!
  • 13. I love XPath if ($tree->root->name == 'methodResponse'){ if (($tree->root->children[0]->name == 'params') && ($tree->root->children[0]->children[0]->name == 'param') && ($tree->root->children[0]->children[0]->children[0]->name == 'value') && ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array') && ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){ $rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0]; } if ($tree->root->children[0]->name == 'fault'){ $fault = $tree->root->children[0]; return $fault; } } $nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()'); if (count($nodes)){ $rsp = array_pop($nodes); }else{ list($fault) = $tree->select_nodes('/methodResponse/fault'); return $fault; }
  • 14. Creating API methods • Stateless method-call APIs are easy to extend • Adding a method requires no knowledge of the transport • Adding a method once makes it available to all the interfaces • Self documenting
  • 15. Red Hot Unicode Action • UTF-8 pages • CJKV support • It’s really cool
  • 16.
  • 17. Unicode for all • It’s really easy – Don’t need PHP support – Don’t need MySQL support – Just need the right headers – UTF-8 is 7-bit transparent – (Just don’t mess with high characters) • Don’t use HtmlEntities()! • But bear in mind… • JavaScript has patchy Unicode support • People using your APIs might be stupid
  • 18. Scaling the beast • Why PHP is great • MySQL scaling • Search scaling • Horizontal scaling
  • 19. Why PHP is great • Stateless – We can bounce people around servers – Everything is stored in the database – Even the smarty cache – “Shared nothing” – (so long as we avoid PHP sessions)
  • 20. MySQL Scaling • Our database server started to slow • Load of 200 • Replication!
  • 21. MySQL Replication • But it only gives you more SELECT’s • Else you need to partition vertically • Re-architecting sucks :(
  • 22. Looking at usage • Snapshot of db1.flickr.com – SELECT’s 44,220,588 – INSERT’s 1,349,234 – UPDATE’s 1,755,503 – DELETE’s 318,439 – 13 SELECT’s per I/U/D
  • 23. Replication is really cool • A bunch of slave servers handle all the SELECT’s • A single master just handles I/U/D’s • It can scale horizontally, at least for a while.
  • 24. Searching • A simple text search • We were using RLIKE • Then switched to LIKE • Then disabled it all together
  • 25. FULLTEXT Indexes • MySQL saves the day! • But they’re only supported my MyISAM tables • We use InnoDb, because it’s a lot faster • We’re doomed
  • 26. But wait! • Partial replication saves the day • Replicate the portion of the database we want to search. • But change the table types on the slave to MyISAM • It can keep up because it’s only handling I/U/D’s on a couple of tables • And we can reduce the I/U/D’s with a little bit of vertical partitioning
  • 27. JOIN’s are slow • Normalised data is for sissies • Keep multiple copies of data around • Makes searching faster • Have to ensure consistency in the application logic
  • 28. Our current setup Slave Farm DB1 Master I/U/D’s SELECT’s Search Slave Farm Search SELECT’s DB3 Main Search slave DB2 Main Slave
  • 29. Horizontal scaling • At the core of our design • Just add hardware! • Inexpensive • Not exponential • Avoid redesigns
  • 30. Talking to the Node Service • Everyone speaks XML (badly) • Just TCP/IP - fsockopen() • We’re issuing commands, not requesting data, so we don’t bother to parse the response – Just substring search for state=“ok” • Don’t rely on it!
  • 31. RSS / Atom / RDF • Different formats • All quite bad • We’re generating a lot of different feeds • Abstract the difference away using templates • No good way to do private feeds. Why is nobody working on this? (WSSE maybe?)
  • 32. Receiving email • Want users to be able to email photos to Flickr • Just get postfix to pipe each mail to a PHP script • Parse the mail and find any photos • Cellular phone companies hate you • Lots of mailers are retarded – Photos as text/plain attachments :/
  • 33. Upload via FTP • PHP isn’t so great at being a daemon • Leaks memory like a sieve • No threads • Java to the rescue • Java just acts as an FTPd and passes all uploaded files to PHP for processing • (This isn’t actually public) • Bricolage does this I think. Maybe Zope?
  • 34. Blogs • Why does everyone loves blogs so much? • Only a few APIs really – Blogger – Metaweblog – Blogger2 – Movable Type – Atom – Live Journal
  • 35. It’s all broken • Lots of blog software has broken interfaces • It’s a support nightmare • Manila is tricky • But it all works, more or less • Abstracted in the application logic • We just call blogs_post_message();
  • 36. Back to those APIs • We opened up the Flickr APIs a few weeks ago • Programmers mainly build tools for other programmers • We have Perl, python, PHP, ActionScript, XMLHTTP and .NET interface libraries • But also a few actual applications
  • 39. So what next? • Much more scaling • PHP 5? • MySQL 5? • Taking over the world
  • 40. Flickr and PHP Cal Henderson