SlideShare a Scribd company logo
Flickr and PHP
Cal Henderson
What’s Flickr
• Photo sharing
• Open APIs
Logical Architecture
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
Physical Architecture
Static Servers Database Servers Node Servers
Web Servers
Users
Where is PHP?
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
Other than PHP?
• Smarty for templating
• PEAR for XML and Email parsing
• Perl for controlling…
• ImageMagick, for image processing
• MySQL (4.0 / InnoDb)
• Java, for the node service
• Apache 2, Redhat, etc. etc.
Big Application?
• One programmer, one designer, etc.
• ~60,000 lines of PHP code
• ~60,000 lines of templates
• ~70 custom smarty functions/modifiers
• ~25,000 DB transactions/second at peak
• ~1000 pages per second at peak
Thinking outside the web app
• Services
– Atom/RSS/RDF Feeds
– APIs
• SOAP
• XML-RPC
• REST
• PEAR::XML::Tree
More cool stuff
• Email interface
– Postfix
– PHP
– PEAR::Mail::mimeDecode
• FTP
• Uploading API
• Authentication API
• Unicode
Even more stuff
• Real time application
• Cool flash apps
• Blogging
– Blogger API (1 & 2)
– Metaweblog API
– Atom
– LiveJournal
APIs are simple!
• Modeled on XML-RPC (Sort of)
• Method calls with XML responses
• SOAP, XML-RPC and REST are just transports
• PHP endpoints mean we can use the same application
logic as the website
XML isn’t simple :(
• PHP 4 doesn’t have good a XML parser
• Expat is cool though (PEAR::XML::Parser)
• Why doesn’t PEAR have XPath?
– Because PEAR is stupid!
– PHP 4 sucks!
I love XPath
if ($tree->root->name == 'methodResponse'){
if (($tree->root->children[0]->name == 'params')
&& ($tree->root->children[0]->children[0]->name == 'param')
&& ($tree->root->children[0]->children[0]->children[0]->name == 'value')
&& ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array')
&& ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){
$rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0];
}
if ($tree->root->children[0]->name == 'fault'){
$fault = $tree->root->children[0];
return $fault;
}
}
$nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()');
if (count($nodes)){
$rsp = array_pop($nodes);
}else{
list($fault) = $tree->select_nodes('/methodResponse/fault');
return $fault;
}
Creating API methods
• Stateless method-call APIs are easy to extend
• Adding a method requires no knowledge of the transport
• Adding a method once makes it available to all the
interfaces
• Self documenting
Red Hot Unicode Action
• UTF-8 pages
• CJKV support
• It’s really cool
Unicode for all
• It’s really easy
– Don’t need PHP support
– Don’t need MySQL support
– Just need the right headers
– UTF-8 is 7-bit transparent
– (Just don’t mess with high characters)
• Don’t use HtmlEntities()!
• But bear in mind…
• JavaScript has patchy Unicode support
• People using your APIs might be stupid
Scaling the beast
• Why PHP is great
• MySQL scaling
• Search scaling
• Horizontal scaling
Why PHP is great
• Stateless
– We can bounce people around servers
– Everything is stored in the database
– Even the smarty cache
– “Shared nothing”
– (so long as we avoid PHP sessions)
MySQL Scaling
• Our database server started to slow
• Load of 200
• Replication!
MySQL Replication
• But it only gives you more SELECT’s
• Else you need to partition vertically
• Re-architecting sucks :(
Looking at usage
• Snapshot of db1.flickr.com
– SELECT’s 44,220,588
– INSERT’s 1,349,234
– UPDATE’s 1,755,503
– DELETE’s 318,439
– 13 SELECT’s per I/U/D
Replication is really cool
• A bunch of slave servers handle all the SELECT’s
• A single master just handles I/U/D’s
• It can scale horizontally, at least for a while.
Searching
• A simple text search
• We were using RLIKE
• Then switched to LIKE
• Then disabled it all together
FULLTEXT Indexes
• MySQL saves the day!
• But they’re only supported my MyISAM tables
• We use InnoDb, because it’s a lot faster
• We’re doomed
But wait!
• Partial replication saves the day
• Replicate the portion of the database we want to search.
• But change the table types on the slave to MyISAM
• It can keep up because it’s only handling I/U/D’s on a
couple of tables
• And we can reduce the I/U/D’s with a little bit of vertical
partitioning
JOIN’s are slow
• Normalised data is for sissies
• Keep multiple copies of data around
• Makes searching faster
• Have to ensure consistency in the application logic
Our current setup
Slave Farm
DB1
Master I/U/D’s
SELECT’s
Search Slave
Farm
Search
SELECT’s
DB3
Main Search
slave
DB2
Main Slave
Horizontal scaling
• At the core of our design
• Just add hardware!
• Inexpensive
• Not exponential
• Avoid redesigns
Talking to the Node Service
• Everyone speaks XML (badly)
• Just TCP/IP - fsockopen()
• We’re issuing commands, not requesting data, so we
don’t bother to parse the response
– Just substring search for state=“ok”
• Don’t rely on it!
RSS / Atom / RDF
• Different formats
• All quite bad
• We’re generating a lot of different feeds
• Abstract the difference away using templates
• No good way to do private feeds. Why is nobody working
on this? (WSSE maybe?)
Receiving email
• Want users to be able to email photos to Flickr
• Just get postfix to pipe each mail to a PHP script
• Parse the mail and find any photos
• Cellular phone companies hate you
• Lots of mailers are retarded
– Photos as text/plain attachments :/
Upload via FTP
• PHP isn’t so great at being a daemon
• Leaks memory like a sieve
• No threads
• Java to the rescue
• Java just acts as an FTPd and passes all uploaded files
to PHP for processing
• (This isn’t actually public)
• Bricolage does this I think. Maybe Zope?
Blogs
• Why does everyone loves blogs so much?
• Only a few APIs really
– Blogger
– Metaweblog
– Blogger2
– Movable Type
– Atom
– Live Journal
It’s all broken
• Lots of blog software has broken interfaces
• It’s a support nightmare
• Manila is tricky
• But it all works, more or less
• Abstracted in the application logic
• We just call blogs_post_message();
Back to those APIs
• We opened up the Flickr APIs a few weeks ago
• Programmers mainly build tools for other programmers
• We have Perl, python, PHP, ActionScript, XMLHTTP and
.NET interface libraries
• But also a few actual applications
Flickr Rainbow
Tag Wallpaper
So what next?
• Much more scaling
• PHP 5?
• MySQL 5?
• Taking over the world
Flickr and PHP
Cal Henderson
Any Questions?

More Related Content

What's hot

Apache Jackrabbit
Apache JackrabbitApache Jackrabbit
Apache Jackrabbit
elliando dias
 
PostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databasePostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty database
Barry Jones
 
JSON REST API for WordPress
JSON REST API for WordPressJSON REST API for WordPress
JSON REST API for WordPress
Taylor Lovett
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
Ngoc Dao
 
Reviewing RESTful Web Apps
Reviewing RESTful Web AppsReviewing RESTful Web Apps
Reviewing RESTful Web Apps
Takuto Wada
 
Go from a PHP Perspective
Go from a PHP PerspectiveGo from a PHP Perspective
Go from a PHP Perspective
Barry Jones
 
The JSON REST API for WordPress
The JSON REST API for WordPressThe JSON REST API for WordPress
The JSON REST API for WordPress
Taylor Lovett
 
Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2
Manish Pandit
 
PLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfPLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring Surf
Alfresco Software
 
OrientDB & Lucene
OrientDB & LuceneOrientDB & Lucene
OrientDB & Lucene
wolf4ood
 
Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch
Taylor Lovett
 
WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013
Curtiss Grymala
 
Best Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseBest Practices for WordPress in Enterprise
Best Practices for WordPress in Enterprise
Taylor Lovett
 
Best Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsBest Practices for Building WordPress Applications
Best Practices for Building WordPress Applications
Taylor Lovett
 
Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3
Wen-Tien Chang
 
WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013
Curtiss Grymala
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache Solr
Shalin Shekhar Mangar
 
Software Development with Open Source
Software Development with Open SourceSoftware Development with Open Source
Software Development with Open Source
OpusVL
 
You Got React.js in My PHP
You Got React.js in My PHPYou Got React.js in My PHP
You Got React.js in My PHP
Taylor Lovett
 

What's hot (19)

Apache Jackrabbit
Apache JackrabbitApache Jackrabbit
Apache Jackrabbit
 
PostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databasePostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty database
 
JSON REST API for WordPress
JSON REST API for WordPressJSON REST API for WordPress
JSON REST API for WordPress
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
 
Reviewing RESTful Web Apps
Reviewing RESTful Web AppsReviewing RESTful Web Apps
Reviewing RESTful Web Apps
 
Go from a PHP Perspective
Go from a PHP PerspectiveGo from a PHP Perspective
Go from a PHP Perspective
 
The JSON REST API for WordPress
The JSON REST API for WordPressThe JSON REST API for WordPress
The JSON REST API for WordPress
 
Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2
 
PLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfPLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring Surf
 
OrientDB & Lucene
OrientDB & LuceneOrientDB & Lucene
OrientDB & Lucene
 
Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch
 
WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013
 
Best Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseBest Practices for WordPress in Enterprise
Best Practices for WordPress in Enterprise
 
Best Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsBest Practices for Building WordPress Applications
Best Practices for Building WordPress Applications
 
Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3
 
WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache Solr
 
Software Development with Open Source
Software Development with Open SourceSoftware Development with Open Source
Software Development with Open Source
 
You Got React.js in My PHP
You Got React.js in My PHPYou Got React.js in My PHP
You Got React.js in My PHP
 

Viewers also liked

Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and ApproachesScalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
adunne
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Cal Henderson
 
Php internal architecture
Php internal architecturePhp internal architecture
Php internal architecture
Elizabeth Smith
 
Building Big on the Web
Building Big on the WebBuilding Big on the Web
Building Big on the Web
Cal Henderson
 
Fifty Rock Solid Templates
Fifty Rock Solid TemplatesFifty Rock Solid Templates
Fifty Rock Solid Templates
Abhishek Shah
 
Head Over Heals - My Love Song
Head Over Heals - My Love SongHead Over Heals - My Love Song
Head Over Heals - My Love Song
Abhishek Shah
 
Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)
Jennifer Bell
 
Nature
NatureNature
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQL
Joe Stump
 
Going Places
Going PlacesGoing Places
Going Places
Abhishek Shah
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Aaron Saray
 
6 3 tier architecture php
6 3 tier architecture php6 3 tier architecture php
6 3 tier architecture php
cefour
 
Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture
AppDynamics
 
Thassos - Greece
Thassos - GreeceThassos - Greece
Thassos - Greece
Manuela Szekely
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
Elizabeth Smith
 
Benefits of Open Government Data
Benefits of Open Government DataBenefits of Open Government Data
Benefits of Open Government Data
Jennifer Bell
 
Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"
Fwdays
 
Aids Out Life In
Aids Out Life InAids Out Life In
Aids Out Life In
Abhishek Shah
 
Slab
SlabSlab
Domain Driven Design using Laravel
Domain Driven Design using LaravelDomain Driven Design using Laravel
Domain Driven Design using Laravel
wajrcs
 

Viewers also liked (20)

Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and ApproachesScalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 
Php internal architecture
Php internal architecturePhp internal architecture
Php internal architecture
 
Building Big on the Web
Building Big on the WebBuilding Big on the Web
Building Big on the Web
 
Fifty Rock Solid Templates
Fifty Rock Solid TemplatesFifty Rock Solid Templates
Fifty Rock Solid Templates
 
Head Over Heals - My Love Song
Head Over Heals - My Love SongHead Over Heals - My Love Song
Head Over Heals - My Love Song
 
Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)
 
Nature
NatureNature
Nature
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQL
 
Going Places
Going PlacesGoing Places
Going Places
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
 
6 3 tier architecture php
6 3 tier architecture php6 3 tier architecture php
6 3 tier architecture php
 
Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture
 
Thassos - Greece
Thassos - GreeceThassos - Greece
Thassos - Greece
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
 
Benefits of Open Government Data
Benefits of Open Government DataBenefits of Open Government Data
Benefits of Open Government Data
 
Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"
 
Aids Out Life In
Aids Out Life InAids Out Life In
Aids Out Life In
 
Slab
SlabSlab
Slab
 
Domain Driven Design using Laravel
Domain Driven Design using LaravelDomain Driven Design using Laravel
Domain Driven Design using Laravel
 

Similar to flickr's architecture & php

Flickr Php
Flickr PhpFlickr Php
Flickr Php
royans
 
Flickr Architecture Presentation
Flickr Architecture PresentationFlickr Architecture Presentation
Flickr Architecture Presentation
web25
 
Flickr and PHP - Cal Henderson
Flickr and PHP - Cal HendersonFlickr and PHP - Cal Henderson
Flickr and PHP - Cal Henderson
kangaro10a
 
Top ten-list
Top ten-listTop ten-list
Top ten-list
Brian DeShong
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
royans
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
royans
 
hibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfhibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdf
Patiento Del Mar
 
New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1
Stefan Schmidt
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Brett Meyer
 
Running MongoDB in the Cloud
Running MongoDB in the CloudRunning MongoDB in the Cloud
Running MongoDB in the Cloud
Tony Tam
 
Hard Coding as a design approach
Hard Coding as a design approachHard Coding as a design approach
Hard Coding as a design approach
Oren Eini
 
Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013
Aaron Blythe
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
DATAVERSITY
 
REST APIs in Laravel 101
REST APIs in Laravel 101REST APIs in Laravel 101
REST APIs in Laravel 101
Samantha Geitz
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
Big Data Developers Moscow Meetup 1  - sql on hadoopBig Data Developers Moscow Meetup 1  - sql on hadoop
Big Data Developers Moscow Meetup 1 - sql on hadoop
bddmoscow
 
Modeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeModeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught Me
David Boike
 
Rails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineRails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search Engine
David Keener
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Sarah Guido
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute Beginner
Ike Ellis
 
Oracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionOracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG Edition
Jeff Smith
 

Similar to flickr's architecture & php (20)

Flickr Php
Flickr PhpFlickr Php
Flickr Php
 
Flickr Architecture Presentation
Flickr Architecture PresentationFlickr Architecture Presentation
Flickr Architecture Presentation
 
Flickr and PHP - Cal Henderson
Flickr and PHP - Cal HendersonFlickr and PHP - Cal Henderson
Flickr and PHP - Cal Henderson
 
Top ten-list
Top ten-listTop ten-list
Top ten-list
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
 
hibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfhibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdf
 
New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
 
Running MongoDB in the Cloud
Running MongoDB in the CloudRunning MongoDB in the Cloud
Running MongoDB in the Cloud
 
Hard Coding as a design approach
Hard Coding as a design approachHard Coding as a design approach
Hard Coding as a design approach
 
Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
 
REST APIs in Laravel 101
REST APIs in Laravel 101REST APIs in Laravel 101
REST APIs in Laravel 101
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
Big Data Developers Moscow Meetup 1  - sql on hadoopBig Data Developers Moscow Meetup 1  - sql on hadoop
Big Data Developers Moscow Meetup 1 - sql on hadoop
 
Modeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeModeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught Me
 
Rails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineRails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search Engine
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute Beginner
 
Oracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionOracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG Edition
 

More from coolpics

J9gumn6ysd33f
J9gumn6ysd33fJ9gumn6ysd33f
J9gumn6ysd33f
coolpics
 
Funny Planes
Funny PlanesFunny Planes
Funny Planes
coolpics
 
Funny Bus Banners
Funny Bus BannersFunny Bus Banners
Funny Bus Banners
coolpics
 
cool pics
cool picscool pics
cool pics
coolpics
 
coolpics
coolpicscoolpics
coolpics
coolpics
 
coolpics
coolpicscoolpics
coolpics
coolpics
 
cool pics
cool picscool pics
cool pics
coolpics
 
awesome pictures of transistors copulating
awesome pictures of transistors copulating awesome pictures of transistors copulating
awesome pictures of transistors copulating coolpics
 
Superb pictures of the earth from space
Superb pictures of the earth from spaceSuperb pictures of the earth from space
Superb pictures of the earth from space
coolpics
 
Cool Pictures
Cool PicturesCool Pictures
Cool Pictures
coolpics
 
Funny pictures
Funny picturesFunny pictures
Funny picturescoolpics
 
Cool art that will mess with your head...don't miss
Cool art that will mess with your head...don't missCool art that will mess with your head...don't miss
Cool art that will mess with your head...don't miss
coolpics
 
The scariest car crash ever
The scariest car crash everThe scariest car crash ever
The scariest car crash ever
coolpics
 
Great snaps from the African continent
Great snaps from the African continentGreat snaps from the African continent
Great snaps from the African continent
coolpics
 
What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?
coolpics
 

More from coolpics (15)

J9gumn6ysd33f
J9gumn6ysd33fJ9gumn6ysd33f
J9gumn6ysd33f
 
Funny Planes
Funny PlanesFunny Planes
Funny Planes
 
Funny Bus Banners
Funny Bus BannersFunny Bus Banners
Funny Bus Banners
 
cool pics
cool picscool pics
cool pics
 
coolpics
coolpicscoolpics
coolpics
 
coolpics
coolpicscoolpics
coolpics
 
cool pics
cool picscool pics
cool pics
 
awesome pictures of transistors copulating
awesome pictures of transistors copulating awesome pictures of transistors copulating
awesome pictures of transistors copulating
 
Superb pictures of the earth from space
Superb pictures of the earth from spaceSuperb pictures of the earth from space
Superb pictures of the earth from space
 
Cool Pictures
Cool PicturesCool Pictures
Cool Pictures
 
Funny pictures
Funny picturesFunny pictures
Funny pictures
 
Cool art that will mess with your head...don't miss
Cool art that will mess with your head...don't missCool art that will mess with your head...don't miss
Cool art that will mess with your head...don't miss
 
The scariest car crash ever
The scariest car crash everThe scariest car crash ever
The scariest car crash ever
 
Great snaps from the African continent
Great snaps from the African continentGreat snaps from the African continent
Great snaps from the African continent
 
What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?
 

Recently uploaded

A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
Fwdays
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Fwdays
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
DianaGray10
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 

Recently uploaded (20)

A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 

flickr's architecture & php

  • 1. Flickr and PHP Cal Henderson
  • 2. What’s Flickr • Photo sharing • Open APIs
  • 3. Logical Architecture Page Logic Application Logic DatabasePhoto Storage API EndpointsTemplates 3rd Party Apps Flickr Apps Node Service Flickr.comEmail Users
  • 4. Physical Architecture Static Servers Database Servers Node Servers Web Servers Users
  • 5. Where is PHP? Page Logic Application Logic DatabasePhoto Storage API EndpointsTemplates 3rd Party Apps Flickr Apps Node Service Flickr.comEmail Users
  • 6. Other than PHP? • Smarty for templating • PEAR for XML and Email parsing • Perl for controlling… • ImageMagick, for image processing • MySQL (4.0 / InnoDb) • Java, for the node service • Apache 2, Redhat, etc. etc.
  • 7. Big Application? • One programmer, one designer, etc. • ~60,000 lines of PHP code • ~60,000 lines of templates • ~70 custom smarty functions/modifiers • ~25,000 DB transactions/second at peak • ~1000 pages per second at peak
  • 8. Thinking outside the web app • Services – Atom/RSS/RDF Feeds – APIs • SOAP • XML-RPC • REST • PEAR::XML::Tree
  • 9. More cool stuff • Email interface – Postfix – PHP – PEAR::Mail::mimeDecode • FTP • Uploading API • Authentication API • Unicode
  • 10. Even more stuff • Real time application • Cool flash apps • Blogging – Blogger API (1 & 2) – Metaweblog API – Atom – LiveJournal
  • 11. APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can use the same application logic as the website
  • 12. XML isn’t simple :( • PHP 4 doesn’t have good a XML parser • Expat is cool though (PEAR::XML::Parser) • Why doesn’t PEAR have XPath? – Because PEAR is stupid! – PHP 4 sucks!
  • 13. I love XPath if ($tree->root->name == 'methodResponse'){ if (($tree->root->children[0]->name == 'params') && ($tree->root->children[0]->children[0]->name == 'param') && ($tree->root->children[0]->children[0]->children[0]->name == 'value') && ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array') && ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){ $rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0]; } if ($tree->root->children[0]->name == 'fault'){ $fault = $tree->root->children[0]; return $fault; } } $nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()'); if (count($nodes)){ $rsp = array_pop($nodes); }else{ list($fault) = $tree->select_nodes('/methodResponse/fault'); return $fault; }
  • 14. Creating API methods • Stateless method-call APIs are easy to extend • Adding a method requires no knowledge of the transport • Adding a method once makes it available to all the interfaces • Self documenting
  • 15. Red Hot Unicode Action • UTF-8 pages • CJKV support • It’s really cool
  • 16.
  • 17. Unicode for all • It’s really easy – Don’t need PHP support – Don’t need MySQL support – Just need the right headers – UTF-8 is 7-bit transparent – (Just don’t mess with high characters) • Don’t use HtmlEntities()! • But bear in mind… • JavaScript has patchy Unicode support • People using your APIs might be stupid
  • 18. Scaling the beast • Why PHP is great • MySQL scaling • Search scaling • Horizontal scaling
  • 19. Why PHP is great • Stateless – We can bounce people around servers – Everything is stored in the database – Even the smarty cache – “Shared nothing” – (so long as we avoid PHP sessions)
  • 20. MySQL Scaling • Our database server started to slow • Load of 200 • Replication!
  • 21. MySQL Replication • But it only gives you more SELECT’s • Else you need to partition vertically • Re-architecting sucks :(
  • 22. Looking at usage • Snapshot of db1.flickr.com – SELECT’s 44,220,588 – INSERT’s 1,349,234 – UPDATE’s 1,755,503 – DELETE’s 318,439 – 13 SELECT’s per I/U/D
  • 23. Replication is really cool • A bunch of slave servers handle all the SELECT’s • A single master just handles I/U/D’s • It can scale horizontally, at least for a while.
  • 24. Searching • A simple text search • We were using RLIKE • Then switched to LIKE • Then disabled it all together
  • 25. FULLTEXT Indexes • MySQL saves the day! • But they’re only supported my MyISAM tables • We use InnoDb, because it’s a lot faster • We’re doomed
  • 26. But wait! • Partial replication saves the day • Replicate the portion of the database we want to search. • But change the table types on the slave to MyISAM • It can keep up because it’s only handling I/U/D’s on a couple of tables • And we can reduce the I/U/D’s with a little bit of vertical partitioning
  • 27. JOIN’s are slow • Normalised data is for sissies • Keep multiple copies of data around • Makes searching faster • Have to ensure consistency in the application logic
  • 28. Our current setup Slave Farm DB1 Master I/U/D’s SELECT’s Search Slave Farm Search SELECT’s DB3 Main Search slave DB2 Main Slave
  • 29. Horizontal scaling • At the core of our design • Just add hardware! • Inexpensive • Not exponential • Avoid redesigns
  • 30. Talking to the Node Service • Everyone speaks XML (badly) • Just TCP/IP - fsockopen() • We’re issuing commands, not requesting data, so we don’t bother to parse the response – Just substring search for state=“ok” • Don’t rely on it!
  • 31. RSS / Atom / RDF • Different formats • All quite bad • We’re generating a lot of different feeds • Abstract the difference away using templates • No good way to do private feeds. Why is nobody working on this? (WSSE maybe?)
  • 32. Receiving email • Want users to be able to email photos to Flickr • Just get postfix to pipe each mail to a PHP script • Parse the mail and find any photos • Cellular phone companies hate you • Lots of mailers are retarded – Photos as text/plain attachments :/
  • 33. Upload via FTP • PHP isn’t so great at being a daemon • Leaks memory like a sieve • No threads • Java to the rescue • Java just acts as an FTPd and passes all uploaded files to PHP for processing • (This isn’t actually public) • Bricolage does this I think. Maybe Zope?
  • 34. Blogs • Why does everyone loves blogs so much? • Only a few APIs really – Blogger – Metaweblog – Blogger2 – Movable Type – Atom – Live Journal
  • 35. It’s all broken • Lots of blog software has broken interfaces • It’s a support nightmare • Manila is tricky • But it all works, more or less • Abstracted in the application logic • We just call blogs_post_message();
  • 36. Back to those APIs • We opened up the Flickr APIs a few weeks ago • Programmers mainly build tools for other programmers • We have Perl, python, PHP, ActionScript, XMLHTTP and .NET interface libraries • But also a few actual applications
  • 39. So what next? • Much more scaling • PHP 5? • MySQL 5? • Taking over the world
  • 40. Flickr and PHP Cal Henderson