Cloud, Cache, and Configs

Cloud, Cache, and Conﬁgs
WordCamp NYC 2012

Saturday, June 9, 12

Scott Taylor
Lead PHP Developer, eMusic
@wonderboymusic
www.scotty-t.com
#projectmanagementsoftware


eMusic

Multisite
Regionalized Content
Regionalized Caching
Shared Content across Sites
Tons of Custom Post Types
Post Formats
Tons of Web Services


eMusic Architecture
• 12-24 Amazon EC2 instances (CentOS)
• 4 MySQL (1 Write, 3 Read)
• 4 Memcached (~28GB RAM)
• Amazon S3 / Cloudfront CDN
• PHP 5.3 / MySQL 5.5
• PECL: APC, Memcached, HTTP
• Batcache

APC = duh

• Essential for PHP
• Opcode cache
• apc.shm_size = 64M or higher


Logs

• Never allow any PHP-related notices,
errors, warnings, exceptions, etc
• Check access logs regularly for 404s, 500s
etc
• make a constant for toggling debug logging


Production Data

• Always pull down, never push up
• We only push imported legacy content up
• output buffering
• code ﬁltering
• more to come on this....


Output Buffering

ob_start( $callback )

ob_start();

echo ‘Daryl’;

$daryl = ob_get_clean();


ob_start( function ( $data ) {

return str_replace(
$urls_array,
EMUSIC_CURRENT_HOST,
$data
);

} );


Configs

• Mandatory machine configs
• HyperDB config
• Overridable sunrise.php


Machine Config
• DB credentials local to that machine
• Amazon S3 bucket
• Web Service Endpoints
• Memcached Servers
if ( file_exists( ‘/wp-config/config.php’ ) ) {
require_once( ‘/wp-config/config.php’ );
} else {
die( ‘You must have a local config!’ );
}

Memcached

• Memcached PHP Extension (not Memcache)
• Can be used locally (127.0.0.1)
• Memcached Redux supports
wp_cache_get_/set_multi( )
• Johnny Cache


Batcache
• Full-page caching
• Can be conﬁgured
• You can partition cache by unique values
• Loads before plugins - any code you need
has to be duped or added early
(sunrise.php)


class batcache {

// This is the base configuration. You can edit these variables or move
them into your wp-config.php file.
var $max_age = 300;

// Expire batcache items aged this many seconds (zero to disable batcache)

var $remote = 0;

// Zero disables sending buffers to remote datacenters (req/sec is never sent)

var $times = 5;

// Only batcache a page after it is accessed this many times... (two or more)

var $seconds = 120;

// ...in this many seconds (zero to ignore this and use batcache immediately)

var $group = 'batcache';

// Name of memcached group. You can simulate a cache flush by changing this.

var $unique = array( BATCACHE_REGION, BATCACHE_COUNTRY );

// If you conditionally serve different content, put the variable values here.

var $headers = array(
'X-nananana' => 'Batcache'
);

. . . .
}


Sunrise
• Used to alter multisite context
• sets $current_blog and $current_site
• ﬁlters all URL functions to resolve all URLs
to your current domain
• registers custom locations for media
• ﬁlters Admin URLs


switch_to_blog( $blog_id )

• All dynamic functions need to account for
this
• Shared content needs to resolve proper
URLs
• Different sites have different media
locations


Fix switch_to_blog()
$current_blog = new stdClass();
$current_blog->site_id = 1;
$current_blog->archived = 0;
$current_blog->mature = 0;
$current_blog->spam = 0;
$current_blog->deleted = 0;
$current_blog->lang_id = 0;
$current_blog->public = 1;
$current_blog->registered = '2011-02-20 03:38:22';
$current_blog->last_updated = $_SERVER['REQUEST_TIME'];
$current_blog->domain = EMUSIC_CURRENT_HOST;

function emusic_switch_to_blog( $blog_id, $prev_blog_id = 0 ) {
if ( $blog_id === $prev_blog_id )
return;

global $current_blog, $emusic_paths;
$current_blog->blog_id = $blog_id;
$current_blog->path = $emusic_paths[$blog_id];
}

emusic_switch_to_blog( $the_id );

add_action( 'switch_blog', 'emusic_switch_to_blog', 10, 2 );

$blog_id = $the_id;
$site_id = 1;

$current_site = new stdClass();
$current_site->blog_id = $the_id;
$current_site->id = 1;
$current_site->domain = EMUSIC_CURRENT_HOST;
$current_site->site_name = 'eMusic';
$current_site->path = $the_path;

Filter URLs

add_filter( 'pre_option_upload_path', function () {
$id = get_current_blog_id();
if ( 1 < $id )
return $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$id}/files";

return $_SERVER['DOCUMENT_ROOT'] . '/' . EMUSIC_UPLOADS;
} );

add_filter( 'pre_option_upload_url_path', function () {
$id = get_current_blog_id();
if ( 1 < $id )
return 'http://' . EMUSIC_CURRENT_HOST . "/blogs/{$id}/files";

return 'http://' . EMUSIC_CURRENT_HOST . '/' . EMUSIC_UPLOADS;
} );

add_filter( 'pre_option_siteurl', function () {
global $current_blog;
$extra = rtrim( $current_blog->path, '/' );
return 'http://' . EMUSIC_CURRENT_HOST . $extra;
} );

add_filter( 'pre_option_home', function () {
global $current_blog;
$extra = rtrim( $current_blog->path, '/' );
return 'http://' . EMUSIC_CURRENT_HOST . $extra;
} );


Plugins
• Filter active network plugins (don’t rely on
database being correct)
• Filter each site’s plugins (if you have a
manageable number)
• Use classes, not a bunch of functions
• Extend before you copy / paste


Site Conﬁgs
require_once( 'site-configs/global.php' );

if ( $the_id > 1 ) {
define( 'UPLOADBLOGSDIR', 0 );
define( 'UPLOADS', 0 );
define( 'BLOGUPLOADDIR', $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$the_id}/files" );

switch ( $the_id ) {
case 2:
require_once( 'site-configs/bbpress.php' );
break;

case 3:
require_once( 'site-configs/dots.php' );
break;

case 5:
require_once( 'site-configs/support.php' );
break;
}

add_filter( 'pre_option_template', function () {
return 'dark';
} );
} else {
require_once( 'site-configs/emusic.php' );
}


Global Plugins
add_filter( 'pre_site_option_active_sitewide_plugins', function () {
return array(
'batcache/batcache.php' => 1,
'akismet/akismet.php' => 1,
'avatar/avatar.php' => 1,
'bundle/bundle.php' => 1,
'cloud/cloud.php' => 1,
'download/download.php' => 1,
'emusic-notifications/emusic-notifications.php' => 1,
'emusic-ratings/emusic-ratings.php' => 1,
'emusic-xml-rpc/emusic-xml-rpc.php' => 1,
'johnny-cache/johnny-cache.php' => 1,
'like-buttons/like-buttons.php' => 1,
'members/members.php' => 1,
//'minify/minify.php' => 1,
'apc-admin/apc-admin.php' => 1
//'debug-bar/debug-bar.php' => 1
);
} );


Site Plugins

add_filter( 'pre_option_active_plugins', function () {
return array(
'artist-images/artist-images.php',
'catalog-comments/catalog-comments.php',
'emusic-post-types/emusic-post-types.php',
'discography.php',
'emusic-radio/emusic-radio.php',
'gravityforms/gravityforms.php',
'super-ghetto/super-ghetto.php'
//,'theme-check/theme-check.php'
);
} );


class MyPlugin {
function init() {
add_action( ‘init’, array( $this, ‘register’ ) );
}

function register() {}
}

$my_plugin = new MyPlugin();
$my_plugin->init();

class MyPlugin {
function init() {
add_action( ‘init’, array( ‘MyPlugin’, ‘register’ ) );
}

function register() {}
}

MyPlugin::init();


Share code when possible

class FeaturePack extends eMusicPostTypes implements PostType {
..........
}


MySQL
• Use 5.5, better handlng of weird multibyte
strings
• Use InnoDB, not MyISAM, pretty much in
all cases
• HyperDB handles scaling for you
• Benchmark queries, don’t be afraid to roll
your own SQL, tables, use $wpdb


HyperDB

• Can be empty locally
• Inherits wp-conﬁg DB defaults
• contains functions for replication lag
detection (when used with mk-heartbeat)


Themes

• Theme setup is a class
• Extend before you repeat
• Classes are better than preﬁxing function
names


Theme Conﬁg in functions.php

class Theme_17Dots extends Regionalization {
function __construct() {
global $dots_regions_tax_map, $dots_regions_map;

$this->regions_map = $dots_regions_map;
$this->regions_tax_map = $dots_regions_tax_map;

parent::__construct();
}

function init() {
add_action( 'init', array( $this, 'register' ) );
add_action( 'after_setup_theme', array( $this, 'setup' ) );
add_action( 'add_meta_boxes_post', array( $this, 'boxes' ) );
add_action( 'save_post', array( $this, 'save' ), 10, 2 );
add_filter( 'embed_oembed_html', '_feature_youtube_add_wmode' );
}
. . . . . . .
}


Use base classes

class Regionalization {
var $regions_map;

add_filter( 'manage_posts_columns', array( $this, 'manage_columns' ) );
add_action( 'manage_posts_custom_column', array( $this,
'manage_custom_column' ), 10, 2 );
add_filter( 'posts_clauses', array( $this, 'clauses' ), 10, 2 );
add_filter( 'manage_edit-post_sortable_columns',array( $this, 'sortables' ) );
add_filter( 'pre_get_posts', array( $this, 'pre_posts' ) );
}
. . . . . .
}


pre_get_posts
function regionalize( $query ) {
global $regions_map;
if ( $query->is_main_query() && !is_admin() && ( is_search() || is_archive() || is_home() ) ) {

$types = get_post_types( array( 'publicly_queryable' => true, '_builtin' => false ) );
$tax_region = array(
'taxonomy' => 'region',
'field' => 'term_id',
'terms' => array(
$regions_map[ 'ALL' ],
$regions_map[ THE_REGION ]
),
'operator' => 'IN'
);

if ( is_home() ) {
$query->set( 'posts_per_page', 10 );

} else if ( is_search() && get_option( 'editorial_search_enabled' ) ) {
$ctx = get_query_var( 'search_context' );
$query->set( 's', stripslashes( urldecode( get_query_var( 's' ) ) ) );

if ( in_array( $ctx, array( 'features', 'features-books' ) ) ) {
} else {
}

if ( in_array( $ctx, array( 'books', 'features-books' ) ) ) {
foreach ( $types as $type )
if ( false === strpos( $type, 'book' ) )
unset( $types[$type] );
}

if ( empty( $query->posts ) && $query->is_paged() )
$query->is_paged = false;

} else if ( is_tag() ) {
if ( empty( $query->posts ) && $query->is_paged() )
$query->is_paged = false;
}

//error_log( 'REGIONALIZING' );

if ( !is_post_type_archive() )
$query->set( 'post_type', array_keys( $types ) );

$query->set( 'tax_query', array( $tax_region ) );
}

return $query;
}


Assets

• Use remote storage
• Use a CDN
• Replace hosts using output buffer
• Cloud - pieces of W3 Total Cache


Minify
• JS / CSS concatenation speed up your
front-end loading / perceived loading
• Minify is automagic
• HTML5 CSS properties are automatically
inﬂated
• Admin tool to cache-bust URLs
• Auto-locking while ﬁles are generated

Web Services
• cURL PHP extension
• curl and curl_multi()
• Memcached is essential
• hooks into parse_request to load data
along with WordPress, allows us to cause
WP to 404 and bail early when required
data response fails
• Parallelization with curl_multi()


class eMusicRequest {
var $request;
var $sub_request;

var $path;
var $page;

function load( $page = '' ) {
if ( !empty( $page ) )
$this->page = $page;

$file = $this->path . $this->page . '.php';
if ( file_exists( $file ) )
require_once( $file );
}

function parse() {
$requests = array( $this->request, $this->sub_request );

foreach ( $requests as $request ) {
if ( !empty( $request ) ) {
$keys = array_keys( get_class_vars( get_class( $request ) ) );
foreach ( $keys as $var ) {
if ( !empty( $request->$var ) ) {
$GLOBALS[$var] = $request->$var;
}
}
}
}
}
}


class DarkRequest extends eMusicRequest {
var $genre;
var $post_type;

function init( $request ) {
global $_GENRES;

$vars =& $request->query_vars;

.........
}

}


switch ( get_current_blog_id() ) {
case 1:
$_dark_request = new DarkRequest();
add_action( 'parse_request', array( $_dark_request, 'init' ) );
break;
case 4:
$_my_emusic_request = new MyEMusicRequest();
add_action( 'parse_request', array( $_my_emusic_request,
'init' ) );
break;
}


<?php
class HomeRequest extends RequestMap {
var $recommendations;

parent::__construct();

if ( !get_option( 'recs_enabled' ) )
return;

if ( is_user_logged_in() && 'US' === THE_REGION ) {
$user = wp_get_current_user();
$user_id = isset( $_GET['user_id'] ) ? $_GET['user_id'] : $user->ID;
$params = array( 'userId' => $user_id, 'return' => true );

$this->add( get_user_recommendations( $params ), array( $this, 'parse_recommendations' ) );
$this->send();
}
}

function parse_recommendations( $data ) {
if ( empty( $data['recommendations'] ) )
return;

$data = $data['recommendations'];

if (
!empty( $data ) &&
isset( $data[key($data)]['items'] ) &&
!empty( $data[key($data)]['items'] )
) {
$this->recommendations = array_slice( $data[key($data)]['items'], 0, 18 );
shuffle( $this->recommendations );

foreach ( $this->recommendations as &$rec ) {
$rec = array( 'work_id' => $rec['catalogId'] );
}
}
}
}


<?php
class RequestMap extends API {
private $requests;
private $responses;
private $ttl;
private $useCache = true;

protected $error = false;

public function __construct() {
$this->flush();
}

public function is_error() {
return $this->error;
}

public function flush() {
$this->requests = array();
}

public function getTtl() {
if ( empty( $this->ttl ) ) {
$this->ttl = CACHE::API_CACHE_TTL;
}
return $this->ttl;
}

public function add( $url, $callback, $vars = array() ) {
$params = new stdClass();
$params->url = $url;
$params->callback = $callback;
$params->params = (array) $vars;
$this->requests[] = $params;
}

private function exec( $item, $response ) {
$params = array_merge( array( $response ), $item->params );
call_user_func_array( $item->callback, $params );
}

public function send() {
if ( !empty( $this->requests ) ) {
$this->responses = self::batch( $this->getRequestUrls(), $this->getTtl(), $this->useCache );

if ( is_array( $this->responses ) ) {
foreach ( $this->responses as $i => $response ) {
if ( !empty( $this->requests[$i] ) ) {
$this->exec( $this->requests[$i], self::parse_response( $response ) );
}
}
}
}
}
}

class API {

public static function batch( $urls, $ttl = '', $usecache = true ) {
$response = array();
ob_start();

if ( empty( $ttl ) ) {
$ttl = CACHE::API_CACHE_TTL;
}

if ( is_array( $urls ) ) {
if ( $usecache ) {
foreach ( $urls as $index => $url ) {
$in = Cache::get( Cache::API, $url );

if ( $in ) {
$response[$index] = $in;
unset( $urls[$index] );
}
}
}

$keys = array_keys( $urls );
}
$calls = self::multi_request( $urls );

. . . . . . . .


if ( is_array( $calls ) && count( $calls ) > 0 ) {
$calls = array_combine( $keys, array_values( $calls ) );

foreach ( $calls as $index => $c ) {
if ( $c ) {
$response[$index] = self::parse_response( $c );
if ( isset( $response[$index]['status']['code'] ) && $response[$index]
['status']['code'] < 400 ) {
if ( isset( $response[$index]['results'] ) && !
empty( $response[$index]['results'] ) ) {
Cache::put( Cache::API, $urls[$index], $response[$index], $ttl );
} else {
}
} else if ( !isset( $response[$index]['status']['code'] ) ) {
}
} else {
$response[$index] = null;
}
}
} else if ( !empty( $calls ) && isset( $urls[0] ) ) {
$data = self::parse_response( $calls );
if ( isset( $data['status']['code'] ) && $data['status']['code'] < 400 ) {
if ( isset( $data['results'] ) && !empty( $data['results'] ) ) {
Cache::put( Cache::API, $urls[0], $data, $ttl );
} else {
}
} else if ( !isset( $data['status']['code'] ) ) {
}
$response[] = $data;
}
ob_end_clean();

return $response;

Because we used
classes, everything is
abstracted


Custom Authentication
• WP stores slashed passwords
• wp_authentication arguments are slashed
• Inherit your base system’s rules
• DO NOT sanitize email / password
(WordPress does by default)
• User tables in this case are really a
transient cache


Questions / complaints?


Cloud, Cache, and Configs

More Related Content

What's hot

Similar to Cloud, Cache, and Configs

More from Scott Taylor

Recently uploaded

Cloud, Cache, and Configs