Cloud, Cache, and Configs                       WordCamp NYC 2012Saturday, June 9, 12
Scott Taylor                       Lead PHP Developer, eMusic                           @wonderboymusic                   ...
eMusic                Multisite          Regionalized Content          Regionalized Caching       Shared Content across Si...
eMusic Architecture                       • 12-24 Amazon EC2 instances (CentOS)                       • 4 MySQL (1 Write, ...
APC = duh                       • Essential for PHP                       • Opcode cache                       • apc.shm_s...
Logs                       • Never allow any PHP-related notices,                         errors, warnings, exceptions, et...
Production Data                       • Always pull down, never push up                       • We only push imported lega...
Output Buffering                        ob_start( $callback )                             ob_start();                     ...
ob_start( function ( $data ) {                          return str_replace(                             $urls_array,      ...
Configs                       • Mandatory machine configs                       • HyperDB config                       • Over...
Machine Config                       • DB credentials local to that machine                       • Amazon S3 bucket       ...
Memcached                       • Memcached PHP Extension (not Memcache)                       • Can be used locally (127....
Batcache                       • Full-page caching                       • Can be configured                       • You ca...
class batcache {                     // This is the base configuration. You can edit these variables or move              ...
Sunrise                       • Used to alter multisite context                       • sets $current_blog and $current_si...
switch_to_blog( $blog_id )                       • All dynamic functions need to account for                         this ...
Fix switch_to_blog()                       $current_blog = new stdClass();                       $current_blog->site_id = ...
Filter URLs                       add_filter( pre_option_upload_path, function () {                           $id = get_cu...
Plugins                       • Filter active network plugins (don’t rely on                         database being correc...
Site Configs  require_once( site-configs/global.php );  if ( $the_id > 1 ) {      define( UPLOADBLOGSDIR, 0 );      define(...
Global Plugins  add_filter( pre_site_option_active_sitewide_plugins, function () {       return array(           batcache/...
Site Plugins                   add_filter( pre_option_active_plugins, function () {                        return array(  ...
class MyPlugin {        function init() {             add_action( ‘init’, array( $this, ‘register’ ) );        }          ...
Share code when possible        class FeaturePack extends eMusicPostTypes implements PostType {              ..........   ...
MySQL                       • Use 5.5, better handlng of weird multibyte                         strings                  ...
HyperDB                       • Can be empty locally                       • Inherits wp-config DB defaults                ...
Themes                       • Theme setup is a class                       • Extend before you repeat                    ...
Theme Config in functions.php    class Theme_17Dots extends Regionalization {        function __construct() {            gl...
Use base classes   class Regionalization {       var $regions_map;       function __construct() {           add_filter( ma...
pre_get_posts                       function regionalize( $query ) {                              global $regions_map;    ...
Assets                       • Use remote storage                       • Use a CDN                       • Replace hosts ...
Minify                       • JS / CSS concatenation speed up your                         front-end loading / perceived ...
Web Services                       • cURL PHP extension                       • curl and curl_multi()                     ...
class eMusicRequest {                          var $request;                          var $sub_request;                   ...
class DarkRequest extends eMusicRequest {                          var $genre;                          var $post_type;   ...
switch ( get_current_blog_id() ) {       case 1:           $_dark_request = new DarkRequest();           add_action( parse...
<?php             class HomeRequest extends RequestMap {                 var $recommendations;                   function ...
<?php        class RequestMap extends API {            private $requests;            private $responses;            privat...
class API {       public static function batch( $urls, $ttl = , $usecache = true ) {           $response = array();       ...
if ( is_array( $calls ) && count( $calls ) > 0 ) {                  $calls = array_combine( $keys, array_values( $calls ) ...
Because we used                       classes, everything is                            abstractedSaturday, June 9, 12
Custom Authentication                       • WP stores slashed passwords                       • wp_authentication argume...
Questions / complaints?Saturday, June 9, 12
Upcoming SlideShare
Loading in …5
×

Cloud, Cache, and Configs

18,718 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
18,718
On SlideShare
0
From Embeds
0
Number of Embeds
12,834
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Cloud, Cache, and Configs

  1. 1. Cloud, Cache, and Configs WordCamp NYC 2012Saturday, June 9, 12
  2. 2. Scott Taylor Lead PHP Developer, eMusic @wonderboymusic www.scotty-t.com #projectmanagementsoftwareSaturday, June 9, 12
  3. 3. eMusic Multisite Regionalized Content Regionalized Caching Shared Content across Sites Tons of Custom Post Types Post Formats Tons of Web ServicesSaturday, June 9, 12
  4. 4. eMusic Architecture • 12-24 Amazon EC2 instances (CentOS) • 4 MySQL (1 Write, 3 Read) • 4 Memcached (~28GB RAM) • Amazon S3 / Cloudfront CDN • PHP 5.3 / MySQL 5.5 • PECL: APC, Memcached, HTTP • BatcacheSaturday, June 9, 12
  5. 5. APC = duh • Essential for PHP • Opcode cache • apc.shm_size = 64M or higherSaturday, June 9, 12
  6. 6. Logs • Never allow any PHP-related notices, errors, warnings, exceptions, etc • Check access logs regularly for 404s, 500s etc • make a constant for toggling debug loggingSaturday, June 9, 12
  7. 7. Production Data • Always pull down, never push up • We only push imported legacy content up • output buffering • code filtering • more to come on this....Saturday, June 9, 12
  8. 8. Output Buffering ob_start( $callback ) ob_start(); echo ‘Daryl’; $daryl = ob_get_clean();Saturday, June 9, 12
  9. 9. ob_start( function ( $data ) { return str_replace( $urls_array, EMUSIC_CURRENT_HOST, $data ); } );Saturday, June 9, 12
  10. 10. Configs • Mandatory machine configs • HyperDB config • Overridable sunrise.phpSaturday, June 9, 12
  11. 11. Machine Config • DB credentials local to that machine • Amazon S3 bucket • Web Service Endpoints • Memcached Servers if ( file_exists( ‘/wp-config/config.php’ ) ) { require_once( ‘/wp-config/config.php’ ); } else { die( ‘You must have a local config!’ ); }Saturday, June 9, 12
  12. 12. Memcached • Memcached PHP Extension (not Memcache) • Can be used locally (127.0.0.1) • Memcached Redux supports wp_cache_get_/set_multi( ) • Johnny CacheSaturday, June 9, 12
  13. 13. Batcache • Full-page caching • Can be configured • You can partition cache by unique values • Loads before plugins - any code you need has to be duped or added early (sunrise.php)Saturday, June 9, 12
  14. 14. class batcache { // This is the base configuration. You can edit these variables or move them into your wp-config.php file. var $max_age = 300; // Expire batcache items aged this many seconds (zero to disable batcache) var $remote = 0; // Zero disables sending buffers to remote datacenters (req/sec is never sent) var $times = 5; // Only batcache a page after it is accessed this many times... (two or more) var $seconds = 120; // ...in this many seconds (zero to ignore this and use batcache immediately) var $group = batcache; // Name of memcached group. You can simulate a cache flush by changing this. var $unique = array( BATCACHE_REGION, BATCACHE_COUNTRY ); // If you conditionally serve different content, put the variable values here. var $headers = array( X-nananana => Batcache ); . . . . }Saturday, June 9, 12
  15. 15. Sunrise • Used to alter multisite context • sets $current_blog and $current_site • filters all URL functions to resolve all URLs to your current domain • registers custom locations for media • filters Admin URLsSaturday, June 9, 12
  16. 16. switch_to_blog( $blog_id ) • All dynamic functions need to account for this • Shared content needs to resolve proper URLs • Different sites have different media locationsSaturday, June 9, 12
  17. 17. Fix switch_to_blog() $current_blog = new stdClass(); $current_blog->site_id = 1; $current_blog->archived = 0; $current_blog->mature = 0; $current_blog->spam = 0; $current_blog->deleted = 0; $current_blog->lang_id = 0; $current_blog->public = 1; $current_blog->registered = 2011-02-20 03:38:22; $current_blog->last_updated = $_SERVER[REQUEST_TIME]; $current_blog->domain = EMUSIC_CURRENT_HOST; function emusic_switch_to_blog( $blog_id, $prev_blog_id = 0 ) { if ( $blog_id === $prev_blog_id ) return; global $current_blog, $emusic_paths; $current_blog->blog_id = $blog_id; $current_blog->path = $emusic_paths[$blog_id]; } emusic_switch_to_blog( $the_id ); add_action( switch_blog, emusic_switch_to_blog, 10, 2 ); $blog_id = $the_id; $site_id = 1; $current_site = new stdClass(); $current_site->blog_id = $the_id; $current_site->id = 1; $current_site->domain = EMUSIC_CURRENT_HOST; $current_site->site_name = eMusic; $current_site->path = $the_path;Saturday, June 9, 12
  18. 18. Filter URLs add_filter( pre_option_upload_path, function () { $id = get_current_blog_id(); if ( 1 < $id ) return $_SERVER[DOCUMENT_ROOT] . "/blogs/{$id}/files"; return $_SERVER[DOCUMENT_ROOT] . / . EMUSIC_UPLOADS; } ); add_filter( pre_option_upload_url_path, function () { $id = get_current_blog_id(); if ( 1 < $id ) return http:// . EMUSIC_CURRENT_HOST . "/blogs/{$id}/files"; return http:// . EMUSIC_CURRENT_HOST . / . EMUSIC_UPLOADS; } ); add_filter( pre_option_siteurl, function () { global $current_blog; $extra = rtrim( $current_blog->path, / ); return http:// . EMUSIC_CURRENT_HOST . $extra; } ); add_filter( pre_option_home, function () { global $current_blog; $extra = rtrim( $current_blog->path, / ); return http:// . EMUSIC_CURRENT_HOST . $extra; } );Saturday, June 9, 12
  19. 19. Plugins • Filter active network plugins (don’t rely on database being correct) • Filter each site’s plugins (if you have a manageable number) • Use classes, not a bunch of functions • Extend before you copy / pasteSaturday, June 9, 12
  20. 20. Site Configs require_once( site-configs/global.php ); if ( $the_id > 1 ) { define( UPLOADBLOGSDIR, 0 ); define( UPLOADS, 0 ); define( BLOGUPLOADDIR, $_SERVER[DOCUMENT_ROOT] . "/blogs/{$the_id}/files" ); switch ( $the_id ) { case 2: require_once( site-configs/bbpress.php ); break; case 3: require_once( site-configs/dots.php ); break; case 5: require_once( site-configs/support.php ); break; } add_filter( pre_option_template, function () { return dark; } ); } else { require_once( site-configs/emusic.php ); }Saturday, June 9, 12
  21. 21. Global Plugins add_filter( pre_site_option_active_sitewide_plugins, function () { return array( batcache/batcache.php => 1, akismet/akismet.php => 1, avatar/avatar.php => 1, bundle/bundle.php => 1, cloud/cloud.php => 1, download/download.php => 1, emusic-notifications/emusic-notifications.php => 1, emusic-ratings/emusic-ratings.php => 1, emusic-xml-rpc/emusic-xml-rpc.php => 1, johnny-cache/johnny-cache.php => 1, like-buttons/like-buttons.php => 1, members/members.php => 1, //minify/minify.php => 1, apc-admin/apc-admin.php => 1 //debug-bar/debug-bar.php => 1 ); } );Saturday, June 9, 12
  22. 22. Site Plugins add_filter( pre_option_active_plugins, function () { return array( artist-images/artist-images.php, catalog-comments/catalog-comments.php, emusic-post-types/emusic-post-types.php, discography.php, emusic-radio/emusic-radio.php, gravityforms/gravityforms.php, super-ghetto/super-ghetto.php //,theme-check/theme-check.php ); } );Saturday, June 9, 12
  23. 23. class MyPlugin { function init() { add_action( ‘init’, array( $this, ‘register’ ) ); } function register() {} } $my_plugin = new MyPlugin(); $my_plugin->init(); class MyPlugin { function init() { add_action( ‘init’, array( ‘MyPlugin’, ‘register’ ) ); } function register() {} } MyPlugin::init();Saturday, June 9, 12
  24. 24. Share code when possible class FeaturePack extends eMusicPostTypes implements PostType { .......... }Saturday, June 9, 12
  25. 25. MySQL • Use 5.5, better handlng of weird multibyte strings • Use InnoDB, not MyISAM, pretty much in all cases • HyperDB handles scaling for you • Benchmark queries, don’t be afraid to roll your own SQL, tables, use $wpdbSaturday, June 9, 12
  26. 26. HyperDB • Can be empty locally • Inherits wp-config DB defaults • contains functions for replication lag detection (when used with mk-heartbeat)Saturday, June 9, 12
  27. 27. Themes • Theme setup is a class • Extend before you repeat • Classes are better than prefixing function namesSaturday, June 9, 12
  28. 28. Theme Config in functions.php class Theme_17Dots extends Regionalization { function __construct() { global $dots_regions_tax_map, $dots_regions_map; $this->regions_map = $dots_regions_map; $this->regions_tax_map = $dots_regions_tax_map; parent::__construct(); } function init() { add_action( init, array( $this, register ) ); add_action( after_setup_theme, array( $this, setup ) ); add_action( add_meta_boxes_post, array( $this, boxes ) ); add_action( save_post, array( $this, save ), 10, 2 ); add_filter( embed_oembed_html, _feature_youtube_add_wmode ); } . . . . . . . }Saturday, June 9, 12
  29. 29. Use base classes class Regionalization { var $regions_map; function __construct() { add_filter( manage_posts_columns, array( $this, manage_columns ) ); add_action( manage_posts_custom_column, array( $this, manage_custom_column ), 10, 2 ); add_filter( posts_clauses, array( $this, clauses ), 10, 2 ); add_filter( manage_edit-post_sortable_columns,array( $this, sortables ) ); add_filter( pre_get_posts, array( $this, pre_posts ) ); } . . . . . . }Saturday, June 9, 12
  30. 30. pre_get_posts function regionalize( $query ) { global $regions_map; if ( $query->is_main_query() && !is_admin() && ( is_search() || is_archive() || is_home() ) ) { $types = get_post_types( array( publicly_queryable => true, _builtin => false ) ); $tax_region = array( taxonomy => region, field => term_id, terms => array( $regions_map[ ALL ], $regions_map[ THE_REGION ] ), operator => IN ); if ( is_home() ) { $query->set( posts_per_page, 10 ); } else if ( is_search() && get_option( editorial_search_enabled ) ) { $ctx = get_query_var( search_context ); $query->set( s, stripslashes( urldecode( get_query_var( s ) ) ) ); if ( in_array( $ctx, array( features, features-books ) ) ) { $query->set( posts_per_page, 48 ); } else { $query->set( posts_per_page, 12 ); } if ( in_array( $ctx, array( books, features-books ) ) ) { foreach ( $types as $type ) if ( false === strpos( $type, book ) ) unset( $types[$type] ); } if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; } else if ( is_tag() ) { if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; } //error_log( REGIONALIZING ); if ( !is_post_type_archive() ) $query->set( post_type, array_keys( $types ) ); $query->set( tax_query, array( $tax_region ) ); } return $query; }Saturday, June 9, 12
  31. 31. Assets • Use remote storage • Use a CDN • Replace hosts using output buffer • Cloud - pieces of W3 Total CacheSaturday, June 9, 12
  32. 32. Minify • JS / CSS concatenation speed up your front-end loading / perceived loading • Minify is automagic • HTML5 CSS properties are automatically inflated • Admin tool to cache-bust URLs • Auto-locking while files are generatedSaturday, June 9, 12
  33. 33. Web Services • cURL PHP extension • curl and curl_multi() • Memcached is essential • hooks into parse_request to load data along with WordPress, allows us to cause WP to 404 and bail early when required data response fails • Parallelization with curl_multi()Saturday, June 9, 12
  34. 34. class eMusicRequest { var $request; var $sub_request; var $path; var $page; function load( $page = ) { if ( !empty( $page ) ) $this->page = $page; $file = $this->path . $this->page . .php; if ( file_exists( $file ) ) require_once( $file ); } function parse() { $requests = array( $this->request, $this->sub_request ); foreach ( $requests as $request ) { if ( !empty( $request ) ) { $keys = array_keys( get_class_vars( get_class( $request ) ) ); foreach ( $keys as $var ) { if ( !empty( $request->$var ) ) { $GLOBALS[$var] = $request->$var; } } } } } }Saturday, June 9, 12
  35. 35. class DarkRequest extends eMusicRequest { var $genre; var $post_type; function init( $request ) { global $_GENRES; $vars =& $request->query_vars; ......... } }Saturday, June 9, 12
  36. 36. switch ( get_current_blog_id() ) { case 1: $_dark_request = new DarkRequest(); add_action( parse_request, array( $_dark_request, init ) ); break; case 4: $_my_emusic_request = new MyEMusicRequest(); add_action( parse_request, array( $_my_emusic_request, init ) ); break; }Saturday, June 9, 12
  37. 37. <?php class HomeRequest extends RequestMap { var $recommendations; function __construct() { parent::__construct(); if ( !get_option( recs_enabled ) ) return; if ( is_user_logged_in() && US === THE_REGION ) { $user = wp_get_current_user(); $user_id = isset( $_GET[user_id] ) ? $_GET[user_id] : $user->ID; $params = array( userId => $user_id, return => true ); $this->add( get_user_recommendations( $params ), array( $this, parse_recommendations ) ); $this->send(); } } function parse_recommendations( $data ) { if ( empty( $data[recommendations] ) ) return; $data = $data[recommendations]; if ( !empty( $data ) && isset( $data[key($data)][items] ) && !empty( $data[key($data)][items] ) ) { $this->recommendations = array_slice( $data[key($data)][items], 0, 18 ); shuffle( $this->recommendations ); foreach ( $this->recommendations as &$rec ) { $rec = array( work_id => $rec[catalogId] ); } } } }Saturday, June 9, 12
  38. 38. <?php class RequestMap extends API { private $requests; private $responses; private $ttl; private $useCache = true; protected $error = false; public function __construct() { $this->flush(); } public function is_error() { return $this->error; } public function flush() { $this->requests = array(); } public function getTtl() { if ( empty( $this->ttl ) ) { $this->ttl = CACHE::API_CACHE_TTL; } return $this->ttl; } public function add( $url, $callback, $vars = array() ) { $params = new stdClass(); $params->url = $url; $params->callback = $callback; $params->params = (array) $vars; $this->requests[] = $params; } private function exec( $item, $response ) { $params = array_merge( array( $response ), $item->params ); call_user_func_array( $item->callback, $params ); } public function send() { if ( !empty( $this->requests ) ) { $this->responses = self::batch( $this->getRequestUrls(), $this->getTtl(), $this->useCache ); if ( is_array( $this->responses ) ) { foreach ( $this->responses as $i => $response ) { if ( !empty( $this->requests[$i] ) ) { $this->exec( $this->requests[$i], self::parse_response( $response ) ); } } } } } }Saturday, June 9, 12
  39. 39. class API { public static function batch( $urls, $ttl = , $usecache = true ) { $response = array(); ob_start(); if ( empty( $ttl ) ) { $ttl = CACHE::API_CACHE_TTL; } if ( is_array( $urls ) ) { if ( $usecache ) { foreach ( $urls as $index => $url ) { $in = Cache::get( Cache::API, $url ); if ( $in ) { $response[$index] = $in; unset( $urls[$index] ); } } } $keys = array_keys( $urls ); } $calls = self::multi_request( $urls ); . . . . . . . .Saturday, June 9, 12
  40. 40. if ( is_array( $calls ) && count( $calls ) > 0 ) { $calls = array_combine( $keys, array_values( $calls ) ); foreach ( $calls as $index => $c ) { if ( $c ) { $response[$index] = self::parse_response( $c ); if ( isset( $response[$index][status][code] ) && $response[$index][status][code] < 400 ) { if ( isset( $response[$index][results] ) && !empty( $response[$index][results] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } else { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else if ( !isset( $response[$index][status][code] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else { $response[$index] = null; } } } else if ( !empty( $calls ) && isset( $urls[0] ) ) { $data = self::parse_response( $calls ); if ( isset( $data[status][code] ) && $data[status][code] < 400 ) { if ( isset( $data[results] ) && !empty( $data[results] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } else { Cache::put( Cache::API, $urls[0], $data, $ttl ); } } else if ( !isset( $data[status][code] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } $response[] = $data; } ob_end_clean(); return $response;Saturday, June 9, 12
  41. 41. Because we used classes, everything is abstractedSaturday, June 9, 12
  42. 42. Custom Authentication • WP stores slashed passwords • wp_authentication arguments are slashed • Inherit your base system’s rules • DO NOT sanitize email / password (WordPress does by default) • User tables in this case are really a transient cacheSaturday, June 9, 12
  43. 43. Questions / complaints?Saturday, June 9, 12

×