Successfully reported this slideshow.

Cloud, Cache, and Configs

1

Share

Loading in …3
×
1 of 43
1 of 43

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Cloud, Cache, and Configs

  1. 1. Cloud, Cache, and Configs WordCamp NYC 2012 Saturday, June 9, 12
  2. 2. Scott Taylor Lead PHP Developer, eMusic @wonderboymusic www.scotty-t.com #projectmanagementsoftware Saturday, June 9, 12
  3. 3. eMusic Multisite Regionalized Content Regionalized Caching Shared Content across Sites Tons of Custom Post Types Post Formats Tons of Web Services Saturday, June 9, 12
  4. 4. eMusic Architecture • 12-24 Amazon EC2 instances (CentOS) • 4 MySQL (1 Write, 3 Read) • 4 Memcached (~28GB RAM) • Amazon S3 / Cloudfront CDN • PHP 5.3 / MySQL 5.5 • PECL: APC, Memcached, HTTP • Batcache Saturday, June 9, 12
  5. 5. APC = duh • Essential for PHP • Opcode cache • apc.shm_size = 64M or higher Saturday, June 9, 12
  6. 6. Logs • Never allow any PHP-related notices, errors, warnings, exceptions, etc • Check access logs regularly for 404s, 500s etc • make a constant for toggling debug logging Saturday, June 9, 12
  7. 7. Production Data • Always pull down, never push up • We only push imported legacy content up • output buffering • code filtering • more to come on this.... Saturday, June 9, 12
  8. 8. Output Buffering ob_start( $callback ) ob_start(); echo ‘Daryl’; $daryl = ob_get_clean(); Saturday, June 9, 12
  9. 9. ob_start( function ( $data ) { return str_replace( $urls_array, EMUSIC_CURRENT_HOST, $data ); } ); Saturday, June 9, 12
  10. 10. Configs • Mandatory machine configs • HyperDB config • Overridable sunrise.php Saturday, June 9, 12
  11. 11. Machine Config • DB credentials local to that machine • Amazon S3 bucket • Web Service Endpoints • Memcached Servers if ( file_exists( ‘/wp-config/config.php’ ) ) { require_once( ‘/wp-config/config.php’ ); } else { die( ‘You must have a local config!’ ); } Saturday, June 9, 12
  12. 12. Memcached • Memcached PHP Extension (not Memcache) • Can be used locally (127.0.0.1) • Memcached Redux supports wp_cache_get_/set_multi( ) • Johnny Cache Saturday, June 9, 12
  13. 13. Batcache • Full-page caching • Can be configured • You can partition cache by unique values • Loads before plugins - any code you need has to be duped or added early (sunrise.php) Saturday, June 9, 12
  14. 14. class batcache { // This is the base configuration. You can edit these variables or move them into your wp-config.php file. var $max_age = 300; // Expire batcache items aged this many seconds (zero to disable batcache) var $remote = 0; // Zero disables sending buffers to remote datacenters (req/sec is never sent) var $times = 5; // Only batcache a page after it is accessed this many times... (two or more) var $seconds = 120; // ...in this many seconds (zero to ignore this and use batcache immediately) var $group = 'batcache'; // Name of memcached group. You can simulate a cache flush by changing this. var $unique = array( BATCACHE_REGION, BATCACHE_COUNTRY ); // If you conditionally serve different content, put the variable values here. var $headers = array( 'X-nananana' => 'Batcache' ); . . . . } Saturday, June 9, 12
  15. 15. Sunrise • Used to alter multisite context • sets $current_blog and $current_site • filters all URL functions to resolve all URLs to your current domain • registers custom locations for media • filters Admin URLs Saturday, June 9, 12
  16. 16. switch_to_blog( $blog_id ) • All dynamic functions need to account for this • Shared content needs to resolve proper URLs • Different sites have different media locations Saturday, June 9, 12
  17. 17. Fix switch_to_blog() $current_blog = new stdClass(); $current_blog->site_id = 1; $current_blog->archived = 0; $current_blog->mature = 0; $current_blog->spam = 0; $current_blog->deleted = 0; $current_blog->lang_id = 0; $current_blog->public = 1; $current_blog->registered = '2011-02-20 03:38:22'; $current_blog->last_updated = $_SERVER['REQUEST_TIME']; $current_blog->domain = EMUSIC_CURRENT_HOST; function emusic_switch_to_blog( $blog_id, $prev_blog_id = 0 ) { if ( $blog_id === $prev_blog_id ) return; global $current_blog, $emusic_paths; $current_blog->blog_id = $blog_id; $current_blog->path = $emusic_paths[$blog_id]; } emusic_switch_to_blog( $the_id ); add_action( 'switch_blog', 'emusic_switch_to_blog', 10, 2 ); $blog_id = $the_id; $site_id = 1; $current_site = new stdClass(); $current_site->blog_id = $the_id; $current_site->id = 1; $current_site->domain = EMUSIC_CURRENT_HOST; $current_site->site_name = 'eMusic'; $current_site->path = $the_path; Saturday, June 9, 12
  18. 18. Filter URLs add_filter( 'pre_option_upload_path', function () { $id = get_current_blog_id(); if ( 1 < $id ) return $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$id}/files"; return $_SERVER['DOCUMENT_ROOT'] . '/' . EMUSIC_UPLOADS; } ); add_filter( 'pre_option_upload_url_path', function () { $id = get_current_blog_id(); if ( 1 < $id ) return 'http://' . EMUSIC_CURRENT_HOST . "/blogs/{$id}/files"; return 'http://' . EMUSIC_CURRENT_HOST . '/' . EMUSIC_UPLOADS; } ); add_filter( 'pre_option_siteurl', function () { global $current_blog; $extra = rtrim( $current_blog->path, '/' ); return 'http://' . EMUSIC_CURRENT_HOST . $extra; } ); add_filter( 'pre_option_home', function () { global $current_blog; $extra = rtrim( $current_blog->path, '/' ); return 'http://' . EMUSIC_CURRENT_HOST . $extra; } ); Saturday, June 9, 12
  19. 19. Plugins • Filter active network plugins (don’t rely on database being correct) • Filter each site’s plugins (if you have a manageable number) • Use classes, not a bunch of functions • Extend before you copy / paste Saturday, June 9, 12
  20. 20. Site Configs require_once( 'site-configs/global.php' ); if ( $the_id > 1 ) { define( 'UPLOADBLOGSDIR', 0 ); define( 'UPLOADS', 0 ); define( 'BLOGUPLOADDIR', $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$the_id}/files" ); switch ( $the_id ) { case 2: require_once( 'site-configs/bbpress.php' ); break; case 3: require_once( 'site-configs/dots.php' ); break; case 5: require_once( 'site-configs/support.php' ); break; } add_filter( 'pre_option_template', function () { return 'dark'; } ); } else { require_once( 'site-configs/emusic.php' ); } Saturday, June 9, 12
  21. 21. Global Plugins add_filter( 'pre_site_option_active_sitewide_plugins', function () { return array( 'batcache/batcache.php' => 1, 'akismet/akismet.php' => 1, 'avatar/avatar.php' => 1, 'bundle/bundle.php' => 1, 'cloud/cloud.php' => 1, 'download/download.php' => 1, 'emusic-notifications/emusic-notifications.php' => 1, 'emusic-ratings/emusic-ratings.php' => 1, 'emusic-xml-rpc/emusic-xml-rpc.php' => 1, 'johnny-cache/johnny-cache.php' => 1, 'like-buttons/like-buttons.php' => 1, 'members/members.php' => 1, //'minify/minify.php' => 1, 'apc-admin/apc-admin.php' => 1 //'debug-bar/debug-bar.php' => 1 ); } ); Saturday, June 9, 12
  22. 22. Site Plugins add_filter( 'pre_option_active_plugins', function () { return array( 'artist-images/artist-images.php', 'catalog-comments/catalog-comments.php', 'emusic-post-types/emusic-post-types.php', 'discography.php', 'emusic-radio/emusic-radio.php', 'gravityforms/gravityforms.php', 'super-ghetto/super-ghetto.php' //,'theme-check/theme-check.php' ); } ); Saturday, June 9, 12
  23. 23. class MyPlugin { function init() { add_action( ‘init’, array( $this, ‘register’ ) ); } function register() {} } $my_plugin = new MyPlugin(); $my_plugin->init(); class MyPlugin { function init() { add_action( ‘init’, array( ‘MyPlugin’, ‘register’ ) ); } function register() {} } MyPlugin::init(); Saturday, June 9, 12
  24. 24. Share code when possible class FeaturePack extends eMusicPostTypes implements PostType { .......... } Saturday, June 9, 12
  25. 25. MySQL • Use 5.5, better handlng of weird multibyte strings • Use InnoDB, not MyISAM, pretty much in all cases • HyperDB handles scaling for you • Benchmark queries, don’t be afraid to roll your own SQL, tables, use $wpdb Saturday, June 9, 12
  26. 26. HyperDB • Can be empty locally • Inherits wp-config DB defaults • contains functions for replication lag detection (when used with mk-heartbeat) Saturday, June 9, 12
  27. 27. Themes • Theme setup is a class • Extend before you repeat • Classes are better than prefixing function names Saturday, June 9, 12
  28. 28. Theme Config in functions.php class Theme_17Dots extends Regionalization { function __construct() { global $dots_regions_tax_map, $dots_regions_map; $this->regions_map = $dots_regions_map; $this->regions_tax_map = $dots_regions_tax_map; parent::__construct(); } function init() { add_action( 'init', array( $this, 'register' ) ); add_action( 'after_setup_theme', array( $this, 'setup' ) ); add_action( 'add_meta_boxes_post', array( $this, 'boxes' ) ); add_action( 'save_post', array( $this, 'save' ), 10, 2 ); add_filter( 'embed_oembed_html', '_feature_youtube_add_wmode' ); } . . . . . . . } Saturday, June 9, 12
  29. 29. Use base classes class Regionalization { var $regions_map; function __construct() { add_filter( 'manage_posts_columns', array( $this, 'manage_columns' ) ); add_action( 'manage_posts_custom_column', array( $this, 'manage_custom_column' ), 10, 2 ); add_filter( 'posts_clauses', array( $this, 'clauses' ), 10, 2 ); add_filter( 'manage_edit-post_sortable_columns',array( $this, 'sortables' ) ); add_filter( 'pre_get_posts', array( $this, 'pre_posts' ) ); } . . . . . . } Saturday, June 9, 12
  30. 30. pre_get_posts function regionalize( $query ) { global $regions_map; if ( $query->is_main_query() && !is_admin() && ( is_search() || is_archive() || is_home() ) ) { $types = get_post_types( array( 'publicly_queryable' => true, '_builtin' => false ) ); $tax_region = array( 'taxonomy' => 'region', 'field' => 'term_id', 'terms' => array( $regions_map[ 'ALL' ], $regions_map[ THE_REGION ] ), 'operator' => 'IN' ); if ( is_home() ) { $query->set( 'posts_per_page', 10 ); } else if ( is_search() && get_option( 'editorial_search_enabled' ) ) { $ctx = get_query_var( 'search_context' ); $query->set( 's', stripslashes( urldecode( get_query_var( 's' ) ) ) ); if ( in_array( $ctx, array( 'features', 'features-books' ) ) ) { $query->set( 'posts_per_page', 48 ); } else { $query->set( 'posts_per_page', 12 ); } if ( in_array( $ctx, array( 'books', 'features-books' ) ) ) { foreach ( $types as $type ) if ( false === strpos( $type, 'book' ) ) unset( $types[$type] ); } if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; } else if ( is_tag() ) { if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; } //error_log( 'REGIONALIZING' ); if ( !is_post_type_archive() ) $query->set( 'post_type', array_keys( $types ) ); $query->set( 'tax_query', array( $tax_region ) ); } return $query; } Saturday, June 9, 12
  31. 31. Assets • Use remote storage • Use a CDN • Replace hosts using output buffer • Cloud - pieces of W3 Total Cache Saturday, June 9, 12
  32. 32. Minify • JS / CSS concatenation speed up your front-end loading / perceived loading • Minify is automagic • HTML5 CSS properties are automatically inflated • Admin tool to cache-bust URLs • Auto-locking while files are generated Saturday, June 9, 12
  33. 33. Web Services • cURL PHP extension • curl and curl_multi() • Memcached is essential • hooks into parse_request to load data along with WordPress, allows us to cause WP to 404 and bail early when required data response fails • Parallelization with curl_multi() Saturday, June 9, 12
  34. 34. class eMusicRequest { var $request; var $sub_request; var $path; var $page; function load( $page = '' ) { if ( !empty( $page ) ) $this->page = $page; $file = $this->path . $this->page . '.php'; if ( file_exists( $file ) ) require_once( $file ); } function parse() { $requests = array( $this->request, $this->sub_request ); foreach ( $requests as $request ) { if ( !empty( $request ) ) { $keys = array_keys( get_class_vars( get_class( $request ) ) ); foreach ( $keys as $var ) { if ( !empty( $request->$var ) ) { $GLOBALS[$var] = $request->$var; } } } } } } Saturday, June 9, 12
  35. 35. class DarkRequest extends eMusicRequest { var $genre; var $post_type; function init( $request ) { global $_GENRES; $vars =& $request->query_vars; ......... } } Saturday, June 9, 12
  36. 36. switch ( get_current_blog_id() ) { case 1: $_dark_request = new DarkRequest(); add_action( 'parse_request', array( $_dark_request, 'init' ) ); break; case 4: $_my_emusic_request = new MyEMusicRequest(); add_action( 'parse_request', array( $_my_emusic_request, 'init' ) ); break; } Saturday, June 9, 12
  37. 37. <?php class HomeRequest extends RequestMap { var $recommendations; function __construct() { parent::__construct(); if ( !get_option( 'recs_enabled' ) ) return; if ( is_user_logged_in() && 'US' === THE_REGION ) { $user = wp_get_current_user(); $user_id = isset( $_GET['user_id'] ) ? $_GET['user_id'] : $user->ID; $params = array( 'userId' => $user_id, 'return' => true ); $this->add( get_user_recommendations( $params ), array( $this, 'parse_recommendations' ) ); $this->send(); } } function parse_recommendations( $data ) { if ( empty( $data['recommendations'] ) ) return; $data = $data['recommendations']; if ( !empty( $data ) && isset( $data[key($data)]['items'] ) && !empty( $data[key($data)]['items'] ) ) { $this->recommendations = array_slice( $data[key($data)]['items'], 0, 18 ); shuffle( $this->recommendations ); foreach ( $this->recommendations as &$rec ) { $rec = array( 'work_id' => $rec['catalogId'] ); } } } } Saturday, June 9, 12
  38. 38. <?php class RequestMap extends API { private $requests; private $responses; private $ttl; private $useCache = true; protected $error = false; public function __construct() { $this->flush(); } public function is_error() { return $this->error; } public function flush() { $this->requests = array(); } public function getTtl() { if ( empty( $this->ttl ) ) { $this->ttl = CACHE::API_CACHE_TTL; } return $this->ttl; } public function add( $url, $callback, $vars = array() ) { $params = new stdClass(); $params->url = $url; $params->callback = $callback; $params->params = (array) $vars; $this->requests[] = $params; } private function exec( $item, $response ) { $params = array_merge( array( $response ), $item->params ); call_user_func_array( $item->callback, $params ); } public function send() { if ( !empty( $this->requests ) ) { $this->responses = self::batch( $this->getRequestUrls(), $this->getTtl(), $this->useCache ); if ( is_array( $this->responses ) ) { foreach ( $this->responses as $i => $response ) { if ( !empty( $this->requests[$i] ) ) { $this->exec( $this->requests[$i], self::parse_response( $response ) ); } } } } } } Saturday, June 9, 12
  39. 39. class API { public static function batch( $urls, $ttl = '', $usecache = true ) { $response = array(); ob_start(); if ( empty( $ttl ) ) { $ttl = CACHE::API_CACHE_TTL; } if ( is_array( $urls ) ) { if ( $usecache ) { foreach ( $urls as $index => $url ) { $in = Cache::get( Cache::API, $url ); if ( $in ) { $response[$index] = $in; unset( $urls[$index] ); } } } $keys = array_keys( $urls ); } $calls = self::multi_request( $urls ); . . . . . . . . Saturday, June 9, 12
  40. 40. if ( is_array( $calls ) && count( $calls ) > 0 ) { $calls = array_combine( $keys, array_values( $calls ) ); foreach ( $calls as $index => $c ) { if ( $c ) { $response[$index] = self::parse_response( $c ); if ( isset( $response[$index]['status']['code'] ) && $response[$index] ['status']['code'] < 400 ) { if ( isset( $response[$index]['results'] ) && ! empty( $response[$index]['results'] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } else { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else if ( !isset( $response[$index]['status']['code'] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else { $response[$index] = null; } } } else if ( !empty( $calls ) && isset( $urls[0] ) ) { $data = self::parse_response( $calls ); if ( isset( $data['status']['code'] ) && $data['status']['code'] < 400 ) { if ( isset( $data['results'] ) && !empty( $data['results'] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } else { Cache::put( Cache::API, $urls[0], $data, $ttl ); } } else if ( !isset( $data['status']['code'] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } $response[] = $data; } ob_end_clean(); return $response; Saturday, June 9, 12
  41. 41. Because we used classes, everything is abstracted Saturday, June 9, 12
  42. 42. Custom Authentication • WP stores slashed passwords • wp_authentication arguments are slashed • Inherit your base system’s rules • DO NOT sanitize email / password (WordPress does by default) • User tables in this case are really a transient cache Saturday, June 9, 12
  43. 43. Questions / complaints? Saturday, June 9, 12

×