Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Master Data PHP BLT #1

27,527 views

Published on

Big Master Data PHP BLT #1

Published in: Technology
  • Be the first to comment

Big Master Data PHP BLT #1

  1. 1. Big ”Master”Data Masahiro Nagano @kazeburo PHP BLT#1
  2. 2. LT but Me • Masahiro Nagano • @kazeburo • Site Reliability Engineer at Mercari, inc. • ISUCON芸人 • a PHP newbie
  3. 3. What is Master Data?
  4. 4. “Master”Data items id name description shipping_payer shipping_method condition category_id brand_id ... payer id name created updated ... method id name created updated ... id name created updated ... condition categories id name disp_order parent_id created updated ... brands id name disp_order pronunciation created updated ...
  5. 5. Some time Master Data cause N+1 • SELECT with JOIN • Cache in File/Memory/Memcached • Embed in PHP source code
  6. 6. Master Data we thought array( 1 => array( 'id' => '1', 'created' => '2014-07-16 03:30:27', 'updated' => '2014-07-16 03:30:27', 'name' => 'Buyer', 'display_order' => '2', ), 2 => array( 'id' => '2', 'created' => '2014-07-16 03:30:27', 'updated' => '2014-07-16 03:30:27', 'name' => 'Seller', 'display_order' => '1', ), );
  7. 7. Big Master Data $ ls -lh 2.0M item_brands_master.php 509K item_categories_master.php
  8. 8. BMD is Heavy $ cat bench.php <?php $time_start = microtime(true); require_once '/item_brands_master.php'; $time_end = microtime(true); $time = ($time_end - $time_start); echo "#1 in $time secn"; $ php bench.php #1 in 0.071656942367554 sec
  9. 9. “Use OPcache!!” you say
  10. 10. How OPcache works Process shared mem Copy! Destroy!! empty full php php php php php php DB API php php php Request Response
  11. 11. 5-10 msec to load a 2.0MB php from shared memory According to strace/benchmark sorry missed logs
  12. 12. Find way to reduce file size
  13. 13. TSV! split the line and create an array on demand
  14. 14. Example public static $keys = array ( 0 => 'id', 1 => 'created', 2 => 'updated', 3 => 'name_ja', 4 => 'name_en', 5 => 'alternate_string', 6 => 'initials', 7 => 'display_order', 8 => 'is_deleted', ); public static $rows = array ( 1 => '1 2014-07-16 03:30:27 2014-07-16 03:30:27 104+ Octane Boost 104+ Octane Boost 1 50 no', 2 => '2 2014-07-16 03:30:27 2014-07-16 03:30:27 20/10 Products 20/10 Products 2 50 no', 3 => '3 2014-07-16 03:30:27 2014-07-16 03:30:27 303 Products 303 Products 3 50 no', 4 => '4 2014-07-16 03:30:27 2014-07-16 03:30:27 3.1 Phillip Lim 3.1 Phillip Lim 3 50 no',
  15. 15. File Size $ ls -lh 2.0M item_brands_master.php 660K item_brands_master.php require_once 70msec => 7msec (without OPCache)
  16. 16. Benchmarkfunction get($master, $id) { return $master::$rows[$id]; } function get2($master, $id) { return array_combine($master::$keys, explode("t",$master::$rows[$id])); } $time_start = microtime(true); for ( $i=1; $i<3000; $i++ ) { $val = get("ItemBrandsMaster",$i); } $time = 3000/(microtime(true) - $time_start); echo "#1-2 in $time get/secn"; $time_start = microtime(true); for ( $i=1; $i<3000; $i++ ) { $val = get2("ItemBrandsMaster2",$i); } $time_end = microtime(true); $time = 3000/(microtime(true) - $time_start); echo "#2-2 in $time get/secn";
  17. 17. Result $ php bench.php #1-2 in 1374881.1188811 get/sec #2-2 in 355409.33227884 get/sec Seems Big speed down, but keeps fast enough. Does not impact real application
  18. 18. Has been in production
  19. 19. END

×