Big Master Data PHP BLT #1

17,045 views

Published on

Big Master Data PHP BLT #1

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
17,045
On SlideShare
0
From Embeds
0
Number of Embeds
11,279
Actions
Shares
0
Downloads
6
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Big Master Data PHP BLT #1

  1. 1. Big ”Master”Data Masahiro Nagano @kazeburo PHP BLT#1
  2. 2. LT but Me • Masahiro Nagano • @kazeburo • Site Reliability Engineer at Mercari, inc. • ISUCON芸人 • a PHP newbie
  3. 3. What is Master Data?
  4. 4. “Master”Data items id name description shipping_payer shipping_method condition category_id brand_id ... payer id name created updated ... method id name created updated ... id name created updated ... condition categories id name disp_order parent_id created updated ... brands id name disp_order pronunciation created updated ...
  5. 5. Some time Master Data cause N+1 • SELECT with JOIN • Cache in File/Memory/Memcached • Embed in PHP source code
  6. 6. Master Data we thought array( 1 => array( 'id' => '1', 'created' => '2014-07-16 03:30:27', 'updated' => '2014-07-16 03:30:27', 'name' => 'Buyer', 'display_order' => '2', ), 2 => array( 'id' => '2', 'created' => '2014-07-16 03:30:27', 'updated' => '2014-07-16 03:30:27', 'name' => 'Seller', 'display_order' => '1', ), );
  7. 7. Big Master Data $ ls -lh 2.0M item_brands_master.php 509K item_categories_master.php
  8. 8. BMD is Heavy $ cat bench.php <?php $time_start = microtime(true); require_once '/item_brands_master.php'; $time_end = microtime(true); $time = ($time_end - $time_start); echo "#1 in $time secn"; $ php bench.php #1 in 0.071656942367554 sec
  9. 9. “Use OPcache!!” you say
  10. 10. How OPcache works Process shared mem Copy! Destroy!! empty full php php php php php php DB API php php php Request Response
  11. 11. 5-10 msec to load a 2.0MB php from shared memory According to strace/benchmark sorry missed logs
  12. 12. Find way to reduce file size
  13. 13. TSV! split the line and create an array on demand
  14. 14. Example public static $keys = array ( 0 => 'id', 1 => 'created', 2 => 'updated', 3 => 'name_ja', 4 => 'name_en', 5 => 'alternate_string', 6 => 'initials', 7 => 'display_order', 8 => 'is_deleted', ); public static $rows = array ( 1 => '1 2014-07-16 03:30:27 2014-07-16 03:30:27 104+ Octane Boost 104+ Octane Boost 1 50 no', 2 => '2 2014-07-16 03:30:27 2014-07-16 03:30:27 20/10 Products 20/10 Products 2 50 no', 3 => '3 2014-07-16 03:30:27 2014-07-16 03:30:27 303 Products 303 Products 3 50 no', 4 => '4 2014-07-16 03:30:27 2014-07-16 03:30:27 3.1 Phillip Lim 3.1 Phillip Lim 3 50 no',
  15. 15. File Size $ ls -lh 2.0M item_brands_master.php 660K item_brands_master.php require_once 70msec => 7msec (without OPCache)
  16. 16. Benchmarkfunction get($master, $id) { return $master::$rows[$id]; } function get2($master, $id) { return array_combine($master::$keys, explode("t",$master::$rows[$id])); } $time_start = microtime(true); for ( $i=1; $i<3000; $i++ ) { $val = get("ItemBrandsMaster",$i); } $time = 3000/(microtime(true) - $time_start); echo "#1-2 in $time get/secn"; $time_start = microtime(true); for ( $i=1; $i<3000; $i++ ) { $val = get2("ItemBrandsMaster2",$i); } $time_end = microtime(true); $time = 3000/(microtime(true) - $time_start); echo "#2-2 in $time get/secn";
  17. 17. Result $ php bench.php #1-2 in 1374881.1188811 get/sec #2-2 in 355409.33227884 get/sec Seems Big speed down, but keeps fast enough. Does not impact real application
  18. 18. Has been in production
  19. 19. END

×