Your SlideShare is downloading. ×
0
Concurrent PHP
in the Etsy API
Matthew Graham
@lapsu
@EtsyAPI Lead
#phpday
2014 April
@lapsu@EtsyAPI
@lapsu@EtsyAPI
$1.3 Billion
Things That Matter
@lapsu@EtsyAPI
@lapsu
Etsy's PHP
www api admin
queues cron
@EtsyAPI
@lapsu
~200 engineers
30+ deploys / day
@EtsyAPI
Also, Rasmus
@lapsu@EtsyAPI
Spoilers
@lapsu
Mobile Clients Are Special
1 Thread != No Concurrency
@EtsyAPI
@lapsu
<motivation>
@EtsyAPI
Premise:
@lapsu
The Future is Mobile
@EtsyAPI
@lapsu
Past
The Future is Mobile
November 2013
@EtsyAPI
@lapsu@EtsyAPI
@EtsyAPI
Mobile Networks Suck
@lapsu
<
@EtsyAPI
Not Mobile
@lapsu
www.etsy.com/shop/AVintageWanderer
@EtsyAPI
Network Performance
@lapsu
3G < 4G
@EtsyAPI
Network Coverage
@lapsu
3G > 4G
@EtsyAPI
Mobile Requests
@lapsu
More != Better
@lapsu@EtsyAPI
@lapsu
1000ms Time To Glass
@EtsyAPI
@lapsu
1000ms
- 900ms
-------------
100ms
Network/Client
------------------------
Server
@EtsyAPI
@lapsu
100ms = Bespoke
+ Concurrent
@EtsyAPI
@lapsu
Single Threads
@EtsyAPI
Concurrency
@lapsu@EtsyAPI
Main “Thread”
Child “Thread” Child “Thread”
@lapsu
</motivation><interface>
@EtsyAPI
Paul goes to Netflix
@lapsu@EtsyAPI
1 View : 1 Bespoke
@lapsu@EtsyAPI
ClientView
Bespoke
View View
Bespoke Bespoke
API
Multiple Clients
@lapsu@EtsyAPI
ClientsView
Bespoke
View View
Bespoke Bespoke
API
Bespoke : Components
@lapsu@EtsyAPI
Bespoke Bespoke Bespoke
API
Item User Shop Favs Tx
Components as REST
@lapsu@EtsyAPI
Bespoke Bespoke
API
Item User Shop Favs Tx
@lapsu
?includes=User
@EtsyAPI
Android User View
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id...
Concurrent Client
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id...
Making Requests
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id);...
@lapsu@EtsyAPI
shop
favs
t0 t1
Inputs
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id);
$items =...
Future Parameters
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id...
@lapsu@EtsyAPI
shop
favs
items
t0 t1 t2 t3 t4
~6 Lines
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id);
$items...
@lapsu
</interface><performance>
@EtsyAPI
@lapsu
Web Pages Are Clients Too
@EtsyAPI
Web First
@lapsu@EtsyAPI
API First
@lapsu@EtsyAPI
Android User View
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->user_id...
Desktop Web User View
@lapsu
function handle($cli, $inp) {
$shop = $cli->shop($inp->user_id);
$favs = $cli->favs($inp->use...
@lapsu
Activity Feed
@EtsyAPI
@lapsu
Activity Feed
Page
@EtsyAPI
Home
Page
30s of HTTP Time
@lapsu@EtsyAPI
30s of HTTP Time
@lapsu
880ms Real Time
@EtsyAPI
Components
@lapsu@EtsyAPI
Bespoke Bespoke Bespoke
API
Item User Shop Favs Tx
Components Cache
@lapsu@EtsyAPI
Bespoke Bespoke Bespoke
Cache
API
Item User Shop Favs Tx
Local Call
@lapsu@EtsyAPI
Me
Long Distance Call
@lapsu@EtsyAPI
Atlantic
Ocean User
Long Distance Call
@lapsu@EtsyAPI
Atlantic
Ocean
API
Templates
User
@lapsu
</performance><internals>
@EtsyAPI
@lapsu
curl_multi_*
@EtsyAPI
@lapsu@EtsyAPI
curl?
General Sequence
@lapsu@EtsyAPI
curl_multi_init();
curl_multi_add();
curl_multi_exec();
while (!$done) {
curl_multi_select...
curl_multi_init
@lapsu@EtsyAPI
$mh = curl_multi_init();
curl_multi_add_handle
@lapsu@EtsyAPI
$mh = curl_multi_init();
$ch = curl_init($url);
curl_setopt_array($ch, $options);
cur...
@lapsu@EtsyAPI
multi handle
handle handle handle
curl_multi_exec
@lapsu@EtsyAPI
do {
$code = curl_multi_exec($mh, $r);
} while ($code == CURLM_CALL_MULTI_PERFORM);
curl_multi_select
@lapsu@EtsyAPI
$cnt = curl_multi_select($mh, $tmout);
curl_multi_info_read
@lapsu@EtsyAPI
$info = curl_multi_info_read($mh);
$ch = $info['handle'];
$content = curl_multi_getcon...
@lapsu@EtsyAPI
localhost: Expected
@lapsu@EtsyAPI
R1
R2
R3
t0
Network: Actual
@lapsu@EtsyAPI
R2
R3
t0 t1
R1
curl Protocols
@lapsu@EtsyAPI
HTTP
LDAP
Gopher POP3
IMAP
TELNET TFTP
curl_get_multi_handle_state
@lapsu@EtsyAPI
$state = CURLM_STATE_FIRST;
curl_multi_get_handle_state(
$mh, $ch, $state);
$se...
Expected, Actual
@lapsu@EtsyAPI
R1
R2
R3
t0
Revised Sequence
@lapsu@EtsyAPI
curl_multi_init();
curl_multi_add();
while (!$sent) {
curl_multi_exec();
curl_multi_get_ha...
Still Headed Upstream
@lapsu@EtsyAPI
Patch URL
@lapsu
bit.ly/etsy_curl_multi_patch
@EtsyAPI
Recursion?
@lapsu@EtsyAPI
Recursion?
@lapsu
No.
@EtsyAPI
Recursion?
@lapsu
No?
Not yet.
@EtsyAPI
Visibility
@lapsu@EtsyAPI
@lapsu
PHP Coroutines
@EtsyAPI
More Code
@lapsu
Available Upon Request
@EtsyAPI
@lapsu
codeascraft.etsy.com
@EtsyAPI
@lapsu
</internals><wrap/>
@EtsyAPI
@lapsu
Address Mobile Challenges
@EtsyAPI
@lapsu
PHP Does Concurrency
@EtsyAPI
PHP Abides
@lapsu@EtsyAPI
Concurrent PHP
in the Etsy API
Matthew Graham
@lapsu
@EtsyAPI Lead
#phpday
2014 April
Thank You
Reminder:
@lapsu
Repeat the questions
@EtsyAPI
@lapsu
SPDY / HTTP 2.0
@EtsyAPI
Upcoming SlideShare
Loading in...5
×

Concurrent PHP in the Etsy API

996

Published on

How we at Etsy are adding concurrent data access to our PHP API.

From PHP Day 2014

1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total Views
996
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
9
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Concurrent PHP in the Etsy API"

  1. 1. Concurrent PHP in the Etsy API Matthew Graham @lapsu @EtsyAPI Lead #phpday 2014 April
  2. 2. @lapsu@EtsyAPI
  3. 3. @lapsu@EtsyAPI $1.3 Billion Things That Matter
  4. 4. @lapsu@EtsyAPI
  5. 5. @lapsu Etsy's PHP www api admin queues cron @EtsyAPI
  6. 6. @lapsu ~200 engineers 30+ deploys / day @EtsyAPI
  7. 7. Also, Rasmus @lapsu@EtsyAPI
  8. 8. Spoilers @lapsu Mobile Clients Are Special 1 Thread != No Concurrency @EtsyAPI
  9. 9. @lapsu <motivation> @EtsyAPI
  10. 10. Premise: @lapsu The Future is Mobile @EtsyAPI
  11. 11. @lapsu Past The Future is Mobile November 2013 @EtsyAPI
  12. 12. @lapsu@EtsyAPI
  13. 13. @EtsyAPI Mobile Networks Suck @lapsu <
  14. 14. @EtsyAPI Not Mobile @lapsu www.etsy.com/shop/AVintageWanderer
  15. 15. @EtsyAPI Network Performance @lapsu 3G < 4G
  16. 16. @EtsyAPI Network Coverage @lapsu 3G > 4G
  17. 17. @EtsyAPI Mobile Requests @lapsu More != Better
  18. 18. @lapsu@EtsyAPI
  19. 19. @lapsu 1000ms Time To Glass @EtsyAPI
  20. 20. @lapsu 1000ms - 900ms ------------- 100ms Network/Client ------------------------ Server @EtsyAPI
  21. 21. @lapsu 100ms = Bespoke + Concurrent @EtsyAPI
  22. 22. @lapsu Single Threads @EtsyAPI
  23. 23. Concurrency @lapsu@EtsyAPI Main “Thread” Child “Thread” Child “Thread”
  24. 24. @lapsu </motivation><interface> @EtsyAPI
  25. 25. Paul goes to Netflix @lapsu@EtsyAPI
  26. 26. 1 View : 1 Bespoke @lapsu@EtsyAPI ClientView Bespoke View View Bespoke Bespoke API
  27. 27. Multiple Clients @lapsu@EtsyAPI ClientsView Bespoke View View Bespoke Bespoke API
  28. 28. Bespoke : Components @lapsu@EtsyAPI Bespoke Bespoke Bespoke API Item User Shop Favs Tx
  29. 29. Components as REST @lapsu@EtsyAPI Bespoke Bespoke API Item User Shop Favs Tx
  30. 30. @lapsu ?includes=User @EtsyAPI
  31. 31. Android User View @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  32. 32. Concurrent Client @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  33. 33. Making Requests @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  34. 34. @lapsu@EtsyAPI shop favs t0 t1
  35. 35. Inputs @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  36. 36. Future Parameters @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  37. 37. @lapsu@EtsyAPI shop favs items t0 t1 t2 t3 t4
  38. 38. ~6 Lines @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  39. 39. @lapsu </interface><performance> @EtsyAPI
  40. 40. @lapsu Web Pages Are Clients Too @EtsyAPI
  41. 41. Web First @lapsu@EtsyAPI
  42. 42. API First @lapsu@EtsyAPI
  43. 43. Android User View @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  44. 44. Desktop Web User View @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); $teams = $cli->teams($inp->user_id); Curl_Orchestrator::run( [$shop,$favs,$items,$teams]); return [$favs,$items,$teams]; } @EtsyAPI
  45. 45. @lapsu Activity Feed @EtsyAPI
  46. 46. @lapsu Activity Feed Page @EtsyAPI Home Page
  47. 47. 30s of HTTP Time @lapsu@EtsyAPI
  48. 48. 30s of HTTP Time @lapsu 880ms Real Time @EtsyAPI
  49. 49. Components @lapsu@EtsyAPI Bespoke Bespoke Bespoke API Item User Shop Favs Tx
  50. 50. Components Cache @lapsu@EtsyAPI Bespoke Bespoke Bespoke Cache API Item User Shop Favs Tx
  51. 51. Local Call @lapsu@EtsyAPI Me
  52. 52. Long Distance Call @lapsu@EtsyAPI Atlantic Ocean User
  53. 53. Long Distance Call @lapsu@EtsyAPI Atlantic Ocean API Templates User
  54. 54. @lapsu </performance><internals> @EtsyAPI
  55. 55. @lapsu curl_multi_* @EtsyAPI
  56. 56. @lapsu@EtsyAPI curl?
  57. 57. General Sequence @lapsu@EtsyAPI curl_multi_init(); curl_multi_add(); curl_multi_exec(); while (!$done) { curl_multi_select(); curl_multi_exec(); curl_multi_info_read(); }
  58. 58. curl_multi_init @lapsu@EtsyAPI $mh = curl_multi_init();
  59. 59. curl_multi_add_handle @lapsu@EtsyAPI $mh = curl_multi_init(); $ch = curl_init($url); curl_setopt_array($ch, $options); curl_multi_add_handle($mh, $ch);
  60. 60. @lapsu@EtsyAPI multi handle handle handle handle
  61. 61. curl_multi_exec @lapsu@EtsyAPI do { $code = curl_multi_exec($mh, $r); } while ($code == CURLM_CALL_MULTI_PERFORM);
  62. 62. curl_multi_select @lapsu@EtsyAPI $cnt = curl_multi_select($mh, $tmout);
  63. 63. curl_multi_info_read @lapsu@EtsyAPI $info = curl_multi_info_read($mh); $ch = $info['handle']; $content = curl_multi_getcontent($ch);
  64. 64. @lapsu@EtsyAPI
  65. 65. localhost: Expected @lapsu@EtsyAPI R1 R2 R3 t0
  66. 66. Network: Actual @lapsu@EtsyAPI R2 R3 t0 t1 R1
  67. 67. curl Protocols @lapsu@EtsyAPI HTTP LDAP Gopher POP3 IMAP TELNET TFTP
  68. 68. curl_get_multi_handle_state @lapsu@EtsyAPI $state = CURLM_STATE_FIRST; curl_multi_get_handle_state( $mh, $ch, $state); $sent = $state > CURLM_STATE_FIRST && $state < CURLM_STATE_PERFORM;
  69. 69. Expected, Actual @lapsu@EtsyAPI R1 R2 R3 t0
  70. 70. Revised Sequence @lapsu@EtsyAPI curl_multi_init(); curl_multi_add(); while (!$sent) { curl_multi_exec(); curl_multi_get_handle_state(); } while (!$done) { curl_multi_select(); curl_multi_exec(); curl_multi_info_read(); }
  71. 71. Still Headed Upstream @lapsu@EtsyAPI
  72. 72. Patch URL @lapsu bit.ly/etsy_curl_multi_patch @EtsyAPI
  73. 73. Recursion? @lapsu@EtsyAPI
  74. 74. Recursion? @lapsu No. @EtsyAPI
  75. 75. Recursion? @lapsu No? Not yet. @EtsyAPI
  76. 76. Visibility @lapsu@EtsyAPI
  77. 77. @lapsu PHP Coroutines @EtsyAPI
  78. 78. More Code @lapsu Available Upon Request @EtsyAPI
  79. 79. @lapsu codeascraft.etsy.com @EtsyAPI
  80. 80. @lapsu </internals><wrap/> @EtsyAPI
  81. 81. @lapsu Address Mobile Challenges @EtsyAPI
  82. 82. @lapsu PHP Does Concurrency @EtsyAPI
  83. 83. PHP Abides @lapsu@EtsyAPI
  84. 84. Concurrent PHP in the Etsy API Matthew Graham @lapsu @EtsyAPI Lead #phpday 2014 April Thank You
  85. 85. Reminder: @lapsu Repeat the questions @EtsyAPI
  86. 86. @lapsu SPDY / HTTP 2.0 @EtsyAPI
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×