Concurrent PHP in the Etsy API
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Concurrent PHP in the Etsy API

  • 784 views
Uploaded on

How we at Etsy are adding concurrent data access to our PHP API. ...

How we at Etsy are adding concurrent data access to our PHP API.

From PHP Day 2014

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
784
On Slideshare
562
From Embeds
222
Number of Embeds
2

Actions

Shares
Downloads
5
Comments
1
Likes
1

Embeds 222

http://home20.intesys.it 213
http://librosweb.es 9

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Concurrent PHP in the Etsy API Matthew Graham @lapsu @EtsyAPI Lead #phpday 2014 April
  • 2. @lapsu@EtsyAPI
  • 3. @lapsu@EtsyAPI $1.3 Billion Things That Matter
  • 4. @lapsu@EtsyAPI
  • 5. @lapsu Etsy's PHP www api admin queues cron @EtsyAPI
  • 6. @lapsu ~200 engineers 30+ deploys / day @EtsyAPI
  • 7. Also, Rasmus @lapsu@EtsyAPI
  • 8. Spoilers @lapsu Mobile Clients Are Special 1 Thread != No Concurrency @EtsyAPI
  • 9. @lapsu <motivation> @EtsyAPI
  • 10. Premise: @lapsu The Future is Mobile @EtsyAPI
  • 11. @lapsu Past The Future is Mobile November 2013 @EtsyAPI
  • 12. @lapsu@EtsyAPI
  • 13. @EtsyAPI Mobile Networks Suck @lapsu <
  • 14. @EtsyAPI Not Mobile @lapsu www.etsy.com/shop/AVintageWanderer
  • 15. @EtsyAPI Network Performance @lapsu 3G < 4G
  • 16. @EtsyAPI Network Coverage @lapsu 3G > 4G
  • 17. @EtsyAPI Mobile Requests @lapsu More != Better
  • 18. @lapsu@EtsyAPI
  • 19. @lapsu 1000ms Time To Glass @EtsyAPI
  • 20. @lapsu 1000ms - 900ms ------------- 100ms Network/Client ------------------------ Server @EtsyAPI
  • 21. @lapsu 100ms = Bespoke + Concurrent @EtsyAPI
  • 22. @lapsu Single Threads @EtsyAPI
  • 23. Concurrency @lapsu@EtsyAPI Main “Thread” Child “Thread” Child “Thread”
  • 24. @lapsu </motivation><interface> @EtsyAPI
  • 25. Paul goes to Netflix @lapsu@EtsyAPI
  • 26. 1 View : 1 Bespoke @lapsu@EtsyAPI ClientView Bespoke View View Bespoke Bespoke API
  • 27. Multiple Clients @lapsu@EtsyAPI ClientsView Bespoke View View Bespoke Bespoke API
  • 28. Bespoke : Components @lapsu@EtsyAPI Bespoke Bespoke Bespoke API Item User Shop Favs Tx
  • 29. Components as REST @lapsu@EtsyAPI Bespoke Bespoke API Item User Shop Favs Tx
  • 30. @lapsu ?includes=User @EtsyAPI
  • 31. Android User View @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 32. Concurrent Client @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 33. Making Requests @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 34. @lapsu@EtsyAPI shop favs t0 t1
  • 35. Inputs @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 36. Future Parameters @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 37. @lapsu@EtsyAPI shop favs items t0 t1 t2 t3 t4
  • 38. ~6 Lines @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 39. @lapsu </interface><performance> @EtsyAPI
  • 40. @lapsu Web Pages Are Clients Too @EtsyAPI
  • 41. Web First @lapsu@EtsyAPI
  • 42. API First @lapsu@EtsyAPI
  • 43. Android User View @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); Curl_Orchestrator::run( [$shop,$favs,$items]); return [$favs,$items]; } @EtsyAPI
  • 44. Desktop Web User View @lapsu function handle($cli, $inp) { $shop = $cli->shop($inp->user_id); $favs = $cli->favs($inp->user_id); $items = $cli->items($shop->shop_id); $teams = $cli->teams($inp->user_id); Curl_Orchestrator::run( [$shop,$favs,$items,$teams]); return [$favs,$items,$teams]; } @EtsyAPI
  • 45. @lapsu Activity Feed @EtsyAPI
  • 46. @lapsu Activity Feed Page @EtsyAPI Home Page
  • 47. 30s of HTTP Time @lapsu@EtsyAPI
  • 48. 30s of HTTP Time @lapsu 880ms Real Time @EtsyAPI
  • 49. Components @lapsu@EtsyAPI Bespoke Bespoke Bespoke API Item User Shop Favs Tx
  • 50. Components Cache @lapsu@EtsyAPI Bespoke Bespoke Bespoke Cache API Item User Shop Favs Tx
  • 51. Local Call @lapsu@EtsyAPI Me
  • 52. Long Distance Call @lapsu@EtsyAPI Atlantic Ocean User
  • 53. Long Distance Call @lapsu@EtsyAPI Atlantic Ocean API Templates User
  • 54. @lapsu </performance><internals> @EtsyAPI
  • 55. @lapsu curl_multi_* @EtsyAPI
  • 56. @lapsu@EtsyAPI curl?
  • 57. General Sequence @lapsu@EtsyAPI curl_multi_init(); curl_multi_add(); curl_multi_exec(); while (!$done) { curl_multi_select(); curl_multi_exec(); curl_multi_info_read(); }
  • 58. curl_multi_init @lapsu@EtsyAPI $mh = curl_multi_init();
  • 59. curl_multi_add_handle @lapsu@EtsyAPI $mh = curl_multi_init(); $ch = curl_init($url); curl_setopt_array($ch, $options); curl_multi_add_handle($mh, $ch);
  • 60. @lapsu@EtsyAPI multi handle handle handle handle
  • 61. curl_multi_exec @lapsu@EtsyAPI do { $code = curl_multi_exec($mh, $r); } while ($code == CURLM_CALL_MULTI_PERFORM);
  • 62. curl_multi_select @lapsu@EtsyAPI $cnt = curl_multi_select($mh, $tmout);
  • 63. curl_multi_info_read @lapsu@EtsyAPI $info = curl_multi_info_read($mh); $ch = $info['handle']; $content = curl_multi_getcontent($ch);
  • 64. @lapsu@EtsyAPI
  • 65. localhost: Expected @lapsu@EtsyAPI R1 R2 R3 t0
  • 66. Network: Actual @lapsu@EtsyAPI R2 R3 t0 t1 R1
  • 67. curl Protocols @lapsu@EtsyAPI HTTP LDAP Gopher POP3 IMAP TELNET TFTP
  • 68. curl_get_multi_handle_state @lapsu@EtsyAPI $state = CURLM_STATE_FIRST; curl_multi_get_handle_state( $mh, $ch, $state); $sent = $state > CURLM_STATE_FIRST && $state < CURLM_STATE_PERFORM;
  • 69. Expected, Actual @lapsu@EtsyAPI R1 R2 R3 t0
  • 70. Revised Sequence @lapsu@EtsyAPI curl_multi_init(); curl_multi_add(); while (!$sent) { curl_multi_exec(); curl_multi_get_handle_state(); } while (!$done) { curl_multi_select(); curl_multi_exec(); curl_multi_info_read(); }
  • 71. Still Headed Upstream @lapsu@EtsyAPI
  • 72. Patch URL @lapsu bit.ly/etsy_curl_multi_patch @EtsyAPI
  • 73. Recursion? @lapsu@EtsyAPI
  • 74. Recursion? @lapsu No. @EtsyAPI
  • 75. Recursion? @lapsu No? Not yet. @EtsyAPI
  • 76. Visibility @lapsu@EtsyAPI
  • 77. @lapsu PHP Coroutines @EtsyAPI
  • 78. More Code @lapsu Available Upon Request @EtsyAPI
  • 79. @lapsu codeascraft.etsy.com @EtsyAPI
  • 80. @lapsu </internals><wrap/> @EtsyAPI
  • 81. @lapsu Address Mobile Challenges @EtsyAPI
  • 82. @lapsu PHP Does Concurrency @EtsyAPI
  • 83. PHP Abides @lapsu@EtsyAPI
  • 84. Concurrent PHP in the Etsy API Matthew Graham @lapsu @EtsyAPI Lead #phpday 2014 April Thank You
  • 85. Reminder: @lapsu Repeat the questions @EtsyAPI
  • 86. @lapsu SPDY / HTTP 2.0 @EtsyAPI