Downloaded 15 times
Uploaded byPremshree Pillai
Building a Scalable System for Tracking Shipping Packages
This document describes the architecture for tracking shipping packages at Etsy. It discusses using APIs from carriers to get tracking status updates, mapping tracking states to queues with different update frequencies, sharding data by shop/user for querying, using Gearman for asynchronous processing, and monitoring the system with StatsD to track metrics.
Building a Scalable System for Tracking Shipping Packages
- 1. BUILDING A SCALABLESYSTEM FOR TRACKING SHIPPING PACKAGES
- 2.
- 3.
- 4.
- 5. SR. ENGINEER, ETSY PREVIOUSLY,YAHOO! @premshree
- 6.
- 7.
- 8.
- 9. empathy noun ˈem-pə-thē : thefeeling that you understand and share another person's experiences and emotions : the ability to share someone else's feelings https://www.etsy.com/listing/26849451/the-jetty-sad-girl-empathy-melancholy
- 10. https://www.etsy.com/listing/110699162/seeker-of-happiness-canvas-tote-bag-you RE-IMAGINE COMMERCE IN WAYS THATBUILD A MORE FULFILLING AND LASTING WORLD
- 12.
- 15.
- 17.
- 19.
- 21.
- 22.
- 23.
- 24.
- 25. 1.5 MILLION ACTIVESELLERS https://www.etsy.com/listing/191371121/jael-green-red-sisal-basket-from-kenya
- 26. 1.5 MILLION ACTIVESELLERS 21.7 MILLION ACTIVE BUYERS https://www.etsy.com/listing/191371121/jael-green-red-sisal-basket-from-kenya
- 27. 1.5 MILLION ACTIVESELLERS 21.7 MILLION ACTIVE BUYERS https://www.etsy.com/listing/191371121/jael-green-red-sisal-basket-from-kenya $1.93 BILLION GMS (2014)
- 33.
- 34.
- 35. OUT FOR DELIVERY10,000+ https://www.etsy.com/listing/129521877/original-world-map-watercolor-print
- 36.
- 37.
- 39.
- 40.
- 51.
- 52.
- 53.
- 54.
- 60.
- 64.
- 65.
- 66. INFO_RECEIVED “acceptance” “processed at uspsorigin facility" “accepted at usps origin sort facility” “picked up”
- 67. INFO_RECEIVED “tendered at fedexoffice” “picked up" “in fedex possession”
- 68. IN_TRANSIT “dispatched from uspssort facility” “sorting complete" “local holiday - will attempt delivery on next working day” “customs clearance processing
- 69. SELECT * FROMtrackings WHERE last_access_date < ?
- 72. SELECT * FROMtrackings WHERE last_access_date < ?
- 73. ? = now()- 15
- 75.
- 76.
- 77.
- 78. SELECT * FROMtrackings WHERE last_access_date < ? LIMIT ?
- 79.
- 81.
- 82.
- 83.
- 84.
- 85.
- 90. SELECT * FROMtrackings WHERE shard = ? AND last_access_date < ?
- 92.
- 93.
- 95.
- 96.
- 97. INFO_RECEIVED => QUEUE_1_DAY PACKAGE_RECEIVED=> QUEUE_1_DAY IN_TRANSIT => QUEUE_1_DAY OUT_FOR_DELIVERY => QUEUE_1_HOUR ... DELIVERED => QUEUE_STOPPED
- 98. INFO_RECEIVED => QUEUE_3_HOUR PACKAGE_RECEIVED=> QUEUE_3_HOUR IN_TRANSIT => QUEUE_3_HOUR OUT_FOR_DELIVERY => QUEUE_1_HOUR ... DELIVERED => QUEUE_STOPPED
- 99.
- 100. QUEUE_3_DAY => 1*60*60*24*3 QUEUE_1_DAY => 1*60*60*24 QUEUE_3_HOUR => 3*60*60 QUEUE_1_HOUR => 1*60*60
- 101. SELECT * FROMtrackings WHERE shard = ? AND queue = ? AND last_access_date < ?
- 102. SELECT * FROMtrackings WHERE shard = ? AND queue = ? AND last_access_date < ?
- 103.
- 104.
- 105.
- 106. unique job key (shard,queue)
- 112.
- 113.
- 114.
- 115.
- 116.
- 117.
- 118. Tracking stats Gearman workercalls Tracking calls by provider Tracking queues CURL errors
- 119.
- 120.
- 121.
- 122. get tracking (canadapost)
- 123. Tracking stats Gearman workercalls Tracking calls by provider Tracking queues CURL errors
- 124.
- 125. Tracking stats Gearman workercalls Tracking calls by provider Tracking queues CURL errors
- 126.
- 127.
- 128.
- 129.
- 130.
- 131.
- 132.
- 133.
- 134.
- 135.