Building a Scalable System for Tracking Shipping Packages
•
1 like•979 views
This document discusses the architecture for tracking shipping packages at Etsy. It describes how Etsy retrieves tracking data from multiple carriers' APIs, standardizes the tracking states, shards the tracking data across multiple databases, uses queues to schedule updates, and implements asynchronous processing and monitoring to handle the large volume of tracking requests at scale.
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Building a Scalable System for Tracking Shipping Packages
- 2. BUILDING A SCALABLE SYSTEM FOR TRACKING SHIPPING PACKAGES PREMSHREE PILLAI,
- 4. SR. ENGINEER, SHIPPING PREVIOUSLY, YAHOO! @premshree
- 5. AGENDA
- 8. em·pa·thynoun ˈem-pə-thē : the feeling that you understand and share another person's experiences and emotions : the ability to share someone else's feelings
- 9. em·pa·thynoun ˈem-pə-thē : the feeling that you understand and share another person's experiences and emotions : the ability to share someone else's feelings
- 10. https://www.etsy.com/listing/110699162/seeker-of-happiness-canvas-tote-bag-you RE-IMAGINE COMMERCE IN WAYS THAT BUILD A MORE FULFILLING AND LASTING WORLD
- 19. ♥ POTTERY
- 21. SCOTTSDALE, AZ
- 22. ANDOVER, MA
- 23. LEFKIS, GREECE
- 24. 1.4 MILLION ACTIVE SELLERS 19.8 MILLION ACTIVE BUYERS 29 MILLION ACTIVE LISTINGS https://www.etsy.com/listing/191371121/jael-green-red-sisal-basket-from-kenya
- 31. WHERE IS MY PACKAGE? ♥ https://www.etsy.com/listing/180703192/johannes-holst-sailboat-at-sea-1959
- 33. OUT FOR DELIVERY 125,000+https://www.etsy.com/listing/129521877/original-world-map-watercolor-print
- 38. LET’S DIVE IN
- 39. mysql
- 50. tracking_events
- 51. SHIPPING APIS
- 59. TRACKING STATES
- 64. STATE MAP
- 65. INFO_RECEIVED “acceptance” “processed at usps origin facility" “accepted at usps origin sort facility” “picked up”
- 66. INFO_RECEIVED “tendered at fedex office” “picked up" “in fedex possession”
- 67. IN_TRANSIT “dispatched from usps sort facility” “sorting complete" “local holiday - will attempt delivery on next working day” “customs clearance processing
- 68. SELECT * FROM trackings WHERE last_access_date < ?
- 69. SELECT * FROM trackings WHERE last_access_date < ?
- 70. ? = now() - 15
- 72. caveat lector
- 74. SIZE
- 75. SELECT * FROM trackings WHERE last_access_date < ? LIMIT ?
- 76. STRAGGLERS
- 78. SHARDING 101
- 87. SELECT * FROM trackings WHERE shard = ? AND last_access_date < ?
- 89. QUEUE
- 90. FAUX QUEUE
- 93. QUEUE MAP
- 94. INFO_RECEIVED => QUEUE_1_DAY PACKAGE_RECEIVED => QUEUE_1_DAY IN_TRANSIT => QUEUE_1_DAY OUT_FOR_DELIVERY => QUEUE_1_HOUR ... DELIVERED => QUEUE_STOPPED
- 95. INFO_RECEIVED => QUEUE_3_HOUR PACKAGE_RECEIVED => QUEUE_3_HOUR IN_TRANSIT => QUEUE_3_HOUR OUT_FOR_DELIVERY => QUEUE_1_HOUR ... DELIVERED => QUEUE_STOPPED
- 96. QUEUE AGE
- 97. QUEUE_3_DAY => 1*60*60*24*3 QUEUE_1_DAY => 1*60*60*24 QUEUE_3_HOUR => 3*60*60 QUEUE_1_HOUR => 1*60*60
- 98. SELECT * FROM trackings WHERE shard = ? AND queue = ? AND last_access_date < ?
- 99. SELECT * FROM trackings WHERE shard = ? AND queue = ? AND last_access_date < ?
- 100. last_access_date < time() - age
- 101. ASYNC
- 102. GEARMAN
- 103. unique job key (shard, queue)
- 105. IN SUMMARY
- 107. MONITORING
- 108. StatsD
- 110. Tracking stats Gearman worker calls Tracking calls by provider Tracking queues CURL errors
- 111. tracking numbers
- 113. get tracking (usps)
- 114. get tracking (canada post)
- 115. curl errors (usps)
- 116. curl errors (fedex)
- 117. Tracking stats Gearman worker calls Tracking calls by provider Tracking queues CURL errors
- 118. get tracking (usps)
- 119. Tracking stats Gearman worker calls Tracking calls by provider Tracking queues CURL errors
- 120. curl errors (usps)
- 124. ActiveMQ/RabbitMQ?
- 125. webhooks?
- 126. your ideas