In this talk I will be covering some lessons learned in dealing with unclear customer requirements on complex ML problems and how to pivot after unpromising model results. From the technical side, I will cover some methods on how to ensemble classification models with confidence probability scores and how to choose the right probability cutoffs. The final point will be a brief note on when to stop tweaking models by considering relevant tradeoffs.

1. Delivering Delivery Time
Prediction
Aid Ahmetovic, 17.11.2022

2. About me
> Work experience
- Machine Learning Engineer - 3 years
- Software Engineer – 4 years
> Educational background
- Master’s Degree in Information and Computer Engineering - TU
Graz
> Contact
- Linkedin
- Email: a.ahmetovic@netconomy.net

3. Agenda

4. Agenda
1.
2.
3
4.
5.
About us
Project overview
Front-end result
Key takeaways
Q & A

5. About us

6. About us
Founded in 2000 with
headquarters in Graz, Austria
Around 500 employees
In 9 locations across Europe
Full-service provider for
ecommerce solutions
About us

7. Locations
Berlin
Dortmund
Graz (HQ)
Vienna
Klagenfurt
Belgrade
Madrid
Novi Sad
Zürich

8. Some of our clients

9. Project overview

10. Project overview
> Why?
- Improve customer experience while ordering in the
Douglas shop
- Increased customer satisfaction and conversion rate
> What?
- Predict the delivery time of purchases made in the
Douglas shop for Germany
> How?
- Calculate the delivery time of each enqueued order on
the Douglas shop
- Estimate the earliest & latest date of delivery
- Show ML model prediction during checkout

11. Problem definition
User orders
an item from
the shop
Package arrives
to the customer
Platform processing
time
Packaging time Shipping time

12. Client requirements
> How many days in the future do we want to predict?
- up to 5 working days in the future
> Testing periodicity and high load seasons (Singles day, Black
Friday, Christmas period)
- shadow mode deployment for the first 2 weeks
> Re-training and deployment frequency
- evaluate models every week, retrain & deploy if model
performance decreases

13. Project pipeline
CC DB
Data
exports MinIO
Data
preprocessing
Feature
engineering
Model
training
Model
evaluation
MinIO
ModelDB
Wrap models
Create docker version for
deployment
Update deployment
manifest
Staging and production
deployments
Container
registry
Data workflow
Training workflow
Deployment workflow
Source: Minio
Source: Argo
Source: ModelDB
Source Seldon

14. ML approaches

15. ML model
1st iteration - Regression
features
8.3
24.
5
15.
1
…
Regression model
deliveryTime (h)

16. ML model
2nd iteration - Classification
features
0
1
2
3
4
5
…
Classification model
deliveryDays

17. ML model
3rd iteration – Multi-model approach
features
0
1
2
3
4
5
…
Binary classification model
deliveryDays
day 0
Binary classification model day N
class 0
class 1
…..

18. ML model
3rd iteration – Multi-model approach -
inference
features
Binary classification model
day 0: 0.9
other: 0.1
Binary classification model
day 1: 0.7
other: 0.3
Binary classification model
day 2: 0.4
other: 0.6
Idea: Include probability
threshold, ex. 0.6
Result is a prediction
range [0,1]

19. Training overview

20. > Overview of training
runs
Training overview
Source: ModelDB

21. Front-end result

22. > Delivery time estimations
are shown on the last
checkout step
Front-end result

23. Key takeaways

24. Key takeaways
> Have regular meetings with the client during the lifecycle
of a PoC/Project
- Domain knowledge clarifications
- Data clarifications
- Integrations/architectural definitions
> Have fast iterations
- With data preprocessing, model training and deployment pipelines
in place
> Have a shadow mode deployment and model monitoring in
production

25. Thank you!
Questions?