Download to read offline
Uploaded byScyllaDB
iFood on Delivering 100 Million Events a Month to Restaurants with Scylla
Thales Biancalana, a senior backend developer at iFood, discusses migrating the company's infrastructure to a microservice event-driven architecture to improve polling services for 500k merchants. The case study outlines challenges and successes in transitioning from various database systems to ScyllaDB, highlighting significant cost reductions and performance enhancements. Key learnings include the importance of understanding database features and managing complexity in solutions.
More Related Content
Similar to iFood on Delivering 100 Million Events a Month to Restaurants with Scylla
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
iFood on Delivering 100 Million Events a Month to Restaurants with Scylla
- 1. Multiple Device Polling EventsSystems Thales Biancalana, Senior Backend Developer A case study from iFood
- 2. Presenter Thales Biancalana, SeniorBackend Developer at iFood Control and Automation Engineer that decided that programming is more exciting than building robots. Worked in multiple applications using .NET, Node, React, Swift and Java, now working as a backend developer at iFood. Always looking for new challenges and different ways to solve them
- 3.
- 4. iFood ■ Food-tech deliverybusiness. ■ Main delivery app in Brazil and also present in Colombia and Mexico > 100k> 20M/month
- 5.
- 6. iFood Infrastructure ■ Migratingto an microservice event driven architecture
- 7.
- 8. Connection Team Responsible fordelivering orders' events to merchants Connection Merchant App Integrations
- 9.
- 10. POST /acks Polling Services Multiple pollingsystems running in parallel Service A Service B Proxy Service App Orders Events Acknowledgements GET /events
- 11. ■ Http requestsevery 30 seconds for each device ■ Database to be invoked for each call ■ Heavy queries on read nodes: “all non-acked events by the device” ■ Mid term goal to support 500k connected merchants with 1 device each Polling Services ■ Why?
- 12. Polling Services Multiple pollingsystems running in parallel: ■ Proxy Service: Kitchen-Polling ■ Service A: Gateway-Core (PostgresQL) - Dead ■ Service B: Connection-Order-Events (Apache Ignite) ■ Service C: Connection-Polling (DynamoDB) - Dying ■ Service D: Connection-Polling (ScyllaDB)
- 13.
- 14. PostgresQL Legacy Service ■Events indexed in one table and the acknowledgements in another ■ Readings (JOINS) were starting to become a problem as the number of events and merchants increased ■ Master node “suffering with increasing load” ■ Single point of failure
- 15.
- 16.
- 17. Apache Ignite Service ■Works really well (reading ~3ms) Problems: ■ Hard to monitor, as service and database are one ■ We need to save events in another database used when adding machines or recovering from disasters (more code to maintain) ■ It takes longer to get the service back up as it needs to fill the cache from the PostgresQL database. That's why we have a fallback system for when it is down
- 18.
- 19. NoSQL Modeling ■ Ourmain query? ● All events that were not acked by a device ■ Orders (and events) belong to merchants, not devices ● We need the merchant devices when saving events ■ What to do with new devices? ● Return all merchant events and save them to the not acked by device table ■ We are only interested in events from the last 8 hours from delivery time
- 20.
- 21.
- 22. DynamoDB Service ■ WhyDynamoDB? ● Try a NoSQL approach ● Most of infrastructure is in AWS ● Fully managed solution
- 23. DynamoDB Service -Issues Issues with DynamoDB for our use: ■ DynamoDB autoscaling was not fast enough for our use case unless we left a high minimum throughput our manage it ourselves ● Defeats the purpose as a fully managed solution ■ DynamoDB new on-demand mode is great, but expensive
- 24.
- 25. ScyllaDB Service A ■Quite easy to migrate from DynamoDB to Scylla with the same modeling. Should be even easier with the new Project Alternator
- 26. ScyllaDB Service A- Results ■ How did it compare with DynamoDB? ● We started with three c5.2xlarge machine cluster that easily held the throughput. This was nearly 9x database cost reduction that could still hold more throughput (around $4.5k to $500/month)
- 27. ScyllaDB Service A- Learnings ■ Scylla uses TTL by column vs DynamoDB expiration time by document ■ Scylla Support: we identified a bug when reading pages from secondary index with prepared statements. After opening a Github issue we had a new build with the fix in less than 4 days (https://github.com/scylladb/scylla/issues/4569)
- 28. Modeling Issues Issues withthis modeling: ■ We need to manage restaurant devices ■ Need to manage old events for new devices ● It may be quite heavy to introduce a new device in the middle of the day
- 29. ScyllaDB Service B Secondmodeling using collections. Drawbacks: ■ Reads are expected to be slower (okay as a fallback system) Advantages: ■ Less complex ■ Events table can be used to populate ignite cache
- 30. ScyllaDB Service B- Catch
- 31. ScyllaDB Service B- Results The good: ■ Nearly 9x database cost reduction when comparing with DynamoDB on-demand ■ Time reduction from ~80ms to ~3ms to index events which resulted in nearly 8x infrastructure reduction for writes ■ Solution complexity reduction from 4 tables and 2 indexes to 2 tables and 1 index and 40% less code The bad: ■ Increase in read times, worth it for now as a fallback system ■ Collections updates are CPU intensive and generate tombstones -> use carefully
- 32.
- 33. Final Thoughts ■ Scyllawas cheaper when comparing with DynamoDB, but we created a cluster on AWS machines ● Take in consideration the cost of maintaining a cluster. Learn from other talks how easy it is to maintain a cluster when choosing between databases. ● But we have had no problems as of now ■ Check what you know about your domain and problem, it can be used to simplify the solution ● Knowing it was a fallback system and the average number of devices per merchant and orders per merchant led me believe it was a good trade off to have collections updates, which should be used carefully
- 34. Final Thoughts ■ Getto know all features of your database before using them ● Collection updates are not cheap! Each update incurs in a tombstone which slowdowns reads and gives more work to the garbage collector. We are still toying with gc_grace to improve performance ● ScyllaDB secondary indexes are global by default which was a good thing for our second solution, where the index has a cardinality as high as the number of merchants (a bit more than 100k merchants online today). It could be achieved in cassandra with Materialized Views. ● Global is the default, but it may not be always the best one to use, so Scylla also supports local indexes and you need to know when to use each.
- 35.
- 36. Next Steps ■ Noacknowledgment polling solution using Scylla ■ Force Scylla to fail ■ Working on MQTT pub/sub solution
- 37. Thank you Stayin touch Any questions? Thales Biancalana thales.biancalana@ifood.com.br 37
Editor's Notes
- #2 Hi everyone, my name is Thales. I’m here today because I’ve seen a lot of Scylla presentations talking about how awesome it is from a tech perspective, like how many ops, how it compares with cassandra as a drop in replacements and things like that, so I'm here to give a different perspective on how was to develop an application using Scylla from a developer perspective. I will not go into how to maintain the infrastructure, just about monitoring and costs.
- #3 (I'll probably skip this slide, but I'll leave it here) So as I said I’m Thales
- #4 Let me start by giving a little bit of context of what we do at iFood
- #5 iFood is a food tech delivery business. It is the main delivery app in Brazil, but we are present in other countries: Colombia and Mexico. We connect over 12 million users, 100 thousand merchants - mostly restaurants today, and deliverymen to deliver a bit over 20 million orders a month as of now, which amounts to a bit over 100 million events going through the platform every month.
- #6 So here is the user app on the left and the merchant web app on the left
- #7 Something relevant is how fast iFood grew. It went from 1 million to 20 million orders a month in a bit over two years. Because of that we still have some legacy services being broken into microservices using java, node, docker and kubernetes. This was only possible using a cloud service, and most of iFood's infrastructure runs on AWS, which is why we are still using SNS and SQS to move events around our platform. We use other technologies, but I'll mostly focus on these for our problem. Even though its size, iFood is not an established tech company as of now, and with the growing issues we are facing we are always looking for new ways to scale the infrastructure, which is what I'll try to share with you guys today. Most of what we have today in our infrastructure database is over PostgresQL and DynamoDB which is not scaling well as it is becoming expensive.
- #8 Now to talk about the project I'll first have to introduce the team I work in: Connection.
- #9 We are responsible, among other things, for delivering order's events to merchants, either directly to our merchant app I showed before or to integrations for huge food companies. One of the ways this is done today is with a polling API.
- #10 So now I'll present the polling services we've worked on until we got to the Scylla solution.
- #11 Events arrive from the platform via SNS-SQS and are indexed in multiple services running in parallel so we can compare them. These events are polled from the app via an GET /events endpoint and are acknowledged via a POST /acks endpoints. The app them sends and acknowledgement for each event it receives as not to receive it again on the next event poll.
- #12 The polling is done every 30 seconds for each device The database will be invoked on each /events call We have heavy queries on reading nodes of: all non acked events by the device The master Something that I want to adress now: why are we using polling instead of something with a pub/sub approach? We do have a MQTT service, which we are still developing, but unfortunately we also need to support external integrations, and a lot of them are not tech savvy, so having a REST API is a strategic advantage for having more merchants without going after them.
- #13 This is just to give names to all the services we developed
- #15 We started with a monolith polling system over a PostgresQL database that was the core of iFood for a long time. Readings were starting to become a problem as we got close to 10 million orders a month. We could solve it for some time by replicating to more databases and scaling the master vetically, but since we were separating the polling system from other functionalities they took this opportunity to work on something better.
- #16 Just to give you a better understanding, this was the relational data format. We had the events table at the top and an acknowledge table with the event id and the device id for the acknowledge. We would then join both tables for the polling result.
- #18 Our second approach was to deliver events using Apache Ignite in-memory database by indexing events and acknowledges. We decided to use Apache Ignite because we already used it in another service. It was put in place around october last year. It works really well and is currently the primary polling system at iFood. When we first deployed it, it had the postgres solution as a fallback. It works wonderfully, but after working with it for some time we had some bones to pick with it. First that service and database are one, so we need to be really careful about deployments and scaling (one machine at a time), and, although not a problem directly with Ignite, we had multiple issues with AWS ELB discovery for the machines to talk with each other. We also need to save the events/acks in another database for when adding machines or recover from disasters. With that in mind and thinking about removing the postgres solution as a fallback we started working on our first NoSQL solution.
- #20 So what do we know about our domain: First that we want all events not acked by a device Second that orders (and events) belong to a merchant, not to the device, so we need to know the merchant devices when saving the device events We need to also index the events by merchant to query them when introducing a new device Also, we are only interested in events from the last 8 hours from the delivery time. When I say delivery time is because we may have scheduled orders
- #21 So this was our first NoSQL model. We have a table for unacked device events, one table for the restaurant or merchant events and another for the restaurant devices. I’m just going to point out that we introduced restaurants as merchants not so long ago, so we sometimes still use the term restaurant.
- #23 Now we get to the good NoSQL part, where I'll get into it a bit more than the other solutions on how we implemented the solution. But first, why did we choose DynamoDB as our first NoSQL solution? First we wanted to try a NoSQL solution, second that we were already in AWS ecosystem, and third because it is a full managed solution.
- #24 As you can see, the solution is quite complex. We need to manage the restaurant devices and events for new devices Other problems with this solution is that DynamoDB autoscaling was not fast enough unless we left a high enough reading and write capacities, which would defeat the purpose of cutting costs. DynamoDB autoscaling only happens every 5 minutes, which is not fast enough for us. Lunch and especially dinner go from 0 to max throughput quite fast. We are currently using on-demand, but it is expensive. We could do the auto scaling ourselves, but it would no longer be a fully managed solution. It was around the time Scylla got in contact with our DBAs and started working on a new Scylla. The main problem we saw was the cost The scaling policy also contains a target utilization—the percentage of consumed provisioned throughput at a point in time. Application Auto Scaling uses a target tracking algorithm to adjust the provisioned throughput of the table (or index) upward or downward in response to actual workloads, so that the actual capacity utilization remains at or near your target utilization. You can set the auto scaling target utilization values between 20 and 90 percent for your read and write capacity. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
- #26 The first implementation using Scylla service was a direct comparison between Scylla and dynamodb solutions, so we implemented the same modeling. Because of this we could use the same code base and only change de DAO
- #27 This CPU load chart was taken from the scylla grafana overview dashboard provided by the scylla team.
- #28 I'll talk about Scylla collections
- #29 As you can see, the solution is quite complex. We need to manage the restaurant devices and events for new devices Other problems with this solution is that DynamoDB autoscaling was not fast enough unless we left a high enough reading and write capacities, which would defeat the purpose of cutting costs. DynamoDB autoscaling only happens every 5 minutes, which is not fast enough for us. Lunch and especially dinner go from 0 to max throughput quite fast. We are currently using on-demand, but it is expensive. We could do the auto scaling ourselves, but it would no longer be a fully managed solution. It was around the time Scylla got in contact with our DBAs and started working on a new Scylla. The main problem we saw was the cost The scaling policy also contains a target utilization—the percentage of consumed provisioned throughput at a point in time. Application Auto Scaling uses a target tracking algorithm to adjust the provisioned throughput of the table (or index) upward or downward in response to actual workloads, so that the actual capacity utilization remains at or near your target utilization. You can set the auto scaling target utilization values between 20 and 90 percent for your read and write capacity. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
- #30 Reads are slower, which is ok for a fallback system. Could be faster if NOT CONTAINS was supported on SETs (which not supported in Cassandra as it is not usually a good approach
- #31 Remember what I said about Scylla TTL? It is column based, not document based, so the new acked devices column would not have the TTL, thus it was never be deleted
- #34 This is probably the most important slide
- #35 This is probably the most important slide https://www.scylladb.com/2019/07/23/global-or-localsecondary-indexes-in-scylla-the-choice-is-now-yours/ https://thelastpickle.com/blog/2018/03/21/hinted-handoff-gc-grace-demystified.html