iFood on Delivering 100 Million Events a Month to Restaurants with Scylla
•Download as PPTX, PDF•
0 likes•1,876 views
iFood is the largest Brazilian-based food delivery app company. It connects users, restaurants, and deliverymen using an event-driven architecture using AWS SQS and SNS, with programming in Java and Node.js. Thales' team is responsible for delivering orders' events to restaurant devices at least once, which is currently done using a REST API polling and acknowledgment system. Learn how their database infrastructure evolved from a PostgreSQL database, but began to show limitations and was a single point of failure. Growing through a few intermediary steps, including Amazon DynamoDB, eventually, turning to Scylla for its data model and collections to condense multiple tables. Using Scylla, iFood reduced the time to process events and acknowledgments (from ~80ms to ~3ms) and reduced costs using Scylla vs DynamoDB by over 9x.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to iFood on Delivering 100 Million Events a Month to Restaurants with Scylla
Similar to iFood on Delivering 100 Million Events a Month to Restaurants with Scylla (20)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
iFood on Delivering 100 Million Events a Month to Restaurants with Scylla
- 1. Multiple Device Polling Events Systems Thales Biancalana, Senior Backend Developer A case study from iFood
- 2. Presenter Thales Biancalana, Senior Backend Developer at iFood Control and Automation Engineer that decided that programming is more exciting than building robots. Worked in multiple applications using .NET, Node, React, Swift and Java, now working as a backend developer at iFood. Always looking for new challenges and different ways to solve them
- 3. Context
- 4. iFood ■ Food-tech delivery business. ■ Main delivery app in Brazil and also present in Colombia and Mexico > 100k> 20M/month
- 5. iFood apps
- 6. iFood Infrastructure ■ Migrating to an microservice event driven architecture
- 8. Connection Team Responsible for delivering orders' events to merchants Connection Merchant App Integrations
- 10. POST /acks Polling Services Multiple polling systems running in parallel Service A Service B Proxy Service App Orders Events Acknowledgements GET /events
- 11. ■ Http requests every 30 seconds for each device ■ Database to be invoked for each call ■ Heavy queries on read nodes: “all non-acked events by the device” ■ Mid term goal to support 500k connected merchants with 1 device each Polling Services ■ Why?
- 12. Polling Services Multiple polling systems running in parallel: ■ Proxy Service: Kitchen-Polling ■ Service A: Gateway-Core (PostgresQL) - Dead ■ Service B: Connection-Order-Events (Apache Ignite) ■ Service C: Connection-Polling (DynamoDB) - Dying ■ Service D: Connection-Polling (ScyllaDB)
- 14. PostgresQL Legacy Service ■ Events indexed in one table and the acknowledgements in another ■ Readings (JOINS) were starting to become a problem as the number of events and merchants increased ■ Master node “suffering with increasing load” ■ Single point of failure
- 15. PostgresQL Legacy Service - Data
- 17. Apache Ignite Service ■ Works really well (reading ~3ms) Problems: ■ Hard to monitor, as service and database are one ■ We need to save events in another database used when adding machines or recovering from disasters (more code to maintain) ■ It takes longer to get the service back up as it needs to fill the cache from the PostgresQL database. That's why we have a fallback system for when it is down
- 18. NoSQL
- 19. NoSQL Modeling ■ Our main query? ● All events that were not acked by a device ■ Orders (and events) belong to merchants, not devices ● We need the merchant devices when saving events ■ What to do with new devices? ● Return all merchant events and save them to the not acked by device table ■ We are only interested in events from the last 8 hours from delivery time
- 20. NoSQL Modeling
- 22. DynamoDB Service ■ Why DynamoDB? ● Try a NoSQL approach ● Most of infrastructure is in AWS ● Fully managed solution
- 23. DynamoDB Service - Issues Issues with DynamoDB for our use: ■ DynamoDB autoscaling was not fast enough for our use case unless we left a high minimum throughput our manage it ourselves ● Defeats the purpose as a fully managed solution ■ DynamoDB new on-demand mode is great, but expensive
- 25. ScyllaDB Service A ■ Quite easy to migrate from DynamoDB to Scylla with the same modeling. Should be even easier with the new Project Alternator
- 26. ScyllaDB Service A - Results ■ How did it compare with DynamoDB? ● We started with three c5.2xlarge machine cluster that easily held the throughput. This was nearly 9x database cost reduction that could still hold more throughput (around $4.5k to $500/month)
- 27. ScyllaDB Service A - Learnings ■ Scylla uses TTL by column vs DynamoDB expiration time by document ■ Scylla Support: we identified a bug when reading pages from secondary index with prepared statements. After opening a Github issue we had a new build with the fix in less than 4 days (https://github.com/scylladb/scylla/issues/4569)
- 28. Modeling Issues Issues with this modeling: ■ We need to manage restaurant devices ■ Need to manage old events for new devices ● It may be quite heavy to introduce a new device in the middle of the day
- 29. ScyllaDB Service B Second modeling using collections. Drawbacks: ■ Reads are expected to be slower (okay as a fallback system) Advantages: ■ Less complex ■ Events table can be used to populate ignite cache
- 30. ScyllaDB Service B - Catch
- 31. ScyllaDB Service B - Results The good: ■ Nearly 9x database cost reduction when comparing with DynamoDB on-demand ■ Time reduction from ~80ms to ~3ms to index events which resulted in nearly 8x infrastructure reduction for writes ■ Solution complexity reduction from 4 tables and 2 indexes to 2 tables and 1 index and 40% less code The bad: ■ Increase in read times, worth it for now as a fallback system ■ Collections updates are CPU intensive and generate tombstones -> use carefully
- 32. Final Thoughts
- 33. Final Thoughts ■ Scylla was cheaper when comparing with DynamoDB, but we created a cluster on AWS machines ● Take in consideration the cost of maintaining a cluster. Learn from other talks how easy it is to maintain a cluster when choosing between databases. ● But we have had no problems as of now ■ Check what you know about your domain and problem, it can be used to simplify the solution ● Knowing it was a fallback system and the average number of devices per merchant and orders per merchant led me believe it was a good trade off to have collections updates, which should be used carefully
- 34. Final Thoughts ■ Get to know all features of your database before using them ● Collection updates are not cheap! Each update incurs in a tombstone which slowdowns reads and gives more work to the garbage collector. We are still toying with gc_grace to improve performance ● ScyllaDB secondary indexes are global by default which was a good thing for our second solution, where the index has a cardinality as high as the number of merchants (a bit more than 100k merchants online today). It could be achieved in cassandra with Materialized Views. ● Global is the default, but it may not be always the best one to use, so Scylla also supports local indexes and you need to know when to use each.
- 35. Next Steps
- 36. Next Steps ■ No acknowledgment polling solution using Scylla ■ Force Scylla to fail ■ Working on MQTT pub/sub solution
- 37. Thank you Stay in touch Any questions? Thales Biancalana thales.biancalana@ifood.com.br 37
Editor's Notes
- Hi everyone, my name is Thales. I’m here today because I’ve seen a lot of Scylla presentations talking about how awesome it is from a tech perspective, like how many ops, how it compares with cassandra as a drop in replacements and things like that, so I'm here to give a different perspective on how was to develop an application using Scylla from a developer perspective. I will not go into how to maintain the infrastructure, just about monitoring and costs.
- (I'll probably skip this slide, but I'll leave it here) So as I said I’m Thales
- Let me start by giving a little bit of context of what we do at iFood
- iFood is a food tech delivery business. It is the main delivery app in Brazil, but we are present in other countries: Colombia and Mexico. We connect over 12 million users, 100 thousand merchants - mostly restaurants today, and deliverymen to deliver a bit over 20 million orders a month as of now, which amounts to a bit over 100 million events going through the platform every month.
- So here is the user app on the left and the merchant web app on the left
- Something relevant is how fast iFood grew. It went from 1 million to 20 million orders a month in a bit over two years. Because of that we still have some legacy services being broken into microservices using java, node, docker and kubernetes. This was only possible using a cloud service, and most of iFood's infrastructure runs on AWS, which is why we are still using SNS and SQS to move events around our platform. We use other technologies, but I'll mostly focus on these for our problem. Even though its size, iFood is not an established tech company as of now, and with the growing issues we are facing we are always looking for new ways to scale the infrastructure, which is what I'll try to share with you guys today. Most of what we have today in our infrastructure database is over PostgresQL and DynamoDB which is not scaling well as it is becoming expensive.
- Now to talk about the project I'll first have to introduce the team I work in: Connection.
- We are responsible, among other things, for delivering order's events to merchants, either directly to our merchant app I showed before or to integrations for huge food companies. One of the ways this is done today is with a polling API.
- So now I'll present the polling services we've worked on until we got to the Scylla solution.
- Events arrive from the platform via SNS-SQS and are indexed in multiple services running in parallel so we can compare them. These events are polled from the app via an GET /events endpoint and are acknowledged via a POST /acks endpoints. The app them sends and acknowledgement for each event it receives as not to receive it again on the next event poll.
- The polling is done every 30 seconds for each device The database will be invoked on each /events call We have heavy queries on reading nodes of: all non acked events by the device The master Something that I want to adress now: why are we using polling instead of something with a pub/sub approach? We do have a MQTT service, which we are still developing, but unfortunately we also need to support external integrations, and a lot of them are not tech savvy, so having a REST API is a strategic advantage for having more merchants without going after them.
- This is just to give names to all the services we developed
- We started with a monolith polling system over a PostgresQL database that was the core of iFood for a long time. Readings were starting to become a problem as we got close to 10 million orders a month. We could solve it for some time by replicating to more databases and scaling the master vetically, but since we were separating the polling system from other functionalities they took this opportunity to work on something better.
- Just to give you a better understanding, this was the relational data format. We had the events table at the top and an acknowledge table with the event id and the device id for the acknowledge. We would then join both tables for the polling result.
- Our second approach was to deliver events using Apache Ignite in-memory database by indexing events and acknowledges. We decided to use Apache Ignite because we already used it in another service. It was put in place around october last year. It works really well and is currently the primary polling system at iFood. When we first deployed it, it had the postgres solution as a fallback. It works wonderfully, but after working with it for some time we had some bones to pick with it. First that service and database are one, so we need to be really careful about deployments and scaling (one machine at a time), and, although not a problem directly with Ignite, we had multiple issues with AWS ELB discovery for the machines to talk with each other. We also need to save the events/acks in another database for when adding machines or recover from disasters. With that in mind and thinking about removing the postgres solution as a fallback we started working on our first NoSQL solution.
- So what do we know about our domain: First that we want all events not acked by a device Second that orders (and events) belong to a merchant, not to the device, so we need to know the merchant devices when saving the device events We need to also index the events by merchant to query them when introducing a new device Also, we are only interested in events from the last 8 hours from the delivery time. When I say delivery time is because we may have scheduled orders
- So this was our first NoSQL model. We have a table for unacked device events, one table for the restaurant or merchant events and another for the restaurant devices. I’m just going to point out that we introduced restaurants as merchants not so long ago, so we sometimes still use the term restaurant.
- Now we get to the good NoSQL part, where I'll get into it a bit more than the other solutions on how we implemented the solution. But first, why did we choose DynamoDB as our first NoSQL solution? First we wanted to try a NoSQL solution, second that we were already in AWS ecosystem, and third because it is a full managed solution.
- As you can see, the solution is quite complex. We need to manage the restaurant devices and events for new devices Other problems with this solution is that DynamoDB autoscaling was not fast enough unless we left a high enough reading and write capacities, which would defeat the purpose of cutting costs. DynamoDB autoscaling only happens every 5 minutes, which is not fast enough for us. Lunch and especially dinner go from 0 to max throughput quite fast. We are currently using on-demand, but it is expensive. We could do the auto scaling ourselves, but it would no longer be a fully managed solution. It was around the time Scylla got in contact with our DBAs and started working on a new Scylla. The main problem we saw was the cost The scaling policy also contains a target utilization—the percentage of consumed provisioned throughput at a point in time. Application Auto Scaling uses a target tracking algorithm to adjust the provisioned throughput of the table (or index) upward or downward in response to actual workloads, so that the actual capacity utilization remains at or near your target utilization. You can set the auto scaling target utilization values between 20 and 90 percent for your read and write capacity. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
- The first implementation using Scylla service was a direct comparison between Scylla and dynamodb solutions, so we implemented the same modeling. Because of this we could use the same code base and only change de DAO
- This CPU load chart was taken from the scylla grafana overview dashboard provided by the scylla team.
- I'll talk about Scylla collections
- As you can see, the solution is quite complex. We need to manage the restaurant devices and events for new devices Other problems with this solution is that DynamoDB autoscaling was not fast enough unless we left a high enough reading and write capacities, which would defeat the purpose of cutting costs. DynamoDB autoscaling only happens every 5 minutes, which is not fast enough for us. Lunch and especially dinner go from 0 to max throughput quite fast. We are currently using on-demand, but it is expensive. We could do the auto scaling ourselves, but it would no longer be a fully managed solution. It was around the time Scylla got in contact with our DBAs and started working on a new Scylla. The main problem we saw was the cost The scaling policy also contains a target utilization—the percentage of consumed provisioned throughput at a point in time. Application Auto Scaling uses a target tracking algorithm to adjust the provisioned throughput of the table (or index) upward or downward in response to actual workloads, so that the actual capacity utilization remains at or near your target utilization. You can set the auto scaling target utilization values between 20 and 90 percent for your read and write capacity. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
- Reads are slower, which is ok for a fallback system. Could be faster if NOT CONTAINS was supported on SETs (which not supported in Cassandra as it is not usually a good approach
- Remember what I said about Scylla TTL? It is column based, not document based, so the new acked devices column would not have the TTL, thus it was never be deleted
- This is probably the most important slide
- This is probably the most important slide https://www.scylladb.com/2019/07/23/global-or-localsecondary-indexes-in-scylla-the-choice-is-now-yours/ https://thelastpickle.com/blog/2018/03/21/hinted-handoff-gc-grace-demystified.html