This document discusses how using Redis and a flat schema improved performance when processing JSON data. Key aspects include: 1) Storing data simply as strings in Redis allows fetching all data in one call and sorting/formatting in code for faster processing than querying a relational database. 2) Hosting Redis locally avoids network latency compared to an external service. 3) Redis is persistent so data is not lost if the server crashes and restarts.

1. Better performance using Redis
and flat schema
@singhal_manu @adhasmana

2. Demo

3. Time taken

4. What we have done in these
milliseconds
• Parsed the input json
• Applied the business logic to combine the
parameters
• Fetch all the data from redis
• Formatted and sorting the output data
• Generated ruby results to suit the output format.
• Handed over the data to axlsx gem for writing to
excel
• Save the excel file to disk (for record keeping)

5. Understanding the domain
• Our schema is defined and is very very specific
to our case.

6. Redis infrastructure
• Redis is hosted locally on each box

7. Redis infrastructure
• Redis is hosted locally on each box
• So we save few ms in the network call as its
not an external call.

8. Redis is persistent if you want
• Since our application is read intensive, we are
fine if Redis writes to db every 5 mins.
• It is configurable
• Extensive details at
http://oldblog.antirez.com/post/redis-
persistence-demystified.html

9. Its smart about writing to disk
• Save after 900 sec if atleast 1 key changed
• Save after 300 sec if atleast 10 keys changed
• Save after 60 sec if atleast 10000 keys changed

10. Redis crashed!!
• First of all its not all that common
• Secondly, its persistent. When the server
restarts, it will loads the data from the disk.

11. Our code
Input params Apply business logic
Arrange the keys in the
order of the output
Make a call to redis
Map the values with
the text
Write to excel
providing the
formatting details
Save the excel to disk
Transfer it over the
wire

12. How the data is stored
• Simple string in redis
• get "UPLOADDATA:15-XXXX-YYYYactual"
"{"2000":"792243","2001":"1040286","2002":"1154559","2003":"1222218",
"2004":"1281314","2005":"1327570","2006":"1427395","2007":"1502816",
"2008":"1537581","2009":"1491956","2010":"1513845","2011":"1600489",
"2012":"1670999","2013":"1716535","2014":"1795339","2015":"1871419",
"2016":"1956448","2017":"2040757","2018":"2123301","2019":"2205923",
"2020":"2287407"}"

13. “Key” to success
• $redis.mget : Gives the output in the input
order
• This enabled us to make just one call to redis
and play with the data in ruby.

14. Other things we used of Redis
• Redis pipeline
– For bulk upload of keys
– Do not wait for the response to come and process
next request.
– For 10K pings, here is the result.

15. I have much bigger data to upload to
redis
• Billions of keys to upload
• cat data.txt | redis-cli --pipe
• That will produce an output similar to

16. Future plans
• Sorted sets