Optimizing cloud firestore reads

Optimizing Cloud Firestore
Reads
By: Ryan Sneyd

Quick Definitions
Firebase: The name of the suite of tools Google uses to provide BaaS (Backend as a Service)
Real Time Database: Document based NoSQL used for smaller projects that require low latency
Cloud FireStore: The new version of Real Time Database that is faster and more scalable
Document: Holds data that contains a key that can be indexed and value associated with that key (Think
table of contents that has a name of a chapter (Key) and a page number (Value))
Collection: List of documents

Pricing Models
Google charges users a fixed fee for every read, write and delete operation
Google also charges for the amount of GB stored on their network
Google offers three plans:
- Spark: Free tier with limited daily usage
- Flame: $25/month plan that stops charging if users go over a specific limit
- Blaze: Pay-as-you-go plan that charges based on usage (See next slide)
See https://firebase.google.com/pricing for more details

Blaze Pricing Model Breakdown
*price in USD
[1]"Understand Cloud Firestore billing | Firebase", Firebase, 2019. [Online]. Available:
https://firebase.google.com/docs/firestore/pricing. [Accessed: 15- Jul- 2019].

Managing Reads and Writes
Google sets the Blaze plan as default but it can be switched to any plan based on the users needs
Since Google charges based on Read, Write and Delete operations there are strategies that can be used
to minimize reads and writes and subsequently optimize your backend
The goal is to give Google as little money as possible and avoid spending “$30,356.56 USD in just 72
hours” [8]
[8]N. Contreras, "How we spent 30k USD in Firebase in less than 72 hours - By", Hackernoon.com, 2019. [Online]. Available:
https://hackernoon.com/how-we-spent-30k-usd-in-firebase-in-less-than-72-hours-307490bd24d. [Accessed: 22- Jul- 2019].

How Reads and Writes works
Reads
- When data is received from a document using get() or exist()
- If data in a document is changed and client reads the update
- If user logs out and logs back in after 30 minutes and reads the same data
Writes
- set() and update() are called
- Everytime the data is manually changed in Cloud Firestore
Deletes
- Anytime a document is deleted or document field is deleted

Strategies for Optimizing Reads and Writes
Strategy 1:
- Minimize hotspotting on Firestore
Strategy 2:
- Use Transactions and Batch Writes along with other Google recommended practices
Strategy 3:
- Follow Document Based NoSQL design patterns when modeling data

Hotspotting
Hotspotting: When one part of a system is being overloaded instead of being distributed across the
whole system
This occurs when:
- Many documents are being created at once with incrementing/decrementing ids
- Generating lots of documents in small collections
- Adding data that frequently changes (i.e timestamps)
- Deleting multiple documents in a collection
- Writing to a document too frequently without gradually increasing traffic
[2]"Best practices | Cloud Firestore | Google Cloud", Google Cloud, 2019. [Online]. Available:
https://cloud.google.com/firestore/docs/best-practices#hotspots. [Accessed: 15- Jul- 2019].

Minimizing hotspotting
Document Ids
- Avoid using the characters . .. and /
- Do not use incrementing ids (i.e. Customer1, Customer2, Customer3 …)
- Best to use a unique identifier such as a Username or email
Field names
- Avoid using periods, brackets, asterisk and backticks (Requires extra processing)
Indexing
- Avoid indexing as it increases storage costs
- Only use indexing to partition or retreive expensive data (i.e large text file, large arrays)

Following Google’s Best Practices
Avoid writing more that one document per second
- This can lead to high latency, timeouts or worse
Use Asynchronous calls over synchronous calls
Use cursors instead of offsets
Use transactions and batch writes for reads and writes

Transactions and Batch Writes
Transactions and batch writes are used to perform atomic operations meaning it “guaranteed to be
isolated from other operations that may be happening at the same time.” [3]
Transaction is a set of reads and writes operations on one or more documents. [4]
Batch write is a set of write operations on one or more documents. [4]
[3]J. Fisher, "What the Heck Is an "Atomic Object"?", Atomic Spin, 2019. [Online]. Available:
https://spin.atomicobject.com/2016/01/06/defining-atomic-object/. [Accessed: 15- Jul- 2019].
[4]"Transactions and batched writes | Cloud Firestore | Google Cloud", Google Cloud, 2019. [Online]. Available:
https://cloud.google.com/firestore/docs/manage-data/transactions#batched-writes. [Accessed: 15- Jul- 2019].

Transactions
A transaction is any get() operation followed by any set(), update() or delete() operation
By using transactions data is guaranteed to be up to date and consistent
Things to note:
- Read operations must come before write operations
- Transaction may be executed more than once if there are concurrent edits
- Transaction should not directly modify the application state
- Transactions will fail if the client is offline

Transaction in Python

Transaction Failure
A transaction will fail if:
- Transaction contains read operations after a write operation
- A document was modified during a transaction. In this case the transaction will retry for a set
number of times
- Transaction size is greater than 10 MB
Failed transactions does not write to firestore

Batch Write
Batch writes allow you to write a combination set(), update() or delete() operations as a single atomic
action.
Batch write can hold up to 500 operations
Other operations include serverTimestamp() , arrayUnion() and increment()
Batch writes are less likely to fail and will not retry like transactions will
Batch writes will execute even if the client is offline

Batch Write in Python

Designing Document Based NoSQL
In traditional database tables have schema, a set on constraints the data must follow
In Firestore, data is schema-less meaning it does not have to follow constraints
[5]Microsoft, LocalDB used in Microsoft Visual Studio. 2019.
[6]Medium, Document used in Firebase Firestore. 2019.

Polymorphic Schema
Because there are no constraints to follow we can put any type of data into a collection which makes the
schema polymorphic or can take “many forms” [7]
An example could be an online store that sells Appliances, CDs and Books
Each item has similar attributes like price, name and quantity but also unique ones like:
- Books have a Page Number
- CDs have a Song Count
- Appliances have a type such as Kitchen
[7]D. Sullivan, NoSQL for mere mortals®. Hoboken [etc.]: Addison-Wesley, 2015, pp. 152 - 217.

Polymorphic Schema
Since our online store will always be displaying the price, name and quantity to our users, the three
products will be retrieved the same way
Instead of a storing each product into separate collections for Books, CDs and Appliances it is better to
have a products collection because the data is retrieved the same
By simplifying our collections using a process known as denormalization, we reduce the number of reads
and writes to our database
Warning: Don’t over-simplify collections as it may reduce performance

One To Many Relationships
One to Many: When an instance of an entity has one or more related instances of another entity [7]
Examples include:
- A Garage contains one or many cars
- A shelf contains one or many books
Suggested Practice: To put the multiple instances as a map or array inside the single instance [7]

One To Many Example
Location Instance 1
Location Instance 2
Single
Instance
Customer

Many to Many Relationships
Many to Many: When multiple instances of one entity are related to multiple instances of another entity
[7]
Examples Include:
- Many students take many classes
- Many doctors have many patients
Suggested Practice: Use separate collections to represent the class of entities. Documents in the
collection contain references to the data they are related to. [7]

Many to Many Example
Courses Collection
Students Collection
Reference To Student Document
Course Document
Student Document Reference To Courses Document

Hierarchy Relationships
Hierarchy: Instances of entities in some kind of parent-child or part-subpart relationship [7]
Examples:
- Creating a recliner, table and desk as parts of a furniture collection
- Creating a lion, tiger and bobcat as children of a cat collection
Suggested Practice: Give child entities a reference to the parent entities [7]

Hierarchy Example
Parent Reference

Conclusion
In my experience, following these guidelines will help:
- Organize your data
- Make faster queries
- Create repeatable quality
- Reduce costs
Overall not just improving your user’s experience but your wallet’s experience as well

References
[3]J. Fisher, "What the Heck Is an "Atomic Object"?", Atomic Spin, 2019. [Online]. Available:
https://spin.atomicobject.com/2016/01/06/defining-atomic-object/. [Accessed: 15- Jul- 2019].
[4]"Transactions and batched writes | Cloud Firestore | Google Cloud", Google Cloud, 2019. [Online].
Available: https://cloud.google.com/firestore/docs/manage-data/transactions#batched-writes.
[Accessed: 15- Jul- 2019].

References
[5]Microsoft, LocalDB used in Microsoft Visual Studio. 2019.
[6]Medium, Document used in Firebase Firestore. 2019.
[8]N. Contreras, "How we spent 30k USD in Firebase in less than 72 hours - By", Hackernoon.com, 2019.
[Online]. Available: https://hackernoon.com/how-we-spent-30k-usd-in-firebase-in-less-than-72-
hours-307490bd24d. [Accessed: 22- Jul- 2019].

Optimizing cloud firestore reads

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizing cloud firestore reads

Similar to Optimizing cloud firestore reads (20)

Recently uploaded

Recently uploaded (20)

Optimizing cloud firestore reads