The progress how we migrate data from Firebase real-time database to PostgreSQL

1. Case Study:
Seamless database migration - from
Firebase real-time database to
PostgreSQL
8th Oct, 2020
Software Development Manager
Pin-Ying Tu
spirit@inline.tw

2. / 23
About
• High-scale software engineer
• US$1M+ spent on cloud
• Hiring Manager :-)
• Ph.D. NTUT
• 8 years software engineering
experience
• inline
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 1
github.com/dbi1463

3. / 23
inline services 1/2
• Start from automated queueing
– Realtime status update
• Booking management
– Multiple sources
– Full capcaity management
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 2

4. / 23
inline services 2/2
• CRM
– Actionable insights
– Campaign SMS
– Survey and coupon
• Takeout and delivery and more
– Menu management
– Integration with multiple delivery
partners
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 3

5. / 23
We are growing, quickly
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 4

6. / 23
Why did we choose Firebase real-time database?
• Less backend work
• Real-time data synchronization is a very important feature
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 5

7. / 23
Sharding for growth
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
main
Shard 1 Shard 2 Cust 1
Cust 2 Cust 3 Cust 4
Without our own API
server, clients need to be
aware of which shard to
connect
Moving data around
shards takes time and
money
Shards got slow again
when data growth
Script to move data from
master to the new shard
and deploy functions to
each shard
6

8. / 23
New synchronization mechanism 1/2
• Minimize memory usage in client-side endpoints
– Firebase iOS SDK consume huge memory if off-line mode is enabled
– Firebase iOS SDK read/write performance become very bad when
the network environment is not good
• We want a faster and smaller memory footprint sync method
– Data synchronization between multiple devices
• 95% reservations should be synchronized in 5 seconds
– Support off-line first
• Still provide good user experience (do not block user operations)
• Can not lose any reservation even the network environment is not good
–Seamless
• No service down time
• The restaurants are not aware of the changes
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 7

9. / 23
New synchronization mechanism 2/2
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
Persistent,
ordered, and
parallel requests
Three long polling
APIs for better UX
experience
Trigger cloud
functions
Publish a sync
event to PubSub
Background
jobs receive
sync events
Background jobs handle the
out of order issue and write
data to Elasticsearch
Background jobs touch
Redis to notify hooked
long polling requests
Monitor the pending
writes requests
Monitor the cost distribution
Unsync reporter
8

10. / 23
Decouple Firebase SDK, finally
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
+request(in path) : InvokableRestRequest
RestClientImpl
+request(in path) : InvokableRestRequest
<<interface>>
RestClient
+query(in name , in value)
+where(in name, in value)
+get(in handler : ResponseHandler)
+post(in handler : ResponseHandler)
+put(in handler : ResponseHandler)
+delete(in handler : ResponseHandler)
+getPlainText(in handler : ResponseHandler)
+invoke(in method : string, in body : object, in handler : ResponseHandler)
InvokableRestRequestImpl
InvokableRestRequest
+create(in reservation)
MutableReservationFirebaseWebService
<<interface>>
MutableReservationWebService
+create(in reservation)
MutableReservationRestfulWebService
+get() : MutableReservationWebServiceProvider
MutableReservationWebServiceProvider
+add(in request : SerializableRequest)
-persist(in request : SerializableRequest)
-remove(in request : SerializableRequest)
+runNext()
SerializableRestRequestQueue
+invoke()
+init()
-id : string
-timestamp : long
-completedURL : string
-httpHeaders : [string: string]
-body : object
-handler : ResponseHandler
SerializableRequest
Codeable
Only serialize the write operations, i.e., POST, PUT, and
DELETE
The requests must be
executed in order
Move the original Firebase
implementation to here
9

11. / 23
Performance improvement
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
Off-line to online Bad network
Create and edit n reservations
and wait all reservations were
synchronized to all devices
X: time
Y: pending writes per second
Blue: Firebase SDK
Red: Firebase REST API
Normal network
10

12. / 23
The device sync dashboard
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
Device-to-device cost by location Upload latency by location
Device-to-device cost by day
11

13. / 23
So far so good, but we want better
• Scaling goals we want to achieve
– 99.99% availability of customer-facing services, with:
– Fast and flexible queries
• Like other database, without index the query is very slow
• Firebase real-time database query only supports one index
• Need to build the indices programmatically by ourselves
– Fully managed DB shard and persistence migrations
• Firebase team migrate the instances without any notification
• We can not schedule the migration
• During the time to rebuild in-memory indices after migration, all accesses to
Firebase database are queued
– Able to scale workers horizontally in one AZ and across Azs
• Firebase cloud functions have 1,000 deployment quota
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
As a result, we decide to migrate our main database.
12

14. / 23
Architecture Evolution
• Current architecture
• New architecture
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
Persistent,
ordered, and
parallel requests
inline APIs write to PostgreSQL and
touch Redis to notify the subscribed
WebSocket handlers
Devices receive events through
WebSocket with some APIs to
improve UX experience
Background jobs to perform
the functionalities of the
original cloud functions
13

15. / 23
Phase 0 – Performance evaluation
• Make sure PostgreSQL can meet our performance goals
– 50 M data set
– Maximum query time
• Current 36ms avg
Goal: 200 rows in 30 ms
– Queries per second
• Current: 117/sec sustained
Goal (5x): 585/sec sustained
– Inserts/Updates per second
• Current: 40/sec sustained
Goal (5x): 200/sec sustained
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 14
Connection
pool

16. / 23
Phase 1 – Sync new reservations to PostgreSQL
• Not breaking any existing system
– Still a few restaurants use the version before long polling
– Not every restaurant upgrade app frequently
• Use feature toggle to control the switch between two methods
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 15

17. / 23
Phase 2 – Make sure WebSocket works
• Use feature toggle (remote control) to switch between
WebSocket and long polling
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 16

18. / 23
Phase 3 – Copy old reservations to PostgreSQL
• Controllable copy scopes
– By brand or by shard
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 17

19. / 23
Phase 4 – Move cloud functions
• Totally 31 cloud functions moved
– Use feature toggle to enable/disable the cloud functions
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 18

20. / 23
Phase 5 – inline APIs
• Implement the new inline APIs and clients adopt APIs
– Also write back to Firebase
• Fully tested with all combination on beta
– Firebase REST API
– inline API
– WebSocket
– Long polling
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 19

21. / 23
Phase 6 – controlled roll out
• Planned roll out
– Start to migrate a company that is willing to try new method
– Start to migrate shard by shard until all shards migrated
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 20

22. / 23
Phase 7 - Remove Firebase dependency
• After migrate all shards and companies, trigger the force
upgrade notification
• Remove Firebase dependency from inline APIs
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 21

23. / 23
Lession learned 1/2
• Abstraction layer
– Try to minimize vendor lock-in for fundamentals of your system
– Even if it costs more in time during the first implementation
• Cost estimation
– Don't forget to turn off the load generator
– Set up a spending rate monitor
• Planning is important
– Help design thinking
• Testing is important
– Unit testing, end-to-end testing, and load testing all help us not
breaking anything important to our customers depend on
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL 22

24. / 23
Lession learned 2/2
• Always contact the cloud service support team
– They have more than the public information and wouldn’t let you
know if you did not ask
• Choose cloud service platform carefully
– Do some evaluation for your needs before choosing a cloud service
platform
• Data synchronization is extreme difficult
– Do not build your own synchronization unless the current solution
can not meet your needs
Case Study: Seamless database migration - from Firebase real-time database to PostgreSQL
Any question?
23