The document discusses using OLTP systems at high scale for inventory management in Flipkart's supply chain. It describes how Flipkart uses: - A high throughput, high concurrency inventory management system to handle reservations for orders - Data encoding and custom transaction processing logic in the data store to improve concurrency - Transaction buffering and slave reduction to work around performance limitations of the data store It also discusses how Flipkart uses the Tungsten replication tool to enable real-time data propagation from MySQL databases to Vertica for operational dashboards, overcoming challenges like schema changes, transaction rollbacks, and operational overhead during database switches.

1. OLTP in Supply Chain
Vijay Babu
Rajesh Kannan

2. OLTP in FSG
● From Promise to Customer Doorstep
● Various Microservices and multiple OLTPs
● Two problems
○ High scale OLTP
○ Change propagation

3. Part 1: High Scale OLTP

4. High Scale OLTP
● OLTP system with high throughput and large dataset
● Inventory Management System in Flipkart
● Manages the Inventory view of all 100s of Millions of
Listings
● Handles reservations for each order that is taken in
Flipkart

5. Inventory Management

6. Inventory Management

7. Inventory Management

8. Inventory Management

9. High Scale OLTP
● High throughput
● High concurrency
● Low latency
● Data consistency
● Large dataset

10. High Scale OLTP
● High throughput
● High concurrency
● Low latency
● Data consistency
● Large dataset

11. High Throughput

12. High Scale OLTP
● High throughput
● High concurrency
● Low latency
● Data consistency
● Large dataset

13. Concurrent Transactions
● More users try to reserve same item
● Problem is more prominent at scale during sale events

14. High Scale OLTP
● High throughput
● High concurrency
● Low latency
● Data consistency
● Large dataset

15. High Scale OLTP
● High throughput
● High concurrency
● Low latency
● Data consistency
● Large dataset

16. High Scale OLTP
● High throughput
● High concurrency
● Low latency
● Data consistency
● Large dataset

17. Traditional solutions improvised
● How to scale for higher throughput at low latency?
● Cache the data
○ Data is too dynamic
○ Data size is too huge to cache
○ Use a in-memory store

18. Traditional solutions improvised
● How to scale for a large data size?
● Shard the data
○ Still data size is too big
○ Key value store with encoded data

19. Data store topology

20. Data Encoding
● Space savings to a scale of 5X
● More prominent when there are more attributes
Simple encoding example:
{"quantity":100,"reservations":50}
Encodes to (with delimiter as “::”)
“100::50”

21. Concurrency with encoding
● Encoding/Decoding at App layer leads to concurrency
issues
● Updating the encoded data with partial tokenizing logic
in data store
○ Exploiting data store capabilities for
serializing/deserializing the tokens and operating on
them
○ Lua for redis/aerospike or mysql procedure

22. Concurrency with encoding

23. Unique Challenges
● Scale for BAU, Super-Scale for Sale events
● Hotspot scenario during such events
● High concurrency on a single resource

24. Hotspot scenario
● Very high throughput in a single resource
● Sharding will not distribute the load
● Impact of Network PPS

25. PPS Impact

26. Working with PPS limits
● The impact of PPS can be limited by two approaches
○ Slave reduction
○ Transaction Buffering

27. Working with PPS limits
● The impact of PPS can be limited by two approaches
○ Slave reduction
○ Transaction Buffering

28. Slave reduction

29. Slave reduction
● Isolate read and write scaling
● Precomputed stores for majority of use cases
● Only important reads use the write store
● Pros
○ PPS limit avoided
● Cons
○ Data consistency issues between write and read
stores

30. Working with PPS limits
● The impact of PPS can be limited by two approaches
○ Slave reduction
○ Transaction Buffering

31. Transaction buffering

32. Transaction buffering
● Buffer transactions at App layer
○ Hit data store in batches
○ Buffer for a limited time limit
○ Buffer for a upper bound of size
● Pros
○ PPS limit avoided
● Cons
○ Increased App latencies
○ More threads and memory consumption at app

33. Other Challenges
● Skewed Throughput
● Atomicity at scale

34. ● Traditional strategies - might not suffice once a certain
scale is hit
● Mix of more than one common strategy
● Variation of a traditional strategy
Learnings summary

35. Part 2: Change Propagation
in OLTP

36. What we do
● City Logistics Team handles
○ First & Last leg in Flipkart Supply Chain
○ Delivering shipments to customer
○ Collects returns
● Built on top of multiple microservices backed by mysql

37. Business Problem => Tech problem
● Need for real-time dashboards to
track ground operations
● Achieve real-time replication
against the traditional ETL Job
● Need querying on unified data
view that is obtained from data-
sources of multiple
microservices
● Ability to easily manage schema
changes and not let change
propagation fail due to this
● Isolation of source and
consumer
● Sending notifications based on
the domain data change
● Ability to alter/enrich from
multiple data streams

38. Vertica for operational dashboards
Vertica as a centralized analytical datastore for all
operational needs
○ Columnar data store. Provides high-compression of
data
○ Supports Massively Parallel Processing (MPP) and
scales linearly.

39. Need for Change Propagation System !

40. Options for Change Data
Propagation
Options Message
generation
Relay API Latency IOPS/ IO
overhead
Others
Outbound
store
Sync Async High High Impl. is
relatively easy
Binary Log
replication
Async Async N/A Low More control!
But complex
to impl

41. Tungsten
● https://github.com/vmware/tungsten-replicator
● Provides eventual consistency with exactly-once
delivery
● Native support for DDLs and DMLs
● Replication connectors available for
○ Source(Extractor): MySQL, Oracle, Amazon RDS
○ Target(Applier): Vertica, MySQL, Oracle , HDFS,
other NoSQL and data-warehouse stores.

42. Tungsten - Contd.
● Supports filters (Javascript/java) where transactional
data can be enriched, altered & monitored.
● Supports parallel replication
● Simplicity to use and rich tools for administration
● Has good code documentation, so adding or enhancing
feature is easy
● Active community support

43. Tungsten Architecture
● Master - Slave topology
● Master (Extractor)
○ pulls binlog from source
○ generates THL
● Slave(Applier)
○ pulls THL from
master
○ applies to target
store using JDBC/native
connector.

44. Operational Dashboard

45. Tungsten Adoption: Key challenges

46. Problem - Hard dependency
on mysql master
Context:
Tungsten service persists checkpoints in the source
database.
Problem:
Not a good idea to compromise on master sanctity for this
purpose. Better to use a read-only slave as a source. But
Tungsten does not support replicating from mysql slave
server

47. Problem - Hard dependency
on mysql master - contd
Added the ability to use any remote jdbc store to
maintain checkpointing details.

48. Problem: No bootstrap support for
onboarding
Problem:
No native support for onboarding an existing mysql database to Vertica
Solution:
Automated scripts for
● Create tables in Vertica
● Export mysql data as csv and import to Vertica(issues with zero
dates, bit type)
● Spawn new tungsten master and slave instances.
● Enable live replication from the binlog position
● Data validation between source & target
Now onboarding process requires just 5 mins of manual intervention

49. Problem: Rollback
transactions in binlog
Problem:
● An app uses temporary tables.
● During txn rollback, reverted rows also applied onto
Vertica
● Hence inconsistency in Vertica.

50. Replication during Statement binlog
format - Transaction Commit
Queries executing at Master Inside the binlog Inside the mysql Slave
<<start Txn1>>
Create temp table tmp1
Insert into tmp1 values(...)
Commit
● All queries logged
● Commit
Tmp1 table created with data
<<start Txn2>>
Insert into mainTable1 select * from
tmp1
All queries logged Insertion to mainTable1
Drop Temp table tmp1 All queries logged Dropped tmp1
Commit Commit
TIME

51. Replication during Statement binlog
format - Transaction Rollback
Queries executing at Master Inside the binlog Inside the mysql Slave
<<start Txn1>>
Create temp table tmp1
Insert into tmp1 values(...)
Commit
● All queries logged
● Commit
Tmp1 table created with
data
<<start Txn2>>
Insert into mainTable1 select * from
tmp1
All queries logged Insertion to mainTable1
Drop Temp table tmp1 All queries logged Dropped tmp1
Rollback Rollback Undo mainTable1 inserts
TIME

52. Problem: Rollback transactions in
binlog - Contd
Context:
● Temp tables are connection scoped; Temp tables are replicated in
binlog for STATEMENT format, but not for ROW.
● Binlog formats can be changed on the fly
● Hence Txn containing drop temp table logged for backward
compatibility.
Solution:
1. Lack of support for ROLLBACK statement for Vertica applier.
Implementing rollback at Vertica Applier needs redesigning
2. So avoided the problem by keeping Drop Temp table in a
separate transaction.

53. Problem: Replication breaks
during schema change
Problem:
● Vertica replication fails when a table is created/altered
● Tungsten does not propagate schema change for cross-
platforms.
Solution:
● Added ability for major and minor DDL replication
● Achieved by extracting table metadata from MySQL and
generating SQL
● Supports most of the DDL commands in Vertica

54. Problem - operational overhead
during mysql servers switch
Problem:
● When Mysql slave is down, switching to another is not
trivial because binlog position is localized
● Causes duplicate key error at Vertica
Solution:
● Introduced skip-replication-error flag, enabled during the
switch window
● It overwrites existing data (INSERT=> UPDATE,
UPDATE => double UPDATE)

55. Problem - Jmx metrics
collection
Problem:
● Tungsten publishes replication metrics as MBean
operation
● Flipkart in-house(Cosmos) metrix collector understands
MBean attribute
● Hence no visibility on the system
Solution:
Encapsulated the metrics as MBean attributes

56. Where we are now
Production Environment details:
● 20 Tungster masters reading from source systems
● Replicating 50 Databases
● dataset of size 5TB
● Processing binlogs of 100GB every day
● Processing throughput 10K row change events/sec

57. Legacy Stack migration - Sync Bridge

58. Reference:
Branch contains enhancements & fixes
https://github.fkinternal.com/Flipkart/ekl-tungsten-replicator/tree/cl_changes
Confluence space
https://confluence.fkinternal.com/display/ECLO/Data+Framework

59. Thank You!!