The document discusses the work of Alexandr Dubovikov, CTO and co-founder of QXIP BV, focusing on developments in VoIP/RTC monitoring technologies and the challenges in handling large volumes of telecom data with ClickHouse's replacingmergetree engine. It outlines the practical implementation of this engine, including versioning and partitioning strategies to improve performance in real-time communications. The conclusion emphasizes that the replacingmergetree engine is production-ready when configured correctly.

2. Hello There
Alexandr Dubovikov
- CTO and Co-Founder @ QXIP BV w/ Lorenzo Mangani (CEO)
- Chief ClickHouse Specialist (HEPIC, QRYN, PROMCASA, etc)
- OpenSource enthusiast, developer in many projects like Kamailio, OpenSIPs, Freeswitch,
Asterisk, Sipp. Creator of Homer, Sipgrep, Captagent and many more. In love with hacking,
communication, brainstorming, BigData analysis….
QXIP BV
- Based in Amsterdam (NL) and Valencia (ES) with Remote Devs/Ops teams over 10 countries
- Research & Development of OSS & Commercial VoIP/RTC Monitoring Technologies
- Running a Healthy, Self-Sustainable OSS Business Model w/ great Technical Partnerships
- Worldwide Customer base ranging from Tiny Startups up to massive Fortune Corporations
- Experienced making OSS patches/plugins/extensions for many popular projects and platforms
- Clickhouse database centric development since around 2017 (and never looked back)
- Altinity Support Partnership providing advanced ClickHouse support our customer base
https://qxip.net

3. Warning: Experimental Content

4. ReplacingMergeTree in Telcom Apps

5. ReplacingMergeTree in Telecom Apps
Telecom Challenge
- Millions of telecom actors (voip phones, mobiles, webrtc browsers, servers, bots, etc)
- Millions of network packets each second across globally distributed infrastructure
- Millions of session status changes each second (calling, ringing, connecting, failing, etc)
ClickHouse Challenge (OLAP)
- UPDATE requires lookup for an existing key resulting in bad performance
- No transactions for best performance, just limited workarounds such as “RMT”
- Duplication (versioning) of {n} data copies during updates
- Using FINAL to avoid duplications = bad performance.

6. Example of ReplacingMergeTree
Practical Example:
CREATE TABLE replace_merge (t DateTime, u Int64, id Int32, value String)
ENGINE = ReplacingMergeTree(u)
PARTITION BY toYYYYMMDD(t)
ORDER BY (id, t);
- u: timestamp in milliseconds for versioning
- t: datetime of “transaction”
- id: transaction id (i.e. user id)
- value: some internal information related to the transaction.
- ReplacingMergeTree(u) set the “versioning” key
- toYYYYMMDD(t) create daily partition based on “datetime” of transaction
- ORDER BY (id, t) make an unique key base on datetime of transaction and transaction id
CREATE TABLE replace_merge (t DateTime, u Int64, id Int32, value String)
ENGINE = ReplacingMergeTree(u)
PARTITION BY toYYYYMMDD(t)
ORDER BY (id, t);

7. ● Telecom and Real Time communications sessions continuously changes states
● Use ReplacingMergeTree Engine to “update” existing records based on UUID and the timestamp
● The timestamp field is used as “versioning” to determine the latest transaction/state
● Developers need to handle “transactions” and update the records each time the status changes.
● To avoid duplications use FINAL, however all table data is merged = very expensive
● PREWHERE cannot be used as per regular MergeTree Engine
ReplacingMergeTree in Telecom Apps

8. ● Use >= 22.6 to leverage parallel FINAL merges. Some tests show that UNION ALL queries on
ReplaceMergeTree shows 500% better performance (great job Team ClickHouse!)
● Use OPTIMIZE table [table_name] PARTITION ‘PARTITION_ID’ FINAL
● SET do_not_merge_across_partitions_select_final
● New feature:
SETTINGS min_age_to_force_merge_seconds and min_age_to_force_merge_on_partition_only
Avoid manual OPTIMIZE!
ReplacingMergeTree in Telecom Apps

10. Materialize It
We can use materialized views on ReplacingMergeTree tables to react on status changes.
Usecase Examples:
● Track Usage / Fraud Detection helpers
● Alerting on any condition or parameter
● Write custom formatted results to Remote APIs using URL Engine

11. Conclusion
ReplacingMergeTree Engine is production ready with the proper knowledge & settings!

12. 👋 blog.qryn.dev
Check out our blog, where we talk about our project and all of our experiments and hacks weekly

13. info@qxip.net
https://qryn.dev https://metrico.in
K qryn + clickhouse + flux
Logs Metrics Traces