Pavel Odintsov introduces FastNetMon DDoS prevention and how they migrated analytics to ClickHouse to handle large data volumes. Pavel is CTO and co-founder of FastNetMon LTD. FastNetMon: https://fastnetmon.com/ Meetup: https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/282872933/

1. FastNetMonandMetrics

2. Hello
2
I’m Pavel Odintsov, the author of FastNetMon: https://fastnetmon.com
Ways to contact me:
● linkedin.com/in/podintsov
● github.com/pavel-odintsov
● twitter.com/odintsov_pavel
● IRC, Libera Chat, pavel_odintsov
● pavel@fastnetmon.com

3. What is FastNetMon
3
● DDoS detection Tool
● Amazon AWS VPC Flow Analytics
● Google Compute VPC Flow Analytics
● BGP Blackhole Automation Tool
● BGP Automation Tool
● Traffic Visibility Tool
● Network Automation Tool
● Network Traffic Engineering Tool

4. How It Works
4

5. How It Looks
5

6. FastNetMon: Detection Logic
6
Detection type:
• Threshold based (based on host’s average traffic)
THRESHOLD TYPES:
• USING TOTAL TRAFFIC
• USING TOTAL PPS RATE
• PER PROTOCOL

7. FastNetMon: Tra ic Reports in Grafana
7

8. FastNetMon: More Tra ic Reports in Grafana
8

9. What is the
problem?
9

10. Our Challenges with Metrics
10
● Cardinality: medium sized network has ~1m IPv4 hosts with 32
metrics each
● Value range from 0 to UINT64_MAX: traffic in mbit / s or packet /s
● 1s precision
● Insert in very large batches
● Customers love top-k queries

11. Graphite As Storage for Metrics, 2015
11
● “.” as delimiter. IPs and prefixes look ugly: 10_1_2_3 , 10_1_2_3_24
● Limited by performance of single CPU core
● Disk space hungry datastore format
● Whisper is simple and easy to implement
● Graphite is not well maintained and broken in recent Debian /
Ubuntu

12. Graphite Web: World Before Grafana
12

13. Graphite: Carbon-cache, 200k/s
13

14. Graphite: GoCarbon, 200k/s
14

15. InfluxDB As Storage for Metrics, 2016
15
● Allows “.” in metric names
● Pretty compact datastore format
● Automated retention
● Native Grafana support
● Can use multiple CPU cores
● Easy installation, just single binary
● May need tens of minutes for loading with large database
● Uses lots of memory
● Top-k query is extremely slow
● Does not scale after 2m metrics per second
● Queries over few days of data are very slow

16. ClickHouse As Storage for Metrics, 2022
16
● Allows “.” in metric names
● Pretty compact datastore format
● Automated retention
● Plugin for Grafana
● Can use multiple CPU cores
● Requires SSE 4.2 :(
● Top-k query is pretty fast
● Supports unlimited cardinality
● Queries over few days of data can be finished in reasonable time
● Ability to store flows!

17. Clickhouse vs InfluxDB
17
InfluxDB ClickHouse
Cardinality < 1m of unique series Battle tested with 16m+ unique series
Metrics / s < 1m per second 10m+ per second
Top-k performance Extremely slow Good
Data format Inefficient, text Very Efficient, binary
Query syntax Counterintuitive Well known SQL
Multi CPU support Limited Brilliant, scales linearly
Grafana Native Plugin based

18. Roadblock with CLickHouse
18

19. ClickHouse as Flow Storage
19

20. Bu ering problem
20

21. Can We Retrieve Flows?
21

22. Can We Show Flows?
22

23. FastNetMon: our community
23
● Site: https://fastnetmon.com
● GitHub: https://github.com/pavel-odintsov/fastnetmon
● IRC: #fastnetmon at Libra Chat
● Telegram: https://t.me/fastnetmon
● Slack: http://bit.ly/2o5Idx8
● LinkedIN: https://www.linkedin.com/company/fastnetmon/
● Facebook: https://www.facebook.com/fastnetmon/
● WhatsApp:
https://chat.whatsapp.com/JjwF855pwZvIIasTUsZ7EO

24. THANKS!
ANY QUESTIONS?
You can find me at:
⬥ @odintsov_pavel
⬥ pavel@fastnetmon.com
⬥ linkedin.com/in/podintsov
24