ReversingLabs is on a mission to deliver threat intelligence to their users by providing complete visibility and insight into every destructive object. To deliver on their commitment, they migrated to Scylla to handle thousands of updates per second in their processing engines. In their talk, they will go over their requirements and show how they tuned the system to handle requests from their API frontend.

1. How ReversingLabs
Serves File Reputation
Service for 20B Files
Goran Cvijanovic, Software Architect

2. Presenter
Goran Cvijanovic, Software Architect
Open Source Databases integration and optimization, Oracle &
Microsoft databases specialist, 20+ years of experience in
information technology with more than 15+ years in database
systems integration, migration and tuning.

3. Services
Overview

4. File Reputation
TitaniumCloud
Reputation Services
Powerful threat intelligence
solutions with up-to-date, threat
classification and rich context on
over 20 billion goodware and
malware files
High Availability Database
It’s all about metadata and SLA

5. Pre-Built Threat Connectors & APIs
■ Provides 50+ APIs and feeds
■ Preempts emerging threats
by monitoring malware “in-
the-wild” using threat-
specific feeds including
Ransomware, APT, CVE,
financial, and retail
information sources
■ Supports advanced search
and targeted queries on large
sample datasets

6. In-House
Database

7. In-House DB distributed key-value store
To be able to insert/update and read large amount of data with low
latency and stable response time, we developed our own distributed key-
value store based on LSM Tree architecture.
■ 3B reads / 1B writes per day
■ Latency in DB system < 2ms
● write < 2ms
● read < 1ms
■ Latency for API < 60ms

8. Requirements for Database Independency
To be database independent, we developed database connection library
which can be transparently connected to different databases
Features we needed from Database System:
■ K-V native protobuf format
■ LZ4 Compression for reduced storage size
■ Latency < 2 ms
■ Support record size 1K to 500M
■ High Availability with replication

9. ScyllaDB

10. Scylla setup
Scylla DB v3 with mc storage format tested with different configs
After POC we successfully implemented Scylla cluster with 8 nodes for
our 2 large volume API-s.
■ Concerns
● Binary K-V structure (sharding, how to query)
● Chunk size (compression, storage size)
● Repairs (take long time to finish)

11. Scylla implementation
All concerns successfully resolved
Parameters are tweaked and tested on cluster with production load
■ Concerns resolved
● Binary K-V structure (blobs working fine)
● Chunk size (achieved improvement in compression 49%)
■ 4K for small records
■ 64K for large records
● Repairs (eliminate repairs)
■ Insert/Update with consistency level all
■ Read with consistency level quorum
■ Delete with consistency level all

12. File Reputation
Statistics
Serving threat classification and
rich context on over 20 billion
goodware and malware files from
ScyllaDB cluster
Graph displays
number of requests handled
by hour
req
hours

13. File Reputation
API latency
Performance test on API
includes complete workflow
300 req/sec 32 workers in parallel
■ User request
■ User auth
■ Request validation
■ Query database
■ Format response
■ Sending response
Test duration: 0:05:20
Samples count: 81645, 0.00 % failures
Average times: total 0.120, latency 0.120
Percentiles:
┌───────────────┬───────────────┐
│ Percentile, % │ Resp. Time, s │
├───────────────┼───────────────┤
│ 0.0 │ 0.080 │
│ 50.0 │ 0.118 │
│ 90.0 │ 0.135 │
│ 95.0 │ 0.142 │
│ 99.0 │ 0.166 │
│ 99.9 │ 0.232 │
│ 100.0 │ 0.331 │
└───────────────┴───────────────┘

14. File Reputation
ScyllaDB latency
Average Latency
■ write < 6ms
■ read < 1 ms
99 Percentile Latency
■ write < 12 ms
■ read < 7 ms

15. Lessons Learned
■ How to save some precious NVMe disk space
● Know your data and test compression with different chunk_size
● Can save half of storage used, in our case 49% less
■ Repairs can impact cluster performance
● Insert/Update with consistency level quorum
● Delete with consistency level all
■ Latency on steroids
● Use NVMe disks in RAID-0 configuration

17. Stay in touch
Goran Cvijanovic
goran.cvijanovic@
reversinglabs.com
@gorancv