Building SIEM: from log collecting to IT security routine automation

1. Large enterprise SIEM:
get ready for oversize
Svetlana/Mona Arkhipova
Qiwi
OWASP Meetup, Moscow, 28 Feb 2015

2. What are we talking about?
• Log collecting != Security Information and
Event Management
• Systems monitoring is not enough
• Logs as a ‘Big Data’
•

3. WTF is qRadar?
Hello IBM!
• Log management
• Network activity/anomaly detection
• SIEM
• Nice API

4. WTF is qRadar?
Administrator’s nightmare:
• Frontend: Java+Tomcat
• Backend: Java daemons
• DB: Ariel for collected+
indexed data, PostgreSQL for ‘static’ data
• Painful performance metrics and load
balancing

5. Architecture

6. To log or not to log
Guides/best practices
• https://www.owasp.org/index.php/Logging_Cheat_
Sheet
• http://www.syslog.org/logged/logging-and-syslog-
best-practices/
• http://sniperforensicstoolkit.squarespace.com/stora
ge/logging/Windows%20Logging%20Cheat%20Shee
t%20v1.1.pdf
• https://zeltser.com/media/docs/security-incident-
log-review-checklist.pdf
• …

7. To log or not to log
Huston, we got a problem:
• Standard syslog message size (RFC 5424)
• Windows security logs permissions on
W7/2008+
• Database audit – what to log?
• Log files on FS (IIS and so on)
• In-house developed apps

8. To log or not to log
Standard sources: Windows
• Event collectors vs. agents
• Extended system audit
• Non-English logs:

9. To log or not to log
Standard sources: *nix, network devices
• Syslog as a standard
• TCP syslog+network issues=pain
(google: “TCP is not reliable”)
• UDP syslog message size
• Auditd – what to log?

10. To log or not to log
Standard sources: Databases
• Is login history enough?
• Syslog vs DB connection

11. To log or not to log
Non-Standard sources:
• Exotic network devices
• In-house developed apps
• 1C (OMG…) and other specific apps
• Integration with other security systems (NGFW,
DBFW, AV, Security scanners…)

12. To log or not to log
When syslog is powerless:
WAF CEF log file

13. Normalizing/indexing
Event at a glance
• Standard properties: timestamp, src IP, dst IP, log
source identifier and so on
• Custom event properties – KISS principle
• No search – no property.
Indexing
• Standard properties – index, index, index!
• Custom event properties indexing: with great
power comes great responsibility…
• BTW, watch your index size.

14. Over(sizing)
Current Qiwi SIEM metrics:
• 1800 log sources
• 10 000 - 24 000 RAW events per second (EPS)
• ~11 600 network flows per second (FPS),
~700 000 flows per minute(FPM)
SIEM system: 39 virtual servers, 2 hardware servers
with Napatech 2x10G cards, 1 archive server

15. Over(sizing)
Expectations (sizing) Reality
vCPU 140 160
vRAM 272 Gb 521 Gb
vHDD 15 TB 61 TB
Once upon a time in a far far galaxy we decided to
build our own SIEM…

16. Online/offline storage
Daily stats:
• 67-145 Gb raw event logs per day
• 37-53 Gb network communication events per
day
• Online storage – fast access (realtime + some
previoius data)
• Offline – archive storage

17. What if...
…EPS or FPM x2 ?

18. Internal security scanners
“Normal paranormal” activity inside and outside.
• Butthurt :(
• Log or drop events?
• Custom rules set for nodes
• Keep an eye on credentials!
• Balancers
NAT/SNAThttps://f5.com/resources/white-
papers/load-balancing-101-nuts-and-bolts

19. Autopilot: ON
• Simple rules
• Chained rules:

20. Autopilot: ON

21. Questions?
Svetlana/Mona Arkhipova
Lead information security expert
QIWI infrastructure security team
mona@qiwi.com
mona.sax m0na_sax