The document discusses how IBM QRadar collects and processes security data such as events and network flows. It describes the key components involved in data collection, normalization, storage, correlation and generation of offenses. Data flows through event collectors, event processors, databases and other components, and is correlated using custom rules to detect anomalies and security threats. The document provides an overview of QRadar's architecture and data flow.

1. IBM Security
QRadar SIEM Foundations

2. How QRadar SIEM Collects
Security Data

3. 3 IBM Security
QRadar Data Flow - Overall

4. 4 IBM Security
From an Appliance Perspective
Event Collector Capabilities
Event Collector
(ecs-ec-ingress)
Event Collector
(ecs-ec)

5. 5 IBM Security
From an Appliance Perspective
Event/Flow Processor Capabilities
Event Collector
(ecs-ec-ingress)
+
Event Collector
(ecs-ec)
Event Processor
(ecs-ep)

6. 6 IBM Security
From an Appliance Perspective
AIO/Console Capabilities
Event Collector
(ecs-ec-ingress)
+
Event Collector
(ecs-ec)
Event Processor
(ecs-ep)
Magistrate

7. 7 IBM Security
High-level component architecture and data stores
• Flow and event data is stored in the Ariel
database on the event processors
̶ If accumulation is required, accumulated data is
stored in Ariel accumulation data tables
̶ As soon as data is stored, it cannot be changed
(tamper proof)
̶ Data can be selectively indexed
• Offenses, assets, and identity information are
stored in the master PostgreSQL database on
the Console
̶ Provides one master database with copies on each
processor for backup and automatic restore
• Secure SSH communication between appliances
in a distributed environment is supported
Console services
User interface
Magistrate
Reporting
Event processor
Flow collector Event collector
Identities
Assets
Offenses
Configuration
Flows
Events
Accumulations
Network packet
interface, sFlow,
and 3rd party
Events from log
sources
7

8. 8 IBM Security
Event and Flow Collection

9. 9 IBM Security
QRadar Data Flow - Overall

10. 10 IBM Security
Collecting and Normalizing raw events
• An event is a record from a device that describes an action on a network or host
• QRadar SIEM normalizes the varied information found in raw events
̶ Normalizing means to map information to common field names, for example
• SRC_IP, Source, IP, and others are normalized to Source IP
• user_name, username, login, and others are normalized to User
̶ Normalized events are mapped to high-level and low-level categories to facilitate
further processing
• After raw events are normalized, it is easy to search, report, and cross-correlate these
normalized events

11. 11 IBM Security
Event data pipeline
Event
Data
Protocols
Throttle
Filter
Licensing
Event
Collector –
Ingress (ecs-
ec-ingress)
Event
Collector
(ecs-ec)
Event data is sent to or pulled by QRadar
Event Collector Ingress – Responsible for collecting data at
all times (zero event loss)
Data is collected and buffered during patch and deploys and
processed once the operation is complete
Protocols – Reads or pulls raw data from network devices
(e.g: Windows Servers, Firewalls, etc)
Throttle Filter - Licensing - On a second-by-second
basis, slows down the incoming rate so it does not
exceed the license on the appliance.
Events are sent to ecs-ec-parse to be parsed

12. 12 IBM Security
Event data pipeline
Qflow
Parsing
(DSM, LSX,
CEP)
Coalescing
Forwarding
Log
Only/Data
Sore
Event
Collector
(ecs-ec)
Event
Processor
Event data is received from the ecs-ec-ingress
Parsing – DSMs / LSX / CEP – take the raw data and
normalize it into a common structure.
Coalescing - “Event Compression”. Find nearly
identical events and delete one and increase the event
count on the record. Key is: source IP, dest IP, dest
port, QID, username
Forwarding - Applies routing rules for the system, such
as sending event data to offsite targets, external Syslog
systems, JSON systems, and other SIEMs.
Log Only/Data Store supports the storage of an
unlimited number of logs without counting against the
EPS License
Events are then sent to the Event Processor
component and pass through the Custom Rules Engine
(CRE). They are tested and correlated against the rules
that are configured
Event
Collector –
Ingress (ecs-
ec-ingress)

13. 13 IBM Security
Events not counted against the EPS licenses
- The list of log source types that do not incur EPS hits are as follows:
- System Notification
- CRE
- SIM Audit
- Anomaly Detection Engine
- Asset Profiler
- Search Results from scheduled searches
- Health Metrics
- Risk Manager questions, Simulations and internal logging
- Log Only/Data Store
- Supports the storage of an unlimited number of logs without counting against the
EPS QRadar SIEM license
- Enables an organization to build custom apps and reports based on this stored
data to gain deeper insights into IT environments.

14. 14 IBM Security
Event Coalescing
- Event Coalescing is a method of reducing the data going through the pipeline.
- As data arrives in the pipeline QRadar will attempt to group like events together into a
single event.
- Coalescing occurs after licensing and parsing
- Coalescing is indexed by Log Source, QID, Source IP, Destination IP, Destination Port
and Username.
- If more than 4 events arrive within a 10 second window with these properties being
identical any additional events beyond the 4th will be collapsed together.
- Coalesced events can be identified by looking at the Event Count column in the log
viewer, if the Event Count is >1 the event has been coalesced.
- Coalescing can be turned on or off per log source or by changing the default setting in
the system setting page.

15. 15 IBM Security
QRadar Data Flow - Overall

16. 16 IBM Security
Flow collection and processing
• A flow is a communication session between two hosts
• QFlow Collectors read packets from the wire or receive flows from other devices
• QFlow Collectors convert all gathered network data to flow records similar normalized
events; they include such details as:
̶ when, who, how much, protocols, and options.

17. 17 IBM Security
Flow pipeline
Flows
Qflow
Asymmetric
Recombination
De-Duplication
Flow Governor
(Licensing)
CFP Parsing
Flow
Forwarding
Event
Collector
Event
Processor
The QFlow component collects and creates flow information from
internal and external flow sources
Event Collector – Responsible for parsing and normalizing incoming
flows
Asymmetric recombination - Responsible for combining two sides of
each flow when data is provided asymmetrically
Deduplication - Flow deduplication is a process that removes
duplicate flows when multiple Flow Collectors provide data
to Flow Processors appliances.
Flow Governor - Monitors the number of incoming flows to the
system to manage input queues and licensing.
Custom flow properties – extracts any properties defined in the
Custom Flow Properties
Forwarding - Applies routing rules for the system, such as
sending flow data to offsite targets, external Syslog systems,
JSON systems, and other SIEMs.
Flows are then sent to the Event Processor component and
pass through the Custom Rules Engine (CRE). They are tested
and correlated against the rules that are configured

18. 18 IBM Security
QRadar Data Flow - Overall

19. 19 IBM Security
Event & Flow Correlation and Processing
Event
Collector
Licensing
CRE
Storage and
Indexing
Host Profiling
Real Time
streaming
Event
Processor
Magistrate
Asset Profiler
Ariel
Licensing is applied again on ingress to the EP
After Events and Flows are normalized they are
then sent to the Event Processor for processing
The CRE or Custom Rules Engine Applies the
correlation rules that were created in the UI.
Flow data is then sent to the Ariel Database
for storage.
Host Profiling – Also called passive profiling or
passive scanning. Watches flows on the network
in order to make educated guesses about which
IPs/assets exist and what ports are open.
Streaming – Responsible for the “real time
(streaming)” view in User Interface
If an event matches a rule, the Magistrate
component generates the response that is
configure in the custom rule

20. 20 IBM Security
QRadar Data Flow - Overall

21. 21 IBM Security
Magistrate
• The Magistrate creates and stores offenses in the PostgreSQL database; these offenses
are then brought to the analyst’s attention in the interface
• The Magistrate instructs the Ariel Proxy Server to gather information about all events
and flows that triggered the creation of an offense
• The Vulnerability Information Server (VIS) creates new assets or adds open ports to
existing assets based on information from the EPs
• The Anomaly Detection Engine (ADE) searches the Accumulator databases for
anomalies, which are then used for offense evaluation

22. 22 IBM Security
Ariel Components
Ariel
Ariel
Ariel
Event
Processor
Accumulator
Historical
Correlation
Ariel Proxy Server Ariel Query Server
Offline Forwarder
Report Runner

23. 23 IBM Security
Asset and Vulnerability Flow
Ariel
ecs-ep
Event Processor
Scanners/QVM/3rd
Party
ecs-ec
(event collector
Asset Profiler
vis
(Vulnerability
integration service)
Identity Data
Passive
Profiling
POSTGRES

24. 24 IBM Security
Active scanners
For vulnerability assessment (VA) and maintaining asset profiles, QRadar SIEM can
also integrate with many active scanners
• You can schedule Nessus, Nmap, and IBM Security QRadar Vulnerability Manager
scanner directly in QRadar SIEM
• For other scanners, you schedule only the collection of scan results in QRadar SIEM
but not the scan itself

25. 25 IBM Security
Gathering asset information
Active scanners
QRadar Vulnerability Manager scanner,
Nessus, Nmap, Qualys, and others
Provide:
• List of hosts with risks and potential
vulnerabilities
• IP and MAC addresses
• Open ports
• Services and versions
• Operating system
Pros
• Detailed host information
• Policy and compliance information
Cons
• Out of date quickly
• Full network scans can take weeks
• Active scanners cannot scan past
firewalls
• User can hide from active scans
Passive detection
Flows from QFlow, or other flow
sources in accounting technologies
such as IPFIX/NetFlow, sFlow, and
others
Provide:
• IP addresses in use
• Open ports in use
Pros
• Real-time asset profile updates
• Firewalls have no impact
• End system cannot hide
• Policy and compliance information
Cons
• Not as detailed as active scans
• Does not detect installed but unused
services or ports

26. 26 IBM Security
Hostcontext
Reporting Executor
Report Runner
Tomcat
“Owns” the host it is responsible for starting and stopping
processes and for overall system health and backups.
A stopwatch responsible for keeping track of reports and
when they should run and then instantiating the report
runner
The process that actually generates the reports, querying
postgres, Ariel, etc..
Process that drives our web UI and serves up web pages.
Historical Correlation
Processor
Process that is responsible for historical correlation. Runs a
specified search, runs the results through CRE rules (based
on QRadar time or device time) and generates offenses
The Remainder

27. 27 IBM Security
QRadar Data Flow - Overall

28. 28 IBM Security
Dissecting the flow of a captured event (Example)
© COPYRIGHT IBM CORPORATION 2017
Event processor
Event collector
No
Overflow filter
(enforce license limit)
License
exceeded?
Buffer overflow events
and feed back into stream
when input below limit
DSM normalization filter
Device Support Module (DSM)
Parser for firewall
Yes
Traffic Analysis
(Log source discovery)
Log source
known?
Create new
log source
No
Yes
Coalescing Filter
FW Deny event
FW Deny event
FW Deny events
1
2
3
4
5
28

29. 29 IBM Security
Firewall Deny
Event
Firewall Deny
Event
Dissecting the flow of a captured event (continued)
© COPYRIGHT IBM CORPORATION 2017
Console
Event processor
No
Event collectors
Overflow filter
(enforce license limit)
License
exceeded?
Buffer overflow events
and feed back into stream
when input below limit
Yes
No
Event Storage
Streaming to Log Activity tab in real time
Ariel DB
Flows
Events
Accumulator
Ariel DB Accumulation
s
Host Profiler
6
Custom Rule Engine (CRE)
New host or
port found?
Yes
Normalized events
Rule Processing and
Correlation
1
2
3
5
4
29

30. 30 IBM Security
Dissecting the flow of a captured event (continued)
© COPYRIGHT IBM CORPORATION 2017
Console
No
Event processors
Overflow filter
(enforce license limit)
License
exceeded?
Buffer overflow events
and feed back into stream
when input below limit
Yes
Ariel DB
Flows
Events
Accumulator
Processed events
Magistrate
Custom Rule Engine
(CRE)
Offenses
(PostgreSQL)
Ariel Proxy
Ariel Query
Server
Anomaly Detection
Engine
Vulnerability
Information Server
Host Profiler
Assets
(PostgreSQL)
1
2
3
4 5 6
30

31. ibm.com/security
securityintelligence.com
xforce.ibmcloud.com
@ibmsecurity
youtube/user/ibmsecuritysolutions
© Copyright IBM Corporation 2016. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. Any statement of direction represents IBM's current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and other IBM products
and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service
marks of others.
Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your
enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others.
No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems,
products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products
or services to be most effective. IBM does not warrant that any systems, products or services are immune from, or will make your enterprise immune from, the malicious or illegal conduct of any party.
FOLLOW US ON:
THANK YOU