The document discusses using Apache NiFi to improve logging capabilities at scale for a Security Operations Center (SOC). It provides an agenda that includes an introduction to the speaker, context on the SOC's logging needs, an overview of Apache NiFi, and a demonstration. The SOC generates terabytes of logging data per day across diverse tools and cloud services. It evaluated tools to better collect, process, enrich, and analyze logs. Apache NiFi was selected for its ability to flexibly route and transform data, integrate with existing systems, query diverse sources, scale horizontally, and provide security and data provenance.

1. Logging at scale
Doing more with less
Presented by Andre Fucs de Miranda

2. Macquarie GovernmentMacquarie Government
1. A bit about me
2. A bit of context
3. Apache NiFi and the SOC
4. Demo
5. Questions
The agenda.
| Logging at scale – Doing more with less
2

3. Macquarie Government
A bit
about me.
| Logging at scale – Doing more with less

4. Macquarie Government
Manager @ Macquarie’s Security
Operations Center
20 years working in information cyber
security
Apache NiFi committer and PMC
member
| Logging at scale – Doing more with less
https://github.com/trixpan
https://twitter.com/trixpan
About me

5. Macquarie Government
A bit
of context
| Logging at scale – Doing more with less

6. Macquarie Government
A bit of context
About Macquarie Government
• 42% of Australian Government agencies are our customers
• 3+ billion events per day;
Our tool stack is diverse and busy:
• We generate TBs of data per day.
• Since 2015 we have been using “Big Data” (i.e. Hadoop ecosystem) for reporting and
analytics.
• We are constantly looking for ways to offer our customers with better insights over the
threats targeting them.
• We also felt that relying exclusively on traditional SIEM wasn’t enough anymore.
| Logging at scale – Doing more with less
6

7. Macquarie Government
A bit of context
Could we leverage “big data” solutions to improve our SOC further?
• Perhaps we could rationalise the way we collect and process log messages?
• Perhaps we could do enrichment against a more diverse set of sources??
• What else?
| Logging at scale – Doing more with less
7

8. Macquarie Government
So we went and evaluated lots of
tools and architectures looking to
map things like:
• Ability to integrate with SIEM pipelines
natively
• Ability to consume cloud services (IaaS,
PaaS and Saas)
• Ability to query odd stuff
• Inbuilt Security
• Ability to Scale out
• How easy to maintain and extend
| Logging at scale – Doing more with less
8
and many more…
A bit of context
All Apache project logos are trademarks of the ASF and the respective projects.
Logstash is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.
fluentd is trademark by Treasure Data

9. And the winner was…

10. Macquarie Government
Sorry, there was a mistake…
| Logging at scale – Doing more with less
10
All Apache project logos are trademarks of the ASF and the respective projects.

11. Macquarie Government
Let’s talk tech.
| Logging at scale – Doing more with less
Apache NiFi and the SOC

12. Macquarie Government
A bit about Apache NiFi – A brief Prologue
When you are start shipping “data” seems like an “easy” task
| Logging at scale – Doing more with less
DC1
DB
12

13. Macquarie Government
A bit about Apache NiFi – A brief Prologue
But as the environment grows, complexity compounds…
…but you keep adjusting your environment
| Logging at scale – Doing more with less
DC1
DB
DC2
DB
HQ
ClientX
AZ1
AZ2
13

14. Macquarie Government
‘til the point you suddenly
realise your pipeline is
missing a bit of cheese.
Or worse…
| Logging at scale – Doing more with less
© Luca Nebuloni
https://www.flickr.com/photos/nebulux/10708289086/
14

15. Macquarie Government
A bit about Apache NiFi – A brief Prologue
| Logging at scale – Doing more with less
Source: https://goo.gl/xKoavI
15

16. Macquarie Government
“Apache NiFi supports powerful and
scalable directed graphs of data
routing, transformation, and system
mediation logic.”
Open sourced by the National
Security Agency in 2014[1] and
submitted to The Apache Software
Foundation for on-going stewardship
[1] https://goo.gl/aZxCIC
| Logging at scale – Doing more with less
• User friendly interface
• Flexible
• Data Agnostic
• Inbuilt mechanisms to balance
between latency and throughput
• Fine grain control of delivery
guarantees (e.g. discard a flowfile once
it becomes too old to be relevant).
• “Secure”
• Data provenance (from where, to
where, changed by, etc.)
• Authorization Policies, TLS, Kerberos,
Encryption and a handful of other
features
• Designed for Extension
A bit about Apache NiFi16

17. Macquarie Government
A bit about Apache NiFi
NiFi allows you easily move data between A and B (and B to A) in a controlled,
secure and reliable way, while still allowing you to process and granularly
apply logic to the data in motion.
| Logging at scale – Doing more with less
17

18. Macquarie Government
A bit about Apache NiFi
A few examples on how NiFi capabilities help a SOC:
• Rationalising the flows of data into your SIEM
• Do you truly need your SIEM to be ingesting all your logs?
• What happens when you run more than one SIEM (because it may well happen…)?
• Enrich data against a diverse range of sources
• ElasticSearch, REST APIs, DNS, Redis, Whois, GeoIP, SQL, MISP (via HTTP)
• (Pull|push) data (from|to) a diverse set of platforms
• Object based stores such as GCS or S3, FTP, SFTP, Mainframes via WebSphere MQ, Files,
SQL and Syslog of course.
| Logging at scale – Doing more with less
18

19. Macquarie Government
Let’s take a
closer look.
| Logging at scale – Doing more with less

20. Macquarie Government | Logging at scale – Doing more with less
DEMO Time

21. Macquarie Government
Let’s talk.
| Logging at scale – Doing more with less
Andre Fucs de Miranda
Macquarie Government
amiranda@macquariegovernment.com
1800 004 943

22. Thank you.