Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Missing Layers: Syslog Collection & HTTP Event Collection

237 views

Published on

An evening focused discussing the [often] missing layers of event collection within Splunk deployments. We'll cover the ins and outs of traditional syslog collection and also explore how the Splunk HTTP Event Collector can be used to similar effect.

Published in: Data & Analytics
  • Be the first to comment

The Missing Layers: Syslog Collection & HTTP Event Collection

  1. 1. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Splunk User Group Edinburgh
  2. 2. © 2018 SPLUNK INC. Harry McLaren ● Alumnus of Edinburgh Napier (Mentor) ● Senior Security Consultant at ECS ● Leader of the Splunk User Group Edinburgh
  3. 3. © 2018 SPLUNK INC. Introduction to ECS Splunk Partner - UK – Type: Security / IT Operations / Managed Services (SOC / Splunk) – Awards: Splunk Revolution Award & Splunk Partner of the Year
  4. 4. © 2018 SPLUNK INC. Agenda • Housekeeping: Event Overview & House Rules • Syslog Collection with rsyslog • HTTP Event Collection with Splunk • User Group Update & Request for Speakers
  5. 5. © 2018 SPLUNK INC. Splunk [Official] User Group “The overall goal is to create an authentic, ongoing user group experience for our users, where they contribute and get involved” ● Technical Discussions ● Sharing Environment ● Build Trust ● No Sales!
  6. 6. © 2018 SPLUNK INC. Syslog Collection with rsyslog Graeme Curtis
  7. 7. Syslog - Analysis and Collection Graeme Curtis Head of Research & Development, ECS Security
  8. 8. agenda • syslog overview • different syslog flavours • modular syslog configuration • debugging your configuration • packaging as a Splunk app • logfile weeding • architecture
  9. 9. what is syslog? • a shared message logging service originally created on BSD in the 1980’s • logging follows a standard format <34>Oct 11 22:14:15 MYMACHINE su: 'su root' failed for gac on /dev/pts/8 priority timestamp hostname tag message RFC316 4<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - - - 'su root' failed for gac on /dev/pts/8 RFC542 4 priority version timestamp hostname processname pid msgid bom message • actually, many vendors do exactly as they please and often ignore the syslog standards
  10. 10. the right way to ingest syslog • simply open a couple of listening ports TCP/UDP514 • choose a unique port per device type • collect syslog sources on dedicated syslog servers recommendation: 1. always implement standalone syslog servers as part of your Splunk Infrastructure 2. ensure that the configuration of the syslog server is a responsibility of the Splunk team
  11. 11. syslog server options • choose a *nix operating system • two common options for syslog servers recommendation: rsyslog is my preferred choice as it doesn’t implement a dual license model ‘advanced’ features of syslog-ng are only available via a commercial license
  12. 12. creating a modular rsyslog config • rsyslog configurations can be broken down into 4 major components: 1. globals – defining global variables such as queue / message size loading modules such as udp and tcp sockets 2. inputs – describes message input types 3. rules – determines what action to take when a message is received 4. templates – sets the output format of the message
  13. 13. creating a modular rsyslog config • rsyslog will allow the use of include files and hence we can readily modularise each of the components /etc/rsyslog.conf /etc/rsyslog.d/splunk.conf /etc/rsyslog.d/splunk-global /etc/rsyslog.d/splunk-inputs /etc/rsyslog.d/splunk-rules /etc/rsyslog.d/splunk-templates Let’s have a look at how we structure our filesystem and what goes where…
  14. 14. testing your configuration 1 whenever you onboard a syslog source, always keep a representative example of events in the ./splunk-tests directory
  15. 15. testing your configuration 2 remember we mentioned vendors sending non-standard syslog… Nov 2 15:04:45 10.91.254.15 devname=SGPDC22-F3-NFW01 devid=FGT1KD3916801084 vd=root date=2017-11-02 time=15:04:42 logid=0100100032002 type=event subtype=system level=alert vd=root logdesc="Admin login failed" sn=0 user="c" ui=console method=console srcip=0.0.0.0 dstip=0.0.0.0 action=login status=failed reason="name_invalid" msg="Administrator c login failed from console because of invalid user name"
  16. 16. testing your configuration 3 consider the rules… Nov 2 15:04:45 10.91.254.15 devname=SGPDC22-F3-NFW01 devid=FGT1KD3916801084 vd=root date=2017-11-02 time=15:04:42 logid=0100100032002 type=event subtype=system level=alert vd=root logdesc="Admin login failed" sn=0 user="c" ui=console method=console srcip=0.0.0.0 dstip=0.0.0.0 action=login status=failed reason="name_invalid" msg="Administrator c login failed from console because of invalid user name" if $msg contains 'devid=FGHA' then { action(type="omfile" DynaFile="syslog_514_fortigateha" ...) stop } if $msg contains 'devid=FG' then { action(type="omfile" DynaFile="syslog_514_fortigate” ...) stop }
  17. 17. testing your configuration 3 consider the templates… Nov 2 15:04:45 10.91.254.15 devname=SGPDC22-F3-NFW01 devid=FGT1KD3916801084 vd=root date=2017-11-02 time=15:04:42 logid=0100100032002 type=event subtype=system level=alert vd=root logdesc="Admin login failed" sn=0 user="c" ui=console method=console srcip=0.0.0.0 dstip=0.0.0.0 action=login status=failed reason="name_invalid" msg="Administrator c login failed from console because of invalid user name" if $msg contains 'devid=FGHA' then { action(type="omfile" DynaFile="syslog_514_fortigateha" ...) stop } if $msg contains 'devid=FG' then { action(type="omfile" DynaFile="syslog_514_fortigate" ...) stop } template (name="syslog_514_fortigate” … string=”…/fortigate/%msg:R,ERE,1,DFLT:devname=([^ ]+)--end%/%fromhost-ip%.log") template (name="syslog_514_fortigateha” … string=”…/fortigate/%msg:R,ERE,1,DFLT:[ ]vd=([^ ]+)--end%/%fromhost-ip%.log") Some template content snipped for ease of reading
  18. 18. testing your configuration 4 debugging is configured in our standard policy via a global variable… # Set debug status to either true or false set $/debug = "false";
  19. 19. testing your configuration 5 this will redirect message flow from the syslog_514 ruleset to the debug ruleset … # Set debug status to either true or false set $/debug = "false"; ruleset(name="syslog_514") { $RulesetCreateMainQueue on # Create ruleset specific main queue for performance benefit if $/debug == 'true' then { call debug stop }
  20. 20. testing your configuration 6 add a couple of variables to allow you to track what’s happening in the config… # Set debug status to either true or false set $/debug = "false"; ruleset(name="syslog_514") { $RulesetCreateMainQueue on # Create ruleset specific main queue for performance benefit if $/debug == 'true' then { call debug stop } if $msg contains 'devid=FG' then { set $!debugrule="fortigate_rule_002"; set $!debugtemplate="fortigate"; action(type="omfile" file="/var/splunk-syslog/debug/debug.log" template="debug") stop }
  21. 21. testing your configuration 7 finally, we output the message as it’s been parsed by rsyslog into JSON … ruleset(name="syslog_514") { $RulesetCreateMainQueue on # Create ruleset specific main queue for performance benefit if $/debug == 'true' then { call debug stop } if $msg contains 'devid=FG' then { set $!debugrule="fortigate_rule_002"; set $!debugtemplate="fortigate"; action(type="omfile" file="/var/splunk-syslog/debug/debug.log" template="debug") stop } template (name="debug" type="list") { property(name="jsonmesg") constant(value="n") }
  22. 22. logfile weeding remember to clean-up any locally stored logfiles… - can use logrotate - or alternatively, use your templates to create a folder structure containing datetime values and schedule a simply deletion script.
  23. 23. typical recommended architecture • consider using a hardware load balancer • what’s with the universal forwarders?
  24. 24. © 2018 SPLUNK INC. HTTP Event Collection (HEC) with Splunk Harry McLaren
  25. 25. © 2018 SPLUNK INC. HTTP Event Collector (HEC) ▶ A token-based JSON API for events/metrics. ▶ Send events directly from anywhere (servers, mobile devices, IOT, cloud). ▶ Easy to configure / works out of the box. ▶ Easy to secure using tokens. ▶ Highly performant, scalable and available.
  26. 26. © 2018 SPLUNK INC. How To Use ▶ Enable HTTP Event Collector (Splunk Enterprise) ▶ Create a unique token ▶ Send events to Splunk using the token • Use HTTP(S) Directly • Create a POST request and set the auth header • POST JSON in our event format to the collector • Use logging libraries • Support for .NET, Java and JavaScript loggers
  27. 27. © 2018 SPLUNK INC. Sending Data with: JSON Event Example curl -k -H "Authorization: Splunk 12345678-1234- 1234-1234-1234567890AB" https://http-inputs- mysplunkcloud.example.com:8088/services/collector/ event -d '{"sourcetype": "mysourcetype", "event": "http auth ftw!"}'
  28. 28. © 2018 SPLUNK INC. Sending Data with: JSON Metric Example curl -k https://localhost:8088/services/collector -H "Authorization: Splunk b0221cd8-c4b4-465a- 9a3c-273e3a75aa29" -d '{"time": 1486683865.000,"event":"metric","source":"disk","h ost":"host_99","fields":{"region":"us-west- 1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","t eam":"LON","service":"6","service_version":"0","se rvice_environment":"test","path":"/dev/sda1","fsty pe":"ext3","_value":1099511627776,"metric_name":"t otal"}}'
  29. 29. © 2018 SPLUNK INC. Sending Data with: Raw Event Example curl https://http- input.splunkcloud.com/services/collector/raw -H "X-Splunk-Request-Channel: FE0ECFAD-13D5-401B- 847D-77833BD77131" -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d 'Jun 10 12:17:50 payroll proftpd[15997]: payroll.acme.com (172.16.0.16[172.16.0.16]) - USER Administrator: no such user found from 172.16.0.1 [172.16.0.16] to 10.1.1.1:21' -v
  30. 30. © 2018 SPLUNK INC. Tips ▶ Create tokens per app, department, component, service. etc. Not per user or device especially if you are talking about a large number (> 10000). ▶ Consider partitioning tokens to different indexes. This will speed up searches and make it easy to archive. ▶ Consider delegating token management using DevOps / Engineering. ▶ Explicitly set allowed indexes on the token. If not set, the token can send data to any index. ▶ Use HTTP over HTTPS when you can. You can get about a 30% performance gain. ▶ Ask your developers to batch events, it greatly improves throughput.
  31. 31. © 2018 SPLUNK INC. AWS Cloud Based Example Use AWS Lambda with HTTP Event Collector
  32. 32. © 2018 SPLUNK INC. Distributed Deployment HTTP Event Collector can scale to meet many of your needs! • Build into splunkd, nothing special to install • Run directly on the indexer • Or run on a dedicated collector instance (heavy forwarder) and forward to an indexer • Uses Deployment Server to to sync tokens across the Collector instances
  33. 33. © 2018 SPLUNK INC. Scale & High Availability: Scenario 1
  34. 34. © 2018 SPLUNK INC. Scale & High Availability: Scenario 2
  35. 35. © 2018 SPLUNK INC. Scale & High Availability: Scenario 3
  36. 36. © 2018 SPLUNK INC. Mid-Size Deployment
  37. 37. © 2018 SPLUNK INC. Shake IT? http://splunk.com/shake
  38. 38. © 2018 SPLUNK INC. ▶ Introduction to Splunk HTTP Event Collector ▶ Set up and use HTTP Event Collector ▶ HTTP Event Collector Walkthrough ▶ Use AWS Lambda with HTTP Event Collector Resources
  39. 39. © 2018 SPLUNK INC. User Group Update & Request for Speakers Harry McLaren
  40. 40. © 2018 SPLUNK INC. Technical Discussion Community Driven Open to All Experience Levels Varied Topics Splunk User Group, Edinburgh
  41. 41. © 2018 SPLUNK INC. Request for Speakers & Topics Experts Novices
  42. 42. © 2018 SPLUNK INC. Get Involved! ● Splunk User Group Edinburgh – https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html – https://www.linkedin.com/groups/12013212 ● Splunk’s Slack Group – Register via http://splunk-usergroups.signup.team/ – Channel: #edinburgh ● Present & Share at the User Group? Connect: ‣ Harry McLaren | harry.mclaren@ecs.co.uk | @cyberharibu | harrymclaren.co.uk ‣ ECS | enquiries@ecs.co.uk | @ECS_IT | ecs.co.uk
  43. 43. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Thank You

×