Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Grokking Grok: Monitorama PDX 2015

3,791 views

Published on

Grokking Grok: A Magic Show of Regular Expressions
Monitorama 2015

Published in: Technology

Grokking Grok: Monitorama PDX 2015

  1. 1. Grokking Grok A Magic Show of Regular Expressions @ferggo (Twitter) GregMefford (GitHub)
  2. 2. Key Take-Away: Check Out Logstash
  3. 3. Logstash Does Lots of Things OutputsInputs Filters Codecs
  4. 4. Grok Grok is Magic
  5. 5. Grok is Magic (thanks, @jordansissel ) Grok
  6. 6. https://flic.kr/p/8zAUi6 TransmutationTurning Lead into Gold
  7. 7. TransmutationSpinning Straw into Gold https://flic.kr/p/j4Jg1u
  8. 8. In TheoryFirewalls are simple
  9. 9. https://xkcd.com/730/http://www.startrek.com/database_article/scott there’s Variety (T_T) In The Enterprise™
  10. 10. “Syslog” “Syslog” “Syslog” “Syslog”
  11. 11. https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay Sparkly Unicorn
  12. 12. https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay Sparkly Unicorn Poo
  13. 13. https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay Sparkly Unicorn Poo sometimes adjustable?
  14. 14. Cisco ASA <134>Sep 02 2014 11:50:10: %ASA-6-302013: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80) <134>Sep 02 2014 11:50:10: %ASA-6-302014: Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
  15. 15. Cisco ASA <134>Sep 02 2014 11:50:10: %ASA-6-302013: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80) <134>Sep 02 2014 11:50:10: %ASA-6-302014: Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
  16. 16. grok { match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_msg}" ] } <134>Sep 02 2014 11:50:10: %ASA-6-302013: […] { "@timestamp" => "2014-09-02T15:50:10.000Z", "cisco_tag" => "ASA-6-302013", "cisco_msg" => "[…]" }
  17. 17. Cisco ASA cisco_msg: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80) Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
  18. 18. Cisco ASA cisco_msg: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80) Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs
  19. 19. http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html
  20. 20. http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html
  21. 21. http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html 730 Pages!?
  22. 22. http://geektyrant.com/news/2013/1/10/the-ultimate-movie-training-montage.html
  23. 23. grok { match => [ "cisco_msg", "%{CISCOFW106014}", "cisco_msg", "%{CISCOFW106015}", "cisco_msg", "%{CISCOFW106021}", "cisco_msg", "%{CISCOFW106023}", "cisco_msg", "%{CISCOFW110002}", # ... "cisco_msg", "%{CISCOFW302010}", "cisco_msg", "%{CISCOFW302013_302014_302015_302016}", "cisco_msg", "%{CISCOFW302020_302021}", "cisco_msg", "%{CISCOFW305011}", "cisco_msg", "%{CISCOFW313001_313004_313008}" ]} http://www.gregmefford.com/blog/2014/09/24/analyzing-cisco-asa-firewall-logs-with-logstash/
  24. 24. 302013: Built {in|out}bound TCP connection <ID> … 302014: Teardown TCP connection <ID> … 302015: Built {in|out}bound UDP connection <ID> … 302016: Teardown UDP connection <ID> … CISCOFW302013_302014_302015_302016
  25. 25. Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80) { "action" => "Built", "direction" => "inbound", "protocol" => "TCP", "src_interface" => "inside", "src_ip" => "10.0.1.1", "src_port" => "1234", "dst_interface" => "outside", "dst_ip" => "10.0.2.2", "dst_port" => "80" }
  26. 26. Ta-da!
  27. 27. Bonus!
  28. 28. Transfooooorm! (ation) http://www.deviantart.com/art/Hungry-Luma-210132138
  29. 29. Check Point FW-1 <166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 rule: 150; rule_uid: {DEADBEEF-4444-5555-6666- DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;
  30. 30. Check Point FW-1 <166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 rule: 150; rule_uid: {DEADBEEF-4444-5555-6666- DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;
  31. 31. Transfooooorm! (ation) grok { match => ["message", "^<%{POSINT:syslog_pri}>%{WORD}: + (?<cp_time>%{MONTHDAY}[a-zA-Z]{3}%{YEAR} %{TIME}) + %{WORD:action} +%{IP} +%{DATA:interface} + %{GREEDYDATA:cp_msg}" ] } <166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 … { "@timestamp" => "2015-05-11T18:48:00.000Z", "action" => "drop", "interface" => ">bond1.5" "cp_msg" => "rule: 150; rule_uid: […]" }
  32. 32. Transfooooorm! (ation) rule: 150; rule_uid: {DEADBEEF-4444-5555-6666- DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network; kv { source => "cp_msg" value_split => ":" field_split => ";" trimkey => " " trim => " " }
  33. 33. Transfooooorm! (ation) rule: 150; rule_uid: {DEADBEEF-4444-5555-6666- DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network; { "rule": "150", "rule_uid": "{DEADBEEF-4444-5555-6666-DECAFBAD1234}", "rule_name": "CleanUp", "src": "5.6.7.8", "dst": "9.10.11.12", "proto": "udp", "product": "VPN-1&FireWall-1", "service": "domain-udp" "s_port": "67890", "product_family": "Network", }
  34. 34. Transfooooorm! (ation) http://www.deviantart.com/art/Hungry-Luma-210132138
  35. 35. Transfooooorm! (ation) mutate { rename => [ "dst", "dst_ip", "src", "src_ip", "s_port", "src_port", "proto", "protocol", "service", "dst_port", "interface", "src_interface" ] }
  36. 36. Transfooooorm! (ation) rule: 150; rule_uid: {DEADBEEF-4444-5555-6666- DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network; { "rule": "150", "rule_uid": "{DEADBEEF-4444-5555-6666-DECAFBAD1234}", "rule_name": "CleanUp", "src_ip": "5.6.7.8", "dst_ip": "9.10.11.12", "protocol": "udp", "product": "VPN-1&FireWall-1", "dst_port": "domain-udp" "src_port": "67890", "product_family": "Network", }
  37. 37. Ta-da!
  38. 38. Thanks! @ferggo (Twitter) GregMefford (GitHub)
  39. 39. { "hostname": "FileServer.example.com", "EventType": "AUDIT_SUCCESS", "Severity": "INFO", "EventID": 5145, "SourceName": "Microsoft-Windows-Security-Auditing", "Channel": "Security", "Category": "Detailed File Share", "SubjectUserName": "somebody", "SubjectDomainName": "DOMAIN1", "ObjectType": "File", "IpAddress": "67.89.12.34", "ShareName": "*MyFiles", "ShareLocalPath": "??E:MyFiles", "RelativeTargetName": "DocumentsSomebody", "AccessMask": "0x80", # … }
  40. 40. grok { match => [ "ShareName", "*%{GREEDYDATA:Share}" ] } mutate { add_field => { "ShareFullPath" => "%{hostname}%{Share}%{RelativeTargetName}" } } grok { match => [ "ShareLocalPath", "??%{DATA:LocalPath}()?$" ] } mutate { add_field => { "LocalFullPath" => "%{LocalPath}%{RelativeTargetName}" } }
  41. 41. { "hostname": "FileServer.example.com", "ShareName": "*MyFiles", "ShareLocalPath": "??E:MyFiles", "RelativeTargetName": "DocumentsSomebody", "ShareFullPath": "FileServer.example.comMyFilesDocumentsSomebody", "LocalFullPath": "E:MyFilesDocumentsSomebody", "AccessMask": "0x80", # … }
  42. 42. Ta-da!
  43. 43. ruby { code => " mask = event['AccessMask'].to_i(16) field_names = { 0 => 'READ_DATA_LIST_DIRECTORY', 1 => 'WRITE_DATA_ADD_FILE', 2 => 'APPEND_DATA_ADD_SUBDIRECTORY', 3 => 'READ_EA', 4 => 'WRITE_EA', 5 => 'EXECUTE_TRAVERSE', 6 => 'DELETE_CHILD', 7 => 'READ_ATTRIBUTES', 8 => 'WRITE_ATTRIBUTES', 16 => 'DELETE', 17 => 'READ_CONTROL', 18 => 'WRITE_DAC', 19 => 'WRITE_OWNER', 20 => 'SYNCHRONIZE' } event['AccessMaskFields'] = Hash.new field_names.each do |index, name| event['AccessMaskFields'][name] = mask[index] unless mask[index].nil? end " } Bonus!
  44. 44. { "AccessMask": "0x80", "AccessMaskFields": { "READ_DATA_LIST_DIRECTORY": 0, "WRITE_DATA_ADD_FILE": 0, "APPEND_DATA_ADD_SUBDIRECTORY": 0, "READ_EA": 0, "WRITE_EA": 0, "EXECUTE_TRAVERSE": 0, "DELETE_CHILD": 0, "READ_ATTRIBUTES": 1, <==== "WRITE_ATTRIBUTES": 0, "DELETE": 0, "READ_CONTROL": 0, "WRITE_DAC": 0, "WRITE_OWNER": 0, "SYNCHRONIZE": 0 } } Bonus!
  45. 45. CISCO_TAGGED_SYSLOG: ^<%{POSINT:syslog_pri}> %{CISCOTIMESTAMP:timestamp} ( %{SYSLOGHOST:sysloghost})?: %%{CISCOTAG:ciscotag}: <134>Sep 02 2014 11:50:10: %ASA-6-302014: [...] Behind the Scenes: Grok is Magic syslog_pri timestamp ciscotag
  46. 46. CISCO_TAGGED_SYSLOG: ^<(?<syslog_pri>b(?:[1-9][0-9]*)b)> (?<timestamp> %{MONTH} + %{MONTHDAY}(?: %{YEAR})? %{TIME}) ((?<sysloghost> (?:%{HOSTNAME}|%{IP})))?: %(?<ciscotag> [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+)): Behind the Scenes: Grok is Magic
  47. 47. CISCO_TAGGED_SYSLOG: ^<(?<syslog_pri>b(?:[1-9][0-9]*)b)> (?<timestamp>b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar( ?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:u st)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|De c(?:ember)?)b + (?:(?:0[1-9])|(?:[12][0- 9])|(?:3[01])|[1-9]) (?: (?>dd){1,2})? (?!<[0- 9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]) (?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0- 9])) ((?<sysloghost>(?:b(?:[0-9A-Za-z][0-9A-Za-z- ]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z- ]{0,62}))*(.?|b)|(?:%{IPV6}|%{IPV4}))))?: %(?<ciscotag>[A-Z0-9]+-(?:[+-]?(?:[0-9]+))-(?:[A- Z0-9_]+)): Behind the Scenes: Grok is Magic
  48. 48. IPV6: ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa- f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]d|1dd|[1- 9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3})|:))|(([0-9A-Fa- f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0- 4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1- 9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa- f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]d|1dd|[1- 9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa- f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa- f]{1,4}){0,2}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0- 4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A- Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0- 4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1- 9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa- f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0- 4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1- 9]?d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa- f]{1,4}){0,5}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0- 4]d|1dd|[1-9]?d)){3}))|:)))(%.+)? Behind the Scenes: Grok is Magic
  49. 49. CISCOFW302013_302014_302015_302016: %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} ( (%{IP:src_mapped_ip}/%{INT:src_mapped_port}))? ((%{DATA:src_fwuser}))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} ( (%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}))? ((%{DATA:dst_fwuser}))? ( duration %{TIME:duration} bytes %{INT:bytes})? (?: %{CISCO_REASON:reason})? ( (%{DATA:user}))? Behind the Scenes: Grok is Magic
  50. 50. 302013: Built {in|out}bound TCP connection <ID> … 302014: Teardown TCP connection <ID> … 302015: Built {in|out}bound UDP connection <ID> … 302016: Teardown UDP connection <ID> … CISCOFW302013_302014_302015_302016 Behind the Scenes: Grok is Magic
  51. 51. Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80) (302013) Behind the Scenes: Grok is Magic action direction protocol src_interface src_ip & src_port src_mapped_ip & _port dst_interface dst_ip & _port dst_mapped_ip & _port connection_id
  52. 52. Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs (302014) Behind the Scenes: Grok is Magic action protocol src_interface dst_interface dst_ip & _port duration src_ip & _port bytes reason connection_id

×