Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Security threat analysis points for enterprise with oss


Published on

Security threat analysis points for enterprise with Elasticsearch

Published in: Technology
  • Be the first to comment

Security threat analysis points for enterprise with oss

  1. 1. 1 2017/06/20 Future Architect, Inc. Hisashi Hibino Security Threat Analysis Points for Enterprise with OSS
  2. 2. 2 Hibino Hisashi Employee at Future Architect Inc. Technology Innovation Group Security Architect Fields: Server OS Database Application Network Data Center Security
  3. 3. 3 × Security
  4. 4. 4 Elastic Stack + X-Pack Kibana Elasticsearch Logstash Beats Elastic Cloud Alert Performance Monitoring Graph Security X-Pack Elastic Stack (Open Source) Paid Subscription Machine Learning Report Normalization Save/Store Visualization Authorization /Encryption Notification Correlation Analysis Anomaly Detection These are the functions useful for security threat analysis with Elastic. Collect
  5. 5. 5 Is your important data in danger? Security threats that endanger important and confidential information lie both outside and within. Server (保護対象システム) External Threat Internal Misconduct Servers OS DB Applications Vulnerabilities Data(Confidential) Network(Defense in Depth) Log Integration Management Base System Firewall IDS/IPS WAF Antivirus Sandbox SpamFilter Client Device Control Encryption Antivirus
  6. 6. 6 Ever heard of these incidents in the news? There is a surge in DDoS attacks that target IoT devices with inadequate security measures 2003 2010 2015 2020 0 63 68 125 250 500 Figure:IoT device trends 100 200 300 400 500 72 76 Source: Cisco Consulting Service WorldPopualation(Billion)/IoTDeviceNumbers(Billion) World Population Number of IoT Devices
  7. 7. 7 70% of enterprises don’t notice the attacks Ever since Japan Pension Service was attacked in 2015, more and more attacks have been reported. Nearly 70% were reported from external parties. 2013 2014 2015 2016 0 Source: 「平成28年におけるサイバー空間をめぐる脅威の情勢について」(Tokyo Metropolitan Police) 492 1,723 4,046 1,000 2,000 3,000 4,000 3,828
  8. 8. 8 And now to our main topic! Enough with the introduction, and on to what you can do with Elastic Stack to counter internal and external security threats. 1.External Threats →Defense against Targeted Threats on Servers 2.Internal Threats →Defense against illegal acquisition of Specific Personal Information
  9. 9. 9 Internal Critical points to analyze external targeted threats on servers It is important to conduct reconnaissance after opening E-mail containing malware and to audit the web communication to the C&C server. Firewall DNS Server Mail Server (MTA) AD Server Malware HackersC&C Server DMZ Employee Web ProxyAccess Log Query Log Mail log Communication Log Authorization Log ① ⑥ ③ ② ④ ⑤ No Phase Attack description Type 1 Preparation Choose target, conduct preliminary survey, prepare malware, prepare C&C server 2 Initial Infiltration Send targeted E-mail【①,②】,Execute malware Infiltration Activity 3 Device Control Remote control through C&C communication 【③~⑤】, check infiltrated environment Infiltration Acivity 4 Information Search let off internal activity tools, search within LAN【⑥】 Internal Activity 5 Information Consolidation Gather beneficial information【⑥,⑦】 Internal Activity 6 Send Information Obtain gathered data Internal Activity Targeted E-mail File Server CIFS Audit log ⑦ Incoming Mail Domain Authorization DNS Query Web Access 【Legend】 CIFS Access
  10. 10. 10 Application Server Critical points to analyze internal illegal acquisition of confidential information How to survey the actions of the server manager in charge of confidential data is especially important Jump ServerDB Server File server OS Authentication Log OS Authentication LogOS Authentication Log SQL Audit Log CIFS Audit Log OS Authentication Log System Manager Users SQL SSH/RDP CIFS Application 【Legend】 Application Audit Log Caution!
  11. 11. 11 An Overview of the Incorporation of Logs Log ManagementDB Log File Database Network Devices Performance Data Event Log (Windows) Metricbeat Filebeat Winlogbeat Logstash input JDBC input tcp/udp (syslog/netflow) Client Could(Application) Network traffic Packet Capture Packetbeat Kibana Elasticsearch 【要正規化】 search ①OS Audit Log(Windows) ①‘OS Audit Log(Linux) ②Proxy Log(ProxySG) ②Proxy Log(i-Filter) ③DB Audit Log(Oracle) ③’CIFS Audit Log(NetApp) ③’CIFS Audit Log(Windows)
  12. 12. 12 ①OS Audit Log
  13. 13. 13 Effective utilization of OS Audit Log Be able to audit all audit logs regardless if it’s internal or external. Server A Server B Server C Jump Server System manager SSH/RDP ①Windows Server: Gather Event logs ②Linux Server: Gather audit.log System A
  14. 14. 14 Log Management DB Before Beats, we had to gather event logs with Logstash Until the release of winlogbeat, Logstash was widely used to gather Windows event logs. Event log 【Normal】 ① ② Logstash gathers logs every time they are added 【logstash.conf】 input { eventlog { tags => "EVENT_LOG" type => 'Win32-EventLog' logfile => 'Security' } } output{ elasticsearch{ host => ‘IP of Elasticsearch' protocol => http } }
  15. 15. 15 Log Management DB In order to make sure no logs are lost on Logstash… However, in certain cases when Logstash shutsdown, the logs added during that time will be forever lost even when the service restarts. Event Log 【Abnormal】 Added logs would be lost 【logstash.conf】 input { eventlog { tags => "EVENT_LOG" type => 'Win32-EventLog' logfile => 'Security' } } output{ elasticsearch{ host => ‘IP of Elasticsearch' protocol => http } }
  16. 16. 16 That’s why Windows Event Logs should be converted to text By converting the logs into text files, the logs can be salvaged in case of any problems which might cause Logstash to stop.This way the Sincedb file would have recorded the last line read, and none of your logs will be lost. Event Log Add Event ① Task Scheduler ② ③ Text Files Batch file ④ Log Management DB ⑤ ⑥ 【logstash.conf】 input { file { tags => "EVENT_LOG“ path => ['D:/srv/logstash/logonlist.csv'] codec => plain { charset => "Shift_JIS" } start_position => beginning } } output{ elasticsearch{ host => ‘IP of Elasticsearch' protocol => http } } Create a batch that adds a line to the text every time an event is added using a task scheduler
  17. 17. 17 winlogbeat can save you from that hassle! With the addition of winlogbeat in 2015, all you have to do is set the eventID. No worries about setting a backup plan for system errors, just draw yourself some nice dashboards on kibana! Event Log ① ② Log Management DB No eventID Meaning 1 21 OS login success 2 23 OS logoff success 3 24 OS session end 4 25 OS session reconnect 5 4625 OS login failure 【winlogbeat.yml】 winlogbeat.event_logs: - name: Microsoft-Windows-TerminalServices-LocalSessionManager/Operational tags: ["login"] event_id: 21,23,24,25 - name: Security tags: ["login"] event_id: 4625 output.elasticsearch: # Array of hosts to connect to. hosts: [“IP for Elasticsearch:9200"]
  18. 18. 18 Windows Jump server Login audit graph looks like this Y axis is the number of logins, and the X axis is the time(daily). For every account the bars will stack up showing who, when, and how many times a login was attempted and or failed. User A User B User C 【Sample: Login success graph】
  19. 19. 19 Next up is the Linux authorization log To audit the Linux OS authorization log, use audit.log for input. Before the Beats series, logs had to be manually modified with Logstash Log Management DB audit.log ① ② Every time a log is added, Logstash gathers the information 【logstash.conf】 input { file { tags => "AUDIT" path => "/var/log/audit/audit.log" start_position => "beginning“ } } } filter { if "AUDIT" in [tags] { kv{} grok { match => { "msg" => "audit¥(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}¥):" } } date { match => [ "audit_epoch", "UNIX" ] timezone => ["Asia/Tokyo"] remove_field => ["audit_epoch"] } ・・・
  20. 20. 20 Have you ever seen the Linux audit.log? Audit.log has different fields for different types. Also, multiple lines are added for one login. # tail -f /var/log/audit/audit.log type=CRYPTO_KEY_USER msg=audit(1497422635.535:3732): pid=4803 uid=0 auid=1000 ses=503 msg='op=destroy kind=server fp=e0:c2:63:99:32:4d:b1:13:f4:ee:6f:0c:b3:e7:5d:d4 direction=? spid=4803 suid=0 exe="/usr/sbin/sshd" hostname=? addr= terminal=? res=success' type=CRYPTO_KEY_USER msg=audit(1497422635.535:3733): pid=4803 uid=0 auid=1000 ses=503 msg='op=destroy kind=server fp=94:33:26:70:a7:bf:95:cc:51:b3:ec:d4:a7:17:1c:a8 direction=? spid=4803 suid=0 exe="/usr/sbin/sshd" hostname=? addr= terminal=? res=success' type=CRED_ACQ msg=audit(1497422635.535:3734): pid=4803 uid=0 auid=1000 ses=503 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="centos" exe="/usr/sbin/sshd" hostname=ip-10-146-1-236.ap-northeast-1.compute.internal addr= terminal=ssh res=success' type=USER_LOGIN msg=audit(1497422635.554:3735): pid=4797 uid=0 auid=1000 ses=503 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=ip-10-146-1-236.ap-northeast-1.compute.internal addr= terminal=/dev/pts/0 res=success' type=USER_START msg=audit(1497422635.554:3736): pid=4797 uid=0 auid=1000 ses=503 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=ip-10-146-1-236.ap-northeast-1.compute.internal addr= terminal=/dev/pts/0 res=success' type=CRYPTO_KEY_USER msg=audit(1497422635.571:3737): pid=4797 uid=0 auid=1000 ses=503 msg='op=destroy kind=server fp=94:33:26:70:a7:bf:95:cc:51:b3:ec:d4:a7:17:1c:a8 direction=? spid=4804 suid=1000 exe="/usr/sbin/sshd" hostname=? addr= terminal=? res=success' type=USER_AUTH msg=audit(1497422639.558:3738): pid=4825 uid=1000 auid=1000 ses=503 msg='op=PAM:authentication grantors=pam_unix acct="root" exe="/usr/bin/su" hostname=? addr=? terminal=pts/0 res=success' type=USER_ACCT msg=audit(1497422639.558:3739): pid=4825 uid=1000 auid=1000 ses=503 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="root" exe="/usr/bin/su" hostname=? addr=? terminal=pts/0 res=success' type=CRED_ACQ msg=audit(1497422639.558:3740): pid=4825 uid=1000 auid=1000 ses=503 msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/bin/su" hostname=? addr=? terminal=pts/0 res=success' type=USER_START msg=audit(1497422639.560:3741): pid=4825 uid=1000 auid=1000 ses=503 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth acct="root" exe="/usr/bin/su" hostname=? addr=? terminal=pts/0 res=success‘ 【Reference】Red Hat Enterprise Linux 6 Security Guide 7.6. Understanding Audit Log Files 【Sample:Log for SSH logins】
  21. 21. 21 Audit.logs are made easier with Filebeat modules released after Filebeat5.4 Filebeat was added on 5.3. It can modify logs and offers templates for Kibana graphs.(Auditdmodule was added in 5.4) audit.log ① ② Every time a log is added, Filebeat gathers the log and it will be saved in Elasticsearch without having to normalize the logs 【filebeat.yml】 #------------------------------- Auditd Module ------------------------------- - module: auditd log: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: # Prospector configuration (advanced). Any prospector configuration option # can be added under this section. #prospector: #================ Outputs ===================== # Configure what outputs to use when sending the data collected by the beat. # Multiple outputs may be used. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: [“IP for Elasticsearch:9200"] Log Management DB
  22. 22. 22 Now, Kibana offers template graphs for your logs Timelion will show a correlation graph of login successes and failures
  23. 23. 23 OS authorization logs are useful when used with worksheets Login numbers are not that useful by itself, but when compared and matched with recorded work times, it helps identify internal misconduct and actions preceding targeted attacks. Management by the system Manual Management logs (system output) worksheets (filled in manually) Daily Monitoring Security audit 【System Manager】 ①Check misconduct (rule violations, skipping work) ②Check mistakes (Forgotten operations and mistakes) 【Security Auditor】 ①Check internal misconduct and information leaks ②Check security functions
  24. 24. 24 ②Proxy Logs
  25. 25. 25 Can you properly analyze proxy logs? To counter targeted server attacks, it is important to analyze the web proxy server logs and firewall logs. In addition to the two, correlation with mail severs, DNS servers, active directories are also important. Source:高度サイバー攻撃への対策における Logの活用と分析方法:JPCERT/CC Logを活用した高度サイバー攻撃の早期発見と分析:JPCERT/CC Firewall Web Proxy Mail Server(MTA) AD/DNSZServer Communication logs Access logs Mail logs audit logs/ query logs Logs for correlation analysis
  26. 26. 26 Key Points to analyzing proxy logs pt.1 Visualize log numbers by time to check for any irregular actions outside normal working hours
  27. 27. 27 Key Points to analyzing proxy logs pt.2 Visualize rankings for IP addresses that are accessing URLs that are blocked by filters The blocked URL here is of the blog category. If logs are detecting communication blocks for C&C servers, expect malware infection!
  28. 28. 28 Key Points to analyzing proxy logs pt.3 Visualize HTTP request size to POST and the number of files by rank according to IP addresses Are there any large data sent out externally? Are there small data sent out but large numbers of access to the C&C server? Be prepared to audit these information!
  29. 29. 29 Key Points to analyzing proxy logs pt.4 Classify by UserAgent and check to see if UserAgents often used by malware has no irregular activity Malwares use suspicious UserAgents to communicate with C&C servers
  30. 30. 30 How to incorporate proxy logs Below is an idea of how to incorporate proxy logs from ProxySG and i-Filter Log Management DB rsyslog ProxySG i-Filter syslog Access log (CSV) Access log (CSV) input file input beats Normalize logs with filter csv
  31. 31. 31 ③DB Audit Log/ CIFS Audit Log
  32. 32. 32 How to incorporate Oracle Database SQL Audit logs Packetbeat cannot obtain the SQL audit log from Oracle DBではOracle DB on it’s own so we use AVDF Log Management DBinput jdbc (60min intervals) DBFirewall Audit Vault Server Oracle Database Database Vault Oracle Client (Audit Vault Agent) 【logstash.conf】 input { jdbc { tags => "JDBC" jdbc_connection_string => "jdbc:oracle:thin:<user名>/<PW>@<IPAddress>:1521/<servicename>" jdbc_user => "<username>" jdbc_password => "<PW>" jdbc_driver_library => "/opt/logstash/vendor/jar/jdbc/ojdbc7.jar" jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" record_last_run => "true" schedule => "00 0-23 * * *" statement => "SELECT …<shortened> } } output{ elasticsearch{ host => ‘IP for Elasticsearch' protocol => http } } SQL Audit Log Table Gather SQL execution logs from multiple routes ① ② ③
  33. 33. 33 By the way, what’s Packetbeat? Packetbeat is an awesome tool that gathers network capture data for Elasticsearch
  34. 34. 34 How to incorporate NetApp CIFS access logs NetApp outputs XML logs for access to personal information files contained in the CIFS volume. With the input file in Logstash normalize the XML file. NetApp FAS NASボリューム02 Users File Access CIFSVolume01 NFSVolume02 (/AUDIT01) personal info File Access Audit Log Log Management DB input file use filter xml to normalize logs Access logs 【NetApp CIFS Audit log settings Sample】 Cluster01::> vserver audit create -vserver <SVM> -destination /AUDIT01 -rotate-schedule-minute 10,20,30,40,50 -rotate-limit 1440 Cluster01::> vserver audit enable -vserver <SVM> Cluster01::> vserver audit modify -vserver <SVM> -format xml ① ② ③
  35. 35. 35 NetApp CIFS Access Log Logstash 「XML filter」 filter { xml { source => "message" store_xml => false xpath => [ "/Event/System/EventID/text()","EventID", "/Event/System/EventName/text()","Event_Name", "/Event/System/Source/text()", "Source", "/Event/System/Opcode/text()","Opcode", "/Event/System/Result/text()", "Result", "/Event/System/TimeCreated/@SystemTime","TimeCreated", "/Event/System/Channel/text()", "Channel", "/Event/System/Computer/text()", "Computer", "/Event/EventData/Data[@Name='SubjectIP']/text()","SubjectIP", "/Event/EventData/Data/@Uid","Uid", "/Event/EventData/Data/@Gid","Gid", "/Event/EventData/Data/@Local","Local", "/Event/EventData/Data[@Name='SubjectUserid']/text()","SubjectUserSid", "/Event/EventData/Data[@Name='SubjectUserIsLocal']/text()","SubjectUserIsLocal", "/Event/EventData/Data[@Name='SubjectDomainName']/text()","SubjectDomainName", "/Event/EventData/Data[@Name='SubjectUserName']/text()","SubjectUserName", "/Event/EventData/Data[@Name='ObjectServer']/text()","ObjectServer", "/Event/EventData/Data[@Name='ObjectType']/text()","ObjectType", "/Event/EventData/Data[@Name='HandleID']/text()","HandleID", "/Event/EventData/Data[@Name='ObjectName']/text()","ObjectName", "/Event/EventData/Data[@Name='AccessList']/text()","AccessList", "/Event/EventData/Data[@Name='AccessMask']/text()","AccessMask", "/Event/EventData/Data[@Name='DesiredAccess']/text()","DesiredAccess", "/Event/EventData/Data[@Name='Attributes']/text()","Attributes", "/Event/EventData/Data[@Name='SearchPattern']/text()","SearchPattern", "/Event/EventData/Data[@Name='InformationRequested']/text()","InfoReq", "/Event/EventData/Data[@Name='OldPath']/text()","OldPath", "/Event/EventData/Data[@Name='NewPath']/text()","NewPath", "/Event/EventData/Data[@Name='InformationSet']/text()","InfoSet", "/Event/EventData/Data[@Name='SearchFilter']/text()","SearchFilter", "/Event/EventData/Data[@Name='ReadOffset']/text()","ReadOffset", "/Event/EventData/Data[@Name='ReadCount']/text()","ReadCount" ] } }
  36. 36. 36 For Windows, winlogbeat always comes in handy Event Log Event Added ① ② ③ Log Management DB No イベントID 内容 1 4656 File Open, File delete 2 4658 File Close 3 4663 File Access 4 4690 File Copy 【winlogbeat.yml】 winlogbeat.event_logs: - name: Security tags: ["cifs"] event_id: 4656,4658,4663,4690 output.elasticsearch: # Array of hosts to connect to. hosts: [“IP for Elasticsearch:9200"] When using the offered CIFS file server for Windows, select the EventID with winlogbeat
  37. 37. 37 Audit different types of access to confidential data Prepare two graphs: One that shows users that accessed files by time and another that sums up the amount of files accessed Access numbers 【Summary】 機密データ Access件数 Catch when files are accessed Count how many times a certain account accessed a file 【Sample: Access rankings to confidential data】
  38. 38. 38 Summary  Elasticsearch is useful for mission critical systems such as financial organizations and public service systems Elastic Stack is capable of auditing everyday logs and for security auditing tasks. It is important to start small and gather the minimum and prioritize logs to begin with. Functions that the OSS can’t cover can be easily covered with X-Pack afterwards.  Just thinking about logs won’t count Imagine what sort of security management to have. Create an architecture that would minimize the operation costs Include Elastic Stack as part of that architecture
  39. 39. 39 Appendix
  40. 40. 40 What sort of logs are important for security? 1.timestamp accuracy 2.Integrity of logs 3.Access permissions to logs Source:Computer Security Log Management Guide:NIST Log timestamps must be accurate and no logs can be lost even if the system fails
  41. 41. 41 The pain in the neck to keep log timestamps aligned Linux Log (UNIX Format) Windows Log (ISO8601 Format) Application Log/DB Log (yyyyMMddHHmmssSSS Format) By default, Elasticsearch will recognize the time logs were received by Logstash as [@timestamp] Timestamp Formats Output Results UNIX 1473123710 ISO8601 2016-09-06T10:01:50.000Z yyyyMMddHHmmssSSS 20160906100150000 ※”2016/9/6 10:01:50” Shown in different timestamp formats There is an issue that the timestamp used for Kibana changes to the timestamp the log was imported into Logstash
  42. 42. 42 Logstash 「date filter」 filter { if “SYSLOG" in [tags] { if [message] !~ "Auth" and [message] !~ "Failed to authenticate" { drop{} } } if "EVENT_LOG" in [tags] { kv { field_split => "," value_split => ":" trim => " ¥r" trimkey => " " } date { match => [ "Date", "ISO8601" ] } } } Use the Windows Event Log timestamp for @timestamp
  43. 43. 43 Logstash 「date filter」 filter { if "AUDIT" in [tags] { kv{} grok { match => { "msg" => "audit¥(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}¥):" } } mutate { rename => { "type" => "audit_type" "homename" => "login_name" } } date { match => [ "audit_epoch", "UNIX" ] } } Use the Linux OS audit log timestamp for @timestamp
  44. 44. 44 Logstash 「date filter」 filter { if "JDBC" in [tags] { date { match => [ "event_time","yyyyMMddHHmmssSSS"] } } } Use the application or database timestamp for @timestamp
  45. 45. 45 The Limit to sql last start DB(M) DB(S) Replication Log Management DB 【Normal】 This is an issue that occurs for Master-Slave modeled database audit logs ① Every hour Logstash acquires logs for the past 60minutes ② 【logstash.conf】 input { jdbc { tags => "JDBC" record_last_run => "true“ schedule => "00 0-23 * * *" statement => “… and AV_TIME> :sql_last_start“ } } output{ elasticsearch{ host => ‘IP for Elasticsearch' protocol => http } }
  46. 46. 46 The Limit to sql last start DB(M) DB(S→M) Failover Log Management DB 【Abnormal】 When there is an issue with the master database, there is a possibility that some logs will be lost because Logstash gathers data based on the last timestamp an SQL was executed 【logstash.conf】 input { jdbc { tags => "JDBC" record_last_run => "true“ schedule => "00 0-23 * * *" statement => “… and AV_TIME> :sql_last_start“ } } output{ elasticsearch{ host => ‘IP for Elasticsearch' protocol => http } }
  47. 47. 47 Points for log normalization when incorporating syslog Instead of gathering the syslogs from devices directly to Logstash, receive them from rsyslogd Network Device A (FortiGate) Network Device B (Cisco Catalyst) Network Device C (BluCoat ProxySG) Network Device A (FortiGate) Network Device B (Cisco Catalyst) Network Device C (BlueCoat ProxySG) rsyslog /var/log/forti/forti.log /var/log/proxy/proxy.log /var/log/cisco/cisco.log input { udp { tags => “SYSLOG“ port => 514 } } filter{ if “SYSLOG” in [tags] { ・・・ With this architecture, logs cannot be separated into different filters based on tags because all the logs for different devices would have come through the same input udp. input { file { tags => “forti“ path => "/var/log/forti/forti.log" start_position => "beginning" } } file { tags => “cisco“ path => "/var/log/cisco/cisco.log" start_position => "beginning" } file { tags => “proxy“ path => "/var/log/proxy/proxy.log" start_position => "beginning" } filter{ if “forti” in [tags] { <Normalization rules for FortiGate> } if “cisco” in [tags] { <Normalization rules for Catalyst> } if “proxy” in [tags] { <Normalization rules for ProxySG> } ・・・ By receiving the logs from rsyslogd, logs will separated according to IP, therefore making it possible to use tags depending on which input file it came through. This way, different normalization filters can be applied to different logs <Sample for rsyslog.conf> :fromhost-ip, isequal, “IP Address” where-your-log-file-is
  48. 48. 48 Devices, log formats and Filters No Category(Large) Category(Small) Product Brand Log Format filter 1 End Point Anti Virus DeepSecurity Trend Micro Customized grok 2 Network Router ISR Series Cisco Systems Customized grok 3 Network Switch Catalyst Series Cisco Systems Customized grok 4 Network Switch Nexus Series Cisco Systems Customized grok 5 Network Firewall(UTM) PA Series Palo Alto Networks csv csv 6 Network Firewall(UTM) ASA Series Cisco Systems Customized grok 7 Network Firewall(UTM) FortiGate Series Fortinet key=value kv 8 Network Load Balancer BIG-IP Series F5Networks Customized grok 9 Network Proxy BlueCoat ProxySG Series Symantec csv csv 10 Network Proxy i-Filter Digital Arts csv csv 11 Network Mail FortiMail Series Fortinet key=value kv 12 Database RDBMS Oracle Database Oracle Database jdbc 13 Server Application File Sharing(CIFS) NetApp FAS Series NetApp XML xml 14 Server Application File Sharing FileZen Series Soliton Systems Customized grok 15 Server Application DHCP Windows Server Microsoft csv csv Many of the logs formats for popular devices have its own unique log format, which means you’ll have to play around with grok a lot
  49. 49. 49 Plotting origins of SPAM mail on a world map Define the origin of an E-mail from it’s IP address using Geoip and plot it on a world map Firewall Mail Sever Log Management DB rsyslog syslog Communication Log/ Mail Log input file Use information of the origin IP with the SPAM detection event on the mail log or the recipient port TCP25 to filter geoip to plot it on a map SPAM Mail search SPAM mails will be blocked by filters syslog 【Tile Map】
  50. 50. 50 The Best Practice for Matching IP Blacklist Using Jdbc_streaming is more efficient for matching than using translate Log Management DB RDB logstash-filter-jdbc_streaming 3rd Party Blacklist Input http output jdbc input beats WebServer#1 Access Log WebServer#2 Access Log WebServer#3 Access Log Match the IP address in the access log on Logstash using the jdbc_streaming plugin against blacklisted IP addresses on RDB. ① ② ③ ④ ⑤ ⑥
  51. 51. 51 How to incorporate logs on AWS Users Internet ELB EC2 RDS VPC Flow Logs S3 Bucket Cloud Front S3 Cloud Trail Cloud Watch Logs input s3 input file Save logs Save logs Save logs Audit Log Communication Log Save logs Web Access DB Access Save logs Object Access Elastic Stack AWS Shield AWS WAF Save logs Save logs S3 Access Log Audit AWS Manager Action Log Audit SQL Misconduct Access Log Audit VPC Misconduct Communication Log Audit CloudFront Access Log Audit ELB Access Log Audit WAF Log Audit DDoS Attack Log Audit
  52. 52. 52