Download as PDF, PPTX
Uploaded byINSIGHT FORENSIC
(Fios#02) 2. elk 포렌식 분석
The document discusses the ELK stack, including Logstash for collecting, centralizing, parsing, storing, and searching logs; Elasticsearch for storing parsed log data from Logstash in a searchable format; and Kibana for visualizing and interacting with logs stored in Elasticsearch. It provides examples of using Logstash to ingest logs from multiple systems and ship the parsed data to Elasticsearch.
More Related Content
![[2D1]Elasticsearch 성능 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/2d1elasticsearch-140929192211-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Elasticsearch] 검색의 연관성 좀 더 높여보기](https://cdn.slidesharecdn.com/ss_thumbnails/searching-with-relevancy-141230230553-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds)
![[오픈소스컨설팅] 프로메테우스 모니터링 살펴보고 구성하기](https://cdn.slidesharecdn.com/ss_thumbnails/oscprometheus-190422050231-thumbnail.jpg?width=640&height=640&fit=bounds)
![[발표자료] 오픈소스 Pacemaker 활용한 zabbix 이중화 방안(w/ Zabbix Korea Community)](https://cdn.slidesharecdn.com/ss_thumbnails/high-availabilityzabbixforopensourcepaceamker-191204120827-thumbnail.jpg?width=640&height=640&fit=bounds)
![[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region](https://cdn.slidesharecdn.com/ss_thumbnails/openstackoscv0-160718105826-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Demo session] 관리형 Kafka 서비스 - Oracle Event Hub Service](https://cdn.slidesharecdn.com/ss_thumbnails/oraclouddevmeetupeventhubpublic-180521084803-thumbnail.jpg?width=640&height=640&fit=bounds)
![[ 2021 AI + X 여름 캠프 ] 1. linux 기초 & 장비 세팅](https://cdn.slidesharecdn.com/ss_thumbnails/1-210731153333-thumbnail.jpg?width=640&height=640&fit=bounds)
(Fios#02) 2. elk 포렌식 분석
- 1. FORENSIC INSIGHT; DIGITAL FORENSICS COMMUNITY IN KOREA ELK Forensics demantos demantos@gmail.com http://malwarel4b.blogspot.kr Cho Hoon
- 2. forensicinsight.org Page Tableof Contents 2 ▪ Introduction
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. forensicinsight.org Page HowIt Works 5 Logstash ElasticSearch Kibana •web
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50. forensicinsight.org Page HowIt Works 6 Shipper
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65. syslog Redis Logstash ElasticsearchKibana Indexer Reads
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.
- 95.
- 96.
- 97.
- 98.
- 99.
- 100.
- 101.
- 102.
- 103.
- 104.
- 105.
- 106.
- 107.
- 108.
- 109.
- 110.
- 111.
- 112.
- 113.
- 114.
- 115.
- 116.
- 117.
- 118.
- 119.
- 120.
- 121.
- 122. forensicinsight.org Page Logstash 8 ▪/etc/default/logstash
- 123.
- 124.
- 125.
- 126.
- 127.
- 128.
- 129.
- 130.
- 131.
- 132.
- 133.
- 134.
- 135.
- 136.
- 137. 사이즈 logstash 18235 1 99 10:15 ? 01:38:31 /usr/bin/java -‐Djava.io.tmpdir=/var/lib/logstash -‐Xmx500m -‐XX:+UseParNewGC -‐XX:+UseConcMarkSweepGC -‐Djava.awt.headless=true -‐ XX:CMSInitiatingOccupancyFraction=75 -‐XX:+UseCMSInitiatingOccupancyOnly -‐jar /opt/logstash/ vendor/jar/jruby-‐complete-‐1.7.11.jar -‐I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -‐f /etc/logstash/conf.d -‐l /var/log/logstash/logstash.log -‐w 4
- 138. forensicinsight.org Page Logstash 9 ▪/etc/logstash/conf.d/*.conf
- 139.
- 140.
- 141.
- 142.
- 143.
- 144.
- 145.
- 146.
- 147.
- 148.
- 149.
- 150.
- 151.
- 152. 것인지 input parse/filter output collectd,drupal_dblog,elasticsearch,eventlog,exec,file, ganglia, gelf, gemfire, generator, graphite, heroku, imap, invalid_input, irc, jmx, log4j, lumberjack, pipe, puppet_facter, rabbitmq, rackspace, redis, relp, s3, snmptrap, sqlite, sqs, stdin, stomp, syslog, tcp, twitter, udp, unix, varnishlog, websocket, wmi, xmpp, zenoss, zeromq advisor, alter, anonymize, checksum, cidr, cipher, clone, collate, csv, date, dns, drop, elapsed, elasticsearch, environment, extractnumbers, fingerprint, gelfify, geoip, grep, grok, grokdiscovery, i18n, json, json_encode, kv, metaevent, metrics, multiline, mutate, noop, prune, punct, railsparallelrequest, range, ruby, sleep, split, sumnumbers, syslog_pri, throttle, translate, unique, urldecode, useragent, uuid, wms, wmts, xml, zeromq boundary, circonus, cloudwatch, csv, datadog, datadog_metrics, elasticsearch, elasticsearch_http, elasticsearch_river, email, exec, file, ganglia, gelf, gemfire, google_bigquery, google_cloud_storage, graphite, graphtastic, hipchat, http, irc, jira, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, rabbitmq, rackspace, redis, redmine, riak, riemann, s3, sns, solr_http, sqs, statsd, stdout, stomp, syslog, tcp, udp, websocket, xmpp, zabbix, zeromq [참고]
- 153.
- 154. forensicinsight.org Page Logstash 10 input { tcp { type = apache port = 18080 } } filter { if [type] == apache { grok { match = { message = %{COMBINEDAPACHELOG} } } date { match = [ timestamp, dd/MMM/yyyy:HH:mm:ss Z ] } } } output { if [type] == apache { elasticsearch { index = logstash-‐iistest02 host = localhost } } } Block Block Block plugin
- 155.
- 156.
- 157.
- 158.
- 159.
- 160.
- 161.
- 162.
- 163.
- 164.
- 165.
- 166. • cat /IIS/W3C1234/ex* | nc -‐vv localhost 18003 input { tcp { type = w3c_extended_iis port = 18003 } } input { syslog { type = syslog port = 5514 } }
- 167. forensicinsight.org Page Logstash 12 ▪Filter filter { if [type] == w3c_extended_iis { # drop comment lines if ([message] =~ /^#/) { drop{} } csv { columns = [date, time, s_ip, cs_method, cs_uri_stem, cs_uri_query, s_port, cs_username, c_ip, cs_user_agent, sc_status, sc_substatus, sc_win32_status, time_taken] separator = } mutate { merge = [date, time] } mutate { join = [date, ] } date { match = [date, YYYY-‐MM-‐dd HH:mm:ss ] timezone = ['UTC'] } geoip { source = c_ip }
- 168. forensicinsight.org Page Logstash 13 ▪Filter # extract macb info if (m in [macb]) { mutate { add_tag = [modified] } } if (a in [macb]) { mutate { add_tag = [accessed] } } if (c in [macb]) { mutate { add_tag = [changed] } } if (b in [macb]) { mutate { add_tag = [birth] } } # extract file extension grok { match = [path, (?filename[^/]+?)?$] } grok { match = [filename, ((.(?ext[^./]+))?)?$] } mutate { lowercase = [ext] remove_field = [message, perms, uid, gid] } } }
- 169.
- 170.
- 171.
- 172.
- 173.
- 174.
- 175.
- 176.
- 177.
- 178.
- 179.
- 180.
- 181.
- 182.
- 183.
- 184.
- 185.
- 186.
- 187.
- 188.
- 189.
- 190.
- 191.
- 192.
- 193.
- 194.
- 195.
- 196.
- 197.
- 198.
- 199.
- 200.
- 201.
- 202.
- 203. 가능 %{syntax:semantic} SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] (?:%{WORD:verb} % {NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest}) %{NUMBER:response} (?:%{NUMBER:bytes}|-‐) COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
- 204.
- 205.
- 206. grok Logstash 15 Dec 26 10:45:01 localhost postfix/pickup[27869]: 841D26FFA8: uid=0 from=root %{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:hostname} postfix/(?component[w._/%-‐]+) (?:[%{POSINT:pid}]): (?queueid[0-‐9A-‐F]{,11}): %{GREEDYDATA:message} Built-in
- 207.
- 208.
- 209.
- 210.
- 211.
- 212.
- 213.
- 214.
- 215.
- 216.
- 217.
- 218.
- 219.
- 220.
- 221.
- 222.
- 223.
- 224.
- 225.
- 226.
- 227.
- 228. useragent if [cs_user_agent] != { useragent { source = cs_user_agent prefix = user_agent. }
- 229.
- 230.
- 231. geoip geoip { source = c_ip
- 232.
- 233.
- 234.
- 235.
- 236.
- 237.
- 238.
- 239.
- 240.
- 241.
- 242.
- 243.
- 244.
- 245.
- 246.
- 247.
- 248.
- 249.
- 250.
- 251.
- 252.
- 253. 불가능 output { if [type] == w3c_extended_iis { elasticsearch { index = logstash-‐%{[type]}-‐%{+YYYY.MM.dd} host = localhost } } }
- 254.
- 255.
- 256.
- 257.
- 258.
- 259.
- 260.
- 261.
- 262.
- 263.
- 264.
- 265.
- 266.
- 267.
- 268.
- 269.
- 270.
- 271.
- 272.
- 273.
- 274.
- 275.
- 276.
- 277.
- 278.
- 279.
- 280.
- 281.
- 282.
- 283.
- 284.
- 285.
- 286.
- 287.
- 288.
- 289.
- 290.
- 291.
- 292.
- 293.
- 294.
- 295.
- 296.
- 297.
- 298.
- 299.
- 300. RDBMS Elasticsearch RDBMS Index Database TypeTable Document Row Field Column (1)
- 301.
- 302.
- 303.
- 304. forensicinsight.org Page Elasticsearch 22 ▪/etc/default/elasticsearch
- 305.
- 306.
- 307.
- 308.
- 309.
- 310.
- 311.
- 312.
- 313.
- 314.
- 315.
- 316.
- 317.
- 318.
- 319.
- 320.
- 321.
- 322.
- 323.
- 324.
- 325.
- 326.
- 327.
- 328.
- 329.
- 330.
- 331.
- 332.
- 333.
- 334.
- 335.
- 336.
- 337.
- 338.
- 339.
- 340. 함 elastic+ 8363 1848 8 15:51 ? 00:02:13 /usr/lib/jvm/java-‐7-‐openjdk-‐amd64//bin/java -‐Xms4g -‐Xmx4g -‐ Xss256k -‐Djava.awt.headless=true -‐XX:+UseParNewGC -‐XX:+UseConcMarkSweepGC -‐XX:CMSInitiatingOccupancyFraction=75 -‐XX:+UseCMSInitiatingOccupancyOnly -‐XX:+HeapDumpOnOutOfMemoryError -‐XX:+DisableExplicitGC -‐Dfile.encoding=UTF-‐8 -‐Delasticsearch -‐Des.pidfile=/var/run/elasticsearch.pid -‐Des.path.home=/usr/share/elasticsearch -‐cp :/usr/ share/elasticsearch/lib/elasticsearch-‐1.4.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/ sigar/* -‐Des.default.config=/etc/elasticsearch/elasticsearch.yml -‐Des.default.path.home=/usr/share/ elasticsearch -‐Des.default.path.logs=/var/log/elasticsearch -‐Des.default.path.data=/var/lib/elasticsearch -‐ Des.default.path.work=/tmp/elasticsearch -‐Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
- 341. forensicinsight.org Page Elasticsearch 23 ▪/etc/elasticsearch/elasticsearch.yml
- 342.
- 343.
- 344.
- 345.
- 346.
- 347.
- 348.
- 349.
- 350.
- 351.
- 352.
- 353.
- 354.
- 355.
- 356.
- 357. • index.number_of_shards •index.number_of_replicas • indices.memory.index_buffer_size • index.store.type • index.translog.flush_threshold_ops • index.refresh_interval • bootstrap.mlockall
- 358. forensicinsight.org Page Elasticsearch 24 ▪Tokenization
- 359.
- 360.
- 361.
- 362.
- 363.
- 364.
- 365.
- 366.
- 367.
- 368.
- 369.
- 370.
- 371.
- 372.
- 373.
- 374.
- 375.
- 376.
- 377.
- 378.
- 379.
- 380.
- 381.
- 382.
- 383.
- 384.
- 385.
- 386.
- 387.
- 388.
- 389.
- 390.
- 391.
- 392.
- 393.
- 394.
- 395.
- 396.
- 397.
- 398.
- 399.
- 400.
- 401.
- 402.
- 403.
- 404.
- 405.
- 406. forensicinsight.org Page ELKfor Analysis 28
- 407. forensicinsight.org Page ELKfor Analysis 29 File A File B File C
- 408. forensicinsight.org Page ELKfor Analysis 30 ▪ ELK는
- 409.
- 410.
- 411.
- 412.
- 413.
- 414.
- 415.
- 416.
- 417.
- 418.
- 419.
- 420.
- 421.
- 422.
- 423.
- 424.
- 425.
- 426.
- 427.
- 428.
- 429.
- 430.
- 431.
- 432.
- 433.
- 434.
- 435.
- 436.
- 437.
- 438.
- 439.
- 440.
- 441.
- 442.
- 443.
- 444.
- 445.
- 446.
- 447.
- 448.
- 449.
- 450.
- 451.
- 452.
- 453.
- 454.
- 455.
- 456.
- 457.
- 458.
- 459. forensicinsight.org Page ELKfor Windows Event Log 31 ▪ nxlog
- 460.
- 461.
- 462.
- 463.
- 464.
- 465.
- 466.
- 467.
- 468.
- 469.
- 470.
- 471.
- 472.
- 473.
- 474. 보유 define ROOT C:Program Files (x86)nxlog Moduledir %ROOT%modules CacheDir %ROOT%data Pidfile %ROOT%datanxlog.pid SpoolDir %ROOT%data LogFile %ROOT%datanxlog.log Extension json Module xm_json /Extension # Windows Event Log Input eventlog Module im_msvistalog # Module im_mseventlog Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); /Input Output out Module om_tcp Host 192.168.1.126 Port 3515 /Output Route 1 Path internal, eventlog = out /Route Input Processor Output im_xxx pm_xxx om_xxx
- 475. forensicinsight.org Page ELKfor Windows Event Log 32 ▪ nxlog
- 476.
- 477.
- 478.
- 479.
- 480.
- 481.
- 482. 가능 Input in Module im_msvistalog Exec if ($TargetUserName == 'SYSTEM') OR ($EventType == 'VERBOSE') drop(); /Input Input unix Module im_uds uds /dev/log /Input Processor filter Module pm_filter Condition $raw_event =~ /failed/ or $raw_event =~ /error/ /Processor Output out Module om_file File /var/log/error /Output Route 1 Path unix = filter = out /Route
- 483. forensicinsight.org Page ELKfor Windows Event Log 33 binary json
- 484. forensicinsight.org Page ELKfor Windows Event Log 34
- 485. forensicinsight.org Page ELKfor Windows Event Log 35
- 486. forensicinsight.org Page ELKfor Windows Event Log 36
- 487. forensicinsight.org Page ELKfor Windows Event Log 37
- 488. forensicinsight.org Page PerformanceTest 38 Case #1 (default) Case #2 Case #2’ Case #3 Case #3’ Case #4 Case #4’ index.number_of_replicas 1 1 1 0 index.number_of_shards 5 3 3 1 index.translog.flush_threshold_ops 5000 50000 50000 50000 index.refresh_interval 5s 30s 30s -1 indices.memory.index_buffer_size 10% 30% 50% 50% index.store.type % mmapfs mmapfs mmapfs bootstrap.mlockall - - TRUE TRUE MAX_LOCKED_MEMORY - - unlimited unlimited indexing duration 180m 168m 135m 174m 143m 165m 145m docs / primary size 9,600,201/3.0GB 9,600,201/2.8GB 9,600,201/2.7GB 9,600,201/2.7GB 9,600,201/2.7GB 9,600,201/2.7GB 9,600,201/2.7GB indexing duration 65m 67m 48m 67m 48m 59m 49m docs / primary size 3,303,249/926.5M 3,303,249/888.1M 3,303,249/889.6M 3,303,249/886.7M 3,303,249/890.4M 3,303,249/894.0M 3,303,249/983.4M • 테스트
- 489.
- 490.
- 491. Intel(R) Core(TM) i5-‐2540M 2.60GHz (4 Core)/ 8G RAM / Ubuntu 14.04.2 LTS 64bit • 공통
- 492.
- 493. /etc/default/elasticsearch : ES_HEAP_SIZE=4g /etc/default/logstash : LS_HEAP_SIZE=500m / LS_OPTS=-‐w 4 raw data 2.2G raw data 828M • 일반
- 494.
- 495.
- 496.
- 497.
- 498.
- 499.
- 500.
- 501.
- 502.
- 503.
- 504.
- 505.
- 506.
- 507.
- 508.
- 509.
- 510.
- 511.
- 512.
- 513.
- 514.
- 515.
- 516.
- 517.
- 518.
- 519.
- 520.
- 521.
- 522.
- 523.
- 524.
- 525.
- 526.
- 527.
- 528.
- 529.
- 530.
- 531.
- 532.
- 533.
- 534.
- 535.
- 536.
- 537.
- 538.
- 539.
- 540.
- 541.
- 542. forensicinsight.org Page PerformanceMonitoring 39 Marvel
- 543. forensicinsight.org Page FutureWork 40 ▪ 현장에서
- 544.
- 545.
- 546.
- 547.
- 548.
- 549.
- 550.
- 551.
- 552.
- 553.
- 554.
- 555.
- 556.
- 557.
- 558.
- 559.
- 560.
- 561.
- 562.
- 563.
- 564.
- 565.
- 566.
- 567.
- 568.
- 569.
- 570.
- 571.
- 572.
- 573.
- 574.
- 575.
- 576.
- 577.
- 578.
- 579.
- 580.
- 581.
- 582.
- 583.
- 584.
- 585.
- 586.
- 587.
- 588.
- 589.
- 590.
- 591.
- 592.
- 593.
- 594.
- 595.
- 596.
- 597.
- 598.
- 599.
- 600.
- 601.
- 602.
- 603.
- 604.
- 605.
- 606.
- 607.
- 608.
- 609.
- 610.
- 611.
- 612.
- 613.
- 614.
- 615.
- 616.
- 617.
- 618. forensicinsight.org Page Reference 41 ▪ElasticSearch
- 619.
- 620.
- 621.
- 622.
- 623.
- 624.
- 625.
- 626.
- 627.
- 628.
- 629.
- 630.
- 631.
- 632.
- 633.
- 634.
- 635.
- 636.
- 637.
- 638.
- 639.
- 640.
- 641.
- 642.
- 643.
- 644.
- 645.
- 646.
- 647.
- 648.
- 649.
- 650.
- 651.
- 652.
- 653.
- 654.
- 655.
- 656.
- 657.
- 658.
- 659.
- 660.
- 661.
- 662.
- 663.
- 664.
- 665.
- 666.
- 667.
- 668.
- 669.
- 670.
- 671.
- 672.
- 673.
- 674.
- 675.
- 676.
- 677.
- 678.
- 679.
- 680.
- 681.
- 682.
- 683.
- 684. forensicinsight.org Page Questionand Answer