Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Como encontrar uma agulha num palheiro de logs

491 views

Published on

Palestra ministrada no FISL16 (2015) em Porto Alegre, RS.

Published in: Software
  • Be the first to comment

Como encontrar uma agulha num palheiro de logs

  1. 1. COMO ENCONTRAR UMA AGULHA NUM PALHEIRO DE LOGS DICKSON S. GUEDES @GUEDIZ FISL16 (2015) - PORTO ALEGRE, RS
  2. 2. UMA ANALOGIA SOBRE AGULHA E O PALHEIRO
  3. 3. QUE FERRAMENTAS PERMITIRIAM ENCONTRAR AGULHAS EM UM PALHEIRO?
  4. 4. QUE CARACTERÍSTICAS A AGULHA DEVE POSSUIR PARA SER ENCONTRADA?
  5. 5. UM REGISTRO ESPECIFICO DE LOG SERIA NOSSA AGULHA
  6. 6. TODOS OS DEMAIS REGISTROS SERIAM NOSSO PALHEIRO
  7. 7. PORQUE REGISTRAMOS EVENTOS? auditoria monitoramento recuperação de dados diagnósticos
  8. 8. MAIS DO QUE ISSO, O LOG É UMA INTERFACE DE USUÁRIO
  9. 9. quando? quem? onde? como? o quê? QUE CARACTERÍSTICAS UM LOG PRECISA TER PARA SER ENCONTRADO?
  10. 10. obter parsear filtrar minerar representar refinar interagir QUE ETAPAS PRECISARÍAMOS SEGUIR?
  11. 11. bash, grep, awk, sed, ruby, python, perl … syslog Usar uma pilha pronta como a ELK: Montar sua própria pilha QUEM SERIA NOSSO IMÃ? Elasticsearch Logstash Kibana PostgreSQL Hadoop Flume Fluentd GNUplot D3.js
  12. 12. e podem nos ajudar a enxergar melhor … apresentar CSS e HTML e imagens é o mínimo que se espera manipulação da DOM manipulação de imagens SVG "OS NAVEGADORES ESTÃO MAIS MODERNOS"
  13. 13. NOSSO EXEMPLO DE HOJE Logstash Elasticsearch Kibana D3.js
  14. 14. pipeline input | filter | output file | grep | csv twitter | grep | json imap | ruby | xmpp file | grok | elasticsearch LOGSTASH
  15. 15. banco de dados não relacional indices documentos fields full text search ELASTICSEARCH
  16. 16. visualização dashboard KIBANA
  17. 17. visualização turbinada D3.JS
  18. 18. SHOW ME THE CODE! cd $work mkdir downloads cd downloads wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsear ch-1.6.0.tar.gz wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.ta r.gz cd .. tar zxvf download/elasticsearch-1.6.0.tar.gz tar zxvf download/kibana-4.1.1-linux-x64.tar.gz tar zxvf download/logstash-1.5.2.tar.gz ./elasticsearch-1.6.0/bin/elasticsearch curl -X GET http://localhost:9200 # { # "status" : 200, # "name" : "Kick-Ass", # "cluster_name" : "elasticsearch", # "version" : { # "number" : "1.6.0", # "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", # "build_timestamp" : "2015-06-09T13:36:34Z", # "build_snapshot" : false, # "lucene_version" : "4.10.4" # }, # "tagline" : "You Know, for Search" # } ./logstash-1.5.2/bin/logstash -e 'input { stdin { } } output { stdout {
  19. 19. } }' # Logstash startup completed teste # 2015-07-08T21:42:43.129Z dba01 teste ola mundo # 2015-07-08T21:42:47.899Z dba01 ola mundo ^C # SIGINT received. Shutting down the pipeline. {:level=>:warn} # Logstash shutdown completed ./logstash-1.5.2/bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }' # Logstash startup completed ola mundo! # { # "message" => "ola mundo!", # "@version" => "1", # "@timestamp" => "2015-07-08T21:44:00.804Z", # "host" => "dba01" # } teste # { # "message" => "teste", # "@version" => "1", # "@timestamp" => "2015-07-08T21:45:00.075Z", # "host" => "dba01" # } ./logstash-1.5.2/bin/logstash -e 'input { stdin { } } output { elasticse arch { } }' # Jul 08, 2015 6:48:49 PM org.elasticsearch.node.internal.InternalNode < init> # INFORMAÇÕES: [logstash-dba01-8559-11620] version[1.5.1], pid[8559], bu ild[5e38401/2015-04-09T13:4 # 1:35Z] # Jul 08, 2015 6:48:49 PM org.elasticsearch.node.internal.InternalNode < init>
  20. 20. # INFORMAÇÕES: [logstash-dba01-8559-11620] initializing ... # Jul 08, 2015 6:48:49 PM org.elasticsearch.plugins.PluginsService <init > # INFORMAÇÕES: [logstash-dba01-8559-11620] loaded [], sites [] # Jul 08, 2015 6:48:50 PM org.elasticsearch.node.internal.InternalNode < init> # INFORMAÇÕES: [logstash-dba01-8559-11620] initialized # Jul 08, 2015 6:48:50 PM org.elasticsearch.node.internal.InternalNode s tart # INFORMAÇÕES: [logstash-dba01-8559-11620] starting ... # Jul 08, 2015 6:48:50 PM org.elasticsearch.transport.TransportService d oStart # INFORMAÇÕES: [logstash-dba01-8559-11620] bound_address {inet[/0:0:0:0: 0:0:0:0:9301]}, publish_add # ress {inet[/10.1.6.88:9301]} # Jul 08, 2015 6:48:50 PM org.elasticsearch.discovery.DiscoveryService d oStart # INFORMAÇÕES: [logstash-dba01-8559-11620] elasticsearch/4htwknhiS2S9swI vVsWTxQ # Jul 08, 2015 6:48:53 PM org.elasticsearch.cluster.service.InternalClus terService$UpdateTask run # INFORMAÇÕES: [logstash-dba01-8559-11620] detected_master [Kick-Ass][-e atoEY8TWecIb34yKjM8w][dba01 # ][inet[/10.1.6.88:9300]], added {[Kick-Ass][-eatoEY8TWecIb34yKjM8w][db a01][inet[/10.1.6.88:9300]] # ,}, reason: zen-disco-receive(from master [[Kick-Ass][-eatoEY8TWecIb34 yKjM8w][dba01][inet[/10.1.6 # .88:9300]]]) # Jul 08, 2015 6:48:53 PM org.elasticsearch.node.internal.InternalNode s tart # INFORMAÇÕES: [logstash-dba01-8559-11620] started # Logstash startup completed # no elasticsearch # [2015-07-08 18:48:53,736][INFO ][cluster.service ] [Kick-Ass] added # {[logstash-dba01-8559-11620][4htwknhiS2S9swIvVsWTxQ][dba01][inet[/10.1
  21. 21. .6.88:9301]]{client=true, # data=false},}, reason: zen-disco-receive(join from # node[[logstash-dba01-8559-11620][4htwknhiS2S9swIvVsWTxQ][dba01][inet[/ 10.1.6.88:9301]]{client=true, # data=false}]) # no logstash teste 123 teste 123 4 # em outro shell curl 'http://localhost:9200/_search?pretty' # { # "took" : 1, # "timed_out" : false, # "_shards" : { # "total" : 5, # "successful" : 5, # "failed" : 0 # }, # "hits" : { # "total" : 3, # "max_score" : 1.0, # "hits" : [ { # "_index" : "logstash-2015.07.08", # "_type" : "logs", # "_id" : "AU5vpqcN6gKLePtWPjrH", # "_score" : 1.0, # "_source":{"message":"teste 123","@version":"1","@timestamp":"20 15-07-08T21:51:09.580Z","host":"dba01"} # }, { # "_index" : "logstash-2015.07.08", # "_type" : "logs", # "_id" : "AU5vpr0F6gKLePtWPjrI", # "_score" : 1.0, # "_source":{"message":"teste 123 4","@version":"1","@timestamp":"
  22. 22. 2015-07-08T21:51:15.204Z","host":"dba01"} # }, { # "_index" : "logstash-2015.07.08", # "_type" : "logs", # "_id" : "AU5vpqDe6gKLePtWPjrG", # "_score" : 1.0, # "_source":{"message":"","@version":"1","@timestamp":"2015-07-08T 21:51:07.436Z","host":"dba01"} # } ] # } # } ./kibana-4.1.1-linux-x64/bin/kibana firefox http://localhost:5601/ cd downloads wget https://www.elastic.co/guide/en/kibana/3.0/snippets/shakespeare.jso n wget https://github.com/bly2k/files/blob/master/accounts.zip?raw=true wget https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl. gz unzip accounts.zip gunzip logs.jsonl.gz curl -XPUT http://localhost:9200/shakespeare -d ' { "mappings" : { "_default_" : { "properties" : { "speaker" : {"type": "string", "index" : "not_analyzed" }, "play_name" : {"type": "string", "index" : "not_analyzed" }, "line_id" : { "type" : "integer" }, "speech_number" : { "type" : "integer" } } }
  23. 23. } } '; # {"acknowledged":true} curl -XPOST 'localhost:9200/accounts/account/_bulk?pretty' --data-binary @accounts.json curl -XPOST 'localhost:9200/shakespeare/_bulk?pretty' --data-binary @sha kespeare.json curl -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl curl 'localhost:9200/_cat/indices?v'
  24. 24. LINKS https://github.com/mbostock/d3/wiki/Gallery https://www.elastic.co/guide/index.html https://www.dashingd3js.com/basic-building-blocks http://www.amazon.com/gp/product/B0028N4WJC/ref=as_li_qf ie=UTF8&camp=1789&creative=9325&creativeASIN=B0028N4 20 http://www.visual-literacy.org/periodic_table/periodic_table.html
  25. 25. OBRIGADO! @guediz guedes(arroba)guedesoft.net http://guedesoft.net

×