KIBANA4SOLR
Markus Klose
mk@shi-gmbh.com

Head of Training & Consulting
Agenda - State Of the Art Logging with Kibana4Solr
•
•
•
•

•

•

Me / SHI GmbH
Motivation of Kibana4Solr
What is Kibana
P...
Markus Klose / SHI GmbH
•

Head of Training & Consulting
– Project management
– Requirements Engineering
– Consulting

•
•...
SHI GmbH - Company
•

•

History
– Founded in 1994
– Since 1995 focus on Search Technologies
– Since 2000 focus on Open So...
SHI GmbH – Search Services
Consulting

Realisation



Search Strategy Consulting



Conception of Search Applications

...
Motivation 1
•

In customer projects there is always the need of monitoring data/content and user
data
– „what are the mos...
Motivation 2
•

Use and combine existing tools to
offer all our customer a good
analyzing infrastructure

•

Good experien...
What is Kibana
•

“Kibana is an open source (Apache Licensed), browser based analytics and
search interface to Logstash an...
What is Kibana
•

Create custoum Dasbords BY SEARCHING with
– bar, line diagrams
– scatter plots
– pie charts
– maps
What is kibana4solr
•

combines the Power of Kibana and Solr

•

Translator elasticsearch <-> solr

•
•

open source
deplo...
The project – kibana4solr / SEMKit
•
•
•
•

•
•
•
•

06/2013 - Idea kibana4solr
07/2013 - POC
08/2013 - Implementation
– T...
basic concept of kibana4solr

Elasticsearch
Kibana

solr4kibana

Solr
the bigger picture
•

•

SEMKit – Search Engine Migration Kit
– Framework for smooth search
engine migrations
– Connectors...
Pitfalls & Challanges
•

Minior
– Settings in Kibana
– Settings in Solr
– Tomcat settings
– To much refactoring because th...
Kibana4Solr Use Cases
•

•

•

Business Aware Monitoring
– Search tracking
– User Tracking - Trends, Conversions, Cancelat...
Prospects
•

Kibana4Solr
– Logs to Solr
– RealTimeLogging & analysis

•

SEMKit
– Search Engine Migrations
• Fast2Solr
• …...
Demo
Resources
•

Kibana4Solr/SEMKit Websites
– GitHub
– SHI GmbH
– @SHIEngineers

•

Kibana Websites
– http://www.elasticsearc...
KIBANA4SOLR
Markus Klose
mk@shi-gmbh.com

Head of Training & Consulting
checklist
•
•
•

Project
Kibana4solr
Spellcheck

•

Links?

(10 slides)
(prepare demonstration)
State of the Art Logging. Kibana4Solr is Here!
Upcoming SlideShare
Loading in...5
×

State of the Art Logging. Kibana4Solr is Here!

4,625

Published on

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin

Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown.
Configure


Published in: Technology

State of the Art Logging. Kibana4Solr is Here!

  1. 1. KIBANA4SOLR Markus Klose mk@shi-gmbh.com Head of Training & Consulting
  2. 2. Agenda - State Of the Art Logging with Kibana4Solr • • • • • • Me / SHI GmbH Motivation of Kibana4Solr What is Kibana Project – What is Kibana4Solr – Implementation / Architecture – Pitfalls – Prospect Kibana4Solr – Use cases – Demo Q &A
  3. 3. Markus Klose / SHI GmbH • Head of Training & Consulting – Project management – Requirements Engineering – Consulting • • LucidWorks Certified Lucene / Solr Trainer LucidWorks Certified Lucene / Solr Developer • mk@shi-gmbh.com
  4. 4. SHI GmbH - Company • • History – Founded in 1994 – Since 1995 focus on Search Technologies – Since 2000 focus on Open Source Solutions – Since 2011 Gold Partnership with LucidWorks – Since 2013 Partnership with elasticsearch Complete Software Service Provider – Consulting, Engineering, Training, Support – Open Source Projects – 5 Consultants, 4 Trainers, 20 Developers • 100+ years of experience in working with search technologies • 200+ customers out of different branches
  5. 5. SHI GmbH – Search Services Consulting Realisation  Search Strategy Consulting  Conception of Search Applications  Search Implementation  Execution of Proof of Concepts   Search Usability Studies Programming (Extensions, Portals, Specialized Applications, Embedded, …)  Architecture & Design Reviews  Sizing & Scaling  Relevancy Tuning  Search Semantics & Lingustics 360° Services Trainings  Classroom and On-site Support  Apache Solr Basics, Unleashed & Under The Hood  Development Support (ExpertLink)  Apache Mahout (planned)  Service Level Agreements (SLA)  Apache Hadoop (planned)  Providing Search Consultants and Developers
  6. 6. Motivation 1 • In customer projects there is always the need of monitoring data/content and user data – „what are the most searched terms?“ – „what hit was selected?“ – „are there searches with no hits?“ – „how are my products categorized?“ – „how many of my products are almost not in stock?“ – … • „…is there nothing out of the box?...“ -> NO not with Solr
  7. 7. Motivation 2 • Use and combine existing tools to offer all our customer a good analyzing infrastructure • Good experience with kibana in our elasticsearch projects  idea: enable kibana to use a solr index
  8. 8. What is Kibana • “Kibana is an open source (Apache Licensed), browser based analytics and search interface to Logstash and other timestamped data sets stored in ElasticSearch ... Kibana strives to be easy to get started with, while also being flexible and powerful”  visualize logs and time-stamped data • Seamless integration with Logstash, Apache Flume, Fluentd and others logs Logstash Elasticsearch Kibana
  9. 9. What is Kibana • Create custoum Dasbords BY SEARCHING with – bar, line diagrams – scatter plots – pie charts – maps
  10. 10. What is kibana4solr • combines the Power of Kibana and Solr • Translator elasticsearch <-> solr • • open source deployable as war file in all major application servers and servlet containers What is kibana4solr NOT • • No indexing / crawling tool No complete monitoring tool
  11. 11. The project – kibana4solr / SEMKit • • • • • • • • 06/2013 - Idea kibana4solr 07/2013 - POC 08/2013 - Implementation – Table 09/2013 – idea SEMKit – Refactoring !!! – Facetting (query…) TODAY - kibana4solr/SEMKit is still in progress 11/2013 - more features (histogram facets) 11/2013 - GitHub … 06/2013 Today
  12. 12. basic concept of kibana4solr Elasticsearch Kibana solr4kibana Solr
  13. 13. the bigger picture • • SEMKit – Search Engine Migration Kit – Framework for smooth search engine migrations – Connectors are plugable – More Connectors can be integrated easily kibana4solr is one configuration of the SEMKit – Elasticsearch <-> Solr Solr ES FAST … FAST … SEMKit Solr ES
  14. 14. Pitfalls & Challanges • Minior – Settings in Kibana – Settings in Solr – Tomcat settings – To much refactoring because the idea change within project • Major – Different syntax and feature spectrum, e.g. • labels on facets • …
  15. 15. Kibana4Solr Use Cases • • • Business Aware Monitoring – Search tracking – User Tracking - Trends, Conversions, Cancelation Perfomance Monitoring – Backend Tracking – Network Tracking Data Analysis – Measure the Development of one Product – Extract knwoledge form social data – Sediment Analysis all that in REAL TIME and NOW with one tool and based on a SOLR INDEX
  16. 16. Prospects • Kibana4Solr – Logs to Solr – RealTimeLogging & analysis • SEMKit – Search Engine Migrations • Fast2Solr • … – Multi-Channel-Monitoring • Kibana4MySQL • Kibana4SAP • …
  17. 17. Demo
  18. 18. Resources • Kibana4Solr/SEMKit Websites – GitHub – SHI GmbH – @SHIEngineers • Kibana Websites – http://www.elasticsearch.org/overview/kibana/ – http://rashidkpc.github.io/Kibana/index.html • Q &A
  19. 19. KIBANA4SOLR Markus Klose mk@shi-gmbh.com Head of Training & Consulting
  20. 20. checklist • • • Project Kibana4solr Spellcheck • Links? (10 slides) (prepare demonstration)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×