State of the Art Logging. Kibana4Solr is Here!
Upcoming SlideShare
Loading in...5
×
 

State of the Art Logging. Kibana4Solr is Here!

on

  • 3,684 views

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin ...

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin

Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown.
Configure


Statistics

Views

Total Views
3,684
Views on SlideShare
2,857
Embed Views
827

Actions

Likes
4
Downloads
11
Comments
0

5 Embeds 827

http://www.scoop.it 625
http://www.lucenerevolution.org 190
https://twitter.com 9
http://webcache.googleusercontent.com 2
http://lucenerevolution.org 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    State of the Art Logging. Kibana4Solr is Here! State of the Art Logging. Kibana4Solr is Here! Presentation Transcript

    • KIBANA4SOLR Markus Klose mk@shi-gmbh.com Head of Training & Consulting
    • Agenda - State Of the Art Logging with Kibana4Solr • • • • • • Me / SHI GmbH Motivation of Kibana4Solr What is Kibana Project – What is Kibana4Solr – Implementation / Architecture – Pitfalls – Prospect Kibana4Solr – Use cases – Demo Q &A
    • Markus Klose / SHI GmbH • Head of Training & Consulting – Project management – Requirements Engineering – Consulting • • LucidWorks Certified Lucene / Solr Trainer LucidWorks Certified Lucene / Solr Developer • mk@shi-gmbh.com
    • SHI GmbH - Company • • History – Founded in 1994 – Since 1995 focus on Search Technologies – Since 2000 focus on Open Source Solutions – Since 2011 Gold Partnership with LucidWorks – Since 2013 Partnership with elasticsearch Complete Software Service Provider – Consulting, Engineering, Training, Support – Open Source Projects – 5 Consultants, 4 Trainers, 20 Developers • 100+ years of experience in working with search technologies • 200+ customers out of different branches
    • SHI GmbH – Search Services Consulting Realisation  Search Strategy Consulting  Conception of Search Applications  Search Implementation  Execution of Proof of Concepts   Search Usability Studies Programming (Extensions, Portals, Specialized Applications, Embedded, …)  Architecture & Design Reviews  Sizing & Scaling  Relevancy Tuning  Search Semantics & Lingustics 360° Services Trainings  Classroom and On-site Support  Apache Solr Basics, Unleashed & Under The Hood  Development Support (ExpertLink)  Apache Mahout (planned)  Service Level Agreements (SLA)  Apache Hadoop (planned)  Providing Search Consultants and Developers
    • Motivation 1 • In customer projects there is always the need of monitoring data/content and user data – „what are the most searched terms?“ – „what hit was selected?“ – „are there searches with no hits?“ – „how are my products categorized?“ – „how many of my products are almost not in stock?“ – … • „…is there nothing out of the box?...“ -> NO not with Solr
    • Motivation 2 • Use and combine existing tools to offer all our customer a good analyzing infrastructure • Good experience with kibana in our elasticsearch projects  idea: enable kibana to use a solr index
    • What is Kibana • “Kibana is an open source (Apache Licensed), browser based analytics and search interface to Logstash and other timestamped data sets stored in ElasticSearch ... Kibana strives to be easy to get started with, while also being flexible and powerful”  visualize logs and time-stamped data • Seamless integration with Logstash, Apache Flume, Fluentd and others logs Logstash Elasticsearch Kibana
    • What is Kibana • Create custoum Dasbords BY SEARCHING with – bar, line diagrams – scatter plots – pie charts – maps
    • What is kibana4solr • combines the Power of Kibana and Solr • Translator elasticsearch <-> solr • • open source deployable as war file in all major application servers and servlet containers What is kibana4solr NOT • • No indexing / crawling tool No complete monitoring tool
    • The project – kibana4solr / SEMKit • • • • • • • • 06/2013 - Idea kibana4solr 07/2013 - POC 08/2013 - Implementation – Table 09/2013 – idea SEMKit – Refactoring !!! – Facetting (query…) TODAY - kibana4solr/SEMKit is still in progress 11/2013 - more features (histogram facets) 11/2013 - GitHub … 06/2013 Today
    • basic concept of kibana4solr Elasticsearch Kibana solr4kibana Solr
    • the bigger picture • • SEMKit – Search Engine Migration Kit – Framework for smooth search engine migrations – Connectors are plugable – More Connectors can be integrated easily kibana4solr is one configuration of the SEMKit – Elasticsearch <-> Solr Solr ES FAST … FAST … SEMKit Solr ES
    • Pitfalls & Challanges • Minior – Settings in Kibana – Settings in Solr – Tomcat settings – To much refactoring because the idea change within project • Major – Different syntax and feature spectrum, e.g. • labels on facets • …
    • Kibana4Solr Use Cases • • • Business Aware Monitoring – Search tracking – User Tracking - Trends, Conversions, Cancelation Perfomance Monitoring – Backend Tracking – Network Tracking Data Analysis – Measure the Development of one Product – Extract knwoledge form social data – Sediment Analysis all that in REAL TIME and NOW with one tool and based on a SOLR INDEX
    • Prospects • Kibana4Solr – Logs to Solr – RealTimeLogging & analysis • SEMKit – Search Engine Migrations • Fast2Solr • … – Multi-Channel-Monitoring • Kibana4MySQL • Kibana4SAP • …
    • Demo
    • Resources • Kibana4Solr/SEMKit Websites – GitHub – SHI GmbH – @SHIEngineers • Kibana Websites – http://www.elasticsearch.org/overview/kibana/ – http://rashidkpc.github.io/Kibana/index.html • Q &A
    • KIBANA4SOLR Markus Klose mk@shi-gmbh.com Head of Training & Consulting
    • checklist • • • Project Kibana4solr Spellcheck • Links? (10 slides) (prepare demonstration)