This document provides an overview of the Elasticsearch search engine. It discusses that Elasticsearch is designed for the cloud and NoSQL generation. It is based on Apache Lucene and hides complexity with RESTful and JSON interfaces. Key points are that Elasticsearch is easy to get started with, scales horizontally by adding nodes, and is powerful with Lucene and parallel processing. The document also covers storing data as documents in types and indexes, and interacting with Elasticsearch via its REST API.
"ElasticSearch in action" by Thijs Feryn.
ElasticSearch is a really powerful search engine, NoSQL database & analytics engine. It is fast, it scales and it's a child of the Cloud/BigData generation. This talk will show you how to get things done using ElasticSearch. The focus is on doing actual work, creating actual queries and achieving actual results. Topics that will be covered: - Filters and queries - Cluster, shard and index management - Data mapping - Analyzers and tokenizers - Aggregations - ElasticSearch as part of the ELK stack - Integration in your code.
Battle of the giants: Apache Solr vs ElasticSearchRafał Kuć
Slides from my talk during ApacheCon EU 2012 - "Battle of the giants: Apache Solr vs ElasticSearch". Video available at http://player.vimeo.com/video/55645629
"ElasticSearch in action" by Thijs Feryn.
ElasticSearch is a really powerful search engine, NoSQL database & analytics engine. It is fast, it scales and it's a child of the Cloud/BigData generation. This talk will show you how to get things done using ElasticSearch. The focus is on doing actual work, creating actual queries and achieving actual results. Topics that will be covered: - Filters and queries - Cluster, shard and index management - Data mapping - Analyzers and tokenizers - Aggregations - ElasticSearch as part of the ELK stack - Integration in your code.
Battle of the giants: Apache Solr vs ElasticSearchRafał Kuć
Slides from my talk during ApacheCon EU 2012 - "Battle of the giants: Apache Solr vs ElasticSearch". Video available at http://player.vimeo.com/video/55645629
ElasticSearch introduction talk. Overview of the API, functionality, use cases. What can be achieved, how to scale? What is Kibana, how it can benefit your business.
Global introduction to elastisearch presented at BigData meetup.
Use cases, getting started, Rest CRUD API, Mapping, Search API, Query DSL with queries and filters, Analyzers, Analytics with facets and aggregations, Percolator, High Availability, Clients & Integrations, ...
From zero to hero - Easy log centralization with Logstash and ElasticsearchRafał Kuć
Presentation I gave during DevOps Days Warsaw 2014 about combining Elasticsearch, Logstash and Kibana together or use our Logsene solution instead of Elasticsearch.
Elasticsearch is a powerful, distributed, open source searching technology. By integrating Elasticsearch into your application, you instantly provide a way to search a lot of data very quickly. Elasticsearch has a RESTful API, it scales, its super fast, you can use plugins to customize it, and much more. In this talk I go over the basics of setting up Elasticsearch, creating a search index, importing your data, and doing some basic searching. I also touch on a few advanced topics that will show the flexibility of this awesome service.
Elasticsearch what is it ? How can I use it in my stack ? I will explain how to set up a working environment with Elasticsearch. The slides are in English.
ElasticSearch in Production: lessons learnedBeyondTrees
With Proquest Udini, we have created the worlds largest online article store, and aim to be the center for researchers all over the world. We connect to a 700M solr cluster for search, but have recently also implemented a search component with ElasticSearch. We will discuss how we did this, and how we want to use the 30M index for scientific citation recognition. We will highlight lessons learned in integrating ElasticSearch in our virtualized EC2 environments, and challenges aligning with our continuous deployment processes.
Deep Dive on ElasticSearch Meetup event on 23rd May '15 at www.meetup.com/abctalks
Agenda:
1) Introduction to NOSQL
2) What is ElasticSearch and why is it required
3) ElasticSearch architecture
4) Installation of ElasticSearch
5) Hands on session on ElasticSearch
Talk given for the #phpbenelux user group, March 27th in Gent (BE), with the goal of convincing developers that are used to build php/mysql apps to broaden their horizon when adding search to their site. Be sure to also have a look at the notes for the slides; they explain some of the screenshots, etc.
An accompanying blog post about this subject can be found at http://www.jurriaanpersyn.com/archives/2013/11/18/introduction-to-elasticsearch/
How EverTrue is building a donor CRM on top of ElasticSearch. We cover some of the issues around scaling ElasticSearch and which aspects of ElasticSearch we are using to deliver value to our customers.
Query DSL in Elasticsearch is a way to perform query on elasticsearch cluster.It is rich flexible query language
We can define queries of elasticsearch in JSON format.In this presentation we will see type of query dsl and its usage.
The ultimate guide for Elasticsearch pluginsItamar
Elasticsearch is a great product - for search, for scale, for analyzing data, and much more. But sometimes you need to do something that is not supported by Elasticsearch out of the box, and that's where plugins come into play.
Join me in this talk to explore the plugins land of Elasticsearch. We will discuss the various ways Elasticsearch can be extended, and the various types of plugins available to do that. By giving concrete examples and browsing the large selection of pre-made plugins, we will see how plugins can help us overcome various challenges. We will also discuss possible issues with plugins, and ways to work around them.
Finally, we will discuss scenarios in which custom plugin development is necessary and can really save the day. By showing a demo of one such scenario, and the way we built and debugged a plugin to solve it, we will complete the picture of the Elasticsearch plugin land, and hopefully inspire you to create your own!
Searching Relational Data with Elasticsearchsirensolutions
Second Galway Data Meetup, 29th April 2015
Elasticsearch was originally developed for searching flat documents. However, as real world data is inherently more complex, e.g., nested json data, relational data, interconnected documents and entities, Elasticsearch quickly evolves to support more advanced search scenarios. In this presentation, we will review existing features and plugins to support such scenarios, discuss their advantages and disadvantages, and understand which one is more appropriate for a particular scenario.
ElasticSearch introduction talk. Overview of the API, functionality, use cases. What can be achieved, how to scale? What is Kibana, how it can benefit your business.
Global introduction to elastisearch presented at BigData meetup.
Use cases, getting started, Rest CRUD API, Mapping, Search API, Query DSL with queries and filters, Analyzers, Analytics with facets and aggregations, Percolator, High Availability, Clients & Integrations, ...
From zero to hero - Easy log centralization with Logstash and ElasticsearchRafał Kuć
Presentation I gave during DevOps Days Warsaw 2014 about combining Elasticsearch, Logstash and Kibana together or use our Logsene solution instead of Elasticsearch.
Elasticsearch is a powerful, distributed, open source searching technology. By integrating Elasticsearch into your application, you instantly provide a way to search a lot of data very quickly. Elasticsearch has a RESTful API, it scales, its super fast, you can use plugins to customize it, and much more. In this talk I go over the basics of setting up Elasticsearch, creating a search index, importing your data, and doing some basic searching. I also touch on a few advanced topics that will show the flexibility of this awesome service.
Elasticsearch what is it ? How can I use it in my stack ? I will explain how to set up a working environment with Elasticsearch. The slides are in English.
ElasticSearch in Production: lessons learnedBeyondTrees
With Proquest Udini, we have created the worlds largest online article store, and aim to be the center for researchers all over the world. We connect to a 700M solr cluster for search, but have recently also implemented a search component with ElasticSearch. We will discuss how we did this, and how we want to use the 30M index for scientific citation recognition. We will highlight lessons learned in integrating ElasticSearch in our virtualized EC2 environments, and challenges aligning with our continuous deployment processes.
Deep Dive on ElasticSearch Meetup event on 23rd May '15 at www.meetup.com/abctalks
Agenda:
1) Introduction to NOSQL
2) What is ElasticSearch and why is it required
3) ElasticSearch architecture
4) Installation of ElasticSearch
5) Hands on session on ElasticSearch
Talk given for the #phpbenelux user group, March 27th in Gent (BE), with the goal of convincing developers that are used to build php/mysql apps to broaden their horizon when adding search to their site. Be sure to also have a look at the notes for the slides; they explain some of the screenshots, etc.
An accompanying blog post about this subject can be found at http://www.jurriaanpersyn.com/archives/2013/11/18/introduction-to-elasticsearch/
How EverTrue is building a donor CRM on top of ElasticSearch. We cover some of the issues around scaling ElasticSearch and which aspects of ElasticSearch we are using to deliver value to our customers.
Query DSL in Elasticsearch is a way to perform query on elasticsearch cluster.It is rich flexible query language
We can define queries of elasticsearch in JSON format.In this presentation we will see type of query dsl and its usage.
The ultimate guide for Elasticsearch pluginsItamar
Elasticsearch is a great product - for search, for scale, for analyzing data, and much more. But sometimes you need to do something that is not supported by Elasticsearch out of the box, and that's where plugins come into play.
Join me in this talk to explore the plugins land of Elasticsearch. We will discuss the various ways Elasticsearch can be extended, and the various types of plugins available to do that. By giving concrete examples and browsing the large selection of pre-made plugins, we will see how plugins can help us overcome various challenges. We will also discuss possible issues with plugins, and ways to work around them.
Finally, we will discuss scenarios in which custom plugin development is necessary and can really save the day. By showing a demo of one such scenario, and the way we built and debugged a plugin to solve it, we will complete the picture of the Elasticsearch plugin land, and hopefully inspire you to create your own!
Searching Relational Data with Elasticsearchsirensolutions
Second Galway Data Meetup, 29th April 2015
Elasticsearch was originally developed for searching flat documents. However, as real world data is inherently more complex, e.g., nested json data, relational data, interconnected documents and entities, Elasticsearch quickly evolves to support more advanced search scenarios. In this presentation, we will review existing features and plugins to support such scenarios, discuss their advantages and disadvantages, and understand which one is more appropriate for a particular scenario.
Elasticsearch as a search alternative to a relational databaseKristijan Duvnjak
The volume of data that we are working with is growing every day, the size of data is pushing us to find new intelligent solutions for problem’s put in front of us. Elasticsearch server has proved it self as an excellent full text search solution for big volume’s of data.
Elasticsearch Introduction to Data model, Search & AggregationsAlaa Elhadba
An overview of Elasticsearch features and explains performing smart search, data aggregations, and relevancy through scoring functions. How Elasticsearch works as a distributed scalable data storage. Finally, showcasing some use cases that are currently becoming core functionalities in Zalando.
During this talk, speaker provided a detailed overview of the Elasticsearch search system, gave an insight into offline search tools, and suggested how to fine-tune Elasticsearch depending on specific goals.
This presentation by Mykhailo Brodskyi (Senior Software Engineer, Сonsultant, GlobalLogic, Kharkiv), was delivered at GlobalLogic Kharkiv Java Conference 2018 on June 10, 2018.
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- ZagrebMichael Hichwa
Oracle APEX: The world's best AppDev platform
With over 400,000 developers, Oracle APEX is the industry leading enterprise low-code application development platform. But where did Oracle APEX come from? and what about it makes it resonate with so many developers? Listen to the full story directly from the creator of APEX and hear the origin story of Oracle APEX, the mission that drives it, and what to look forward to next.
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages DATAVERSITY
This webinar will cover the latest trends in advanced query languages for NoSQL databases. We’ll look at how innovations in vendor-independent standardized query languages allow NoSQL developers to query multiple types of data and multiple NoSQL databases using a single query language. We’ll see how using the right NoSQL query language promotes portability across multiple NoSQL databases, avoids vendor lock-in, and keeps your developers productive at the same time. We will be interviewing Matthias Brantner from 28msec and see on how they use JSONiq as a basis for a modern ETL framework that works on a diverse number of data sources.
Java EE microservices architecture - evolving the monolithMarkus Eisele
With the ascent of DevOps, microservices, containers, and cloud-based development platforms, the gap between state-of-the-art solutions and the technology that enterprises typically support has greatly increased. But some enterprises are now looking to bridge that gap by building microservices-based architectures on top of Java EE.
In this webcast, Red Hat Developer Advocate Markus Eisele explores the possibilities for enterprises that want to move ahead with this architecture. However, the issue is complex: Java EE wasn't built with the distributed application approach in mind, but rather as one monolithic server runtime or cluster hosting many different applications. If you're part of an enterprise development team investigating the use of microservices with Java EE, this webcast will guide you to answers for getting started.
Data Analytics Week at the San Francisco Loft
Using Data Lakes
A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data.
Speakers:
John Mallory - Principal Business Development Manager Storage (Object), AWS
Hemant Borole - Sr. Big Data Consultant, AWS
The web has changed! Users spend more time on mobile than on desktops and expect to have an amazing user experience on both. APIs are the heart of the new web as the central point of access data, encapsulating logic and providing the same data and same features for desktops and mobiles. In this workshop, Antonio will show you how to create complex APIs in an easy and quick way using API Platform built on Symfony.
A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data.
Level: Intermediate
Speakers:
Tony Nguyen - Senior Consultant, ProServe, AWS
Hannah Marlowe - Consultant - Federal, AWS
ElasticSearch - index server used as a document databaseRobert Lujo
Presentation held on 5.10.2014 on http://2014.webcampzg.org/talks/.
Although ElasticSearch (ES) primary purpose is to be used as index/search server, in its featureset ES overlaps with common NoSql database; better to say, document database.
Why this could be interesting and how this could be used effectively?
Talk overview:
- ES - history, background, philosophy, featureset overview, focus on indexing/search features
- short presentation on how to get started - installation, indexing and search/retrieving
- Database should provide following functions: store, search, retrieve -> differences between relational, document and search databases
- it is not unusual to use ES additionally as an document database (store and retrieve)
- an use-case will be presented where ES can be used as a single database in the system (benefits and drawbacks)
- what if a relational database is introduced in previosly demonstrated system (benefits and drawbacks)
ES is a nice and in reality ready-to-use example that can change perspective of development of some type of software systems.
NoSQL (Not Only SQL) is believed to be a superset of, or sometimes an intersecting set with, relational SQL databases. The concept itself is still shaping, but already now we can say for sure: NoSQL addresses the task of storing and retrieving the data of large volumes in the systems with high load. There is another very important angle in perceiving the concept:
NoSQL systems can allow storing and efficient searching of the unstructured or semi-unstructured data, like completely raw or preprocessed documents. Using the example of one world-class document retrieval system Apache SOLR (performant HTTP wrapper around Apache Lucene) as a reference we will check upon its use cases, horizontal and vertical scalability, faceted search, distribution and load balancing, crawling, extendability, linguistic support, integration with relational databases and much more.
Dmitry Kan will shortly touch upon *hot* topic of cloud computing using the famous project Apache Hadoop and will help the audience to see whether SOLR shines through the cloud.
These are the slides of our talk at Fosdem 2008 (http://fosdem.org).
We explain what REST is, how it is used in Rails, and finally we give some tips on how to architecture your application so that it follows the REST philosophy.
Building an intelligent big data application in 30 minutesClaudiu Barbura
Strata Barcelona presentation slides, a live demo of building an intelligent big data application from a web console. The tools and APIs behind are built on top of Spark, Spark SQL/Shark, Tachyon, Mesos, Cassandra, SolrCloud, iPython and include: ELT pipeline (ingestion and transformation), data warehouse explorer, export to NoSql and generated APIs, export to SolrCloud and generated APIs, predictive model building, training and publishing, dashboard UI, monitoring and instrumentation.
2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...David Pilato
Vous cherchez toujours dans vos données avec des SELECT * FROM person WHERE name like '%david%pilato%" ?
Au delà des performances obtenues, êtes-vous certain de renvoyer les résultats les plus pertinents pour vos utilisateurs d'abord ?
Venez découvrir comment un moteur de recherche vous aidera à répondre aux questions posées par vos utilisateurs, de manière pertinente et efficace, tout en apportant des fonctionnalités d'analyse des résultats et ce, quelque soit le volume...
Managing your black friday logs Voxxed LuxembourgDavid Pilato
Surveiller une application complexe n'est pas une tâche aisée, mais avec les bons outils, ce n'est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type "Black Friday" (Vendredi noir) ou période de Noël peuvent pousser votre application aux limites de ce qu'elle peut supporter, ou pire, la faire crasher. Parce que le système est fortement sollicité, il génère encore davantage de logs qui peuvent également mettre à mal votre système de supervision.
Dans cette session, j'aborderai les bonnes pratiques d'utilisation de la suite Elastic pour centraliser et monitorer vos logs. Je partagerai également avec vous quelques trucs et astuces pour vous aider à passer sans souci vos Vendredis noirs !
Nous verrons :
* Les architectures de monitoring
* Trouver la taille optimale pour l'API _bulk
* Distribuer la charge
* Taille des index et des shards
* Optimiser les E/S disque
Vous ressortirez de la session avec : des bonnes pratiques pour bâtir son système de monitoring avec la suite Elastic, le tuning avancé pour optimiser les performances d'ingestion et de recherche.
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...David Pilato
Vous cherchez toujours dans vos données avec des SELECT * FROM person WHERE name like '%david%pilato%" ?
Au delà des performances obtenues, êtes-vous certain de renvoyer les résultats les plus pertinents pour vos utilisateurs d'abord ?
Venez découvrir comment un moteur de recherche vous aidera à répondre aux questions posées par vos utilisateurs, de manière pertinente et efficace, tout en apportant des fonctionnalités d'analyse des résultats et ce, quelque soit le volume...
Managing your Black Friday Logs NDC OsloDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
Managing your black friday logs - Code EuropeDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
Managing your black Friday logs - CloudConf.ITDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Présentation d'Elasticsearch lors de Devoxx France 2012
Contenu en français
English translation available here : http://www.slideshare.net/dadoonet/elasticsearch-devoxx-france-2012-english-version
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
3. Abstract
• The need for a search engine ?
• Elasticsearch : a complete, simple and performant solution
• What about indexing Twitter ?
Make some noise on @DevoxxFR
with the #elasticsearch hashtag !
3
5. Usual use case with « SQL old school »
Having a document persisted in database :
• date attribute : 19/04/2012
• coded attribute country : FR
• Association table code/label
• Code : FR
• Label : France
• comment attribute : "There is a type error in the comment for this
product. We should call David."
Engine Elasticsearch Rivers Facets Demo Architecture Community
5
6. Usual use case with « SQL old school »
Having a document persisted in database : doc country
• date attribute : 19/04/2012 date code
• coded attribute country : FR country label
• Association table code/label comment
• Code : FR
• Label : France
• comment attribute : "There is a type error in the comment for this
product. We should call David."
Engine Elasticsearch Rivers Facets Demo Architecture Community
5
7. Usual need with « SQL old school »
• Find a document from december 2011 about france containing
error and david
• SQL :
Engine Elasticsearch Rivers Facets Demo Architecture Community
6
8. Usual need with « SQL old school »
• Find a document from december 2011 about france containing
error and david
• SQL :
SELECT
doc.*, pays.*
FROM
doc, pays
WHERE
doc.pays_code = pays.code AND
doc.date_doc > to_date('2011-12', 'yyyy-mm') AND
doc.date_doc < to_date('2012-01', 'yyyy-mm') AND
lower(pays.libelle) = 'france' AND
lower(doc.commentaire) LIKE ‘%error%' AND
lower(doc.commentaire) LIKE ‘%david%';
Engine Elasticsearch Rivers Facets Demo Architecture Community
6
9. Performance impact of like ‘%’
Engine Elasticsearch Rivers Facets Demo Architecture Community
7
10. Performance impact of like ‘%’
See also : http://www.cestpasdur.com/2012/04/01/elasticsearch-vs-mysql-recherche
Engine Elasticsearch Rivers Facets Demo Architecture Community
7
11. What is a search engine ?
Engine Elasticsearch Rivers Facets Demo Architecture Community
8
12. What is a search engine ?
• A search engine is :
• an index engine for documents
• a search engine on indexes
Engine Elasticsearch Rivers Facets Demo Architecture Community
8
13. What is a search engine ?
• A search engine is :
• an index engine for documents
• a search engine on indexes
• A search engine is more powerful to do searches :
Engine Elasticsearch Rivers Facets Demo Architecture Community
8
14. What is a search engine ?
• A search engine is :
• an index engine for documents
• a search engine on indexes
• A search engine is more powerful to do searches :
it’s designed for it !
Engine Elasticsearch Rivers Facets Demo Architecture Community
8
18. Elasticsearch
• Search engine for the NoSQL generation
Engine Elasticsearch Rivers Facets Demo Architecture Community
10
19. Elasticsearch
• Search engine for the NoSQL generation
• Based on the standard Apache Lucene library
Engine Elasticsearch Rivers Facets Demo Architecture Community
10
20. Elasticsearch
• Search engine for the NoSQL generation
• Based on the standard Apache Lucene library
• Hide the Java / Lucene complexity with standard HTTP / RESTful /
JSON services
Engine Elasticsearch Rivers Facets Demo Architecture Community
10
21. Elasticsearch
• Search engine for the NoSQL generation
• Based on the standard Apache Lucene library
• Hide the Java / Lucene complexity with standard HTTP / RESTful /
JSON services
• You can use it from whatever language or platform
Engine Elasticsearch Rivers Facets Demo Architecture Community
10
22. Elasticsearch
• Search engine for the NoSQL generation
• Based on the standard Apache Lucene library
• Hide the Java / Lucene complexity with standard HTTP / RESTful /
JSON services
• You can use it from whatever language or platform
• Add the cloud layer that Lucene miss
Engine Elasticsearch Rivers Facets Demo Architecture Community
10
23. Elasticsearch
• Search engine for the NoSQL generation
• Based on the standard Apache Lucene library
• Hide the Java / Lucene complexity with standard HTTP / RESTful /
JSON services
• You can use it from whatever language or platform
• Add the cloud layer that Lucene miss
• It’s an engine, not a graphical user interface !
Engine Elasticsearch Rivers Facets Demo Architecture Community
10
25. Key points
• Easy ! In some minutes (Zero Conf), you will get a full search engine
ready to get your documents and perform your searches.
Engine Elasticsearch Rivers Facets Demo Architecture Community
11
26. Key points
• Easy ! In some minutes (Zero Conf), you will get a full search engine
ready to get your documents and perform your searches.
• Efficient ! Just start new Elasticsearch nodes to scale horizontally
with replication and load balancing.
Engine Elasticsearch Rivers Facets Demo Architecture Community
11
27. Key points
• Easy ! In some minutes (Zero Conf), you will get a full search engine
ready to get your documents and perform your searches.
• Efficient ! Just start new Elasticsearch nodes to scale horizontally
with replication and load balancing.
• Powerful ! Lucene based product, with parallel processing to get
acceptable response time (mainly less than 100ms).
Engine Elasticsearch Rivers Facets Demo Architecture Community
11
28. Key points
• Easy ! In some minutes (Zero Conf), you will get a full search engine
ready to get your documents and perform your searches.
• Efficient ! Just start new Elasticsearch nodes to scale horizontally
with replication and load balancing.
• Powerful ! Lucene based product, with parallel processing to get
acceptable response time (mainly less than 100ms).
• Complete ! Many features : analysis and facets, percolation, rivers,
plugins, …
Engine Elasticsearch Rivers Facets Demo Architecture Community
11
30. Storing your data
• Document : A full object containing all your data (NoSQL meaning).
To think "search", you have to forget RDBMS and think "Documents"
Engine Elasticsearch Rivers Facets Demo Architecture Community
12
31. Storing your data
• Document : A full object containing all your data (NoSQL meaning).
To think "search", you have to forget RDBMS and think "Documents"
{
"text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
"created_at": "2012-04-06T20:45:36.000Z",
"source": "Twitter for iPad",
"truncated": false,
A tweet
"retweet_count": 0,
"hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
{ "text": "devoxxfr", "start": 47, "end": 55 } ],
"user": { "id": 51172224, "name": "David Pilato",
"screen_name": "dadoonet", "location": "France",
"description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL
world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
}
Engine Elasticsearch Rivers Facets Demo Architecture Community
12
32. Storing your data
• Document : A full object containing all your data (NoSQL meaning).
To think "search", you have to forget RDBMS and think "Documents"
{
"text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
"created_at": "2012-04-06T20:45:36.000Z",
"source": "Twitter for iPad",
"truncated": false,
A tweet
"retweet_count": 0,
"hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
{ "text": "devoxxfr", "start": 47, "end": 55 } ],
"user": { "id": 51172224, "name": "David Pilato",
"screen_name": "dadoonet", "location": "France",
"description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL
world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
}
• Type : Includes all documents of the same type
Engine Elasticsearch Rivers Facets Demo Architecture Community
12
33. Storing your data
• Document : A full object containing all your data (NoSQL meaning).
To think "search", you have to forget RDBMS and think "Documents"
{
"text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
"created_at": "2012-04-06T20:45:36.000Z",
"source": "Twitter for iPad",
"truncated": false,
A tweet
"retweet_count": 0,
"hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
{ "text": "devoxxfr", "start": 47, "end": 55 } ],
"user": { "id": 51172224, "name": "David Pilato",
"screen_name": "dadoonet", "location": "France",
"description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL
world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
}
• Type : Includes all documents of the same type
• Index : Logical storage of related document types
Engine Elasticsearch Rivers Facets Demo Architecture Community
12
34. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
35. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
36. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
• curl -XGET http://localhost:9200/twitter/tweet/1
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
37. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
• curl -XGET http://localhost:9200/twitter/tweet/1
• curl -XDELETE http://localhost:9200/twitter/tweet/1
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
38. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
• curl -XGET http://localhost:9200/twitter/tweet/1
• curl -XDELETE http://localhost:9200/twitter/tweet/1
Search
• curl -XGET http://localhost:9200/twitter/tweet/_search
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
39. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
• curl -XGET http://localhost:9200/twitter/tweet/1
• curl -XDELETE http://localhost:9200/twitter/tweet/1
Search
• curl -XGET http://localhost:9200/twitter/tweet/_search
• curl -XGET http://localhost:9200/twitter/_search
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
40. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
• curl -XGET http://localhost:9200/twitter/tweet/1
• curl -XDELETE http://localhost:9200/twitter/tweet/1
Search
• curl -XGET http://localhost:9200/twitter/tweet/_search
• curl -XGET http://localhost:9200/twitter/_search
• curl -XGET http://localhost:9200/_search
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
41. Playing with Elasticsearch
REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Documents
• curl -XPUT http://localhost:9200/twitter/tweet/1
• curl -XGET http://localhost:9200/twitter/tweet/1
• curl -XDELETE http://localhost:9200/twitter/tweet/1
Search
• curl -XGET http://localhost:9200/twitter/tweet/_search
• curl -XGET http://localhost:9200/twitter/_search
• curl -XGET http://localhost:9200/_search
Elasticsearch Meta Data
• curl -XGET http://localhost:9200/twitter/_status
Engine Elasticsearch Rivers Facets Demo Architecture Community
13
42. Let’s index a document
$ curl -XPUT localhost:9200/twitter/tweet/1 -d '
{
"text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
"created_at": "2012-04-06T20:45:36.000Z",
"source": "Twitter for iPad",
"truncated": false,
"retweet_count": 0,
"hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
{ "text": "devoxxfr", "start": 47, "end": 55 } ],
"user": { "id": 51172224, "name": "David Pilato",
"screen_name": "dadoonet", "location": "France",
"description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying
NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
}'
Engine Elasticsearch Rivers Facets Demo Architecture Community
14
43. Let’s index a document
$ curl -XPUT localhost:9200/twitter/tweet/1 -d '
{
"text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
"created_at": "2012-04-06T20:45:36.000Z",
"source": "Twitter for iPad",
"truncated": false,
"retweet_count": 0,
"hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
{ "text": "devoxxfr", "start": 47, "end": 55 } ],
"user": { "id": 51172224, "name": "David Pilato",
"screen_name": "dadoonet", "location": "France",
"description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying
NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
}'
{
"ok":true,
"_index":"twitter",
"_type":"tweet",
"_id":"1"
}
Engine Elasticsearch Rivers Facets Demo Architecture Community
14
44. Let’s search for documents
$ curl localhost:9200/twitter/tweet/_search?q=elasticsearch
Engine Elasticsearch Rivers Facets Demo Architecture Community
15
51. Search results
• Elasticsearch gives you the 10 first results (even on many millions) :
pagination
• You can move in the resultset
$ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&from=10&size=10"
Engine Elasticsearch Rivers Facets Demo Architecture Community
16
52. Search results
• Elasticsearch gives you the 10 first results (even on many millions) :
pagination
• You can move in the resultset
$ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&from=10&size=10"
• Scoring is computed with term frequency in a document relative to the
term frequency in the index
$ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&explain=true"
Engine Elasticsearch Rivers Facets Demo Architecture Community
16
53. Searches
QueryDSL for advanced searches
Type Description
Search for everything (useful combined with filters)
Search with term analysis, wildcards (Lucene syntax* +, -, FROM, TO, ^)
Search for individual term without analysis
Search for a text with analysis (OR is applied between tokens by default)
Wildcard search (*, ?)
Combine many criteria (MUST, MUST NOT, SHOULD)
Range search (>, >=, <, <=)
Useful for autocomplete requirements
Filtering queries
Useful to find documents that are “like” provided text
Useful to find documents that are “like” provided text with a minimal constraint on found terms
Engine Elasticsearch Rivers Facets Demo Architecture Community
17
54. Searches
QueryDSL for advanced searches
Type Description
Match All Search for everything (useful combined with filters)
QueryString Search with term analysis, wildcards (Lucene syntax* +, -, FROM, TO, ^)
Term Search for individual term without analysis
Text Search for a text with analysis (OR is applied between tokens by default)
Wildcard Wildcard search (*, ?)
Bool Combine many criteria (MUST, MUST NOT, SHOULD)
Range Range search (>, >=, <, <=)
Prefix Useful for autocomplete requirements
Filtered Filtering queries
Fuzzy like this Useful to find documents that are “like” provided text
More like this Useful to find documents that are “like” provided text with a minimal constraint on found terms
* http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html
Engine Elasticsearch Rivers Facets Demo Architecture Community
17
69. Rivers
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
70. Rivers
• CouchDB River
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
71. Rivers
• CouchDB River
• MongoDB River
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
72. Rivers
• CouchDB River
• MongoDB River
• Wikipedia River
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
73. Rivers
• CouchDB River
• MongoDB River
• Wikipedia River
• Twitter River
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
74. Rivers
• CouchDB River
• MongoDB River
• Wikipedia River
• Twitter River
• RabbitMQ River
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
75. Rivers
• CouchDB River
• MongoDB River
• Wikipedia River
• Twitter River
• RabbitMQ River
• RSS River
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
76. Rivers
• CouchDB River
• MongoDB River
• Wikipedia River
• Twitter River
• RabbitMQ River
• RSS River
• Dick Rivers
Engine Elasticsearch Rivers Facets Demo Architecture Community
25
77. Looking at your data from different points of views
RESULT ANALYSIS (IN NEAR REAL TIME)
26
103. Near Real Time Data Visualization
• Perform a matchAll search on all data
• Update screen every x seconds
• While indexing new documents
Date histogram
Term
Engine Elasticsearch Rivers Facets Demo Architecture Community
37
112. Let’s go further : sharding / replica / scalabilty
ARCHITECTURE
40
113. Glossary
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
114. Glossary
• Node : An Elasticsearch instance (~ server ?)
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
115. Glossary
• Node : An Elasticsearch instance (~ server ?)
• Cluster : A set of nodes
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
116. Glossary
• Node : An Elasticsearch instance (~ server ?)
• Cluster : A set of nodes
• Shard : an index shard where you distribute documents
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
117. Glossary
• Node : An Elasticsearch instance (~ server ?)
• Cluster : A set of nodes
• Shard : an index shard where you distribute documents
• Replica : One or more shard copy in the cluster
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
118. Glossary
• Node : An Elasticsearch instance (~ server ?)
• Cluster : A set of nodes
• Shard : an index shard where you distribute documents
• Replica : One or more shard copy in the cluster
• Primary shard : shard elected as primary in the cluster. Lucene
index documents there.
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
119. Glossary
• Node : An Elasticsearch instance (~ server ?)
• Cluster : A set of nodes
• Shard : an index shard where you distribute documents
• Replica : One or more shard copy in the cluster
• Primary shard : shard elected as primary in the cluster. Lucene
index documents there.
• Secondary shard : store replicas of primary shards
Engine Elasticsearch Rivers Facets Demo Architecture Community
41
120. Let’s create an index
Cluster
Nœud 1
Client
CURL
Engine Elasticsearch Rivers Facets Demo Architecture Community
42
121. Let’s create an index
$ curl -XPUT localhost:9200/twitter -d '{ Cluster
"index" : {
"number_of_shards" : 2,
Nœud 1
"number_of_replicas" : 1 Shard 0
}
}' Shard 1
replication rule is not satisfied
Client
CURL
Engine Elasticsearch Rivers Facets Demo Architecture Community
42
Points abord&#xE9;s :\nA quels besoins essaye t on de r&#xE9;pondre ? A quoi servirait un moteur de recherche dans mon SI ?\nComment Elasticsearch r&#xE9;pond &#xE0; ces besoins et &#xE0; bien d'autres encore\nD&#xE9;mo Live : indexation de messages Twitter ! Faites du bruit en twittant sur @devoxxfr et #elasticsearch\n