This document provides an overview of using Elasticsearch with .NET, including the Elasticsearch.NET and NEST clients. It discusses connecting to Elasticsearch, mapping types, indexing, searching, updating, deleting, and aggregation. The Elasticsearch.NET client exposes low-level APIs while NEST provides a higher-level fluent API. Mapping can be done automatically, with attributes, or fluently. Searching supports structured, unstructured, and combined queries, while aggregations return averaged, summed, or counted results.

1. Getting Started With
Elasticsearch In .NET
Eng. Ahmed Abd Ellatif

2. Agenda
 Getting Started With Elasticsearch
 Different clients
 Elasticsearch.net
 NEST
 Connecting
 Mapping and its types
 Indexing
 Searching
 Updating
 Deleting
 Aggregation

3. Elasticsearch
 Elasticsearch is an open source distributed, RESTful search and analytics engine
capable of solving a growing number of use cases
 Provide REST APIs for performing tasks such as checking cluster health,
performing CRUD (Create, Read, Update, and Delete) and search operations
against indices, and executing advanced search operations such as filtering and
aggregations.
 Elasticsearch supports a variety of languages and official clients are available for
 .NET (C#)
 Java
 JavaScript (Node.js)
 Go
 PHP
 Perl
 Python
 Ruby

4. Using Elasticsearch
 on premise
 download files and extract then run (very easy)
 Using docker
 docker run -d --name elasticsearch --net elasticnetwork -p 9200:9200 -p 9300:9300 -e
"discovery.type=single-node" elasticsearch:tag
 docker run --link acce042c10fe:elasticsearch -p 5601:5601
docker.elastic.co/kibana/kibana:7.6.1
 Elastic Cloud (3 minutes or less)
 14-day trial. All free. No credit card required
 elasticsearch and kibana
 https://www.elastic.co/cloud/elasticsearch-service/signup

5. There are two official clients
 Elasticsearch.Net
 very low level
 dependency free
 Exposes all the Elasticsearch API endpoints but leaves you in control of building the
request and response bodies.
 It also comes with built in, configurable/overridable, cluster failover retry
mechanisms
 NEST
 high level client that maps all requests and responses as types
 Strongly typed interface to Elasticsearch. Fluent and classic object initializer
mappings of requests and responses.
 Uses and exposes Elasticsearch.Net
 Recommended to use NEST
 low level client if you already have the JSON that represents the request that you
wish to send and don’t wish to translate it over to the Fluent API or Object Initializer
syntax

6. Elasticsearch.Net – Low Level Client
 Install-Package Elasticsearch.Net
 to connect instantiating a new instance of the client ElasticLowLevelClient
 Connection Configuration
 Elasticsearch address
 request timeout
 basic authentication
 proxy settings

7. Indexing in Elasticsearch.Net
 Create model class
 Index by sync or async
 response.Body as byte[]
 Serialize as you want

8. Searching
 Elasticsearch Query DSL can be expressed using an anonymous type within the
request

9. NEST – High Level Client
 Install-Package NEST
 provides a strongly typed query DSL that maps one-to-one with the
Elasticsearch query DSL.
 NEST internally uses and still exposes the low level client
 ElasticLowLevelClient, from Elasticsearch.Net can use in NEST by
.LowLevel property on ElasticClient.

10. Connecting
Connection pool
send requests in a
round-robin fashion.

11. Connection options
 ApiKeyAuthentication – BasicAuthentication – ClientCertificate –
ConnectionLimit –MaxDeadTimeout –MaximumRetries - Proxy …

12. Mapping
 POCO types map to JSON documents and fields stored within the inverted
index in Elasticsearch.
 There are several ways to control mapping in NEST
 Auto mapping
 Attribute mapping
 Fluent mapping

13. Auto Mapping
 NEST has inferred mapping support for the following .NET types
.Net Type NEST
Int32 maps to "integer"
Byte maps to "short"
Decimal maps to "double"
Boolean maps to "boolean"
String maps to "text"
Guid maps to "keyword"
DateTime maps to "date"
Auto map using generic
method
Or using the non-generic
method
AutoMap(typeof(Employee))
Inferred .NET type mapping List

14. Auto mapping

15. Attribute mapping
 when you want to map differently to the inferred mapping

16. Attribute mapping
GET /posts/_mapping
When you use attributes, you must also call
.AutoMap() for the attributes to be applied

17. Fluent mapping
 each property of the POCO is explicitly mapped to an Elasticsearch type field
mapping.

18. Fluent mapping
PostDescription not mapped

19. Ignoring properties
 Using the Ignore property on PropertyNameAttribute

20. Indexing -Single Document
 synchronous method
 IndexDocumentAsync asynchronous method that returns a
Task<IIndexResponse>

21. Indexing –Multiple Document
 IndexMany - IndexManyAsync
 These methods are specific to the NEST client and wrap calls to the _bulk
endpoint

22. Multiple documents with bulk
 Bulk and BulkAsync methods and use the descriptors to customise the bulk
calls.
 Advanced bulk indexing
 BufferToBulk allows for the customisation of individual operations within the bulk
request before it is dispatched to the server.
 RetryDocumentPredicate enables fine control on deciding if a document that failed
to be indexed should be retried.
 DroppedDocumentCallback in the event a document is not indexed, even after
retrying, this delegate is called.

23. Searching
 execute a search query and get back search hits that match the query
 DSL is based on JSON and is exposed in NEST in the form of both a Fluent API
and an Object Initializer syntax
 By default, documents will be returned in _score descending order, where
the _score for each hit is the relevancy score calculated for how well the
document matched the query criteria.
 Match All query syntax
 MatchAll() return first 10 documents

24. query operations
 three types of query operations that users typically want to perform
 Structured search
 Unstructured search
 Combining queries

25. Structured search
 Structured search is about querying data that has inherent structure. Dates,
times and numbers are all structured and it is common to want to query
against fields of these types to look for exact matches
 Text can also be structured, for example, the keyword tags applied to a blog
post.
 answer to a query is always yes or no

26. Structured search
 executing a query in a filter context is that Elasticsearch is able to forgo
calculating a relevancy score, as well as cache filters for faster subsequent
performance
 When a field is only to be used for exact matching, you should consider
indexing it as a keyword datatype

27. Unstructured search
 search within full text fields in order to find the most relevant documents.
 have an analysis phase, that is, the query input is analyzed, and the
resulting terms from query analysis are compared to the terms in the inverted
index.

28. Combining queries
 extremely common scenario is to combine separate queries together
1. match documents where lead developer
first name contains ahmed
2. and where the lead developer last name
contains mohamed
3. and where the project started in 2017

29. Paging
 Pagination of results can be done by using the from and size parameters
 from defaults to 0, and size defaults to 10

30. Result window size limit
 from + size can not be more than the index.max_result_window index setting
which defaults to 10,000
 search requests take heap memory
and time proportional to from + size
 User don’t need to navigate
across 1000 page if page size 10
 Recommended to use more filters
In search criteria

31. Deep scrolling by Search After
 The idea is to use the results from the previous page to help the retrieval of the
next page.
 search_after looks for the first document which
fully or partially matches tiebreaker’s provided
value. if a document has a tiebreaker value
of "654323" and you search_after for "654" would still
match that document and return results found after it
 search_after is not a solution to jump freely to
a random page but rather to scroll many queries in parallel.
 always resolved against the latest version of the searcher

32. Deep scrolling by Scroll API
 Scroll api is recommended for efficient deep scrolling but scroll contexts are costly
and it is not recommended to use it for real time user requests
 search request returns a single page of results, the scroll API can be used to
retrieve large numbers of results (or even all results) from a single search
request, in much the same way as you would use a cursor on a traditional
database.
 e.g. in order to reindex the contents of one index into a new index use scroll
 The results that are returned from a scroll request reflect the state of the index at
the time that the initial search request was made, like a snapshot in time.
Subsequent changes to documents (index, update or delete) will only affect later
search requests.

33. Sorting
 Allows to add one or more sort on specific fields. Each sort can be reversed as
well.
 Ascending And Descending

34. Selecting fields to return
 Sometimes you don’t need to return all of the fields of a document from a search query
 Source filtering (includes – excludes)
 .Documents will now contain partial
documents, materialized from source
fields specified to include

35. Update document
 The update API allows to update a document based on a script provided.
 this operation still means a re-index of the document, it just removes some
network roundtrips and reduces chances of version conflicts between the get and
index operations. The _source field need to be enabled for this feature to work.

36. Update document By Script

37. Update by partial documents
 The update API also has a .Update<T, K> variant, where T is the document
type to update, and K is the partial document to merge.
 just use an anonymous object as your partial document

38. Update by anonymous partial documents

39. Upserting
 You can insert the partial object passed to Doc into your index if it doesn’t
already exist by using the DocAsUpsert method

40. Delete Documents based on select
 delete over multiple types and indexes

41. Delete Index
 The delete index API allows you to delete an existing index.
 The delete index API can also be applied to more than one index, or on _all
indices (be careful!).

42. Aggregations
 The aggregations framework helps provide aggregated data based on a search
query
 the Aggegrations available like
 Average
 Max
 Min
 Sum
 Count
 https://www.elastic.co/guide/en/elasticsearch/client/net-
api/1.x/aggregations.html

43. Avg aggregation
 A single-value metrics aggregation that computes the average of numeric
values that are extracted from the aggregated documents.

44. Max - Min
 Returns the maximum or minimum value among numeric values extracted
from the aggregated documents

45. Summary
 Elasticsearch.Net,NEST Clients
 Connect to elasticsearch (ElasticLowLevelClient , ElasticClient)
 Mapping (Auto , Attribute, Fluent )
 Indexing (single, multi , bulk)
 Searching (structured , unstructured , combining queries)
 Updating (script, partial document , anonymous partial , upsert)
 Delete documents
 Delete index
 Aggrgations (average , max, min)

46. Naming conventions
 All fields must be lower case.
 Use snake case (underscores) for combining words
 Group related fields into subdocuments by using dot (.) notation. Groups
typically have common prefixes. For example, if you have fields called
CPULoad and CPUSystem in a service, you would convert them into cpu.load
and cpu.system in the event.
 Avoid repeating the namespace in field names. If a word or abbreviation
appears in the namespace, it’s not needed in the field name. For example,
instead of cpu.cpu_load, use cpu.load.
 Do not use dots (.) in individual field names. Dots are reserved for grouping
related fields into subdocuments.
 Use units suffix when the metric matches one of the known units.

47. Ideas
 Index log or Aduting tables
 Stackoverflow Crawling
 Tweets analytics

48. Useful links
 https://www.elastic.co/guide/en/elasticsearch/client/net-api/6.x/nest.html
 https://github.com/elastic/elasticsearch-net/issues
 https://stackoverflow.com/questions/tagged/nest