Elastic Migration Challenges What is new in Elastic 6.5 Data Migration Strategy

1. ElasticSearch
Migration 2.x to 6.x
Presented by Pankaj Gajjar
1st Dec 2018.

2. 2
Agenda
1. Intro & UseCase
2. ElasticSearch Migration Challenges
3. Migration
4. What is changes in 6.5?
5. AMA

3. Intro & UseCase
3

4. Pankaj Gajjar.
Enterprise Architect (TOGAF® Certified)
13+ years experience in Product Development.
10 years experience in CONTENTSERV
Contact me :
https://about.me/pankajgajjar

5. What We Do ?

6. 6
• We used the ElasticSearch for Application Search
• Only Searchable Data Store into Elastic Search.
• Search Data is updated to Elastic using Open Source Messaging System.
• OLTP Data is store into Open Source RDBMS
UseCase - Application Search

7. ElasticSearch Migration Challenges
7

8. 8
ElasticSearch Migration Challenges
1. Indices creation
2. Multiple mapping types
3. Client
4. Cluster
5. Mapping
6. Document Apis
7. Security

9. 9
• Elasticsearch 6.x can read indices created in version 5.x
• An Elasticsearch 6.x node will not start in the presence of indices created in a version of
Elasticsearch before 5.x
• Indices created in Elasticsearch 2.x or before will need to be reindexed with Elasticsearch 5.x
1. Indices creation

10. 10
• Indices created in 2.x has support multiple mapping types
• Multiple mapping types per index has been removed in 6.x
• New indices will be restricted to a single type in 6.x
• Indices created in 5.x will continue to support multiple mapping types.
2. Multiple mapping types

11. 11
• Java High Level REST Client is designed in 5.6.x
• Works on top of the Java Low Level REST client.
• Release in 2016
• API can be called synchronously or asynchronously
• request marshalling and response un-marshalling is handled by the client itself
• Plan to replace the TransportClient in a near future
• Exists since 8 Feb 2010 (First commit
• Deprecated in 7.0.0.
• Removed in 8.0.0
3. Client Changes

12. 12
• Cluster name no longer allowed in path.data
4. Cluster Changes

13. 13
• Elasticsearch 2.x/5.x recognized the strings true, false, on, off, yes, no, 0, 1 as booleans
• Elasticsearch 6.x recognizes only the strings true and false as booleans and will throw an error
otherwise
• The _all meta field is now disabled by default
• The query_string and simple_query_string queries that previously used _all to search
• Unrecognized match_mapping_type options not silently ignored and will throw an error.
• Elasticsearch max field size
• If there is one object mapping, then the depth is 2 , etc. The default is 20 .
index.mapping.nested_fields.limit. The maximum number of nested fields in an index, defaults to
50
• String fields replaced by text/keyword fields.
• This allows to perform full-text search on the original field name and to
sort and run aggregations on the sub keyword field
5. Mapping Changes

14. 14
• Version type force removed
• Upserts no longer support versions
• created field removed in the Index API
• found field removed in the Delete API
6. Document Apis Changes

15. 15
• Opened X-Pack (Starting with version 6.3)
• X-Pack features: security, alerting, monitoring, reporting, graph analytics, dedicated APM UIs,
and machine learning
• it is open under custom elastic license
7. Security Change

16. Migration
16

17. 17
• Migration between minor versions — e.g. 6.x to 6.y — can be performed by upgrading one node at
a time.
• Migration between consecutive major versions — e.g. 5.x to 6.x — requires a full cluster restart.
• Migration between non-consecutive major versions — e.g. 2.x to 6.x — is not supported mean
reindex need
General rule

18. 18
• Review the breaking changes for changes that affect your application.
• Check the deprecation log to see if you are using any deprecated features.
• If you use custom plugins, make sure compatible versions are available.
• Test upgrades in a dev environment before upgrading your production cluster.
• Back up your data before upgrading.
• You cannot roll back to an earlier version unless you have a backup of your data.
Before upgrading Elasticsearch

19. 19
_reindex api or Logstash
2.x -> 6.x

20. 20
Rolling upgrade
5.6.x -> 6.0.x
• Disable shard allocation
• Shut down a single node
• Upgrade the node you shut down
• Start the upgraded node
• Reenable shard allocation
• Wait for the node to recover
• Repeat - Cluster - all nodes

21. 21
Rolling upgrade
5.6.x -> 6.0.x

22. 22
Full cluster restart upgrade
5.0-5.5 -> 6.x

23. 23
snapshot & restore
2.x -> 6.x
• snapshot & restore _snapshot in the 2.x cluster
• restore 2.x to 5.x in the new cluster 5.x
• 5.x to 6.x enabled
• 2.x to 6.x not allowed

24. 24
Elastic Stack 6.x(3) Upgrade Guide

25. 25
Elastic Stack 6.x(3) Upgrade Guide

26. 26
https://www.elastic.co/products/upgrade_guide
Elastic Stack 6.x(3) Upgrade Guide

27. ElasticSearch 6.5 Changes ?
27

28. 28
Major Changes :
• Replicate data from one cluster to another with cross-cluster replication (beta)(commercial).
• You'll also find _source only snapshots,
• support for Java 11 & ODBC driver (alpha)
• G1 Garbage Collector
Logging Changes
%node_name in logging pattern
Search changes
Scroll
SQL plugin changes
Grouping by columns with missing values will create an additional group
Client changes
Deprecate Retry.withBackoff methods that take Settings
Security changes
Custom role providers interface change
Settings changes
Audit logfile local node info
Highlights :: Changes in ES 6.5

29. 29

30. AMA
30

31. 31
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.0.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html
https://www.elastic.co/blog/elastic-stack-6-5-0-released
Ref. for you

32. Thank You.

33. 33