SlideShare a Scribd company logo
1 of 24
Download to read offline
Tomasz Gintowt
Elastic Stack
upgrade
03-06-2020
2
Architekt, DevOps, SysOps oraz
DBA. Jestem skupiony na
dostarczaniu rozwiązań
składowania i przetwarzania
danych. Nie są mi obce wszelkiej
maści bazy danych, systemy real
time data i streamingu.
Doradzam, wdrażam, projektuje
jak efektywnie wykorzystywać
technologie.
Tomasz Gintowt
https://www.linkedin.com/in/tomasz-gintowt/
Kibana
Beats
Version 6.4.6 Filebeat, metricbeat
ElasticSearch
Version 6.4.6 - 3 x master, 3 x
hot, 5 x cold, 1 x ingest
3
Version 6.4.6 +
ReadOnlyRest plugin
Patient
Elastic Stack cluster
4
21870 1T
2501583
5707
Active Shards
Shards per Hot node CPU
RAM
Shards per Cold node
Game plan
01 UPGRADE FROM 6.4 TO 6.8
02 UPGRADE FROM 6.8 TO 7.X
6
Check
Breaking
Changes
7
Back up
your
DATA!
Warning
Back up
your
DATA!
Back up
your
DATA!
Kibana Beats
ElasticSearch Logstash
8
1 2 3 4
Order
Rolling
Upgrade
9
Full
Cluster
Restart
Upgrade
curl -XPUT
localhost:9200/_cluster/settings -H
'Content-Type: application/json' -d
'{
"transient" :{
"cluster.routing.allocation.exclude.
_ip" : "111.111.111.111"
}
}'
10
Testing on DEV
● elasticsearch-plugin list
● elasticsearch-plugin remove
XXX --purge
● elasticsearch-plugin install
file:////tmp/
11
ES plugins
Change allocation to
"primaries"
Start ES node
/_flush/synced
Install new
plugins
Cluster Status
GREEN
12
Ansible
Stop ES node
Remove old
plugins
Upgrade ES
package
Change allocation to
"all"
- name: Enable allocation
uri:
url: http://{{ es.host }}:{{ es.port }}/_cluster/settings
method: PUT
body_format: json
body: "{{ es.enable_allocation }}"
delegate_to: "{{ es.master }}"
register: enable_allocation
until: enable_allocation.status == 200
13
Ansible
- name: Wait till relocation finish
uri:
url: http://{{ es.host }}:{{ es.port }}/_cluster/health
method: GET
register: response_shards
until: response_shards.json.unassigned_shards
== 0 and response_shards.json.initializing_shards ==
0
retries: 200
delay: 20
delegate_to: "{{ es.master }}"
14
Ansible
https://github.com/elastic/ansible-elasticsearch
15
Ansible
16
Back up
.kibana index
Check
configuration
file
Stop all
kibana
instances
Kibana upgrade
17
Problems
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
18
Upgrade Assistant
{"type":"log","@timestamp":"2020-06-01T20:33:17Z","tags":["info","mig
rations"],"pid":11550,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-06-01T20:33:24Z","tags":["info","mig
rations"],"pid":11550,"message":"Migrating .kibana_1 saved objects to
.kibana_2"}
{"type":"error","@timestamp":"2020-06-01T20:33:27Z","tags":["warning"
,"stats-collection"],"pid":11550,"level":"error","error":{"message":"
mapping set to strict, dynamic introduction of [managed] within
[index-pattern] is not allowed","name":"Error","stack":"Error:
mapping set to strict, dynamic introduction of [managed] within
[index-pattern] is not allowedn at Object.write
(/usr/share/kibana/src/server/saved_objects/migrations/core/elastic_i
ndex.js:110:23)"},"message":"mapping set to strict, dynamic
introduction of [managed] within [index-pattern] is not allowed"} 19
Errors
{"type":"log","@timestamp":"2020-06-01T20:22:18Z","tags":["fatal","r
oot"],"pid":9805,"message":"{ Error: listen EACCES: permission
denied 127.0.0.1:443n at Server.setupListenHandle [as _listen2]
(net.js:1260:19)n at listenInCluster (net.js:1325:12)n at
GetAddrInfoReqWrap.doListen (net.js:1458:7)n at
GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:62:10)n code:
'EACCES',n errno: 'EACCES',n syscall: 'listen',n address:
'127.0.0.1',n port: 443 }"}
20
Errors
https://www.elastic.co/guide/en/logstash/current/upgrading-logstash.html
If any Logstash
plugin is not
compatible with
ver 7.0, then wait
until it’s ready.
Elasticsearch 7.x does not
require Logstash 7.x.
Indexing fail if you
attempt to send the
7.x template to a
cluster ES 6.x
21
Logstash upgrade
Upgrading between
non-consecutive major
versions (e.g. 5.x to 7.x) is
not supported.
If you’re on a previous 6.x
version of Beats, upgrade
to version 6.8 first.
22
Upgrading to 6.8 is
required because the
Elasticsearch index
template was modified to
be compatible with
Elasticsearch 7.0 (the _type
setting changed from doc
to _doc).
Starting with 7.0, the fields
exported by Beats conform to
the Elastic Common Schema
(ECS).
migration.6_to_7.enabled: true
Beats
23
?
Questions
?
?
24
THANKS
Tomasz Gintowt
https://www.linkedin.com/in/tomasz-gintowt/

More Related Content

What's hot

MongoDB and the Internet of Things
MongoDB and the Internet of ThingsMongoDB and the Internet of Things
MongoDB and the Internet of Things
MongoDB
 
MongoDB Tick Data Presentation
MongoDB Tick Data PresentationMongoDB Tick Data Presentation
MongoDB Tick Data Presentation
MongoDB
 

What's hot (20)

Análisis del roadmap del Elastic Stack
Análisis del roadmap del Elastic StackAnálisis del roadmap del Elastic Stack
Análisis del roadmap del Elastic Stack
 
Druid meetup @ Netflix (11/14/2018 )
Druid meetup @ Netflix  (11/14/2018 )Druid meetup @ Netflix  (11/14/2018 )
Druid meetup @ Netflix (11/14/2018 )
 
Au cœur de la roadmap de la Suite Elastic
Au cœur de la roadmap de la Suite ElasticAu cœur de la roadmap de la Suite Elastic
Au cœur de la roadmap de la Suite Elastic
 
Building a Real-Time Gaming Analytics Service with Apache Druid
Building a Real-Time Gaming Analytics Service with Apache DruidBuilding a Real-Time Gaming Analytics Service with Apache Druid
Building a Real-Time Gaming Analytics Service with Apache Druid
 
Data Modeling IoT and Time Series data in NoSQL
Data Modeling IoT and Time Series data in NoSQLData Modeling IoT and Time Series data in NoSQL
Data Modeling IoT and Time Series data in NoSQL
 
Elastic Stack Roadmap
Elastic Stack RoadmapElastic Stack Roadmap
Elastic Stack Roadmap
 
Analytics over Terabytes of Data at Twitter
Analytics over Terabytes of Data at TwitterAnalytics over Terabytes of Data at Twitter
Analytics over Terabytes of Data at Twitter
 
Log analysis using elk
Log analysis using elkLog analysis using elk
Log analysis using elk
 
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский ДмитрийTweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
 
ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов
ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья СвиридовManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов
ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов
 
An indepth look at Google BigQuery Architecture by Felipe Hoffa of Google
An indepth look at Google BigQuery Architecture by Felipe Hoffa of GoogleAn indepth look at Google BigQuery Architecture by Felipe Hoffa of Google
An indepth look at Google BigQuery Architecture by Felipe Hoffa of Google
 
Python and MongoDB as a Market Data Platform by James Blackburn
Python and MongoDB as a Market Data Platform by James BlackburnPython and MongoDB as a Market Data Platform by James Blackburn
Python and MongoDB as a Market Data Platform by James Blackburn
 
MongoDB and the Internet of Things
MongoDB and the Internet of ThingsMongoDB and the Internet of Things
MongoDB and the Internet of Things
 
MongoDB Tick Data Presentation
MongoDB Tick Data PresentationMongoDB Tick Data Presentation
MongoDB Tick Data Presentation
 
Webinar: Managing Real Time Risk Analytics with MongoDB
Webinar: Managing Real Time Risk Analytics with MongoDB Webinar: Managing Real Time Risk Analytics with MongoDB
Webinar: Managing Real Time Risk Analytics with MongoDB
 
Dataspace presentatie
Dataspace presentatieDataspace presentatie
Dataspace presentatie
 
Eagle6 Enterprise Situational Awareness
Eagle6 Enterprise Situational AwarenessEagle6 Enterprise Situational Awareness
Eagle6 Enterprise Situational Awareness
 
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauWebinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
 
Gh registry day_1_edited
Gh registry day_1_editedGh registry day_1_edited
Gh registry day_1_edited
 
Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
 

Similar to Elastic stack upgrade

扩展世界上最大的图片Blog社区
扩展世界上最大的图片Blog社区扩展世界上最大的图片Blog社区
扩展世界上最大的图片Blog社区
yiditushe
 
Presentation Brucon - Anubisnetworks and PTCoresec
Presentation Brucon - Anubisnetworks and PTCoresecPresentation Brucon - Anubisnetworks and PTCoresec
Presentation Brucon - Anubisnetworks and PTCoresec
Tiago Henriques
 

Similar to Elastic stack upgrade (20)

Managing your Black Friday Logs NDC Oslo
Managing your  Black Friday Logs NDC OsloManaging your  Black Friday Logs NDC Oslo
Managing your Black Friday Logs NDC Oslo
 
Managing your black friday logs - Code Europe
Managing your black friday logs - Code EuropeManaging your black friday logs - Code Europe
Managing your black friday logs - Code Europe
 
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018
 
Managing your Black Friday Logs
Managing your Black Friday LogsManaging your Black Friday Logs
Managing your Black Friday Logs
 
Letgo Data Platform: A global overview
Letgo Data Platform: A global overviewLetgo Data Platform: A global overview
Letgo Data Platform: A global overview
 
Black friday logs - Scaling Elasticsearch
Black friday logs - Scaling ElasticsearchBlack friday logs - Scaling Elasticsearch
Black friday logs - Scaling Elasticsearch
 
Streaming Operational Data with MariaDB MaxScale
Streaming Operational Data with MariaDB MaxScaleStreaming Operational Data with MariaDB MaxScale
Streaming Operational Data with MariaDB MaxScale
 
Managing your black friday logs Voxxed Luxembourg
Managing your black friday logs Voxxed LuxembourgManaging your black friday logs Voxxed Luxembourg
Managing your black friday logs Voxxed Luxembourg
 
Yahoo’s next generation user profile platform
Yahoo’s next generation user profile platformYahoo’s next generation user profile platform
Yahoo’s next generation user profile platform
 
Yahoo's Next Generation User Profile Platform
Yahoo's Next Generation User Profile PlatformYahoo's Next Generation User Profile Platform
Yahoo's Next Generation User Profile Platform
 
扩展世界上最大的图片Blog社区
扩展世界上最大的图片Blog社区扩展世界上最大的图片Blog社区
扩展世界上最大的图片Blog社区
 
Fotolog: Scaling the World's Largest Photo Blogging Community
Fotolog: Scaling the World's Largest Photo Blogging CommunityFotolog: Scaling the World's Largest Photo Blogging Community
Fotolog: Scaling the World's Largest Photo Blogging Community
 
AWS re:Invent re:Cap - 데이터 분석: Amazon EC2 C4 Instance + Amazon EBS - 김일호
AWS re:Invent re:Cap - 데이터 분석: Amazon EC2 C4 Instance + Amazon EBS - 김일호AWS re:Invent re:Cap - 데이터 분석: Amazon EC2 C4 Instance + Amazon EBS - 김일호
AWS re:Invent re:Cap - 데이터 분석: Amazon EC2 C4 Instance + Amazon EBS - 김일호
 
Introducing the Jahia Log Analyzer
Introducing the Jahia Log AnalyzerIntroducing the Jahia Log Analyzer
Introducing the Jahia Log Analyzer
 
Presentation Brucon - Anubisnetworks and PTCoresec
Presentation Brucon - Anubisnetworks and PTCoresecPresentation Brucon - Anubisnetworks and PTCoresec
Presentation Brucon - Anubisnetworks and PTCoresec
 
Tracking Huge Files with Git LFS
Tracking Huge Files with Git LFSTracking Huge Files with Git LFS
Tracking Huge Files with Git LFS
 
Javantura v3 - ELK – Big Data for DevOps – Maarten Mulders
Javantura v3 - ELK – Big Data for DevOps – Maarten MuldersJavantura v3 - ELK – Big Data for DevOps – Maarten Mulders
Javantura v3 - ELK – Big Data for DevOps – Maarten Mulders
 
IOOF IT System Modernisation
IOOF IT System ModernisationIOOF IT System Modernisation
IOOF IT System Modernisation
 
Logstash and Maxmind: not just for GEOIP anymore
Logstash and Maxmind: not just for GEOIP anymoreLogstash and Maxmind: not just for GEOIP anymore
Logstash and Maxmind: not just for GEOIP anymore
 
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-BayesOSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
 

More from Anna Ossowski

[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...
[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...
[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...
Anna Ossowski
 

More from Anna Ossowski (14)

Load testing Elasticsearch with Gatling
Load testing Elasticsearch with GatlingLoad testing Elasticsearch with Gatling
Load testing Elasticsearch with Gatling
 
How Elastic Security Meets SOC Needs
How Elastic Security Meets SOC NeedsHow Elastic Security Meets SOC Needs
How Elastic Security Meets SOC Needs
 
Elastic Morocco Meetup Nov 2020
Elastic Morocco Meetup Nov 2020Elastic Morocco Meetup Nov 2020
Elastic Morocco Meetup Nov 2020
 
Elastic stockholm-meetup
Elastic stockholm-meetupElastic stockholm-meetup
Elastic stockholm-meetup
 
Elastic Morocco user group meetup June
Elastic Morocco user group meetup JuneElastic Morocco user group meetup June
Elastic Morocco user group meetup June
 
Elastic Morocco user group meetup July
Elastic Morocco user group meetup JulyElastic Morocco user group meetup July
Elastic Morocco user group meetup July
 
Visualise Covid-19 Data Using the Elastic Stack
Visualise Covid-19 Data Using the Elastic StackVisualise Covid-19 Data Using the Elastic Stack
Visualise Covid-19 Data Using the Elastic Stack
 
Tworzenie wyszukiwarek w elastic search
Tworzenie wyszukiwarek w elastic searchTworzenie wyszukiwarek w elastic search
Tworzenie wyszukiwarek w elastic search
 
Elastic South Africa Meetup September 2020
Elastic South Africa Meetup September 2020Elastic South Africa Meetup September 2020
Elastic South Africa Meetup September 2020
 
Elastic South Africa Meetup July 2020
Elastic South Africa Meetup July 2020Elastic South Africa Meetup July 2020
Elastic South Africa Meetup July 2020
 
Elastic South Africa Meetup June 2020
Elastic South Africa Meetup June 2020Elastic South Africa Meetup June 2020
Elastic South Africa Meetup June 2020
 
Elastic South Africa Meetup April 2020
Elastic South Africa Meetup April 2020Elastic South Africa Meetup April 2020
Elastic South Africa Meetup April 2020
 
[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...
[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...
[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...
 
Tracking and visualizing COVID-19 with Elastic stack
Tracking and visualizing COVID-19 with Elastic stackTracking and visualizing COVID-19 with Elastic stack
Tracking and visualizing COVID-19 with Elastic stack
 

Recently uploaded

Recently uploaded (20)

Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 

Elastic stack upgrade