SlideShare a Scribd company logo

Using AWS Elasticsearch for fast feedback on business data

My presentation to the Wellington AWS User Group on giving the business situational awareness, anomaly detection, and process monitoring. A how to guide on using "traditional" IT tools on the generally-more-important problems of the business.

1 of 23
Download to read offline
Using the 

AWS Elasticsearch Service 

to provide
fast feedback 

for your business
Wellington AWS User Group
December 7, 2017
steven@claritycloudworks.com
+64 27 620 1237
claritycloudworks.com
Steven Ensslen
IT knows the value 

of fast feedback
The Second Way 

of DevOps is 

“shorten and amplify
feedback loops”.
https://itrevolution.com/the-three-ways-principles-underpinning-devops/
Feedback tools in software development
1. IDE feedback
2. Unit tests
3. Continuous Integration
4. SecOps tests in build pipeline
5. Application Performance Monitoring
IT has sub-second feedback
Most business leaders have the
same feedback cycle as they did
50 years ago.
“War is the realm of uncertainty;
three quarters of the factors on
which action in war is based are
wrapped in a fog of greater or
lesser uncertainty. A sensitive and
discriminating judgment is called
for; a skilled intelligence to scent
out the truth.”
–Carl von Clausewitz

19th Century Prussian General
Ad

Recommended

New Relic Plugin for Cassandra | Blue Medora
New Relic Plugin for Cassandra | Blue MedoraNew Relic Plugin for Cassandra | Blue Medora
New Relic Plugin for Cassandra | Blue MedoraBlue Medora
 
New Relic Plugin for Cassandra | Blue Medora
New Relic Plugin for Cassandra | Blue MedoraNew Relic Plugin for Cassandra | Blue Medora
New Relic Plugin for Cassandra | Blue MedoraBlue Medora
 
Palestra de abertura: Evolução e visão do Elastic Observability
Palestra de abertura: Evolução e visão do Elastic ObservabilityPalestra de abertura: Evolução e visão do Elastic Observability
Palestra de abertura: Evolução e visão do Elastic ObservabilityElasticsearch
 
Case-Study: AWS Cloud Cost Optimization with insisive cloud
Case-Study: AWS Cloud Cost Optimization with insisive cloudCase-Study: AWS Cloud Cost Optimization with insisive cloud
Case-Study: AWS Cloud Cost Optimization with insisive cloudinsisiv Labs
 
Empower your security practitioners with the Elastic Stack
Empower your security practitioners with the Elastic StackEmpower your security practitioners with the Elastic Stack
Empower your security practitioners with the Elastic StackElasticsearch
 
_Search? Made Simple: Elastic + App Search
_Search? Made Simple: Elastic + App Search_Search? Made Simple: Elastic + App Search
_Search? Made Simple: Elastic + App SearchElasticsearch
 
Security sizing meetup
Security sizing meetupSecurity sizing meetup
Security sizing meetupDaliya Spasova
 
Automatize a detecção de ameaças e evite falsos positivos
Automatize a detecção de ameaças e evite falsos positivosAutomatize a detecção de ameaças e evite falsos positivos
Automatize a detecção de ameaças e evite falsos positivosElasticsearch
 

More Related Content

What's hot

Reinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic StackReinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic StackElasticsearch
 
Construção de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizadaConstrução de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizadaElasticsearch
 
O monitoramento da infraestrutura facilitado, da ingestão ao insight
O monitoramento da infraestrutura facilitado, da ingestão ao insightO monitoramento da infraestrutura facilitado, da ingestão ao insight
O monitoramento da infraestrutura facilitado, da ingestão ao insightElasticsearch
 
Combining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityCombining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityElasticsearch
 
Azure databricks ml
Azure databricks mlAzure databricks ml
Azure databricks mlVishwas N
 
Reinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic StackReinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic StackElasticsearch
 
Business Intelligence in Minutes with Amazon Athena and Amazon QuickSight
Business Intelligence in Minutes with Amazon Athena and Amazon QuickSightBusiness Intelligence in Minutes with Amazon Athena and Amazon QuickSight
Business Intelligence in Minutes with Amazon Athena and Amazon QuickSightAmazon Web Services
 
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityCombining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityElasticsearch
 
Amazon Athena now supports new Lake Formation fine-grained security and relia...
Amazon Athena now supports new Lake Formation fine-grained security and relia...Amazon Athena now supports new Lake Formation fine-grained security and relia...
Amazon Athena now supports new Lake Formation fine-grained security and relia...Dhaval Soni
 
Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...
Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...
Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...Amazon Web Services
 
Overview of Blue Medora - New Relic Plugin for Cisco UCS
Overview of Blue Medora - New Relic Plugin for Cisco UCSOverview of Blue Medora - New Relic Plugin for Cisco UCS
Overview of Blue Medora - New Relic Plugin for Cisco UCSBlue Medora
 
Elastic Cloud and FedRAMP
Elastic Cloud and FedRAMPElastic Cloud and FedRAMP
Elastic Cloud and FedRAMPElasticsearch
 
Cloud storage providers comparative study
Cloud storage providers   comparative studyCloud storage providers   comparative study
Cloud storage providers comparative studyDayanand Shanmugham
 
Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...
Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...
Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...Dhaval Soni
 
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...Amazon Web Services
 
Meetup ilm virtual emea
Meetup ilm virtual emeaMeetup ilm virtual emea
Meetup ilm virtual emeaDaliya Spasova
 

What's hot (20)

Reinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic StackReinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic Stack
 
Construção de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizadaConstrução de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizada
 
O monitoramento da infraestrutura facilitado, da ingestão ao insight
O monitoramento da infraestrutura facilitado, da ingestão ao insightO monitoramento da infraestrutura facilitado, da ingestão ao insight
O monitoramento da infraestrutura facilitado, da ingestão ao insight
 
Women Who Code
Women Who CodeWomen Who Code
Women Who Code
 
Combining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityCombining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observability
 
Azure databricks ml
Azure databricks mlAzure databricks ml
Azure databricks ml
 
Reinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic StackReinventing enterprise defense with the Elastic Stack
Reinventing enterprise defense with the Elastic Stack
 
Business Intelligence in Minutes with Amazon Athena and Amazon QuickSight
Business Intelligence in Minutes with Amazon Athena and Amazon QuickSightBusiness Intelligence in Minutes with Amazon Athena and Amazon QuickSight
Business Intelligence in Minutes with Amazon Athena and Amazon QuickSight
 
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityCombining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified Observability
 
Amazon Athena now supports new Lake Formation fine-grained security and relia...
Amazon Athena now supports new Lake Formation fine-grained security and relia...Amazon Athena now supports new Lake Formation fine-grained security and relia...
Amazon Athena now supports new Lake Formation fine-grained security and relia...
 
Enhancing Databases with Search
Enhancing Databases with SearchEnhancing Databases with Search
Enhancing Databases with Search
 
AWS Terms
AWS TermsAWS Terms
AWS Terms
 
Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...
Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...
Keeping Pace With The Cloud: Managing and Optimizing as You Scale - ENT316 - ...
 
Overview of Blue Medora - New Relic Plugin for Cisco UCS
Overview of Blue Medora - New Relic Plugin for Cisco UCSOverview of Blue Medora - New Relic Plugin for Cisco UCS
Overview of Blue Medora - New Relic Plugin for Cisco UCS
 
Elastic Cloud and FedRAMP
Elastic Cloud and FedRAMPElastic Cloud and FedRAMP
Elastic Cloud and FedRAMP
 
Cloud storage providers comparative study
Cloud storage providers   comparative studyCloud storage providers   comparative study
Cloud storage providers comparative study
 
Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...
Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...
Amazon s3 adds new s3 event notifications for s3 lifecycle, s3 intelligent ti...
 
Keynote
KeynoteKeynote
Keynote
 
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
 
Meetup ilm virtual emea
Meetup ilm virtual emeaMeetup ilm virtual emea
Meetup ilm virtual emea
 

Similar to Using AWS Elasticsearch for fast feedback on business data

BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceAmazon Web Services
 
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceAmazon Web Services
 
Amazon Web Services (AWS) Presentation
Amazon Web Services (AWS) PresentationAmazon Web Services (AWS) Presentation
Amazon Web Services (AWS) PresentationSunil Jagani
 
Building Data Analytics pipelines in the cloud using serverless technology
Building Data Analytics pipelines in the cloud using serverless technologyBuilding Data Analytics pipelines in the cloud using serverless technology
Building Data Analytics pipelines in the cloud using serverless technologyDomino Data Lab
 
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAmazon Web Services
 
Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...
Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...
Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...Amazon Web Services
 
2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개
2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개 2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개
2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개 Amazon Web Services Korea
 
Log Analytics with Amazon Elasticsearch Service & Kibana
Log Analytics with Amazon Elasticsearch Service & KibanaLog Analytics with Amazon Elasticsearch Service & Kibana
Log Analytics with Amazon Elasticsearch Service & KibanaAmazon Web Services
 
Enterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionEnterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionDmitry Anoshin
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Amazon Web Services
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Amazon Web Services
 
Filebeat Elastic Search Presentation.pptx
Filebeat Elastic Search Presentation.pptxFilebeat Elastic Search Presentation.pptx
Filebeat Elastic Search Presentation.pptxKnoldus Inc.
 
Managing Data with Voume Velocity, and Variety with Amazon ElastiCache for Redis
Managing Data with Voume Velocity, and Variety with Amazon ElastiCache for RedisManaging Data with Voume Velocity, and Variety with Amazon ElastiCache for Redis
Managing Data with Voume Velocity, and Variety with Amazon ElastiCache for RedisAmazon Web Services
 
Qui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in ActionQui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in ActionGlobalLogic Ukraine
 
Using Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SFUsing Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SFAmazon Web Services
 
Scaling web application in the Cloud
Scaling web application in the CloudScaling web application in the Cloud
Scaling web application in the CloudFederico Feroldi
 

Similar to Using AWS Elasticsearch for fast feedback on business data (20)

BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
 
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
 
Amazon Web Services (AWS) Presentation
Amazon Web Services (AWS) PresentationAmazon Web Services (AWS) Presentation
Amazon Web Services (AWS) Presentation
 
Building Data Analytics pipelines in the cloud using serverless technology
Building Data Analytics pipelines in the cloud using serverless technologyBuilding Data Analytics pipelines in the cloud using serverless technology
Building Data Analytics pipelines in the cloud using serverless technology
 
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
 
Amazon quicksight
Amazon quicksightAmazon quicksight
Amazon quicksight
 
Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...
Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...
Managing Data with Amazon ElastiCache for Redis - August 2016 Monthly Webinar...
 
2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개
2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개 2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개
2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개
 
AWS Big Data Solution Days
AWS Big Data Solution DaysAWS Big Data Solution Days
AWS Big Data Solution Days
 
Using Data Lakes
Using Data Lakes Using Data Lakes
Using Data Lakes
 
Log Analytics with Amazon Elasticsearch Service & Kibana
Log Analytics with Amazon Elasticsearch Service & KibanaLog Analytics with Amazon Elasticsearch Service & Kibana
Log Analytics with Amazon Elasticsearch Service & Kibana
 
Enterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionEnterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
 
Filebeat Elastic Search Presentation.pptx
Filebeat Elastic Search Presentation.pptxFilebeat Elastic Search Presentation.pptx
Filebeat Elastic Search Presentation.pptx
 
Managing Data with Voume Velocity, and Variety with Amazon ElastiCache for Redis
Managing Data with Voume Velocity, and Variety with Amazon ElastiCache for RedisManaging Data with Voume Velocity, and Variety with Amazon ElastiCache for Redis
Managing Data with Voume Velocity, and Variety with Amazon ElastiCache for Redis
 
Qui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in ActionQui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in Action
 
Using Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SFUsing Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SF
 
Scaling web application in the Cloud
Scaling web application in the CloudScaling web application in the Cloud
Scaling web application in the Cloud
 
Amazon Redshift Deep Dive
Amazon Redshift Deep Dive Amazon Redshift Deep Dive
Amazon Redshift Deep Dive
 

Recently uploaded

UGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptxUGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptxRitesh Sahu
 
Model Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfModel Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfgalfinprihardiputra0
 
Augmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical ProfessionalsAugmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical Professionalsthirdeyegen65
 
AWS Overview of AWS Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS  Clarify, Feature Store, Hyper parameter TuningAWS Overview of AWS  Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS Clarify, Feature Store, Hyper parameter TuningVarun Garg
 
Red shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's CyberspaceRed shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's Cyberspacesttyk
 
Augmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & DefenseAugmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & Defensethirdeyegen65
 
Biometrics Technology Intresting PPT
Biometrics Technology Intresting PPTBiometrics Technology Intresting PPT
Biometrics Technology Intresting PPTPraveenKumarThota7
 
Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Damar Juniarto
 
Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...ssuser7b7f4e
 

Recently uploaded (9)

UGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptxUGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptx
 
Model Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfModel Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdf
 
Augmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical ProfessionalsAugmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical Professionals
 
AWS Overview of AWS Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS  Clarify, Feature Store, Hyper parameter TuningAWS Overview of AWS  Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS Clarify, Feature Store, Hyper parameter Tuning
 
Red shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's CyberspaceRed shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's Cyberspace
 
Augmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & DefenseAugmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & Defense
 
Biometrics Technology Intresting PPT
Biometrics Technology Intresting PPTBiometrics Technology Intresting PPT
Biometrics Technology Intresting PPT
 
Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023
 
Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...
 

Using AWS Elasticsearch for fast feedback on business data

  • 1. Using the 
 AWS Elasticsearch Service 
 to provide fast feedback 
 for your business Wellington AWS User Group December 7, 2017
  • 2. steven@claritycloudworks.com +64 27 620 1237 claritycloudworks.com Steven Ensslen
  • 3. IT knows the value 
 of fast feedback The Second Way 
 of DevOps is 
 “shorten and amplify feedback loops”. https://itrevolution.com/the-three-ways-principles-underpinning-devops/
  • 4. Feedback tools in software development 1. IDE feedback 2. Unit tests 3. Continuous Integration 4. SecOps tests in build pipeline 5. Application Performance Monitoring
  • 5. IT has sub-second feedback Most business leaders have the same feedback cycle as they did 50 years ago.
  • 6. “War is the realm of uncertainty; three quarters of the factors on which action in war is based are wrapped in a fog of greater or lesser uncertainty. A sensitive and discriminating judgment is called for; a skilled intelligence to scent out the truth.” –Carl von Clausewitz
 19th Century Prussian General
  • 7. The fog of business Information is late, disconnected, and vague.
  • 8. Totals Hide Information If your analysis uses fixed- length periods you will miss trends. Weekly hours Monday Tuesday Wednesday Thursday Friday Saturday Sunday Total Constant 8 8 8 8 8 0 0 40 Spiky 2 5 3 12 11 5 2 40 Constant Spiky
  • 9. Arbitrary Boundaries Hide Trends 0 20 40 60 80 2017-9-1 2017-9-10 2017-9-19 2017-9-28 2017-10-7 2017-10-16 2017-10-25 September Total = 1865 October Total = 1880 November 
 << 
 October
  • 10. “If it moves, graph it. 
 If it doesn’t move, 
 graph it anyway, 
 just in case it does.” –Etsy
  • 11. Elasticsearch 1. Makes fast feedback easy,
 both for IT and business people 2. Makes awesome graphs 3. Is super fast and massively scalable
  • 12. What is ELK? Elasticsearch is a RESTful API and clustering software over Apache Lucene, which is a document database optimised for search. Logstash is a data ingestion tool. It transforms and ships data across networks. Beats are a lighter, less-capable agent for Elasticsearch. Kibana is a powerful ad hoc query tool that quickly creates beautiful graphs.
  • 15. Business Intelligence Tips 1. Work closely with a champion customer 2. Start small, both in scope and audience 3. Reuse the language and labels of your audience 4. Reuse the time periods that are already part of your processes 
 (i.e. financial quarters) 5. Transform data and index the things that your audience think about, like sessions, products, and orders, especially if your raw data doesn’t quite map to them
  • 16. Test Driven Design 1. Use Kinesis Firehose to save all of your production stream to S3, 
 then apply lifecycle policies 2. At the very beginning, play a static, fake data set. Replay feature of the Logstash sleep plugin. Do not develop or test with a random generator! 3. Whenever you encounter undesirable behaviour, 
 add the recording segment to your test suite. 4. Test Elasticsearch with xUnit in your code pipleline 5. Monitor Kibana and Elasticsearch with your APM
  • 17. Elasticsearch tips 1. Predefine your index mapping 2. Only use one type per index (ES6.x removes support for many types per index) 3. Partition your index by time, typically by day 4. There are no joins, use Lambda to enrich data before loading it into Elasticsearch 5. Ideally an ES cluster has 3 small masters and < 10 workers, 
 above 10 nodes scale-up before scale-out
  • 18. AWS ES tips 1. The Elasticsearch port is 80, not 9200 2. Do NOT expose ES or Kibana to the public internet! 3. Start bigger, then shrink (IMHO, seven M4.large is big) 4. Do not use ES as a data store; use RDS, or DynamoDB, or Redshift, or S3 with Athena
  • 19. Cloudwatch vs ES 1. Only fixed thresholds for alerts 2. Much easier to use 3. Much less admin 4. Scales elastically
  • 20. Kinesis Analytics vs ES 1. Simpler for detection 2. Elastic scaling 3. No graphs 4. MillisBehindLatest can be minutes!
  • 21. Athena & Quicksight vs ES 1. Massive, admin-free scaling 2. Need to add Lambda, even then runs periodically not event driven 3. Worse latency 4. Conceivably could be more expensive 
 (1440 scheduled queries * ?)
  • 22. Photo: Micheal Filion, https://www.flickr.com/photos/mike9alive/ Situational Awareness 100% uptime on the GPS of this car isn’t going to help anything
  • 23. Clarity Cloudworks illuminating issues before they become problems