SlideShare a Scribd company logo
C L O U D Y W I T H A
C H A N C E O F S C A L E
M I T C H P I R T L E
E L A S T I C S E A R C H M E E T U P
C A P I TA L O N E L A B S , A R L I N G T O N VA
D E C E M B E R 2 0 1 6
A B O U T M E
• FOSS founder/contributor
• Startupper
• Technology Fellow, Capital One
• Skate punk
• Musician
• Football coach
A B O U T Y O U
• Just a user
• In production by hand
• Fully automated in production
• I should be giving this talk
A B O U T T H I S
S E S S I O N
• ES from an Ops perspective
• The good, bad and truncated
• What to expect
H O W I T A L L S TA R T E D
– M E , T H E E A R LY- O N C L U E L E S S V E R S I O N
“All I want is a web crawler, this can’t be too hard to setup.”
– M E , T H E E A R LY- O N C L U E L E S S V E R S I O N
“All I want is a web crawler, this can’t be too hard to setup.”
F E S S U P !
• Fess: http://fess.codelibs.org/
• FOSS
• Simple web UI for setup
• Multiple sources
• Multiple types
• Supports threads, throttling
• ES as persistence store
S C E N A R I O S .
L O C A L S E T U P
• Fess comes as a self-contained package
• All you need is a java runtime and you’re good
I ’ M T H I N K I N G B I G G E R T H A N T H AT.
S C R E W L O C A L H O S T,
Cloudy with a chance of scale
W H Y P U B L I C C L O U D
• Horizontal scale
• Access to integrated services - storage, load balancing, etc
• Opportunity to automate. All. The. Things.
S I N G L E I N S TA N C E .
• Quick to setup, uses embedded
Elasticsearch + plugins
• Quick to duplicate
• Easy to maintain
S I N G L E I N S TA N C E : 

I S S U E S
• Single point of failure
• Zero scale opportunity
• Fully manual effort
• Good golly filesystem access is
SLOW
C L U S T E R E D
S E R V I C E .
• Horizontal scale
• Easier to expand
C L U S T E R E D
S E R V I C E : I S S U E S
• Needs separate instance of
Elasticsearch, requires plugin
installation (version specific)
• Even harder to setup: Shards or
replicas? Master or data or both, or
neither?
• Even harder to automate
• Requires additional tooling for
operations (logs, events)
H O W D O I D O
T H I S ?
• ElasticSearch Cluster: Configuration
& Best Practices

(http://www.xmsxmx.com/
elasticsearch-cluster-configuration-
best-practices/)
A M A Z O N E L A S T I C
S E R V I C E
• No setup
• Simple scale
• Fully automated
• Advanced configuration by default
A M A Z O N E L A S T I C
S E R V I C E : I S S U E S
• No custom plugins (no Fess!)
• Service limits (number of nodes per
cluster, etc)
• Lack of customization options
T H E L I N K S
• https://aws.amazon.com/elasticsearch-service/
• http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/
aes-limits.html
Cloudy with a chance of scale
S T U M P T H E S P E A K E R
M Y FA V O R I T E G A M E
T H A N K Y O U V E RY
G R A Z I E .
• @mitchitized
• mitch.pirtle@capitalone.com
• github.com/spacemonkey
• about.me/mitchitized
• www.slideshare.net/
spacemonkeylabs

More Related Content

Similar to Cloudy with a chance of scale

Serverless WordPress & next Interface of WordPress
Serverless WordPress & next Interface of WordPressServerless WordPress & next Interface of WordPress
Serverless WordPress & next Interface of WordPress
Hidetaka Okamoto
 
Data Modelling at Scale
Data Modelling at ScaleData Modelling at Scale
Data Modelling at Scale
David Simons
 
WordPress for the 99%
WordPress for the 99%WordPress for the 99%
WordPress for the 99%
Stephanie Leary
 
ITB2016 - ColdBox 4 Modules
ITB2016 - ColdBox 4 ModulesITB2016 - ColdBox 4 Modules
ITB2016 - ColdBox 4 Modules
Ortus Solutions, Corp
 
Data Scientist's Daily Life
Data Scientist's Daily LifeData Scientist's Daily Life
Data Scientist's Daily Life
Bryan Yang
 
Best practices-wordpress-enterprise
Best practices-wordpress-enterpriseBest practices-wordpress-enterprise
Best practices-wordpress-enterprise
Taylor Lovett
 
Four Architectural Patterns
Four Architectural Patterns Four Architectural Patterns
Four Architectural Patterns
David Simons
 
Decoupled APIs through Microservices
Decoupled APIs through MicroservicesDecoupled APIs through Microservices
Decoupled APIs through Microservices
David Simons
 
From Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dotsFrom Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dots
Ronald Ashri
 
From Content Strategy to Drupal Site Building - Connecting the Dots
From Content Strategy to Drupal Site Building - Connecting the DotsFrom Content Strategy to Drupal Site Building - Connecting the Dots
From Content Strategy to Drupal Site Building - Connecting the Dots
Ronald Ashri
 
200,000 Lines Later: Our Journey to Manageable Puppet Code
200,000 Lines Later: Our Journey to Manageable Puppet Code200,000 Lines Later: Our Journey to Manageable Puppet Code
200,000 Lines Later: Our Journey to Manageable Puppet Code
David Danzilio
 
Get Speed 100 - Optimization techniques to make your website fast
Get Speed 100 - Optimization techniques to make your website fastGet Speed 100 - Optimization techniques to make your website fast
Get Speed 100 - Optimization techniques to make your website fast
Alexander Dos Santos
 
You Created a Plugin. Now What?
You Created a Plugin. Now What?You Created a Plugin. Now What?
You Created a Plugin. Now What?
Adam W. Warner
 
FI_SiliconValleySummer2016_ProductDevelopment_AdeoRessi
FI_SiliconValleySummer2016_ProductDevelopment_AdeoRessiFI_SiliconValleySummer2016_ProductDevelopment_AdeoRessi
FI_SiliconValleySummer2016_ProductDevelopment_AdeoRessi
Cory Wang
 
What is spatial sql
What is spatial sqlWhat is spatial sql
What is spatial sql
shawty_ds
 
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB
 
Lessons learned from building Demand Side Platform
Lessons learned from building Demand Side PlatformLessons learned from building Demand Side Platform
Lessons learned from building Demand Side Platform
bbogacki
 
Robust WordPress Installation using L2MP Stack
Robust WordPress Installation using L2MP StackRobust WordPress Installation using L2MP Stack
Robust WordPress Installation using L2MP Stack
Alex Bertens
 
Monitoring and Logging in Wonderland
Monitoring and Logging in WonderlandMonitoring and Logging in Wonderland
Monitoring and Logging in Wonderland
Paul Seiffert
 
You Created a Plugin. Now What? WordCamp Sacramento
You Created a Plugin. Now What? WordCamp SacramentoYou Created a Plugin. Now What? WordCamp Sacramento
You Created a Plugin. Now What? WordCamp Sacramento
Adam W. Warner
 

Similar to Cloudy with a chance of scale (20)

Serverless WordPress & next Interface of WordPress
Serverless WordPress & next Interface of WordPressServerless WordPress & next Interface of WordPress
Serverless WordPress & next Interface of WordPress
 
Data Modelling at Scale
Data Modelling at ScaleData Modelling at Scale
Data Modelling at Scale
 
WordPress for the 99%
WordPress for the 99%WordPress for the 99%
WordPress for the 99%
 
ITB2016 - ColdBox 4 Modules
ITB2016 - ColdBox 4 ModulesITB2016 - ColdBox 4 Modules
ITB2016 - ColdBox 4 Modules
 
Data Scientist's Daily Life
Data Scientist's Daily LifeData Scientist's Daily Life
Data Scientist's Daily Life
 
Best practices-wordpress-enterprise
Best practices-wordpress-enterpriseBest practices-wordpress-enterprise
Best practices-wordpress-enterprise
 
Four Architectural Patterns
Four Architectural Patterns Four Architectural Patterns
Four Architectural Patterns
 
Decoupled APIs through Microservices
Decoupled APIs through MicroservicesDecoupled APIs through Microservices
Decoupled APIs through Microservices
 
From Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dotsFrom Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dots
 
From Content Strategy to Drupal Site Building - Connecting the Dots
From Content Strategy to Drupal Site Building - Connecting the DotsFrom Content Strategy to Drupal Site Building - Connecting the Dots
From Content Strategy to Drupal Site Building - Connecting the Dots
 
200,000 Lines Later: Our Journey to Manageable Puppet Code
200,000 Lines Later: Our Journey to Manageable Puppet Code200,000 Lines Later: Our Journey to Manageable Puppet Code
200,000 Lines Later: Our Journey to Manageable Puppet Code
 
Get Speed 100 - Optimization techniques to make your website fast
Get Speed 100 - Optimization techniques to make your website fastGet Speed 100 - Optimization techniques to make your website fast
Get Speed 100 - Optimization techniques to make your website fast
 
You Created a Plugin. Now What?
You Created a Plugin. Now What?You Created a Plugin. Now What?
You Created a Plugin. Now What?
 
FI_SiliconValleySummer2016_ProductDevelopment_AdeoRessi
FI_SiliconValleySummer2016_ProductDevelopment_AdeoRessiFI_SiliconValleySummer2016_ProductDevelopment_AdeoRessi
FI_SiliconValleySummer2016_ProductDevelopment_AdeoRessi
 
What is spatial sql
What is spatial sqlWhat is spatial sql
What is spatial sql
 
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
 
Lessons learned from building Demand Side Platform
Lessons learned from building Demand Side PlatformLessons learned from building Demand Side Platform
Lessons learned from building Demand Side Platform
 
Robust WordPress Installation using L2MP Stack
Robust WordPress Installation using L2MP StackRobust WordPress Installation using L2MP Stack
Robust WordPress Installation using L2MP Stack
 
Monitoring and Logging in Wonderland
Monitoring and Logging in WonderlandMonitoring and Logging in Wonderland
Monitoring and Logging in Wonderland
 
You Created a Plugin. Now What? WordCamp Sacramento
You Created a Plugin. Now What? WordCamp SacramentoYou Created a Plugin. Now What? WordCamp Sacramento
You Created a Plugin. Now What? WordCamp Sacramento
 

More from Mitch Pirtle

My life with MongoDB
My life with MongoDBMy life with MongoDB
My life with MongoDB
Mitch Pirtle
 
PHP Cloud Deployment Toolkits
PHP Cloud Deployment ToolkitsPHP Cloud Deployment Toolkits
PHP Cloud Deployment Toolkits
Mitch Pirtle
 
MongoDB, Node.js, and You: PART III
MongoDB, Node.js, and You: PART IIIMongoDB, Node.js, and You: PART III
MongoDB, Node.js, and You: PART III
Mitch Pirtle
 
MongoDB, Node.js And You: PART II
MongoDB, Node.js And You: PART IIMongoDB, Node.js And You: PART II
MongoDB, Node.js And You: PART II
Mitch Pirtle
 
Mongodb, Node.js and You: PART I
Mongodb, Node.js and You: PART IMongodb, Node.js and You: PART I
Mongodb, Node.js and You: PART I
Mitch Pirtle
 
MongoDB, Development and You
MongoDB, Development and YouMongoDB, Development and You
MongoDB, Development and You
Mitch Pirtle
 
MongoTorino 2013 Opening Keynote
MongoTorino 2013 Opening KeynoteMongoTorino 2013 Opening Keynote
MongoTorino 2013 Opening Keynote
Mitch Pirtle
 
Data as Documents: Overview and intro to MongoDB
Data as Documents: Overview and intro to MongoDBData as Documents: Overview and intro to MongoDB
Data as Documents: Overview and intro to MongoDB
Mitch Pirtle
 
Cloud conference - mongodb
Cloud conference - mongodbCloud conference - mongodb
Cloud conference - mongodb
Mitch Pirtle
 
Unified Content Model and Joomla!
Unified Content Model and Joomla!Unified Content Model and Joomla!
Unified Content Model and Joomla!
Mitch Pirtle
 
Gridfs and MongoDB
Gridfs and MongoDBGridfs and MongoDB
Gridfs and MongoDB
Mitch Pirtle
 
Joomla - an Overview
Joomla - an OverviewJoomla - an Overview
Joomla - an Overview
Mitch Pirtle
 
Operational MongoDB
Operational MongoDBOperational MongoDB
Operational MongoDB
Mitch Pirtle
 
Joomla Extreme Performance
Joomla Extreme PerformanceJoomla Extreme Performance
Joomla Extreme Performance
Mitch Pirtle
 
Joomla and MongoDB
Joomla and MongoDBJoomla and MongoDB
Joomla and MongoDB
Mitch Pirtle
 
Mongodb and Totsy: An e-commerce case study
Mongodb and Totsy: An e-commerce case studyMongodb and Totsy: An e-commerce case study
Mongodb and Totsy: An e-commerce case study
Mitch Pirtle
 
Mongodb and Totsy - E-commerce Case Study
Mongodb and Totsy - E-commerce Case StudyMongodb and Totsy - E-commerce Case Study
Mongodb and Totsy - E-commerce Case Study
Mitch Pirtle
 
PHP, Lithium and MongoDB
PHP, Lithium and MongoDBPHP, Lithium and MongoDB
PHP, Lithium and MongoDB
Mitch Pirtle
 
Content Management Systems and MongoDB
Content Management Systems and MongoDBContent Management Systems and MongoDB
Content Management Systems and MongoDB
Mitch Pirtle
 
MongoDB: Built for Speed
MongoDB: Built for SpeedMongoDB: Built for Speed
MongoDB: Built for Speed
Mitch Pirtle
 

More from Mitch Pirtle (20)

My life with MongoDB
My life with MongoDBMy life with MongoDB
My life with MongoDB
 
PHP Cloud Deployment Toolkits
PHP Cloud Deployment ToolkitsPHP Cloud Deployment Toolkits
PHP Cloud Deployment Toolkits
 
MongoDB, Node.js, and You: PART III
MongoDB, Node.js, and You: PART IIIMongoDB, Node.js, and You: PART III
MongoDB, Node.js, and You: PART III
 
MongoDB, Node.js And You: PART II
MongoDB, Node.js And You: PART IIMongoDB, Node.js And You: PART II
MongoDB, Node.js And You: PART II
 
Mongodb, Node.js and You: PART I
Mongodb, Node.js and You: PART IMongodb, Node.js and You: PART I
Mongodb, Node.js and You: PART I
 
MongoDB, Development and You
MongoDB, Development and YouMongoDB, Development and You
MongoDB, Development and You
 
MongoTorino 2013 Opening Keynote
MongoTorino 2013 Opening KeynoteMongoTorino 2013 Opening Keynote
MongoTorino 2013 Opening Keynote
 
Data as Documents: Overview and intro to MongoDB
Data as Documents: Overview and intro to MongoDBData as Documents: Overview and intro to MongoDB
Data as Documents: Overview and intro to MongoDB
 
Cloud conference - mongodb
Cloud conference - mongodbCloud conference - mongodb
Cloud conference - mongodb
 
Unified Content Model and Joomla!
Unified Content Model and Joomla!Unified Content Model and Joomla!
Unified Content Model and Joomla!
 
Gridfs and MongoDB
Gridfs and MongoDBGridfs and MongoDB
Gridfs and MongoDB
 
Joomla - an Overview
Joomla - an OverviewJoomla - an Overview
Joomla - an Overview
 
Operational MongoDB
Operational MongoDBOperational MongoDB
Operational MongoDB
 
Joomla Extreme Performance
Joomla Extreme PerformanceJoomla Extreme Performance
Joomla Extreme Performance
 
Joomla and MongoDB
Joomla and MongoDBJoomla and MongoDB
Joomla and MongoDB
 
Mongodb and Totsy: An e-commerce case study
Mongodb and Totsy: An e-commerce case studyMongodb and Totsy: An e-commerce case study
Mongodb and Totsy: An e-commerce case study
 
Mongodb and Totsy - E-commerce Case Study
Mongodb and Totsy - E-commerce Case StudyMongodb and Totsy - E-commerce Case Study
Mongodb and Totsy - E-commerce Case Study
 
PHP, Lithium and MongoDB
PHP, Lithium and MongoDBPHP, Lithium and MongoDB
PHP, Lithium and MongoDB
 
Content Management Systems and MongoDB
Content Management Systems and MongoDBContent Management Systems and MongoDB
Content Management Systems and MongoDB
 
MongoDB: Built for Speed
MongoDB: Built for SpeedMongoDB: Built for Speed
MongoDB: Built for Speed
 

Recently uploaded

Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 

Recently uploaded (20)

Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 

Cloudy with a chance of scale

  • 1. C L O U D Y W I T H A C H A N C E O F S C A L E M I T C H P I R T L E E L A S T I C S E A R C H M E E T U P C A P I TA L O N E L A B S , A R L I N G T O N VA D E C E M B E R 2 0 1 6
  • 2. A B O U T M E • FOSS founder/contributor • Startupper • Technology Fellow, Capital One • Skate punk • Musician • Football coach
  • 3. A B O U T Y O U • Just a user • In production by hand • Fully automated in production • I should be giving this talk
  • 4. A B O U T T H I S S E S S I O N • ES from an Ops perspective • The good, bad and truncated • What to expect
  • 5. H O W I T A L L S TA R T E D
  • 6. – M E , T H E E A R LY- O N C L U E L E S S V E R S I O N “All I want is a web crawler, this can’t be too hard to setup.”
  • 7. – M E , T H E E A R LY- O N C L U E L E S S V E R S I O N “All I want is a web crawler, this can’t be too hard to setup.”
  • 8. F E S S U P ! • Fess: http://fess.codelibs.org/ • FOSS • Simple web UI for setup • Multiple sources • Multiple types • Supports threads, throttling • ES as persistence store
  • 9. S C E N A R I O S .
  • 10. L O C A L S E T U P • Fess comes as a self-contained package • All you need is a java runtime and you’re good
  • 11. I ’ M T H I N K I N G B I G G E R T H A N T H AT. S C R E W L O C A L H O S T,
  • 13. W H Y P U B L I C C L O U D • Horizontal scale • Access to integrated services - storage, load balancing, etc • Opportunity to automate. All. The. Things.
  • 14. S I N G L E I N S TA N C E . • Quick to setup, uses embedded Elasticsearch + plugins • Quick to duplicate • Easy to maintain
  • 15. S I N G L E I N S TA N C E : 
 I S S U E S • Single point of failure • Zero scale opportunity • Fully manual effort • Good golly filesystem access is SLOW
  • 16. C L U S T E R E D S E R V I C E . • Horizontal scale • Easier to expand
  • 17. C L U S T E R E D S E R V I C E : I S S U E S • Needs separate instance of Elasticsearch, requires plugin installation (version specific) • Even harder to setup: Shards or replicas? Master or data or both, or neither? • Even harder to automate • Requires additional tooling for operations (logs, events)
  • 18. H O W D O I D O T H I S ? • ElasticSearch Cluster: Configuration & Best Practices
 (http://www.xmsxmx.com/ elasticsearch-cluster-configuration- best-practices/)
  • 19. A M A Z O N E L A S T I C S E R V I C E • No setup • Simple scale • Fully automated • Advanced configuration by default
  • 20. A M A Z O N E L A S T I C S E R V I C E : I S S U E S • No custom plugins (no Fess!) • Service limits (number of nodes per cluster, etc) • Lack of customization options
  • 21. T H E L I N K S • https://aws.amazon.com/elasticsearch-service/ • http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ aes-limits.html
  • 23. S T U M P T H E S P E A K E R M Y FA V O R I T E G A M E
  • 24. T H A N K Y O U V E RY G R A Z I E . • @mitchitized • mitch.pirtle@capitalone.com • github.com/spacemonkey • about.me/mitchitized • www.slideshare.net/ spacemonkeylabs