Elastic Stackにハマった話

•Download as PPTX, PDF•

0 likes•486 views

This document summarizes Kazuhiro Kosaka's talk about being stuck using the Elastic Stack. Some key points included being stuck with data loss between Filebeat and Logstash due to Logstash's lack of a persistent queue in versions below 5.1. Kosaka was also stuck with Elasticsearch writing issues due to JVM memory pressure on a single m3.medium instance that was resolved by upgrading to 3 m3.large instances. Restrictions of the Amazon Elasticsearch Service were also discussed, as well as next steps of migrating to Elastic Stack 5.x and considering Elastic Cloud or a private cluster.

Software

Been Stuck In Elastic Stack
Kazuhiro Kosaka
Engineer, MDH, CyberAgent, Inc.
Feb 10, 2017

About Me
• 小坂和弘 - Kazuhiro Kosaka @hyperdash
• 2009- CyberAgent, Inc.
• Designer/Front-end Engineer/Back-end Engineer
• Motion Graphics, VJ

Summary
What To Talk?
• Amazon Elasticsearch Service
• v1.x v2.x

Summary
What NOT To Talk?
• Private Elasticsearch Cluster / Elastic Cloud
• v5.x

Summary
Elastic Stack?
• Had been called ELK Stack
• Elasticsearch - Distributed Search/Analytics Engine
• Logstash - Data Processing Pipeline
• Kibana - Data Visualizer
• Beats - Data Shipping Agents

Our Case
• Elasticsearch v1.5 on Amazon Elasticsearch Service
• Logstash v2.3.1
• Filebeat v1.2.1
• m3.large / 3 instances
• 400,000 docs / day
• 95% of logs are by 日予算自動アロケーション Batch
• Will be separated into another index
• 主にユーザの操作履歴調査

Stuck #1 / Settings
DataLossを防ぐには？
• Filebeat uses a back-pressure sensitive protocol
• Logstash < v5.1 has NO Persistent Queue

Stuck #1 / Settings
DataLossを防ぐには？
Source: https://www.elastic.co/guide/en/logstash/5.0/deploying-and-scaling.html

Stuck #1 / Settings
DataLossを防ぐには？
• Use Message Queue e.g. Kafka
• Delete Filebeat registrar file and Re-ship data
• >= v5.1 Persistent Queue (beta) is enabled

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
• m3.medium / 1 instance
• JVMMemoryPressure ≒ 92-94 %
• FreeStorageSpace looks okay

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
Source: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html
• NOT a t2 instance
• ???

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
m3.medium / 1 instance
m3.large / 3 instances
Worked!

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
• indices.fielddata.cache.size: unbounded (default)
This allows ES to allocate unlimited memory for fielddata
• indices.fielddata.cache.size: 50 (%)

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
Source: https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
• ES v1.x
• Doc Values: Default OFF
• Set “doc_values”: true in index template - mappings

Stuck #2 / JVMMemoryPressure
ESにWriteできなくなったんですけど…
• ES >= v2.x
• Doc Values: Default ON

Stuck #3 / Restrictions
Amazon Elasticsearch Serviceの制限
• Dynamic Scripting is not supported
• Custom Plugins are not supported
• Access Control
• Amazon ES is out of VPC
• IP-Based, IAM-Based Access Control
• logstash-output-amazon_es plugin is not stable??

Next Step
• Migrate to Elastic Stack v5.x
• Elastic Cloud or Private Cluster?
• 管理画面上での操作ログ閲覧

Conclusion
• Amazon Elasticsearch Service
• Elastic Cloud
• Easy & Handy (Fully-Managed)
• The latest version is available anytime
• Less Restrictions, or Elastic Cloud Enterprise (alpha)
• Supported by Elastic
• Easy & Handy (Fully-Managed)
• Restrictions
• Supported by AWS

The document discusses the architecture of EC2 which allows running multiple operating systems on a single machine using hypervisors to measure performance and enable features like rapid provisioning and fault tolerance. It also covers the performance benefits of EC2 in being able to achieve higher performance than physical machines and how support is offered through forums and paid support plans at the silver and gold levels.

Backbone.js slides

Ambert Ho

Backbone.js abstracts remote resources into models and collections that emit events on state changes. This allows views to bind and respond to those changes. It provides concepts like models, collections, views and controllers to deal with common problems in web development like organizing rendering of views, supporting navigation, and avoiding callback soup. By abstracting functionality, separating concerns, and decoupling code, Backbone follows best practices and theoretical definitions of frameworks.

SQL Server Dev ToolKit

Kirsten Benzel

Scaling AWS With Scalr

Ram Viswanadha

Amazon Elastic Beanstalk

Eberhard Wolff

Amazon Elastic Beanstalk provides a platform for deploying and scaling web applications easily on AWS. It automatically handles the deployment, capacity provisioning, load balancing, auto-scaling, and application health monitoring of applications hosted on AWS. It supports popular programming languages such as Java, PHP, Python and .NET and integrates with other AWS services including EC2, S3, RDS and DynamoDB. It allows deploying applications easily via a web console or API and handles tasks such as provisioning servers, load balancing, automatic scaling and application health monitoring.

Project development - preparing hell dish together – Oleksii Dashkevych

Ruby Meditation

Immutable servers with Packer/Chef/AWS

Pavel Gabriel

The document discusses immutable servers and how to implement them using tools like Packer and Chef. Immutable servers are never modified after being deployed, but are replaced with new updated instances. Packer can be used to build identical machine images for different platforms from a single configuration. Chef recipes and cookbooks help configure application servers. The overall architecture involves load balancers directing traffic to volatile application servers while databases and logs are persisted separately.

Azure Web Pub Sub Service is a real-time messaging service that uses websockets. It allows building real-time web applications by facilitating high-volume requests and simultaneous connections. The service is currently in preview. Key concepts include services, servers, clients, groups, and messages. Servers generate access tokens and can create groups. Clients connect using websockets and standard APIs to send and receive messages within groups. The service can also be used with Azure Functions for event-driven programming.

Spring Boot. Boot up your development

Strannik_2013

Spring Boot allows developers to create stand-alone, production-grade Spring-based applications that can be launched using Java -main() without the need for XML configuration. It embeds Tomcat or Jetty servers directly and handles automatic configuration and dependencies. Spring Boot includes features like starters for common dependencies, hot swapping of code without restarts, and Actuator endpoints for monitoring and management.

Spring.new hope.1.3

Alex Tumanoff

This document discusses Spring Boot, a framework for creating stand-alone, production-grade Spring based applications that can be "just run". Spring Boot focuses on using sensible default configurations and automatic configuration so that developers can focus on the business problem rather than infrastructure. It provides features like embedded Tomcat/Jetty servers, auto configuration of Spring and third party libraries, actuator endpoints for monitoring apps, and works with properties files, environment variables and JNDI. The document also covers Spring configuration, annotations, issues, Groovy, environment configuration, initialization, auto-configuration classes, properties, and the health and metrics endpoints of Spring Boot Actuator.

Javascript Frameworks Comparison

Deepu S Nath

WordPress Bootcamp Quiz

Metronet

WordPress was officially released in 2003. The current stable version is 3.2.1. WordPress uses The Loop as its main process. Code should not be placed directly in core files and instead can be extended through plugins, themes, or mu-plugins. The two types of hooks in WordPress are actions and filters. Debugging can be turned on by defining WP_DEBUG as true. JavaScript and CSS should be enqueued properly using wp_register_script, wp_enqueue_script, wp_register_style, and wp_enqueue_style. Transient API allows temporary data storage. A minimum of style.css and index.php are required for a WordPress theme. The widgets_init hook

Intro to the Express Web Framework

jasonsich

This document provides an introduction to Express, a web application framework for Node.js. It demonstrates how to quickly generate an Express app, explain its default structure and files, and shows examples of serving static files, using views, and defining routes and APIs to handle requests. Key aspects covered include using middleware like the body parser and static file serving, as well as rendering views and sending JSON responses.

Sails Framework Instroduction

Leo Berdu

Sails is an MVC framework for Node.js that provides features like real-time functionality using WebSockets, an ORM, policies for authentication and authorization, and asset bundling with support for tools like Grunt. It generates RESTful routes and APIs for models and has a similar MVC structure to frameworks like Rails. An example app is created using the sails command line interface to generate a Product model and controller and demonstrate basic CRUD operations and real-time updates. The Sails community is growing rapidly and the framework is approaching production readiness.

Search and analyze your data with elasticsearch

Anton Udovychenko

The importance of search for modern applications is evident and nowadays it is higher than ever. A lot of projects use search forms as a primary interface for communication with a user. Though implementation of an intelligent search functionality is still a challenge and we need a good set of tools. In this presentation, I will talk through the high-level architecture and benefits of Elasticsearch with some examples. Aside from that, we will also take a look at its existing competitors, their similarities, and differences.

Understand & develop ember addons

Dilip Kushwaha

Ember addons allow developers to extend Ember applications by providing reusable functionality. Addons can be consumed by Ember apps, wrap non-Ember libraries, and add preprocessors. To create an addon, use the `ember addon` command and include important files like index.js. The tests/dummy folder allows testing the addon independently of an app. The index.js file acts as the entry point and allows customizing templates. Addons can include blueprints to generate boilerplate code. The default blueprint runs on install to integrate the addon. Additional blueprints provide extra functionality. Published addons are listed on EmberObserver.com.

Consuming REST services with ActiveResource

Wolfram Arnold

EmberJS

ryanstout

This document provides an overview of the Ember.js JavaScript framework. It discusses key concepts like classes and instances, bindings, computed properties, templates, models, controllers, views, and routing. Sections explain how Ember provides object orientation, data binding, and an MVC structure. The document also lists some organizations using Ember.js and provides additional resources for learning more.

WordPress hooks - WPLDN July 2013 Meetup

l3rady

WordPress hooks allow developers to modify the WordPress page lifecycle without changing core code. There are two types of hooks: actions and filters. Actions allow developers to add code at specific points during page loading, while filters allow modifying values by passing them through custom functions. Some examples given include changing the excerpt length on the home page using a filter hook, enqueueing custom scripts conditionally using an action hook, and overriding the main query on the home page using the pre_get_posts action. Hooks are essential for extending and customizing WordPress functionality without modifying core files.

(WEB203) Building a Website That Costs Pennies to Operate | AWS re:Invent 2014

Amazon Web Services

Developing Rest services with SailsJs by Andrey Kolodnitskiy

Lohika_Odessa_TechTalks

This document provides an overview of SailsJS, an MVC framework for Node.js. It discusses that SailsJS is built on Express and inspired by Ruby on Rails. It also summarizes key features such as convention over configuration, automatic REST API generation, support for various databases, and easy integration of web sockets. The document outlines the project structure in SailsJS including controllers, models, policies, and views. It provides examples of generating an API, model definition, controllers, and policies. Finally, it briefly discusses services, assets, configuration, and views in SailsJS projects.

Scalr - Open Source Cloud Management

Arvind Palanisamy

Scalr is a cloud management platform that automates scaling, security, availability, and cost management for infrastructure on cloud platforms like Amazon Web Services. It is trusted by companies like Samsung and helps users avoid time-consuming manual tasks while ensuring their cloud resources are optimized for performance, redundancy, and cost-efficiency. Press quotes praise Scalr for making AWS more powerful and providing fully redundant, self-scaling hosting capabilities.

AWS: Introduction

Sachin Dole

전 세계 팬들이 모일 수 있는 플랫폼 만들기 - 강진우 (beNX) :: AWS Community Day 2020

AWSKRUG - AWS한국사용자모임

jQuery 3.0 Alpha Versions

Osama Quboh

The document announces the alpha release of jQuery version 3.0 and jQuery Compat version 3.0. jQuery 3.0 only supports modern browsers while jQuery Compat 3.0 maintains backwards compatibility. Some key changes in jQuery 3.0 include simplified .show() and .hide() methods, handling of special characters in .data() names, decimal values for width/height, and major speed improvements for custom selectors and animations.

Elasticsearch 5.0 les nouveautés

Mathieu Elie

Introduction au langage Go

Sylvain Wallez

What's hot

Introducing Apache Stratos, your choice for an open PaaS Solution

WSO2

Real World Fun with ActiveResource

Rob C

Azure Web PubSub Serviceを触ってみた

DevTakas

Spring Boot. Boot up your development

Strannik_2013

Spring.new hope.1.3

Alex Tumanoff

Javascript Frameworks Comparison

Deepu S Nath

WordPress Bootcamp Quiz

Metronet

Intro to the Express Web Framework

jasonsich

Sails Framework Instroduction

Leo Berdu

Search and analyze your data with elasticsearch

Anton Udovychenko

Understand & develop ember addons

Dilip Kushwaha

Consuming REST services with ActiveResource

Wolfram Arnold

EmberJS

ryanstout

WordPress hooks - WPLDN July 2013 Meetup

l3rady

(WEB203) Building a Website That Costs Pennies to Operate | AWS re:Invent 2014

Amazon Web Services

Developing Rest services with SailsJs by Andrey Kolodnitskiy

Lohika_Odessa_TechTalks

Scalr - Open Source Cloud Management

Arvind Palanisamy

AWS: Introduction

Sachin Dole

전 세계 팬들이 모일 수 있는 플랫폼 만들기 - 강진우 (beNX) :: AWS Community Day 2020

AWSKRUG - AWS한국사용자모임

jQuery 3.0 Alpha Versions

Osama Quboh

What's hot (20)

Introducing Apache Stratos, your choice for an open PaaS Solution

Real World Fun with ActiveResource

Azure Web PubSub Serviceを触ってみた

Spring Boot. Boot up your development

Spring.new hope.1.3

Javascript Frameworks Comparison

WordPress Bootcamp Quiz

Intro to the Express Web Framework

Sails Framework Instroduction

Search and analyze your data with elasticsearch

Understand & develop ember addons

Consuming REST services with ActiveResource

EmberJS

WordPress hooks - WPLDN July 2013 Meetup

(WEB203) Building a Website That Costs Pennies to Operate | AWS re:Invent 2014

Developing Rest services with SailsJs by Andrey Kolodnitskiy

Scalr - Open Source Cloud Management

AWS: Introduction

전 세계 팬들이 모일 수 있는 플랫폼 만들기 - 강진우 (beNX) :: AWS Community Day 2020

jQuery 3.0 Alpha Versions

Viewers also liked

Elasticsearch 5.0 les nouveautés

Mathieu Elie

Introduction au langage Go

Sylvain Wallez

Elasticsearch 5.0

Matias Cascallares

Rootconf

akbarabi

LinkedIn uses Kafka for log aggregation and monitoring. It ingests over 500,000 metrics per minute from applications into Kafka, which then writes to storage clusters. It also uses Kafka to transport log data to ELK (Elasticsearch, Logstash, Kibana) for real-time search, analysis and visualization of metrics and logs. Kafka provides benefits like horizontal scalability, multiple consumer groups, and no overhead on client applications.

From Config Management Sucks to #cfgmgmtlove

Kris Buytaert

This document summarizes Kris Buytaert's talk on the evolution of config management from the 1990s to present day. It discusses early approaches like manual installations and system imaging tools. It then covers the rise of infrastructure as code using tools like Puppet, Chef, and Docker. The talk addresses challenges like getting operations teams to adopt new methods and complexities that can arise from dependencies and modules. It promotes treating infrastructure like code with development practices for versioning, testing, and continuous integration/deployment.

Mesoscon 2015

Skand Gupta

This document discusses Mesos implementation at Bloomberg. It notes that Bloomberg runs one of the largest private networks and was an early adopter of cloud computing and software as a service. It describes how Mesos is used to provide elastic data processing and analytics across Bloomberg's 3000+ developers. Key parts of the Mesos implementation include using Marathon for application deployment, Kafka for processing topologies, and ELK/InfluxDB/Grafana for centralized monitoring. The document also discusses lessons learned around access control, Zookeeper protection, and cleaning up sandbox data.

Building Product from ground up using Open Source Technologies

Amit Goel

This is a presentation given by Amit Goel, Founder & CEO, Patterbuzz at Open Source India 2014 conference in Bangalore. . He spoke about the story of Patterbuzz and its journey of 2 years since its beginning to explain the product development with the pace of a start up. It covered all the necessary evils and building the technology form scratch using open source softwares and available technologies and how to leverage its advantage. Patterbuzz builds on the core philosophy of unbundling content from the publishing industry and making it available to consumers in a device reading friendly format with pay as you go model. For example: users can buy single articles if they like rather than purchasing the complete magazine. Currently, Patterbuzz enables magazines from India / Indian publishing industry. Most popular magazines available are Popular Science, Mother & Baby, Electronics for you, open source for you, computer active, Gfiles, Power Politics, Hard News, Car India, Bike India etc...

Data science team, a practice to setup

Omid Mogharian

This document outlines roles and processes for building an effective data science team. It recommends that teams include people with skills in machine learning, data engineering, analytics, mathematics, and visualization. The document describes the CRISP-DM process for data science projects and a lambda architecture for continuous analysis of big data using a pipeline of jobs, messaging queues, and data processing applications. It emphasizes building cross-disciplinary teams that can work together to ask important questions and create useful data products.

Прямая выгода BigData для бизнеса

Alexey Lustin

DUMP-2012 - Только хардкор! - "Архитектура и запуск облачного сервиса в Amazo...it-people

Send Balls Into Orbit with Python3, AsyncIO, WebSockets and React

Taras Lyapun

This document discusses building a mini game challenge for a soccer app using Python, AsyncIO, WebSockets, and Redis. It introduces the speaker as the organizer of KhmelnitskiyPy and CTO of Turfmapp, a soccer-focused meetup app. It then explains how AsyncIO allows asynchronous programming in Python without additional libraries. WebSockets enable real-time communication between server and clients. Redis is used as a high performance cache and store for publishing game state updates to clients over WebSockets.

Send that (damn) elevator down !

Ekta Grover

Behind the Scenes at Coolblue - Feb 2017

Pat Hermens

PyCon Poland 2016: Maintaining a high load Python project: typical mistakes

Viach Kakovskyi

Designing Invisible Software at x.ai

Alex Poon

Pycon UA 2016

Taras Lyapun

The document summarizes the Python user community in Ukraine, including regional user groups, past PyCon Ukraine conferences, and Django Girls events. It notes that PyCon Ukraine has seen annual attendance of 300-350 participants from 2010-2012 and around 300 in 2014. It is also held in multiple cities across Ukraine with the largest events in Kyiv, Kharkiv, and Lviv. Django Girls events have attracted hundreds of candidates for only 30 spots in various cities. PyCon Ukraine 2016 was held in Lviv in April 2016. LvivPy #5 is scheduled for late November. Contact information is provided for the PyCon Ukraine and regional user group social media and websites.

Introducing ELK

AllBits BVBA (freelancer)

This document introduces the (B)ELK stack, which consists of Beats, Elasticsearch, Logstash, and Kibana. It describes each component and how they work together. Beats are lightweight data shippers that collect data from logs and systems. Logstash processes and transforms data from inputs like Beats. Elasticsearch stores and indexes the data. Kibana provides visualization and analytics capabilities. The document provides examples of using each tool and tips for working with the ELK stack.

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Surasak Sanguanpong

This document discusses experiences using the ELK stack (Elasticsearch, Logstash, Kibana) and D3.js for large log analysis and visualization. It begins with an overview of network traffic logging at Kasetsart University, which generates over 30 terabytes of log data per day. It then demonstrates setting up an ELK testbed to index these logs in real-time for fast search and exploration in Kibana. Finally, it shows how D3.js can be used to create dynamic, real-time visualizations of the logged data.

Kibana + timelion: time series with the elastic stack

Sylvain Wallez

The document discusses Kibana and Timelion, which are tools for visualizing and analyzing time series data in the Elastic Stack. It provides an overview of Kibana's evolution and capabilities for creating dashboards. Timelion is introduced as a scripting language that allows users to transform, aggregate, and calculate on time series data from multiple sources to create visualizations. The document demonstrates Timelion's expression language, which includes functions, combinations, filtering, and attributes to process and render time series graphs.

Автоматизация анализа логов на базе Elasticsearch

Positive Hack Days

Viewers also liked (20)

Elasticsearch 5.0 les nouveautés

Introduction au langage Go

Elasticsearch 5.0

Rootconf

From Config Management Sucks to #cfgmgmtlove

Mesoscon 2015

Building Product from ground up using Open Source Technologies

Data science team, a practice to setup

Прямая выгода BigData для бизнеса

DUMP-2012 - Только хардкор! - "Архитектура и запуск облачного сервиса в Amazo...

Send Balls Into Orbit with Python3, AsyncIO, WebSockets and React

Send that (damn) elevator down !

Behind the Scenes at Coolblue - Feb 2017

PyCon Poland 2016: Maintaining a high load Python project: typical mistakes

Designing Invisible Software at x.ai

Pycon UA 2016

Introducing ELK

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Kibana + timelion: time series with the elastic stack

Автоматизация анализа логов на базе Elasticsearch

Similar to Elastic Stackにハマった話

(WEB301) Operational Web Log Analysis | AWS re:Invent 2014

Amazon Web Services

Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.

Microservices, Continuous Delivery, and Elasticsearch at Capital One

Noriaki Tatsumi

ENT309 Scaling Up to Your First 10 Million Users

Amazon Web Services

This document provides guidance on scaling a web application from 1 user to over 10 million users on AWS. It recommends starting simply with a single EC2 instance and Route 53, then adding redundancy with multiple instances, load balancing, and SQL databases. As users grow over 1,000 techniques like caching, NoSQL, and auto scaling are introduced. Above 500,000 users more services are split out and automated. Reaching over 1 million requires database sharding or federation. The key strategies emphasized are redundancy, automation, splitting services, and leveraging managed AWS services over custom solutions.

ENT309 Scaling Up to Your First 10 Million Users

Amazon Web Services

Cloud computing gives you a number of advantages, such as the ability to scale your web application or website on demand. If you have a new web application and want to use cloud computing, you might be asking yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We show you how to best combine different AWS services, how to make smarter decisions for architecting your application, and how to scale your infrastructure in the cloud.

Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...

Amazon Web Services

Learn how you can use Amazon ElastiCache to easily deploy a Memcached or Redis compatible, in-memory caching system to speed up your application performance. We show you how to use Amazon ElastiCache to improve your application latency and reduce the load on your database servers. We'll also show you how to build a caching layer that is easy to manage and scale as your application grows. During this session, we go over various scenarios and use cases that can benefit by enabling caching, and discuss the features provided by Amazon ElastiCache.

Where Django Caching Bust at the Seams

Concentric Sky

ENT309 scaling up to your first 10 million users

Amazon Web Services

DjangoCon 2010 Scaling Disqus

zeeg

DISQUS is a comment system that handles high volumes of traffic, with up to 17,000 requests per second and 250 million monthly visitors. They face challenges in unpredictable spikes in traffic and ensuring high availability. Their architecture includes over 100 servers split between web servers, databases, caching, and load balancing. They employ techniques like vertical and horizontal data partitioning, atomic updates, delayed signals, consistent caching, and feature flags to scale their large Django application.

Azug - successfully breeding rabits

Yves Goeleven

This document discusses successfully breeding rabbits in the cloud. It describes rabbits as small public apps that want to live outside and scale quickly. It discusses how the cloud provides rabbits with global reach, fast time to market, performance, scalability, availability and cost efficiency. The document emphasizes that the weakest link limits overall scalability and that caches, queues and separating concerns can help address this. It stresses understanding cloud capacity and efficiently using resources to keep costs low. Finally, it discusses how rabbits can achieve self-service, marketing, support, education and testing at scale.

A Practical Multi-Tenant Cluster

Command Prompt., Inc

Rod Anderson For the small business support person being able to provide PostgreSQL hosting for a small set of specific applications without having to build and support several Pg installations is necessary. By building a multi-tenant Pg cluster with one tenant per database and each application in it's own schema maintenance and support is much simpler. The issues that present themselves are how to provide and control dba and user access to the database and get the applications into their own schema. With this comes need to make logging in to the database (pg_hba.conf) as non-complex as possible.

CMP301_Deep Dive on Amazon EC2 Instances

Amazon Web Services

This document provides an overview of Amazon EC2 instances and best practices for performance optimization. It discusses factors to consider when choosing an EC2 instance type, how EC2 delivers performance while providing flexibility, and how to optimize performance through various instance types. The document covers topics like virtual CPUs, resource allocation, benchmarking, timekeeping, P-state and C-state control, T2 instance burstable performance, X1 instances for large memory workloads, and NUMA considerations. It provides tips on using the TSC clocksource, controlling P-states for AVX2, monitoring T2 credit balances, disabling NUMA awareness, and using numactl to control process placement.

(STG402) Amazon EBS Deep Dive

Amazon Web Services

Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.

Running your Java EE 6 applications in the clouds

Arun Gupta

Geek Sync | Performance Tune Like an MVP

IDERA Software

You can watch the replay for this Geek Sync webcast, Performance Tune Like an MVP, in the IDERA Resource Center, http://ow.ly/aDE250A4qdF. The life of a DBA is evolving and your tuning skills should always be sharp. Tuning is one of the key components of a great DBA and developer. In this demo rich session we'll deep dive into performance tuning for on-premises, PaaS (platform as a service), and IaaS (infrastructure as a service). We'll discuss tips and techniques for troubleshooting bottlenecks and how to remediate them for hardware, OS, and the database. Speaker: Daniel Janik has been supporting SQL Server for over 18 years. Six of those years were at Microsoft Corporation supporting SQL Server as a Senior Premier Field Engineer (PFE) where he supported over 287 different clients with both reactive and proactive database needs. Daniel has presented at many community events and SQL Saturdays.

BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service

Amazon Web Services

Everything generates logs. Applications, infrastructure, security ... everything. Keeping track of the flood of log data is a big challenge, yet critical to your ability to understand your systems and troubleshoot (or prevent) issues. In this session, we will use both Amazon CloudWatch and application logs to show you how to build an end-to-end log analytics solution. First, we cover how to configure an Amazon Elaticsearch Service domain and ingest data into it using Amazon Kinesis Firehose, demonstrating how easy it is to transform data with Firehose. We look at best practices for choosing instance types, storage options, shard counts, and index rotations based on the throughput of incoming data and configure a secure analytics environment. We demonstrate how to set up a Kibana dashboard and build custom dashboard widgets. Finally, we dive deep into the Elasticsearch query DSL and review approaches for generating custom, ad-hoc reports.

STG330_Case Study How Experian Leverages Amazon EC2, EBS, and S3 with Clouder...

Amazon Web Services

Experian gathers, analyzes, and processes credit data at massive scale to help businesses make smarter decisions, individuals gain access to financial services, and lenders to minimize risk. The company built its petabyte-scale data-ingestion and analytics solution using CDH (Cloudera Distribution Including Apache Hadoop) running on Amazon EC2, with data stored in Amazon EBS and Amazon S3. This next generation big data platform aims to improve the data accuracy by moving away from traditional batch uploads to a real-time API-based ingestion process. In this talk, you will learn how Experian has leveraged different AWS compute and storage services for agility and quicker time to market. We will discuss lessons learned and best practices for success throughout.

AWS October Webinar Series - Introducing Amazon Elasticsearch Service

Amazon Web Services

Running Elasticsearch often requires specialized expertise and significant resources to operate and manage infrastructure and Elasticsearch software. Amazon Elasticsearch Service makes it easy to deploy, operate, and scale Elasticsearch in AWS. In this webinar, we will walk through how to launch a fully functional Amazon Elasticsearch domain, load your data, and analyze it using the built-in Kibana integration. We will also cover the CloudWatch Logs integration, which enables you to have your log data, such as VPC logs, automatically loaded into your Amazon Elasticsearch domain for analysis and exploration.

Caching your rails application

ArrrrCamp

Threads Needles Stacks Heaps - Java edition

Ovidiu Dimulescu

Apache Mesos at Twitter (Texas LinuxFest 2014)

Chris Aniszczyk

Chris Aniszczyk presented on Apache Mesos at Twitter. Mesos is an open source cluster management system that provides efficient resource isolation and sharing across distributed applications or frameworks. It allows applications to share computing resources like CPU, memory, storage and networks. Mesos supports high availability with a master-slave architecture and pluggable isolation mechanisms like Docker. A growing Mesos ecosystem includes frameworks for cron jobs, services, batch processing and more. Mesos enables multi-tenancy, high utilization and scalability for complex distributed systems.

Similar to Elastic Stackにハマった話 (20)

(WEB301) Operational Web Log Analysis | AWS re:Invent 2014

Microservices, Continuous Delivery, and Elasticsearch at Capital One

ENT309 Scaling Up to Your First 10 Million Users

Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...

Where Django Caching Bust at the Seams

ENT309 scaling up to your first 10 million users

DjangoCon 2010 Scaling Disqus

Azug - successfully breeding rabits

A Practical Multi-Tenant Cluster

CMP301_Deep Dive on Amazon EC2 Instances

(STG402) Amazon EBS Deep Dive

Running your Java EE 6 applications in the clouds

Geek Sync | Performance Tune Like an MVP

BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service

STG330_Case Study How Experian Leverages Amazon EC2, EBS, and S3 with Clouder...

AWS October Webinar Series - Introducing Amazon Elasticsearch Service

Caching your rails application

Threads Needles Stacks Heaps - Java edition

Apache Mesos at Twitter (Texas LinuxFest 2014)

Recently uploaded

How to write a program in any programming language

Rakesh Kumar R

Transform Your Communication with Cloud-Based IVR Solutions

TheSMSPoint

Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony

Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...

kalichargn70th171

A dynamic process unfolds in the intricate realm of software development, dedicated to crafting and sustaining products that effortlessly address user needs. Amidst vital stages like market analysis and requirement assessments, the heart of software development lies in the meticulous creation and upkeep of source code. Code alterations are inherent, challenging code quality, particularly under stringent deadlines.

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement. Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.

LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx

lorraineandreiamcidl

WhatsApp offers simple, reliable, and private messaging and calling services for free worldwide. With end-to-end encryption, your personal messages and calls are secure, ensuring only you and the recipient can access them. Enjoy voice and video calls to stay connected with loved ones or colleagues. Express yourself using stickers, GIFs, or by sharing moments on Status. WhatsApp Business enables global customer outreach, facilitating sales growth and relationship building through showcasing products and services. Stay connected effortlessly with group chats for planning outings with friends or staying updated on family conversations.

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

timtebeek1

KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD

rodomar2

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

Empowering Growth with Best Software Development Company in Noida - Deuglo

Deuglo Infosystem Pvt Ltd

Do you want Software for your Business? Visit Deuglo Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions. Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC). Requirement — Collecting the Requirements is the first Phase in the SSLC process. Feasibility Study — after completing the requirement process they move to the design phase. Design — in this phase, they start designing the software. Coding — when designing is completed, the developers start coding for the software. Testing — in this phase when the coding of the software is done the testing team will start testing. Installation — after completion of testing, the application opens to the live server and launches! Maintenance — after completing the software development, customers start using the software.

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Google

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-fusion-buddy-review AI Fusion Buddy Review: Key Features ✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini ✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique! ✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs! ✅Fully automated AI articles bulk generation! ✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more. ✅With one keyword or URL, generate complete websites, landing pages, and more… ✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7. ✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches. ✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all! ✅Save over $5000 per year and kick out dependency on third parties completely! ✅Brand New App: Not available anywhere else! ✅ Beginner-friendly! ✅ZERO upfront cost or any extra expenses ✅Risk-Free: 30-Day Money-Back Guarantee! ✅Commercial License included! See My Other Reviews Article: (1) AI Genie Review: https://sumonreview.com/ai-genie-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review #AIFusionBuddyReview, #AIFusionBuddyFeatures, #AIFusionBuddyPricing, #AIFusionBuddyProsandCons, #AIFusionBuddyTutorial, #AIFusionBuddyUserExperience #AIFusionBuddyforBeginners, #AIFusionBuddyBenefits, #AIFusionBuddyComparison, #AIFusionBuddyInstallation, #AIFusionBuddyRefundPolicy, #AIFusionBuddyDemo, #AIFusionBuddyMaintenanceFees, #AIFusionBuddyNewbieFriendly, #WhatIsAIFusionBuddy?, #HowDoesAIFusionBuddyWorks

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Crescat

Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry. Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events. With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use. Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements. If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io

Fundamentals of Programming and Language Processors

Rakesh Kumar R

Artificia Intellicence and XPath Extension Functions

Octavian Nadolu

E-commerce Development Services- Hornet Dynamics

Hornet Dynamics

Graspan: A Big Data System for Big Code Analysis

Aftab Hussain

We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs. We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations. These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18. - Accepted in ASPLOS ‘17, Xi’an, China. - Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17. - Invited for presentation at SoCal PLS ‘16. - Invited for poster presentation at PLDI SRC ‘16.

OpenMetadata Community Meeting - 5th June 2024

OpenMetadata

The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features. * How to run your own data quality framework * What is the performance impact of running data quality frameworks * How to run the test cases in your own ETL pipelines * How the Incident Manager is integrated * Get notified with alerts when test cases fail Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E

GreenCode-A-VSCode-Plugin--Dario-Jurisic

Green Software Development

E-commerce Application Development Company.pdf

Hornet Dynamics

Recently uploaded (20)

How to write a program in any programming language

Transform Your Communication with Cloud-Based IVR Solutions

Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Atelier - Innover avec l’IA Générative et les graphes de connaissances

LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD

openEuler Case Study - The Journey to Supply Chain Security

Empowering Growth with Best Software Development Company in Noida - Deuglo

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Fundamentals of Programming and Language Processors

Artificia Intellicence and XPath Extension Functions

E-commerce Development Services- Hornet Dynamics

Graspan: A Big Data System for Big Code Analysis

OpenMetadata Community Meeting - 5th June 2024

GreenCode-A-VSCode-Plugin--Dario-Jurisic

E-commerce Application Development Company.pdf

Elastic Stackにハマった話

1. Been Stuck In Elastic Stack Kazuhiro Kosaka Engineer, MDH, CyberAgent, Inc. Feb 10, 2017

2. About Me • 小坂和弘 - Kazuhiro Kosaka @hyperdash • 2009- CyberAgent, Inc. • Designer/Front-end Engineer/Back-end Engineer • Motion Graphics, VJ

3. About Me • I love hiking

4. Summary What To Talk? • Amazon Elasticsearch Service • v1.x v2.x

5. Summary What NOT To Talk? • Private Elasticsearch Cluster / Elastic Cloud • v5.x

6. Summary Elastic Stack? • Had been called ELK Stack • Elasticsearch - Distributed Search/Analytics Engine • Logstash - Data Processing Pipeline • Kibana - Data Visualizer • Beats - Data Shipping Agents

7. Our Case 広告運用・配信設定管理画面ユーザ操作ログ

8. Our Case • Elasticsearch v1.5 on Amazon Elasticsearch Service • Logstash v2.3.1 • Filebeat v1.2.1 • m3.large / 3 instances • 400,000 docs / day • 95% of logs are by 日予算自動アロケーション Batch • Will be separated into another index • 主にユーザの操作履歴調査

9. Our Case

10. Stuck #1 / Settings DataLossを防ぐには？ • Filebeat uses a back-pressure sensitive protocol • Logstash < v5.1 has NO Persistent Queue

11. Stuck #1 / Settings DataLossを防ぐには？ Source: https://www.elastic.co/guide/en/logstash/5.0/deploying-and-scaling.html

12. Stuck #1 / Settings DataLossを防ぐには？ • Use Message Queue e.g. Kafka • Delete Filebeat registrar file and Re-ship data • >= v5.1 Persistent Queue (beta) is enabled

13. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… • m3.medium / 1 instance • JVMMemoryPressure ≒ 92-94 % • FreeStorageSpace looks okay

14. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… Source: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html • NOT a t2 instance • ???

15. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… m3.medium / 1 instance m3.large / 3 instances Worked!

16. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… • indices.fielddata.cache.size: unbounded (default) This allows ES to allocate unlimited memory for fielddata • indices.fielddata.cache.size: 50 (%)

17. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… Source: https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html

18. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… • ES v1.x • Doc Values: Default OFF • Set “doc_values”: true in index template - mappings

19. Stuck #2 / JVMMemoryPressure ESにWriteできなくなったんですけど… • ES >= v2.x • Doc Values: Default ON

20. Stuck #3 / Restrictions Amazon Elasticsearch Serviceの制限 • Dynamic Scripting is not supported • Custom Plugins are not supported • Access Control • Amazon ES is out of VPC • IP-Based, IAM-Based Access Control • logstash-output-amazon_es plugin is not stable??

21. Next Step • Migrate to Elastic Stack v5.x • Elastic Cloud or Private Cluster? • 管理画面上での操作ログ閲覧

22. Conclusion • Amazon Elasticsearch Service • Elastic Cloud • Easy & Handy (Fully-Managed) • The latest version is available anytime • Less Restrictions, or Elastic Cloud Enterprise (alpha) • Supported by Elastic • Easy & Handy (Fully-Managed) • Restrictions • Supported by AWS

23. END THANK YOU

Editor's Notes

Elastic Stackにハマった話をいくつかさせていただきたいと思います。
自己紹介です。小坂和弘ともうします。2009年にサイバーエージェントに中途入社して、主にサーバサイドの開発をしています。最近はScala、Golangで広告配信設定管理まわりの開発をしています。以前いたPigg、ゲーム部門の方では、Flash、JS、Unityでフロントの開発もやっていました。それ以前はデザイナーをしていたり、映像制作、VJとかやってました。 2009年にサイバーエージェントに中途入社サーバ: scala, golang, 広告配信設定管理まわりの開発フロント: Pigg, Game部門 Flash, JS, Unity デザイナー、映像、VJ
ハイキングが好きでグランドキャニオンの谷底まで降りたり、砂漠の山を登ったりしました。山最高。あと、アメリカ英語発音の習得が趣味です。 - 趣味の話
今日話すことですが、主にAmazon Elasticsearch Serviceの話です。 Elasticsearchは1.5、Logstashは2.3、あとはFilebeatを使っていますが、少し古いバージョンの話になります。 Amazon Elasticsearch Service ES 1.5 Logstash 2.3 Filebeat やや古いバージョンの話
話さないことですが、自前クラスタでの運用、Elastic Cloudでの運用、バージョン5系の話は今回はあまりしません。自前クラスタでの運用 Elasic Cloudでの運用 v5.x系
まず最初に、Elastic Stackについて軽く説明しますと、以前はES、Logstash、Kibanaの頭文字を取って、ELK Stackと呼ばれていました。 Beatsというコンポーネントが加わってElastic Stackと呼ばれるようになりました。 ESが検索分析エンジン、 Logstashがデータを処理、加工するフィルタパイプライン、 Kibanaがデータを可視化するビジュアライザ、 BeatsがデータをESやLogstashに送るエージェントです。 Beatsにはログファイルを監視して送信するFilebeatなどを始めとして、様々な用途に応じたものがあります。 ES, Logstash, Kibana = ELK Stack + Beats = Elastic Stack ES = 検索分析エンジン LS = データを処理、加工するフィルタパイプライン Kibana = データを可視化するビジュアライザ Beats = データをESやLogstashに送るエージェント Beats = Filebeat, etc
私達が現在Elastic Stackを使っているのは、主に自社メディアへの広告配信設定を運用管理するためのWEB画面です。こちらでのユーザ操作ログの収集と検索になります。このシステムのバックエンドログを一元的に検索できる必要があったのですが、一番早く、簡単に実現できる方法としてAmazon ESを選択しました。サイバーエージェント内ではFluentdが使われることが多かったので、 Filebeat、Logstashの流行りに乗って知見を溜めておきたいという理由もありました。この他にシステムのパフォーマンス測定、ジョブ実行結果の集計にも一部使われてますが、今回はこの操作ログの収集、検索においての話になります。自社メディアへの広告配信設定を運用管理するためのWEB画面ユーザ操作ログの収集と検索 Amazon ES = 最速で構築 Elastic Stackの流行り、知見目的
環境ですが、Amazon ES上でバージョン1.5を使っています。構築当初は2.3が選べなかったんですが、今では5.1も選べるようになっています。あとは、Logstash、Filebeat。現在はこういった構成で運用しています。ドキュメント数は1日40万件ほどありますが、このうち約95%は広告キャンペーンの日予算を自動で調整するバッチからのリクエストによります。単純にユーザの操作履歴を検索する際にはノイズになっているので、別インデックスに分けようという予定です。 v1.5 on Amazon ES 2.3/5.1 選択可能になった LogstashとFilebeat 40万件/日でも95%は人間の操作じゃない → 分離したい - 広告配信ログ = 配信サーバ→flume→Hive/S3/ISE
一部プロキシが入ってる箇所など簡略化してありますが、大まかな構成図はこんな感じになります。バックエンドにResource Serverと呼ばれるGeraphQLのAPIサーバがあり、その各台でFilebeatが動いていて、ログファイルを送信しています。送信先は、Log Aggregatorと呼んでいるLogstashサーバになります。 LogstashではBeats Inputで受け取ったログを一旦ほぼ生の状態でローカルファイルに書き込んでいます。それらをスクリプトでS3にもコピーしています。また、そのローカルファイルをLogstashのFile Inputを使って再度読み込み、不要な情報の削除や、加工を行ってESにoutputしているというシンプルな構成になっています。現在は特にメッセージキューを入れたりはしていません。構成図 RS Genisys Logstash -> ES / S3 （スクリプト）
いま、キューは特に使っていないと言ったんですが、特にLogstashなんですが、これを使うと決めた際に最初に悩んだことが、どうやってデータロスを防ぐかということでした。 Filebeatはバックプレッシャーに対応しているのと、どこまで送信が完了したかをインデックス管理していて、ログを送り損ねることはなさそうでした。問題はLogstashで、v5.1でキューをディスクにバッファする機能がbetaリリースされましたが、当時はv.2.3が最新バージョンで、その機能はありませんでした。データロスをどう防ぐか Filebeat = バックプレッシャー対応、インデクス Logstash = 5.1で内部キューのファイルバッファ機能、2.3にはなかった
次に、Logstashのパイプラインを見てみるとこのようになっています。今回はFilebeatからBeats Input経由でデータを受け取っているので、下の図のようなパイプラインになります。 Inputからデータが入ってきて、Filterを通って、Outputから出ていきます。この際、InputとFilterの間、FilterとOutputの間はメモリ上にデータがキューされます、このキューにデータが入っている状態でLogstashのプロセスがクラッシュするとキュー内部のデータが失われる可能性があります。パイプラインの流れ内部キュー in メモリ Logstashがクラッシュするとキュー内部のデータがロストする可能性
今回はFilebeatを使ったので、オリジナルのデータが失われることはないのですが、特にオリジナルのデータがファイル等に残らないようなデータソースを使う場合には、Logstashの前になんらかのメッセージキューを入れる必要があると思います。例えば、Kafkaを使った場合はオフセット情報を元にデータの再送信が可能なようですが、使うキューによっては、LogstashのPersistent Queueを併用する必要があるかもしれません。この辺はまだ調べきれてないです。 Persistent Queueが使えない場合でも、データが再送信できればデータロスを回復することができます。再送信することで、ESにストアされる際にドキュメント重複する問題が出てきますが、 Logstashのフィルタを使ってhash値取り、それをドキュメントのIDとして使うことで、すでに同じIDのドキュメントがあっても、上書きされるだけで、重複の発生を防げます。今回のケースでは、そこまでクリティカルなデータを扱っていないことから、Logstashがクラッシュした場合は Filebeatのレジストラファイルを削除してログを再送信させる方法でデータロス時の対応をしていますが、 Logstash自体のクラッシュは今まで起こっていません。 Filebeat オリジナルログ残るメッセージキュー Kafka オフセット情報から再送信それ以外、Persistent Queue データ再送信でロス防止重複はhashで回避今回は都度再送信前提で、ただしLogstashのクラッシュはほとんど起こってない
2つめのハマりです。突然ESにWriteができなくなりました。モニタリングで状態を調べてみると、JVMMemoryPressureメトリクスが高い状態で張り付いているものの、FreeStorageSpaceは問題ない数値。当初はメモリは検索には影響が出ても、書き込みには影響しないだろうと考えていたので、原因がよくわかりませんでした。突然Wirteできなくなったこのときの構成このときの状態
AWSのドキュメントをよく読んでみると、t2インスタンスの場合はJVMMemoryPressureが92%に達するとWriteをブロックしますよという記述が見つかりました。 t2インスタンスは使っていませんでしたが、JVMMemoryPressureは確かに高い状態だったので、メモリの圧迫が原因ではないかと推測して対応してみることにしました。 AWSのドキュメントにt2インスタンスでJVMMemoryPressure 92%の記述同じ現象？メモリ周りを疑ってみる
メモリを増やすか、メモリ上に載るデータを減らして解決するか試すために、手っ取り早い方法を取りました。よりメモリを載せたインスタンスでノード数も増やしたところ、Writeが再開されました。JVMMemoryPressureメトリクス値も70%あたりまで下がりました。 - インスタンスタイプを変えて、ノードも増やした
コンソールから設定できる項目は数少ないが、Amazon ESの設定を見直したら… indices.fielddata.cache.size が未設定になっていました。。。この設定値が未設定の場合、制限なくメモリを確保するようです。 fielddataって何かというと、ドキュメントの各フィールドの値をメモリ上にキャッシュして高速化しているのがfielddata cacheのようです。 indices.fielddata.cache.size 未設定だった
50%に設定して様子を見ているが、Elasticのドキュメントを見ると、十分なメモリが残っていないと、Luceneのファイルシステムアクセスのパフォーマンスに影響が出るので、RAMの50%はそのために空けておいた方がよい的なことを言っているので、もっと少ない値を設定すべきかもしれません。この辺はパフォーマンスの様子とAPIでステータスを見ながら調整かなと思っています。 50%に設定しているが、最適な設定値はAPI叩きながら調整？
さらに、ES v1.x では Doc Values という設定がデフォルトでオフになっていることがわかりました。これはLucene側の仕組みで、メモリ上に持っていた情報を代わりにディスクを使うことでメモリ使用量を減らすことができるというもののようです。インデックステンプレートのマッピング設定で明示的にオンに設定することで、さらにメモリ使用量を抑えることができています。 - Doc Values
この設定ですが、バージョンv2.x系からはデフォルトでONになっているので、あまり考える必要はないかなと思います。 - v2.xからはデフォルト
3つめのハマりポイントとして、Amazon ESを使うにあたって、事前の調査が不足していたこともあるんですが、使ってみると、いくつかできないことがあって、ぐぬぬとなると言ったことがありました。スクリプトが使えないので、例えば、ある条件式に当てはまるドキュメントをフィルタするといった場合には工夫が必要です。あとはカスタムプラグインをサポートしていません。 Amazon ESがVPC外部にある関係で、アクセスコントロールはIPベース、またはIAMベースになっています。この認証をLogstashから簡単に行えるようにするため、LogstashのoutputプラグインがAWSから提供されていますが、同じ環境で同じ設定なのに、あるときを境に、突如接続できなくなったりといったことがありました。結局原因は判らずじまいで、悩んだあげく、結局IPベースでのアクセス許可に切り替えることで乗り切っています。 Amazon ESの制限事前調査不足 = 使ってるうちにぐぬぬスクリプト使えないプラグイン使えないアクセスコントロール、IP、IAM、VPC外 logstash-output-amazon_es
これらのハマりポイントを踏まえて、今後の予定ですが、バージョン5系への移行、Amazon Elasticsearch Serviceからの卒業。 Elastic Cloudか自前クラスタでの運用への移行ですね。。 Kibanaを介さず、非エンジニアなユーザが操作履歴の表示をWEBの画面上で閲覧したいという要望があるので、そちらへの対応を予定しています。 5.x系への移行 - Amazon Elasticsearch Serviceからの卒業 - ユーザが操作履歴見れるようWEB画面に統合
今日のまとめです。まとめとしてはAmazon ESやめてバージョンあげたいということなんですが。。。 Amazon ESはあまり設定のことを考えなくても、とりあえずすぐ使えるようになるので立ち上げるのは簡単です。細かい設定なしでも安定して使えるよう、負荷になるような機能に制限がされています。これによって、ESを使いこなしてくると物足りなさを感じるかもしれません。 Elastic社の公式クラウドサービスであるElastic Cloudは、Amazon ESと同じく簡単に立ち上げられるのはもちろんですが、最新バージョンに簡単にアップデートできて、制限も少ないです。完全に制限のない環境が必要な場合には、まだalpha版ですが、Enterprise版という選択肢もでてきました。もちろん自前でクラスタ構築するのもありだと思いますが、そこはメンテナンスやチューニングのコストをどう捉えるか、なのかなと思っています。 Amazon ES = 簡単、チューニングあまり考えずに済む、そのかわり、負荷になる操作が制限されている、物足りなさ Elastic Cloud = 公式、Amazon ES同様簡単、フルマネージド。常に最新バージョンが使えて、簡単にアップデートできる、制限は少ない Enterprise = 制限ない alpha メンテナンス、チューニングコストの捉え方次第
Elastic Stack使ってるという方がいらっしゃれば、懇親会の方でお声がけいただけるとありがたいです。まだまだ初心者なので、是非いろいろ教えていただければと思います。発表は以上になります。ありがとうございました。

Elastic Stackにハマった話

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Elastic Stackにハマった話

Similar to Elastic Stackにハマった話 (20)

Recently uploaded

Recently uploaded (20)

Elastic Stackにハマった話

Editor's Notes