Вячеслав Крюков, Ivinco

•

1 like•1,957 views

The document describes a multi-terabyte Sphinx HA (high availability) cluster. It discusses requirements for the cluster including tolerance, balancing, redundancy, and easy deployment. It provides details on the cluster architecture, configuration options for balancing strategies and monitoring agent status. Metrics for monitoring performance and availability are also outlined.

Technology

Multi-Terabyte Sphinx HA cluster
Vyacheslav Kryukov
vkrukov@ivinco.com

Sphinx HA cluster, requrements

●

Incident tolerance and availability level

●

Adaptive balancing

●

Resources redundancy utilisation

●

Easy deployment of new resources

Sphinx HA cluster, ha_strategy

●

●

Simple balancing
●
random
●
roundrobin
Adaptive balancing
●
nodeads
●
noerrors

http://sphinxsearch.com/docs/current.html#conf-ha-strategy

Sphinx HA cluster, adaptive balancing
●

Latency

●

Query timeouts

●

Connect timeouts

●

Connect failures

●

Network errors

●

Wrong replies

●

Unexpected closings

●

Warnings

$Sphinx HA cluster, configuration index some_index { type = distributed agent = se01-1:3312|se01-2:3312:some_index_se01 agent = se02-1:3312|se02-2:3312:some_index_se02 agent = se03-1:3312|se03-2:3312:some_index_se03 agent = se04-1:3312|se04-2:3312:some_index_se04 ha_strategy = nodeads } searchd { ... ha_ping_interval = 1000 ha_period_karma = 60 ... } http://sphinxsearch.com/docs/current.html#conf-ha-ping-interval http://sphinxsearch.com/docs/current.html#conf-ha-period-karma$

Sphinx HA cluster, SHOW AGENT STATUS
mysql> SHOW AGENT STATUS;
+-------------------------------------+--------------------+
| Key
| Value
|
+-------------------------------------+--------------------+
| status_period_seconds
| 60
|
| status_stored_periods
| 15
|
...
| ag_19_hostname
| se02-1:3312
|
| ag_19_references
| 13
|
| ag_19_lastquery
| 1.91
|
| ag_19_lastanswer
| 1.86
|
| ag_19_lastperiodmsec
| 51
|
| ag_19_errorsarow
| 0
|
| ag_19_1periods_query_timeouts
| 0
|
| ag_19_1periods_connect_timeouts
| 0
|
| ag_19_1periods_connect_failures
| 0
|
| ag_19_1periods_network_errors
| 0
|
| ag_19_1periods_wrong_replies
| 0
|
| ag_19_1periods_unexpected_closings | 0
|
| ag_19_1periods_warnings
| 0
|
| ag_19_1periods_succeeded_queries
| 101
|
| ag_19_1periods_msecsperqueryy
| 83.92
|
(the same for 5periods_ and 15periods_)
| ag_20_hostname
| se02-2:3312
|
| ag_20_references
| 13
|
| ag_20_lastquery
| 0.55
|
| ag_20_lastanswer
| 0.49
|
| ag_20_lastperiodmsec
| 55
|
| ag_20_errorsarow
| 0
|
| ag_20_1periods_query_timeouts
| 0
|
| ag_20_1periods_connect_timeouts
| 0
|
| ag_20_1periods_connect_failures
| 0
|
| ag_20_1periods_network_errors
| 0
|
| ag_20_1periods_wrong_replies
| 0
|
| ag_20_1periods_unexpected_closings | 0
|
| ag_20_1periods_warnings
| 0
|
| ag_20_1periods_succeeded_queries
| 55
|
| ag_20_1periods_msecsperqueryy
| 86.08
|
(the same for 5periods_ and 15periods_)
...

Sphinx HA cluster, balancing in real time

Sphinx HA cluster, balancing in real time

# cd /mnt/data
# iozone -i0 -i2 -s16g -r32k -f iozone.tmp

Sphinx HA cluster, data processing

●

Data loading to permanent store

●

Data indexig

Indexes validation and synchronization (Rsync and
NetCat)
●

●

Update indexes from application

Sphinx HA cluster, performance and
availability
●

Provide performance with band wide

●

What to monitor
●

SHOW AGENT STATUS, nodes performance, disc
space, io and cpu usage

●

Errors, warnings, crashes

●

Indexes synchronization, validity, freshness

Sphinx HA cluster, distributed indexer
●

Automated
●

distributed indexing

●

Indexes validation

●

indexes delivery

●

Failover

●

Centralised Sphinx indexes configuration management

●

Indexes rebalancing

Resources consumption accounting

●

io ops

●

io size

●

fetched_docs

●

fetched_hits

●

fetched_skips

●

total_found

Rosette Linguistics Platform

●

Used for analysis of unstructured text in CJK languages

●

Better quality then using ngram options

●

Slow indexer performance

http://www.basistech.com/text-analytics/rosette/

The document provides an overview of improvements and new features in Nagios Core 4. Key points include: - Bottlenecks in Nagios Core 3 were analyzed and improvements were made to configuration parsing, event queue insertion, macro resolution, and check processing. These improved performance and scalability. - A new query handler and NERD (Nagios Event Radio Dispatcher) were added to provide real-time data to external addons. - Support for service parents and new check result variables were added. Deprecation notices were provided for removed or changed features.

Nginx conf.compressed

Mauricio Roman

Mauricio Roman discusses detecting anomalies in Nginx log data through multi-dimensional analysis. He explores his company's Nginx logs, extracting over 100 features to identify unexpected error patterns. Parsing logs with open source tools, he sends data to Kafka and finds: 1) 408 errors correlate with large GET payloads, 2) most 4xx errors come from Opera, 3) Opera 4xx errors originate from specific countries. His vision is to automate such exploration and correlate HTTP and application logs in real time to monitor error rates and identify true anomalies.

Nmap2Nessus Presentation Slides at Black Hat Asia Arsenal 2015

Keith Lee

This document describes nmap2nessus, a tool that takes Nmap scan results as input in XML format and uses them to configure and launch a targeted vulnerability scan with Nessus. It explains how nmap2nessus works by extracting open ports and live IPs from the Nmap file, modifying an existing Nessus policy, and starting a scan with that policy. The document also provides instructions for using nmap2nessus and includes a demo of it used against a Metasploitable2 virtual machine.

RSA NetWitness Log Decoder

Susam Pal

This document provides instructions on configuring and troubleshooting the NetWitness Log Decoder. It begins with basic configuration steps like adding a log decoder, setting the capture interface, and starting capture. It then covers potential capture failures due to issues like disk space, parser errors, and describes how to analyze logs and configurations to diagnose the problems. The document also demonstrates how to ingest logs and view them in the log decoder and concentrator. It provides tips on troubleshooting log ingestion issues and checking that the log decoder is capturing logs received on port 514.

Scaling for India's Cricket Hungry Population (Bhavesh Raheja & Namit Mahuvak...

confluent

Hotstar is a media entertainment platform, and has a large user base in India. Last year during the Indian Premier League, Hotstar introduced "Watch N Play", a real-time cricket prediction game in which over 33 million unique users answered over 2 billion questions, won more than a 100 million rewards, built with Kafka as the backbone. In the game, the user guesses the outcome of the next ball. If he/she guesses right before the actual outcome, they score points to climb up the ladder and receive rewards along the way. Supporting potentially millions of users with differing stream times & device latencies; we used topics to separate logical streams & partitions to scale to support 1M requests/second. We'll talk about how we reduced the -end latency to make the user-experience as real-time as possible. We'll focus on the operational challenges that we faced and how we overcame them by building automation around it to make our operational cum lives easier. We also have a war story around ghost topic creation where after deletion; topics would get magically created without any create topic requests. This led us to some interesting revelations and very important learnings of how long-living producers and consumers with very short-lived topics are a recipe for disaster.

GDG Cloud Taipei meetup #50 - Build go kit microservices at kubernetes with ...

KAI CHU CHUNG

Lab: JVM Production Debugging 101

Tomer Gabel

масштабируемый Sphinx кластер. вячеслав крюков. зал 1rit2011

Monal Daxini presented on the declarative benchmarking tool NDBench and its Cassandra plugin. The tool allows users to define performance test profiles that specify the Cassandra schema, queries, load patterns, and other parameters. It executes the queries against Cassandra clusters and collects metrics to analyze performance. The plugin supports all Cassandra data types and allows testing different versions. Netflix uses it to validate data models and certify Cassandra upgrades. Future enhancements include adding more data generators and supporting other data stores.

Gnocchi v4 - past and present

Gordon Chung

Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming

Apache Apex

Presenter: Devendra Tagare - DataTorrent Engineer, Contributor to Apex, Data Architect experienced in building high scalability big data platforms. Apache Apex is a next generation native Hadoop big data platform. This talk will cover details about how it can be used as a powerful and versatile platform for big data. Apache Apex is a native Hadoop data-in-motion platform. We will discuss architectural differences between Apache Apex features with Spark Streaming. We will discuss how these differences effect use cases like ingestion, fast real-time analytics, data movement, ETL, fast batch, very low latency SLA, high throughput and large scale ingestion. We will cover fault tolerance, low latency, connectors to sources/destinations, smart partitioning, processing guarantees, computation and scheduling model, state management and dynamic changes. We will also discuss how these features affect time to market and total cost of ownership.

[CB20] Vulnerabilities of Machine Learning Infrastructure by Sergey Gordeychik

CODE BLUE

The boom of AI brought to the market a set of impressive solutions both on the hardware and software side. On the other hand, massive implementation of AI in various areas brings about problems, and security is one of the greatest concerns. In this talk we will present results of hands-on vulnerability research of different components of AI infrastructure including NVIDIA DGX GPU servers, ML frameworks such as Pytorch, Keras and Tensorflow, data processing pipelines and specific applications, including Medical Imaging and face recognition powered CCTV. Updated Internet Census toolkit based on the Grinder framework will be introduced.

Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...

DataStax

At Knewton we operate across five different VPCs a total of 29 clusters, each ranging from 3 nodes to 24 nodes. For a team of three to maintain this is not herculean, however good tools to diagnose issues and gather information in a distributed manner are vital to moving quickly and minimizing engineering time spent. The database team at Knewton has been successfully using a combination of Ansible and custom open sourced tools to maintain and improve the Cassandra deployment at Knewton. I will be talking about several of these tools and giving examples of how we are using them. Specifically I will discuss the cassandra-tracing tool, which analyzes the contents of the system_traces keyspace, and the cassandra-stat tool, which gives real-time output of the operations of a cassandra cluster. Distributed administration with ad-hoc Ansible will also be covered and I will walk through examples of using these commands to identify and remediate clusterwide issues. About the Speaker Jeffrey Berger Lead Database Engineer, Knewton Dr. Jeffrey Berger is currently the lead database engineer at Knewton, an education tech startup in NYC. He joined the tech scene in NYC in 2013 and spent two years working with MongoDB, becoming a certified MongoDB administrator and a MongoDB Master. He received his Cassandra Administrator certification at Cassandra Summit 2015. He holds a Ph.D. in Theoretical Physics from Penn State and spent several years working on high energy nuclear interactions.

Intro to Apache Apex - Next Gen Platform for Ingest and Transform

Apache Apex

Introduction to Apache Apex - The next generation native Hadoop platform. This talk will cover details about how Apache Apex can be used as a powerful and versatile platform for big data processing. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch alerts, real-time actions, threat detection, etc. Bio: Pramod Immaneni is Apache Apex PMC member and senior architect at DataTorrent, where he works on Apache Apex and specializes in big data platform and applications. Prior to DataTorrent, he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs.

Network Automation with Salt and NAPALM: Introuction

Cloudflare

This document discusses how CloudFlare uses Salt and NAPALM for network automation and configuration management across their global network of over 80 points of presence. Some key points: - CloudFlare routes web traffic through their global network of over 80 locations serving over 4 million domains and handling over 43 billion DNS queries per day. - They use Salt and NAPALM to automate the deployment, configuration, monitoring and maintenance of their large network, including replacing equipment and deploying new points of presence. - NAPALM integrates directly with Salt and provides vendor-agnostic modules to control network devices, retrieve information and enforce configurations across different device types and vendors.

Update on OpenTSDB and AsyncHBase

HBaseCon

Scaling Up Logging and Metrics

Ricardo Lourenço

This document discusses scaling up logging and metrics in OpenShift Container Platform (OCP). It provides an overview of the logging stack including Elasticsearch, Fluentd, and Kibana. It also summarizes the metrics stack including Cassandra, Heapster, and Hawkular. The document outlines testing done to evaluate limits and scaling of these components on large OCP clusters with thousands of nodes and pods. It provides recommendations for configuring and deploying the infrastructure to support high throughput logging and metrics collection.

Comparison between zookeeper, etcd 3 and other distributed coordination systems

Imesha Sudasingha

This is a comparison between popular distributed coordination systems including zookeeper (which powers Apache Hadoop), etcd 3 (which powers Kubernetes), consul and hazelcast. This comparison was made in second half of 2016. Therefore, please note that some of these technologies have improved immensely over the time. Anyway, this presentation will provide an initial idea of each distributed coordination systems.

NetflixOSS Open House Lightning talks

Ruslan Meshenberg

This summary provides an overview of the lightning talks presented at the NetflixOSS Open House: - Jordan Zimmerman from Netflix presented on several NetflixOSS projects he works on including Curator, a Java library that makes using ZooKeeper easier, and Blitz4j, an asynchronous logging library that improves performance over Log4j. - Additional talks covered Eureka, a REST service for discovering middle-tier services; Ribbon for load balancing between middle-tier instances; Archaius for dynamic configuration; Astyanax for interacting with Cassandra; and various other NetflixOSS projects. - The talks highlighted the motivation for these projects including addressing challenges of scaling for Netflix's large data

FOSDEM 2012: MySQL synchronous replication in practice with Galera

FromDual GmbH

This document provides an overview of MySQL synchronous replication using Galera. It discusses why Galera is used over other replication options, how to set up a Galera cluster with 3 nodes, the configuration required, and how to perform operations like starting/stopping nodes and performing a state snapshot transfer. It also covers characteristics of Galera like synchronous multi-master replication, automatic membership control, and parallel replication. The document is meant to facilitate questions and discussion about using Galera in practice.

Clug 2012 March web server optimisation

grooverdan

Speed up R with parallel programming in the Cloud

Revolution Analytics

There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe several techniques available in R to speed up workloads like these, by running multiple iterations simultaneously, in parallel. Many of these techniques require the use of a cluster of machines running R, and I'll provide examples of using cloud-based services to provision clusters for parallel computations. In particular, I will describe how you can use the SparklyR package to distribute data manipulations using the dplyr syntax, on a cluster of servers provisioned in the Azure cloud. Presented by David Smith at Data Day Texas in Austin, January 27 2018.

Incrementalism: An Industrial Strategy For Adopting Modern Automation

Sean Chittenden

Update on OpenTSDB and AsyncHBase

HBaseCon

This document summarizes an update on OpenTSDB, an open source time series database. It discusses OpenTSDB's ability to store trillions of data points at scale using HBase, Cassandra, or Bigtable as backends. Use cases mentioned include systems monitoring, sensor data, and financial data. The document outlines writing and querying functionality and describes the data model and table schema. It also discusses new features in OpenTSDB 2.2 and 2.3 like downsampling, expressions, and data stores. Community projects using OpenTSDB are highlighted and the future of OpenTSDB is discussed.

Andrii Rodionov: What can go wrong in a distributed system – experience from ...

Lviv Startup Club

Kubernetes Navigation Stories – DevOpsStage 2019, Kyiv

Aleksey Asiutin

thredUP team shares key learnings from after-migration processes. We tell you about what technologies and solutions worked best for us and where we spent time troubleshooting and improving. In particular we have focused on development and staging experience, user authentication, cloud-native CI pipelines, applications telemetry and service mesh. We also share our experience with Kubernetes security hardening, autoscaling and tell you about a new service creation within our infrastructure.

Machine learning at Scale with Apache Spark

Martin Zapletal

This document discusses scaling machine learning using Apache Spark. It covers several key topics: 1) Parallelizing machine learning algorithms and neural networks to distribute computation across clusters. This includes data, model, and parameter server parallelism. 2) Apache Spark's Resilient Distributed Datasets (RDDs) programming model which allows distributing data and computation across a cluster in a fault-tolerant manner. 3) Examples of very large neural networks trained on clusters, such as a Google face detection model using 1,000 servers and a IBM brain-inspired chip model using 262,144 CPUs.

Shall we play a game?

Maciej Lasyk

How to run system administrator recruitment process? By creating platform based on open source parts in just 2 nights! I gave this talk in Poland / Kraków OWASP chapter meeting on 17th October 2013 at our local Google for Entrepreneurs site. It's focused on security and also shows how to create recruitment process in CTF / challenge way. This story covers mostly security details of this whole platform. There's great chance, that I will give another talk about this system but this time focusing on technical details. Stay tuned ;)

One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...

Вячеслав Крюков, Ivinco

Recommended

Recommended

More Related Content

Similar to Вячеслав Крюков, Ivinco

Similar to Вячеслав Крюков, Ivinco (20)

More from Ontico

More from Ontico (20)

Recently uploaded

Recently uploaded (20)

Вячеслав Крюков, Ivinco