Monitoring MySQL at scale

•

1 like•2,119 views

This document discusses monitoring MySQL databases at scale. It begins with background on Lithium Technologies and their MySQL architecture. It then covers the challenges of monitoring in dynamic environments, monitoring 101 theory, and a real incident triage at Lithium. Key points discussed include the need for automation, metrics collection from all layers, and sharing knowledge. Monitoring hundreds or thousands of MySQL instances requires collecting 350+ metrics from each to gain necessary insights.

Technology

Who We Are
Ilan Rabinovitch
Dir. Technical Community
Datadog
Ovais Tariq
Storage SRE
Uber
(formerly at Lithium & Percona)

Agenda
1. About Lithium and MySQL
2. Background: Monitoring Challenges in a Dynamic World
3. Theory: Monitoring 101
4. Practical: Triaging a Real Incident at Lithium

About Lithium Technologies
Lithium’s platform helps brands connect,
engage and understand their customers

MySQL Architecture / Data Flow
•Multi-Tenant SaaS applications
•Typical Master-slave replication setup
•MySQL running
○ On bare metal
○ In AWS public cloud
○ In OpenStack

Culture
Automation
Metrics
Sharing
Damon Edwards and John Willis
DevOps Day LA

You’re in the cloud and it's everything
you dreamed of!
Autoscaling Infinite StorageManaged
Databases
Container
Orchestration
Private Clouds

Collecting data is cheap;
not having it when you
need it can be expensive

Operational Complexity Increases with..
• Number of things to measure
• Velocity of change

How much we measure?
1 instance
• 10 metrics from CloudWatch
1 operating system (e.g., Linux)
• 100 metrics
MySQL Instance
• 350~ metrics

460
metrics per host
46,000
100
instances

•Earlier - typical Nagios and Cacti setup
•Static config and lack of context
•No correlation between alerts and
graphs
•No self-service for developers
•In-house tooling has high cost

• Query Time
• Queries Per Second
Data Sources
• Performance Schema
• MySQL Status Variables

• Query Time
• Queries Per Second
Sources:
• Performance Schema

• Disk Space Usage
• Threads_connected
• Threads_running
• Connection_errors_ internal
• Aborted_connects
• Connection_errors_ max_connections
Sources:
● Server Status Variables

• Configuration Change
• Code Deployment
• Service Started / Stopped
• MySQL Upgrades
• Failovers
• etc

Change in workload without an increase in
workload affected the schema ‘groupecasino’
• Workload characteristics change to make it more CPU bound
• No increase in IO activity
• Increase in number of read operations
• No change in types of read operations
• Similar number of range queries reading more rows

Monitoring 101: Alerting
https://www.datadoghq.com/blog/monitoring-101-alerting/
Monitoring 101: Collecting the Right Data
https://www.datadoghq.com/blog/monitoring-101-collecting-data/
Monitoring 101: Investigating performance issues
https://www.datadoghq.com/blog/monitoring-101-investigation/
Monitoring MySQL Performance Metrics
https://www.datadoghq.com/blog/monitoring-mysql-performance-metrics/
Collecting MySQL Metrics
https://www.datadoghq.com/blog/collecting-mysql-statistics-and-metrics/

The Hadoop Distributed File System is the foundational storage layer in typical Hadoop deployments. Performance and stability of HDFS are crucial to the correct functioning of applications at higher layers in the Hadoop stack. This session is a technical deep dive into recent enhancements committed to HDFS by the entire Apache contributor community. We describe real-world incidents that motivated these changes and how the enhancements prevent those problems from reoccurring. Attendees will leave this session with a deeper understanding of the implementation challenges in a distributed file system and identify helpful new metrics to monitor in their own clusters.

Data Lake and the rise of the microservices

Bigstep

By simply looking at structured and unstructured data, Data Lakes enable companies to understand correlations between existing and new external data - such as social media - in ways traditional Business Intelligence tools cannot. For this you need to find out the most efficient way to store and access structured or unstructured petabyte-sized data across your entire infrastructure. In this meetup we’ll give answers on the next questions: 1. Why would someone use a Data Lake? 2. Is it hard to build a Data Lake? 3. What are the main features that a Data Lake should bring in? 4. What’s the role of the microservices in the big data world?

C*ollege Credit: Is My App a Good Fit for Cassandra?DataStax

Real time analytics

Leandro Totino Pereira

DATA LAKE AND THE RISE OF THE MICROSERVICES - ALEX BORDEI

Big Data Week

Alex Bordei is a developer turned Product Manager. He has been developing infrastructure products for over nine years. Before becoming Bigstep’s Product Manager, he was one of the core developers for Hostway Corporation’s provisioning platform. He then focused on defining and developing products for Hostway’s EMEA market and was one of the pioneers of virtualization in the company. After successfully launching two public clouds based on VMware software, he created the first prototype of Bigstep’s Full Metal Cloud in 2011. He now focuses on guaranteeing that the Full Metal Cloud is the highest performance cloud in the world, for big data applications.

Making Every Drop Count: How i20 Addresses the Water Crisis with the IoT and ...

DataStax

Lean Enterprise, Microservices and Big DataStylight

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Data Con LA

Druid is a high performance, column-oriented distributed data store that is widely used at Oath for big data analysis. Druid has a JSON schema as its query language, making it difficult for new users unfamiliar with the schema to start querying Druid quickly. The JSON schema is designed to work with the data ingestion methods of Druid, so it can provide high performance features such as data aggregations in JSON, but many are unable to utilize such features, because they not familiar with the specifics of how to optimize Druid queries. However, most new Druid users at Yahoo are already very familiar with SQL, and the queries they want to write for Druid can be converted to concise SQL. We found that our data analysts wanted an easy way to issue ad-hoc Druid queries and view the results in a BI tool in a way that's presentable to nontechnical stakeholders. In order to achieve this, we had to bridge the gap between Druid, SQL, and our BI tools such as Apache Superset. In this talk, we will explore different ways to query a Druid datasource in SQL and discuss which methods were most appropriate for our use cases. We will also discuss our open source contributions so others can utilize our work. GURUGANESH KOTTA, Software Dev Eng, Oath and JUNXIAN WU, Software Engineer, Oath Inc.

ProtectWise Revolutionizes Enterprise Network Security in the Cloud with Data...

DataStax Academy

Transforms Document Management at Scale with Distributed Database Solution wi...

DataStax Academy

SpringCM will discuss how they achieve massive scalability and blazing performance with DataStax Enterprise and HP Moonshot Solutions; to transform enterprise document management forever. They will dive into: how they broke through their capacity to run millions of workloads per hour, why relational database technologies simply cannot handle the scalability demands of document management SaaS, and how they deliver up to 70% more energy savings than traditional rack server architecture.

Building a Digital Bank

DataStax

Continuous Applications at Scale of 100 Teams with Databricks Delta and Struc...

Databricks

Zalando strives to be a fully data-driven company that utilizes AI to make decisions fast and accurately. For this reason we have built a Data Lake that contains all data of the company. To provide easy access to that data and enable the company to make use of it, we have established an internal platform that offers Databricks as a service for all departments and teams. Making Databricks Delta tables available to all clients of the Data Lake enabled them to leverage Structured Streaming and to build continuous applications on top of it. Big part of this journey was solving challenges in governance, security and access management. In this talk we want to share our experience in productionizing and operating Databricks at scale and in making data-driven continuous applications feasible out of the box.

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Data Con LA

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes per day of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. This session is especially recommended for data infrastructure engineers and architects planning, building, or maintaining similar systems.

How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...

DataWorks Summit

For over 30 years, Parametric has been a leading provider of model-based portfolios to institutional and private investors, with unique implementation and customization expertise. Much like other cutting-edge financial services providers, Parametric operates with highly diverse, fast moving data from which they glean insights. Data sources range from benchmark providers to electronic trading participants to stock exchanges etc. The challenge is to not just onboard the data but also to figure out how to monetize it when the schemas are fast changing. This presents a problem to traditional architectures where large teams are needed to design the new ETL flow. Organizations that are able to quickly adapt to new schemas and data sources have a distinct competitive advantage. In this presentation and demo, Architects from Parametric , Chris Gambino & Vamsi Chemitiganti will present the data architecture designed in response to this business challenge. We discuss the approach (and trade-offs) to pooling, managing, processing the data using the latest techniques in data ingestion & pre-processing. The overall best practices in creating a central data pool are also discussed. Quantitative analysts to have the most accurate and up to date information for their models to work on. Attendees will be able to draw on their experiences both from a business and technology standpoint on not just creating a centralized data platform but also being able to distribute it to different units.

Cloudstate - Towards Stateful Serverless

Lightbend

Webinar: Don't Leave Your Data in the Dark

DataStax

As new types of data sources emerge from cloud, mobile devices, social media and machine sensor devices, traditional databases hit the ceiling due to today’s dynamic, data-volume driven business culture. Join us in this online webinar and learn how you can incorporate a modern, NoSQL platform into daily operations to optimize and simplify data performance. DataStax recently announced DataStax Enterprise 4.0, a production-certified version of Apache Cassandra with an in-memory option, enterprise search, advanced security features and visual management tools. Give your developers a simple and powerful way to deliver the information your customers care about most—unconstrained by the complexities and high costs of traditional database systems. Learn how to: - Easily assign data based on its performance needs on traditional spinning disk, SSD or in-memory. All in the same database instance - Leverage DataStax’s built-in enhancements for broader information search and analysis even with many thousands of concurrent requests - Visually monitor, manage, and fine-tune your environment to get the most of your online data

Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Todd Fritz

Bigger Faster Easier: LinkedIn Hadoop Summit 2015

Shirshanka Das

Azure + DataStax Enterprise Powers Office 365 Per User Store

DataStax Academy

Databus - LinkedIn's Change Data Capture Pipeline

Sunil Nagaraj

Redash: Open Source SQL Analytics on Data Lakes

Databricks

Webinar: Get On-Demand Education Anytime, Anywhere with Coursera and DataStaxDataStax

Webinar | Introducing DataStax Enterprise 4.6

DataStax

DataStax Enterprise 4.6, the fastest, most scalable distributed database now integrates Apache Spark analytics on streaming data while providing enterprise-grade backup and restore capabilities to safeguard critical and distributed customer information. Join established database expert and DataStax's VP of Products, Robin Schumacher, as he explores new capabilities in DataStax Enterprise 4.6 including security enhancements, analytics on streaming data and increased performance for modern web, mobile and IoT applications. Robin will discuss how the new OpsCenter 5.1 makes backup and restore processes push-button simple with the option of restoring critical data to and from the cloud taking the burden off database administrators. Watch to learn how - Faster and easier analytics with Spark SQL and Spark Streaming and simplified search make it easy to build scalable fault-tolerant streaming applications - Enhanced server security with LDAP and Active Directory integration for easier external security management - An automated high availability option allows a secondary OpsCenter service to take over, should a failure occur so your maintenance operations are always running

Webinar: Eventual Consistency != Hopeful Consistency

DataStax

Automated Metadata Management in Data Lake – A CI/CD Driven Approach

Databricks

We as data engineers are aware of trade off’s between development speed, metadata governance and schema evolution (or restriction) in rapidly evolving organization. Our day to day activities involve adding/removing/updating tables, protecting PII Information, curating and exposing data to our consumers. While our data lake keeps growing exponentially, there is equal increase in our downstream consumers. Struggle is to maintain balance between quickly promoting metadata changes with robust validation for downstream systems stability. In relational world DDL, DML changes can be managed through numerous options available for every kind of database from the vendor or 3rd party. As engineers we developed a tool which uses centralized git managed repository of data schemas in yml structure with ci/cd capabilities which maintains stability of our data lake and downstream systems. In this presentation Northwestern Mutual Engineers, will discuss how they designed and developed new end-to-end ci/cd driven metadata management tool to make introduction of new tables/views, managing access requests etc in a more robust, maintainable and scalable way, all with only checking in yml files. This tool can be used by people who have no or minimal knowledge of spark. Key focus will be: Need for metadata management tool in a data lake Architecture and Design of the tool Maintaining information on databases/tables/views like schema, owner, PII, description etc in simple to understand yml structure Live demo of creating a new table with CI/CD promotion to production

More Data, More Problems: Scaling Kafka-Mirroring Pipelines at LinkedIn

confluent

(Celia Kung, LinkedIn) Kafka Summit SF 2018 For several years, LinkedIn has been using Kafka MirrorMaker as the mirroring solution for copying data between Kafka clusters across data centers. However, as LinkedIn data continued to grow, mirroring trillions of Kafka messages per day across data centers uncovered the scale limitations and operability challenges of Kafka MirrorMaker. To address such issues, we have developed a new mirroring solution, built on top our stream ingestion service, Brooklin. Brooklin MirrorMaker aims to provide improved performance and stability, while facilitating better management through finer control of data pipelines. Through flushless Kafka produce, dynamic management of data pipelines, per-partition error handling and flow control, we are able to increase throughput, better withstand consume and produce failures and reduce overall operating costs. As a result, we have eliminated the major pain points of Kafka MirrorMaker. In this talk, we will dive deeper into the challenges LinkedIn has faced with Kafka MirrorMaker, how we tackled them with Brooklin MirrorMaker and our plans for iterating further on this new mirroring solution.

Logical-DataWarehouse-Alluxio-meetupGianmario Spacagna

Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015

Datadog

Alerting: more signal, less noise, less pain

Datadog

What's hot

Querying Druid in SQL with Superset

DataWorks Summit

ProtectWise Revolutionizes Enterprise Network Security in the Cloud with Data...

DataStax Academy

Transforms Document Management at Scale with Distributed Database Solution wi...

DataStax Academy

Building a Digital Bank

DataStax

Continuous Applications at Scale of 100 Teams with Databricks Delta and Struc...

Databricks

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Data Con LA

How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...

DataWorks Summit

Cloudstate - Towards Stateful Serverless

Lightbend

Webinar: Don't Leave Your Data in the Dark

DataStax

Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Todd Fritz

Bigger Faster Easier: LinkedIn Hadoop Summit 2015

Shirshanka Das

Azure + DataStax Enterprise Powers Office 365 Per User Store

DataStax Academy

Databus - LinkedIn's Change Data Capture Pipeline

Sunil Nagaraj

Redash: Open Source SQL Analytics on Data Lakes

Databricks

Webinar: Get On-Demand Education Anytime, Anywhere with Coursera and DataStaxDataStax

Webinar | Introducing DataStax Enterprise 4.6

DataStax

Webinar: Eventual Consistency != Hopeful Consistency

DataStax

Automated Metadata Management in Data Lake – A CI/CD Driven Approach

Databricks

More Data, More Problems: Scaling Kafka-Mirroring Pipelines at LinkedIn

confluent

Logical-DataWarehouse-Alluxio-meetupGianmario Spacagna

What's hot (20)

Querying Druid in SQL with Superset

ProtectWise Revolutionizes Enterprise Network Security in the Cloud with Data...

Transforms Document Management at Scale with Distributed Database Solution wi...

Building a Digital Bank

Continuous Applications at Scale of 100 Teams with Databricks Delta and Struc...

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...

Cloudstate - Towards Stateful Serverless

Webinar: Don't Leave Your Data in the Dark

Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Bigger Faster Easier: LinkedIn Hadoop Summit 2015

Azure + DataStax Enterprise Powers Office 365 Per User Store

Databus - LinkedIn's Change Data Capture Pipeline

Redash: Open Source SQL Analytics on Data Lakes

Webinar: Get On-Demand Education Anytime, Anywhere with Coursera and DataStax

Webinar | Introducing DataStax Enterprise 4.6

Webinar: Eventual Consistency != Hopeful Consistency

Automated Metadata Management in Data Lake – A CI/CD Driven Approach

More Data, More Problems: Scaling Kafka-Mirroring Pipelines at LinkedIn

Logical-DataWarehouse-Alluxio-meetup

Viewers also liked

Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015

Datadog

Alerting: more signal, less noise, less pain

Datadog

Events and metrics the Lifeblood of Webops

Datadog

I <3 graphs in 20 slides

Datadog

Fact based monitoringDatadog

Just enough web ops for web developers

Datadog

Treating Infrastructure as Garbage

Datadog

Deep dive into Nagios analytics

Datadog

The Data Mullet: From all SQL to No SQL back to Some SQLDatadog

Big (IT) data

Datadog

DevOps, continuous delivery, & the new composable enterprise

Donnie Berkholz

Making Cassandra Perform as a Time Series Database - Cassandra Summit 15

SignalFx

SignalFx engineer Paul Ingram presented these slides at Cassandra Summit 2015. SignalFx ingests, processes runs analytics against, (and ultimately stores) massive numbers of time series streaming in parallel into our service which provides an analytics-based monitoring platform for modern applications. We've chose to build our time series database (TSDB) on Cassandra for it's read and write performance at high load. This presentation will go over our evolution of optimizations to squeeze the most performance out of the TSDB to date and some steps we'll be taking in the future. Read more: http://blog.signalfx.com/making-cassandra-perform-as-a-tsdb

Customer Ops: DevOps <3 customer support

Datadog

Effective monitoring with StatsD

Datadog

Monitoring Docker containers - Docker NYC Feb 2015

Datadog

Monitoring NGINX (plus): key metrics and how-to

Datadog

NGINX just works and that's why we use it. That does not mean that it should be left unmonitored. As a web server, it plays a central role in a modern infrastructure. As a gatekeeper, it sees every interaction with the application. If you monitor it properly it can explain a lot about what is happening in the rest of your infrastructure. In this talk you will learn more about NGINX (plus) metrics, what they mean and how to use them. You will also learn different methods (status, statsd, logs) to monitor NGINX with their pros and cons, illustrated with real data coming from real servers.

PyData NYC 2015 - Automatically Detecting Outliers with Datadog

Datadog

Monitoring even a modestly-sized systems infrastructure quickly becomes untenable without automated alerting. For many metrics it is nontrivial to define ahead of time what constitutes “normal” versus “abnormal” values. This is especially true for metrics whose baseline value fluctuates over time. To make this problem more tractable, Datadog provides outlier detection functionality to automatically identify any host (or group of hosts) that is behaving abnormally compared to its peers. These slides cover the algorithms we use for outlier detection, and show how easy they are to implement using Python. This presentation also covers the lessons we've learned from using outlier detection on our own systems, along with some real-life examples on how to avoid false positives and negatives. Learn more at www.datadoghq.com.

How to measure everything - a million metrics per second with minimal develop...

Jos Boumans

Krux is an infrastructure provider for many of the websites you use online today, like NYTimes.com, WSJ.com, Wikia and NBCU. For every request on those properties, Krux will get one or more as well. We grew from zero traffic to several billion requests per day in the span of 2 years, and we did so exclusively in AWS. To make the right decisions in such a volatile environment, we knew that data is everything; without it, you can't possibly make informed decisions. However, collecting it efficiently, at scale, at minimal cost and without burdening developers is a tremendous challenge. Join me in this session to learn how we overcame this challenge at Krux; I will share with you the details of how we set up our global infrastructure, entirely managed by Puppet, to capture over a million data points every second on virtually every part of the system, including inside the web server, user apps and Puppet itself, for under $2000/month using off the shelf Open Source software and some code we've released as Open Source ourselves. In addition, I’ll show you how you can take (a subset of) these metrics and send them to advanced analytics and alerting tools like Circonus or Zabbix. This content will be applicable for anyone collecting or desiring to collect vast amounts of metrics in a cloud or datacenter setting and making sense of them.

Application Monitoring using Datadog

Mukta Aphale

Viewers also liked (19)

Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015

Alerting: more signal, less noise, less pain

Events and metrics the Lifeblood of Webops

I <3 graphs in 20 slides

Fact based monitoring

Just enough web ops for web developers

Treating Infrastructure as Garbage

Deep dive into Nagios analytics

The Data Mullet: From all SQL to No SQL back to Some SQL

Big (IT) data

DevOps, continuous delivery, & the new composable enterprise

Making Cassandra Perform as a Time Series Database - Cassandra Summit 15

Customer Ops: DevOps <3 customer support

Effective monitoring with StatsD

Monitoring Docker containers - Docker NYC Feb 2015

Monitoring NGINX (plus): key metrics and how-to

PyData NYC 2015 - Automatically Detecting Outliers with Datadog

How to measure everything - a million metrics per second with minimal develop...

Application Monitoring using Datadog

Similar to Monitoring MySQL at scale

Webinar: SQL for Machine Data?

Crate.io

By 2020, 50% of all new software will process machine-generated data of some sort (Gartner). Historically, machine data use cases have required non-SQL data stores like Splunk, Elasticsearch, or InfluxDB. Today, new SQL DB architectures rival the non-SQL solutions in ease of use, scalability, cost, and performance. Please join this webinar for a detailed comparison of machine data management approaches.

John adams talk cloudy

John Adams

Serverless SQL

Torsten Steinbach

Microservices - opportunities, dilemmas and problems

Łukasz Sowa

Ankus, bigdata deployment and orchestration framework

Ashrith Mekala

Microservices, Continuous Delivery, and Elasticsearch at Capital One

Noriaki Tatsumi

Demystifying Data Warehouse as a Service (DWaaS)

Kent Graziano

This is from the talk I gave at the 30th Anniversary NoCOUG meeting in San Jose, CA. We all know that data warehouses and best practices for them are changing dramatically today. As organizations build new data warehouses and modernize established ones, they are turning to Data Warehousing as a Service (DWaaS) in hopes of taking advantage of the performance, concurrency, simplicity, and lower cost of a SaaS solution or simply to reduce their data center footprint (and the maintenance that goes with that). But what is a DWaaS really? How is it different from traditional on-premises data warehousing? In this talk I will: • Demystify DWaaS by defining it and its goals • Discuss the real-world benefits of DWaaS • Discuss some of the coolest features in a DWaaS solution as exemplified by the Snowflake Elastic Data Warehouse.

Log Monitoring and Anomaly Detection at Scale at ORNL

Elasticsearch

Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost

Zilliz

If you are building a RAG application that serves millions of users, you should consider how to scale your system seamlessly and cost-efficiently. The Zilliz Serverless tier represents a significant innovation in the field of vector search, enabling you to rapidly scale to millions of tenants and billions of vectors, while fully leveraging the hot/cold characteristics across tenants to reduce data storage costs. It enables vector storage at costs comparable to S3 and facilitates vector search times in the hundreds of milliseconds for tens of millions of data points! In this talk, we will delve into the implementation details, usage patterns, and performance metrics of Zilliz Serverless. We will discuss how it empowers AI-native applications to achieve rapid business growth by providing a cost-effective and scalable vector storage and search solution.

Stay productive_while_slicing_up_the_monolith

Markus Eisele

Cloud-native Data

cornelia davis

<November 2017 Updated from earlier presentations on Cloud-native Data> Cloud-native applications form the foundation for modern, cloud-scale digital solutions, and the patterns and practices for cloud-native at the app tier are becoming widely understood – statelessness, service discovery, circuit breakers and more. But little has changed in the data tier. Our modern apps are often connected to monolithic shared databases that have monolithic practices wrapped around them. As a result, the autonomy promised by moving to a microservices application architecture is compromised. What we need are patterns and practices for cloud-native data. The anti-patterns of shared databases and simple proxy-style web services to front them give way to approaches that include use of caches (Netflix calls caching their hidden microservice), database per service and polyglot persistence, modern versions of ETL and data integration and more. In this session, aimed at the application developer/architect, Cornelia will look at those patterns and see how they serve the needs of the cloud-native application.

Cloud-Native-Data with Cornelia Davis

VMware Tanzu

Suning OpenStack Cloud and Heat

Qiming Teng

Estimating the Total Costs of Your Cloud Analytics Platform

DATAVERSITY

Organizations today need a broad set of enterprise data cloud services with key data functionality to modernize applications and utilize machine learning. They need a platform designed to address multi-faceted needs by offering multi-function Data Management and analytics to solve the enterprise’s most pressing data and analytic challenges in a streamlined fashion. They need a worry-free experience with the architecture and its components.

Maplelabs scalable-field-device-cloud-native

Ganeshkumar Sundararajan

Chirp 2010: Scaling Twitter

John Adams

Scaling Systems: Architectures that grow

Gibraltar Software

It's harder than ever to predict the load your application will need to handle in advance, so how do you design your architecture so you can afford to implement as you go and be ready for whatever comes your way. It's easy to focus on optimizing each part of your application but your application architecture determines the options you have to make big leaps in scalability. In this talk we'll cover practical patterns you can build today to meet the needs of rapid development while still creating systems that can scale up and out. Specific code examples will focus on .NET but the principles apply across many technologies. Real world systems will be discussed based on our experience helping customers around the world optimize their enterprise applications.

25 snowflake

剑飞陈

Automated Cluster Management and Recovery for Large Scale Multi-Tenant Sea...

Lucidworks

Scaling Your Database in the Cloud

RightScale

Rightscale Webinar: The number one cause of poor scalable web application performance is the database. This problem is magnified in cloud environments where I/O and bandwidth are generally slower and less predictable than in dedicated data centers. Database sharding is a highly effective method of removing the database scalability barrier by operating on top of proven RDBMS products such as MySQL and PostgreSQL. In this webinar, you'll learn what it really takes to implement sharding, the role it plays in the effective end-to-end lifecycle management of your entire database environment, and why it is crucial for ensuring reliability. In this webinar, we will: - Guide you on how to choose the best technology for your specific application - Show you how to shard your existing database - Review a case study on a Top 20 Facebook application built on dbShards

Similar to Monitoring MySQL at scale (20)

Webinar: SQL for Machine Data?

John adams talk cloudy

Serverless SQL

Microservices - opportunities, dilemmas and problems

Ankus, bigdata deployment and orchestration framework

Microservices, Continuous Delivery, and Elasticsearch at Capital One

Demystifying Data Warehouse as a Service (DWaaS)

Log Monitoring and Anomaly Detection at Scale at ORNL

Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost

Stay productive_while_slicing_up_the_monolith

Cloud-native Data

Cloud-Native-Data with Cornelia Davis

Suning OpenStack Cloud and Heat

Estimating the Total Costs of Your Cloud Analytics Platform

Maplelabs scalable-field-device-cloud-native

Chirp 2010: Scaling Twitter

Scaling Systems: Architectures that grow

25 snowflake

Automated Cluster Management and Recovery for Large Scale Multi-Tenant Sea...

Scaling Your Database in the Cloud

Recently uploaded

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

JMeter webinar - integration with InfluxDB and Grafana

RTTS

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Recently uploaded (20)

Key Trends Shaping the Future of Infrastructure.pdf

UiPath Test Automation using UiPath Test Suite series, part 3

Leading Change strategies and insights for effective change management pdf 1.pdf

FIDO Alliance Osaka Seminar: Overview.pdf

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Neuro-symbolic is not enough, we need neuro-*semantic*

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Accelerate your Kubernetes clusters with Varnish Caching

When stars align: studies in data quality, knowledge graphs, and machine lear...

JMeter webinar - integration with InfluxDB and Grafana

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Epistemic Interaction - tuning interfaces to provide information for AI support

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

"Impact of front-end architecture on development cost", Viktor Turskyi

Connector Corner: Automate dynamic content and events by pushing a button

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

The Art of the Pitch: WordPress Relationships and Sales

Designing Great Products: The Power of Design and Leadership by Chief Designe...

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Monitoring MySQL at scale

1. Monitoring MySQL at SCALE

2. Who We Are Ilan Rabinovitch Dir. Technical Community Datadog Ovais Tariq Storage SRE Uber (formerly at Lithium & Percona)

3. Agenda 1. About Lithium and MySQL 2. Background: Monitoring Challenges in a Dynamic World 3. Theory: Monitoring 101 4. Practical: Triaging a Real Incident at Lithium

4. About Lithium Technologies Lithium’s platform helps brands connect, engage and understand their customers

5. MySQL Architecture / Data Flow •Multi-Tenant SaaS applications •Typical Master-slave replication setup •MySQL running ○ On bare metal ○ In AWS public cloud ○ In OpenStack

7. Culture Automation Metrics Sharing Damon Edwards and John Willis DevOps Day LA

8. Culture Automation Metrics Sharing Damon Edwards and John Willis DevOps Day LA

10. You’re in the cloud and it's everything you dreamed of! Autoscaling Infinite StorageManaged Databases Container Orchestration Private Clouds

11.

12. Collecting data is cheap; not having it when you need it can be expensive

13.

14. Instrument all the things!

15. Operational Complexity Increases with.. • Number of things to measure • Velocity of change

16. How much we measure? 1 instance • 10 metrics from CloudWatch 1 operating system (e.g., Linux) • 100 metrics MySQL Instance • 350~ metrics

17. 460 metrics per host 46,000 100 instances

18.

19.

20.

21.

22.

23.

24.

25. •Earlier - typical Nagios and Cacti setup •Static config and lack of context •No correlation between alerts and graphs •No self-service for developers •In-house tooling has high cost

26.

27.

28.

29. When to let a sleeping engineer lie?

30.

31. Recurse until you find root cause

32. • Query Time • Queries Per Second Data Sources • Performance Schema • MySQL Status Variables

33. • Query Time • Queries Per Second Sources: • Performance Schema

34. • Disk Space Usage • Threads_connected • Threads_running • Connection_errors_ internal • Aborted_connects • Connection_errors_ max_connections Sources: ● Server Status Variables

35. • Configuration Change • Code Deployment • Service Started / Stopped • MySQL Upgrades • Failovers • etc

36.

37.

38.

39.

40.

41.

42.

43.

44.

45. Change in workload without an increase in workload affected the schema ‘groupecasino’ • Workload characteristics change to make it more CPU bound • No increase in IO activity • Increase in number of read operations • No change in types of read operations • Similar number of range queries reading more rows

46.

47. Monitoring 101: Alerting https://www.datadoghq.com/blog/monitoring-101-alerting/ Monitoring 101: Collecting the Right Data https://www.datadoghq.com/blog/monitoring-101-collecting-data/ Monitoring 101: Investigating performance issues https://www.datadoghq.com/blog/monitoring-101-investigation/ Monitoring MySQL Performance Metrics https://www.datadoghq.com/blog/monitoring-mysql-performance-metrics/ Collecting MySQL Metrics https://www.datadoghq.com/blog/collecting-mysql-statistics-and-metrics/

Monitoring MySQL at scale

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Monitoring MySQL at scale

Similar to Monitoring MySQL at scale (20)

Recently uploaded

Recently uploaded (20)

Monitoring MySQL at scale