Troubleshooting Cassandra 2.1: A Guided Tour of nodetool and system.log. From Cassandra Summit 2015. Download and check out the presenter notes for tips!
I’ll give a general lay of the land for troubleshooting Cassandra. Then I’ll take you on a deep dive through nodetool and system.log and give you a guided tour of the useful information they provide for troubleshooting. I’ll devote special attention to monitoring the various processes that Cassandra uses to do its work and how to effectively search for information about specific error messages online.
Slides from my talk at Cassandra Summit 2016 on troubleshooting Cassandra. This is a reprise of my popular talk from last summit, reorganized, expanded, and updated for Cassandra 3.0. In it I share the secrets I've learned in four years of supporting hundreds of customers using Apache Cassandra and DataStax Enterprise. Be sure to check out presenter notes for additional tips and links to further resources.
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
As part of the Tungsten project, Spark has started an ongoing effort to dramatically improve performance to bring the execution closer to bare metal. In this talk, we’ll go over the progress that has been made so far and the areas we’re looking to invest in next. This talk will discuss the architectural changes that are being made as well as some discussion into how Spark users can expect their application to benefit from this effort. The focus of the talk will be on Spark SQL but the improvements are general and applicable to multiple Spark technologies.
Understanding of Apache kafka metrics for monitoring SANG WON PARK
2019 kafka conference seould에서 발표한 "Apache Kafka 모니터링을 위한 Metrics 이해" 슬라이드 자료
기존 2018년 자료에서 모니터링 관점에서 중요한 metrcis를 중심으로 정리하였고, 2019년 기준으로 추가/변경된 metrics를 반영하였다.
주용 내용은
- 업무에 최적화된 apache kafka 모니터링을 하려면?
- 어떤 정보를 모니터링 해야 할까?
- 적시성 관점의 모니터링 지표 (TotalTimeMs에 대한 세부 구조 이해)
- 안정성 관점의 모니터링 지표 (데이터 유실이 없이 중단없는 서비스)
- 언제 apache kafka 클러스터를 확장해야 할까? (어떤 지표를 봐야 할까?)
위 모든 지표는 producer/broker/consumer 3가지 측면에서 검토하였다.
컨퍼런스 영상 링크 : https://www.youtube.com/watch?v=p2RGsTOCHAg
Cloud dw benchmark using tpd-ds( Snowflake vs Redshift vs EMR Hive )SANG WON PARK
몇년 전부터 Data Architecture의 변화가 빠르게 진행되고 있고,
그 중 Cloud DW는 기존 Data Lake(Hadoop 기반)의 한계(성능, 비용, 운영 등)에 대한 대안으로 주목받으며,
많은 기업들이 이미 도입했거나, 도입을 검토하고 있다.
본 자료는 이러한 Cloud DW에 대해서 개념적으로 이해하고,
시장에 존재하는 다양한 Cloud DW 중에서 기업의 환경에 맞는 제품이 어떤 것인지 성능/비용 관점으로 비교했다.
- 왜기업들은 CloudDW에주목하는가?
- 시장에는어떤 제품들이 있는가?
- 우리Biz환경에서는 어떤 제품을 도입해야 하는가?
- CloudDW솔루션의 성능은?
- 기존DataLake(EMR)대비 성능은?
- 유사CloudDW(snowflake vs redshift) 대비성능은?
앞으로도 Data를 둘러싼 시장은 Cloud DW를 기반으로 ELT, Mata Mesh, Reverse ETL등 새로운 생테계가 급속하게 발전할 것이고,
이를 위한 데이터 엔지니어/데이터 아키텍트 관점의 기술적 검토와 고민이 필요할 것 같다.
https://blog.naver.com/freepsw/222654809552
Slides from my talk at Cassandra Summit 2016 on troubleshooting Cassandra. This is a reprise of my popular talk from last summit, reorganized, expanded, and updated for Cassandra 3.0. In it I share the secrets I've learned in four years of supporting hundreds of customers using Apache Cassandra and DataStax Enterprise. Be sure to check out presenter notes for additional tips and links to further resources.
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
As part of the Tungsten project, Spark has started an ongoing effort to dramatically improve performance to bring the execution closer to bare metal. In this talk, we’ll go over the progress that has been made so far and the areas we’re looking to invest in next. This talk will discuss the architectural changes that are being made as well as some discussion into how Spark users can expect their application to benefit from this effort. The focus of the talk will be on Spark SQL but the improvements are general and applicable to multiple Spark technologies.
Understanding of Apache kafka metrics for monitoring SANG WON PARK
2019 kafka conference seould에서 발표한 "Apache Kafka 모니터링을 위한 Metrics 이해" 슬라이드 자료
기존 2018년 자료에서 모니터링 관점에서 중요한 metrcis를 중심으로 정리하였고, 2019년 기준으로 추가/변경된 metrics를 반영하였다.
주용 내용은
- 업무에 최적화된 apache kafka 모니터링을 하려면?
- 어떤 정보를 모니터링 해야 할까?
- 적시성 관점의 모니터링 지표 (TotalTimeMs에 대한 세부 구조 이해)
- 안정성 관점의 모니터링 지표 (데이터 유실이 없이 중단없는 서비스)
- 언제 apache kafka 클러스터를 확장해야 할까? (어떤 지표를 봐야 할까?)
위 모든 지표는 producer/broker/consumer 3가지 측면에서 검토하였다.
컨퍼런스 영상 링크 : https://www.youtube.com/watch?v=p2RGsTOCHAg
Cloud dw benchmark using tpd-ds( Snowflake vs Redshift vs EMR Hive )SANG WON PARK
몇년 전부터 Data Architecture의 변화가 빠르게 진행되고 있고,
그 중 Cloud DW는 기존 Data Lake(Hadoop 기반)의 한계(성능, 비용, 운영 등)에 대한 대안으로 주목받으며,
많은 기업들이 이미 도입했거나, 도입을 검토하고 있다.
본 자료는 이러한 Cloud DW에 대해서 개념적으로 이해하고,
시장에 존재하는 다양한 Cloud DW 중에서 기업의 환경에 맞는 제품이 어떤 것인지 성능/비용 관점으로 비교했다.
- 왜기업들은 CloudDW에주목하는가?
- 시장에는어떤 제품들이 있는가?
- 우리Biz환경에서는 어떤 제품을 도입해야 하는가?
- CloudDW솔루션의 성능은?
- 기존DataLake(EMR)대비 성능은?
- 유사CloudDW(snowflake vs redshift) 대비성능은?
앞으로도 Data를 둘러싼 시장은 Cloud DW를 기반으로 ELT, Mata Mesh, Reverse ETL등 새로운 생테계가 급속하게 발전할 것이고,
이를 위한 데이터 엔지니어/데이터 아키텍트 관점의 기술적 검토와 고민이 필요할 것 같다.
https://blog.naver.com/freepsw/222654809552
Apache Kafka lies at the heart of the largest data pipelines, handling trillions of messages and petabytes of data every day. Learn the right approach for getting the most out of Kafka from the experts at LinkedIn and Confluent. Todd Palino and Gwen Shapira demonstrate how to monitor, optimize, and troubleshoot performance of your data pipelines—from producer to consumer, development to production—as they explore some of the common problems that Kafka developers and administrators encounter when they take Apache Kafka from a proof of concept to production usage. Too often, systems are overprovisioned and underutilized and still have trouble meeting reasonable performance agreements.
Topics include:
- What latencies and throughputs you should expect from Kafka
- How to select hardware and size components
- What you should be monitoring
- Design patterns and antipatterns for client applications
- How to go about diagnosing performance bottlenecks
- Which configurations to examine and which ones to avoid
This presentation provides an overview of the Dell PowerEdge R730xd server performance results with Red Hat Ceph Storage. It covers the advantages of using Red Hat Ceph Storage on Dell servers with their proven hardware components that provide high scalability, enhanced ROI cost benefits, and support of unstructured data.
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
Talk for Kernel Recipes 2017 by Brendan Gregg. "Linux perf is a crucial performance analysis tool at Netflix, and is used by a self-service GUI for generating CPU flame graphs and other reports. This sounds like an easy task, however, getting perf to work properly in VM guests running Java, Node.js, containers, and other software, has been at times a challenge. This talk summarizes Linux perf, how we use it at Netflix, the various gotchas we have encountered, and a summary of advanced features."
BlueStore, A New Storage Backend for Ceph, One Year InSage Weil
BlueStore is a new storage backend for Ceph OSDs that consumes block devices directly, bypassing the local XFS file system that is currently used today. It's design is motivated by everything we've learned about OSD workloads and interface requirements over the last decade, and everything that has worked well and not so well when storing objects as files in local files systems like XFS, btrfs, or ext4. BlueStore has been under development for a bit more than a year now, and has reached a state where it is becoming usable in production. This talk will cover the BlueStore design, how it has evolved over the last year, and what challenges remain before it can become the new default storage backend.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
EMR 플랫폼 기반의 Spark 워크로드 실행 최적화 방안 - 정세웅, AWS 솔루션즈 아키텍트:: AWS Summit Online Ko...Amazon Web Services Korea
발표영상 다시보기: https://youtu.be/hPvBst9TPlI
S3 기반의 데이터레이크에서 대량의 데이터 변환과 처리에 사용될 수 있는 가장 대표적인 솔루션이 Apache Spark 입니다. EMR 플랫폼 환경에서 쉽게 적용 가능한 Apache Spark의 성능 향상 팁을 소개합니다. 또한 데이터의 레코드 레벨 업데이트, 리소스 확장, 권한 관리 및 모니터링과 같은 다양한 데이터 워크로드 관리 최적화 방안을 함께 살펴봅니다.
Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. At this scale, output committers that create extra copies or can’t handle task failures are no longer practical. This talk will explain the problems that are caused by the available committers when writing to S3, and show how Netflix solved the committer problem.
In this session, you’ll learn:
– Some background about Spark at Netflix
– About output committers, and how both Spark and Hadoop handle failures
– How HDFS and S3 differ, and why HDFS committers don’t work well
– A new output committer that uses the S3 multi-part upload API
– How you can use this new committer in your Spark applications to avoid duplicating data
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...confluent
Kafka is notoriously tricky for multi-dc use cases. The log abstraction and client failover breaks down when you cannot at least guarantee offset consistency. In this talk, we define the current state of Kafka in terms of multi-dc usage, how different approaches provide different guarantees as well as examining the missing gaps, and how the community is addressing them.
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward
Let’s be honest: Running a distributed stateful stream processor that is able to handle terabytes of state and tens of gigabytes of data per second while being highly available and correct (in an exactly-once sense) does not work without any planning, configuration and monitoring. While the Flink developer community tries to make everything as simple as possible, it is still important to be aware of all the requirements and implications In this talk, we will provide some insights into the greatest operations mysteries of Flink from a high-level perspective: - Capacity and resource planning: Understand the theoretical limits. - Memory and CPU configuration: Distribute resources according to your needs. - Setting up High Availability: Planning for failures. - Checkpointing and State Backends: Ensure correctness and fast recovery For each of the listed topics, we will introduce the concepts of Flink and provide some best practices we have learned over the past years supporting Flink users in production.
Cassandra Troubleshooting (for 2.0 and earlier)J.B. Langston
I’ll give a general lay of the land for troubleshooting Cassandra. Then I’ll take you on a deep dive through nodetool and system.log and give you a guided tour of the useful information they provide for troubleshooting. I’ll devote special attention to monitoring the various processes that Cassandra uses to do its work and how to effectively search for information about specific error messages online.
This is the old version of this presentation for Cassandra 2.0 and earlier. Check out the updated slide deck for Cassandra 2.1.
Apache Kafka lies at the heart of the largest data pipelines, handling trillions of messages and petabytes of data every day. Learn the right approach for getting the most out of Kafka from the experts at LinkedIn and Confluent. Todd Palino and Gwen Shapira demonstrate how to monitor, optimize, and troubleshoot performance of your data pipelines—from producer to consumer, development to production—as they explore some of the common problems that Kafka developers and administrators encounter when they take Apache Kafka from a proof of concept to production usage. Too often, systems are overprovisioned and underutilized and still have trouble meeting reasonable performance agreements.
Topics include:
- What latencies and throughputs you should expect from Kafka
- How to select hardware and size components
- What you should be monitoring
- Design patterns and antipatterns for client applications
- How to go about diagnosing performance bottlenecks
- Which configurations to examine and which ones to avoid
This presentation provides an overview of the Dell PowerEdge R730xd server performance results with Red Hat Ceph Storage. It covers the advantages of using Red Hat Ceph Storage on Dell servers with their proven hardware components that provide high scalability, enhanced ROI cost benefits, and support of unstructured data.
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
Talk for Kernel Recipes 2017 by Brendan Gregg. "Linux perf is a crucial performance analysis tool at Netflix, and is used by a self-service GUI for generating CPU flame graphs and other reports. This sounds like an easy task, however, getting perf to work properly in VM guests running Java, Node.js, containers, and other software, has been at times a challenge. This talk summarizes Linux perf, how we use it at Netflix, the various gotchas we have encountered, and a summary of advanced features."
BlueStore, A New Storage Backend for Ceph, One Year InSage Weil
BlueStore is a new storage backend for Ceph OSDs that consumes block devices directly, bypassing the local XFS file system that is currently used today. It's design is motivated by everything we've learned about OSD workloads and interface requirements over the last decade, and everything that has worked well and not so well when storing objects as files in local files systems like XFS, btrfs, or ext4. BlueStore has been under development for a bit more than a year now, and has reached a state where it is becoming usable in production. This talk will cover the BlueStore design, how it has evolved over the last year, and what challenges remain before it can become the new default storage backend.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
EMR 플랫폼 기반의 Spark 워크로드 실행 최적화 방안 - 정세웅, AWS 솔루션즈 아키텍트:: AWS Summit Online Ko...Amazon Web Services Korea
발표영상 다시보기: https://youtu.be/hPvBst9TPlI
S3 기반의 데이터레이크에서 대량의 데이터 변환과 처리에 사용될 수 있는 가장 대표적인 솔루션이 Apache Spark 입니다. EMR 플랫폼 환경에서 쉽게 적용 가능한 Apache Spark의 성능 향상 팁을 소개합니다. 또한 데이터의 레코드 레벨 업데이트, 리소스 확장, 권한 관리 및 모니터링과 같은 다양한 데이터 워크로드 관리 최적화 방안을 함께 살펴봅니다.
Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. At this scale, output committers that create extra copies or can’t handle task failures are no longer practical. This talk will explain the problems that are caused by the available committers when writing to S3, and show how Netflix solved the committer problem.
In this session, you’ll learn:
– Some background about Spark at Netflix
– About output committers, and how both Spark and Hadoop handle failures
– How HDFS and S3 differ, and why HDFS committers don’t work well
– A new output committer that uses the S3 multi-part upload API
– How you can use this new committer in your Spark applications to avoid duplicating data
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...confluent
Kafka is notoriously tricky for multi-dc use cases. The log abstraction and client failover breaks down when you cannot at least guarantee offset consistency. In this talk, we define the current state of Kafka in terms of multi-dc usage, how different approaches provide different guarantees as well as examining the missing gaps, and how the community is addressing them.
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward
Let’s be honest: Running a distributed stateful stream processor that is able to handle terabytes of state and tens of gigabytes of data per second while being highly available and correct (in an exactly-once sense) does not work without any planning, configuration and monitoring. While the Flink developer community tries to make everything as simple as possible, it is still important to be aware of all the requirements and implications In this talk, we will provide some insights into the greatest operations mysteries of Flink from a high-level perspective: - Capacity and resource planning: Understand the theoretical limits. - Memory and CPU configuration: Distribute resources according to your needs. - Setting up High Availability: Planning for failures. - Checkpointing and State Backends: Ensure correctness and fast recovery For each of the listed topics, we will introduce the concepts of Flink and provide some best practices we have learned over the past years supporting Flink users in production.
Cassandra Troubleshooting (for 2.0 and earlier)J.B. Langston
I’ll give a general lay of the land for troubleshooting Cassandra. Then I’ll take you on a deep dive through nodetool and system.log and give you a guided tour of the useful information they provide for troubleshooting. I’ll devote special attention to monitoring the various processes that Cassandra uses to do its work and how to effectively search for information about specific error messages online.
This is the old version of this presentation for Cassandra 2.0 and earlier. Check out the updated slide deck for Cassandra 2.1.
Webinar: Diagnosing Apache Cassandra Problems in ProductionDataStax Academy
This session covers diagnosing and solving common problems encountered in production, using performance profiling tools. We’ll also give a crash course to basic JVM garbage collection tuning. Viewers will leave with a better understanding of what they should look for when they encounter problems with their in-production Cassandra cluster.
Performance tuning - A key to successful cassandra migrationRamkumar Nottath
In last few years, technology has seen a major drift in the dominance of traditional / RDMBS databases across different domains. Expeditious adoption of NoSQL databases especially Cassandra in the industry opens up a lot more discussions on what are the major challenges that are faced during implementation of Cassandra and how to mitigate it. Many a times we conclude that migration or POC (proof of concept) is not successful; however the real flaw might be in the data modeling, identifying the right hardware configurations, database parameters, right consistency level and so on. There's no one good model or configuration which fits all use cases and all applications. Performance tuning an application is truly an art and requires perseverance. This paper delve into different performance tuning considerations and anti-patterns that need to be considered during Cassandra migration / implementation to make sure we are able to reap the benefits of Cassandra, what makes it a ‘Visionary’ in 2014 Gartner’s Magic Quadrant for Operational Database Management Systems.
I'm going to cover something which could be seen as essential for Cassandra but which hasn't gotten much attention in the Cassandra community and literature. It's schema migrations--how you go about pushing out and versioning changes to your keyspace and table definitions across environments. This is an area that has established solutions in the relational database world, with tools like Liquibase(http://www.liquibase.org/) and Flyway (http://flywaydb.org/) and in web frameworks like Rails and Grails.
I'll explain the different types of migrations but then focus, for most of the talk, on schema migrations. I'll explain how schema migrations have been done in the Cassandra community and the roadblocks teams have faced trying to use Liquibase and Flyway to manage Cassandra migrations.
Then I'll share an elegant, lightweight schema migrations system that we at GridPoint built on top of Flyway. I'll use our system as a context for discussing schema migration best practices for Cassandra and the various choices teams have for their migrations and table definitions, including when NOT to use a tool like Flyway. I'll also touch on the other types of migrations besides keyspace and table definitions that can be versioned and driven off source control.
This will be a reprise of my popular talk from last year, updated and expanded for Cassandra 3.0. I'll discuss the general approach to troubleshooting Cassandra, then give a guided tour what to look for in nodetool, logs, and OpsCenter, highlighting the most useful topics for troubleshooting real-world Cassandra issues.
About the Speaker
J.b. Langston Principal Support Engineer, DataStax
I've been with DataStax support for over 4 years, helping customers troubleshoot problems and keep their Cassandra clusters running smoothly. Prior to that, I had 8 years of experience supporting a Java-based grid computing platform.
Webinar: Untethering Compute from StorageAvere Systems
Enterprise storage infrastructures are gradually sprawling across the globe and consumers of data increasingly require access to remote storage resources. Solutions for mitigating the pain associated with this growth are out there, but performance varies. This Webinar will take a look at these challenges, review available solutions, and compare tests of performance.
What do data center operators need to know when deploying Hadoop in the Data Center? Multi-tenancy, network topology, workload types, and myriad other factors affect the way applications run and perform in the data center. Understanding performance characteristics of the distributed system is key to not only optimize for Hadoop, but allows Hadoop to seamlessly operate side-by-side existing applications.
Enterprise data centers house numerous workloads. With Hadoop growing in these data centers, IT departments need tools to avoid creating silos, while maintaining SLAs, reporting and charge-back requirements. We present a completely open source reference architecture including Apache Hadoop, Linux cgroups and namespace isolation, Gluster and HTCondor. Topics to be covered – . Augmenting existing HDFS and MapReduce infrastructure with dynamically provisioned resources . On-demand creating, growing and shrinking MapReduce infrastructure for user workload . Isolating workloads to enable multi-tenant access to resources . Publishing of resource utilization and accounting information for ingest into charge-back systems
There are many aspects of tuning Cassandra for production and a lot can go wrong: network splits and latency, hardware issues and failure, data corruption, etc. Most are mitigated with Cassandra's architecture but there are use cases where we need to dig deep and tune all layers to get the result we need to achieve specific business goals.
We will explore such case where we had to tune Cassandra for performance but also have consistent results on 99.999% of the queries. Getting even to 99 percent was relatively easy, but pushing those extra nines involved a lot of work. There are many nuts and bolts to turn and tune in order to get consistent results.
We will cover biggest latency-inducing factors and see how to set up metrics and tackle inevitable issues when doing cloud-based deployments. We will get into one of the major "sins" regarding AWS deployment by demystifying EBS based storage and talk about how we can leverage OS properties while tuning for high read performance.
About the Speaker
Matija Gobec CTO, SmartCat
Experienced software engineer interested in distributed streaming systems and real time analytics. In love with Cassandra since early versions.
Avoiding Chaos: Methodology for Managing Performance in a Shared Storage A...brettallison
Scope - The primary focus of this presentation is on the methodology we use for managing performance in a very large shared Storage Area Network environment with a Primary focus on Distributed Systems and IBM Enterprise Storage Server. The focus on this presentation is methodology and NOT measurement. There are numerous excellent presentations already out there on measurement. However, there are several references in the back of the presentation to measurement tools.
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAlluxio, Inc.
Alluxio Tech Talk
Jul 17, 2019
Speakers:
Brien Porter, Intel
Alex Ma, Alluxio
The ever increasing challenge to process and extract value from exploding data with AI and analytics workloads makes a memory centric architecture with disaggregated storage and compute more attractive. This decoupled architecture enables users to innovate faster and scale on-demand. Enterprises are also increasingly looking towards object stores to power their big data & machine learning workloads in a cost-effective way. However, object stores don’t provide big data compatible APIs as well as the required performance.
In this webinar, the Intel and Alluxio teams will present a proposed reference architecture using Alluxio as the in-memory accelerator for object stores to enable modern analytical workloads such as Spark, Presto, Tensorflow, and Hive. We will also present a technical overview of Alluxio.
Practice and challenges from building IaaSShawn Zhu
It is an invited presentation for NCSC2012 (China National Conference on Social Computing) on cloud computing from industry.
It summarized what we learn on developing and operating an Infrastructure as a Service in a highly scalable manner. The service described inside the corporation is kind of dogfood that engineers work with in their daily work.
AI Genie Review: World’s First Open AI WordPress Website CreatorGoogle
AI Genie Review: World’s First Open AI WordPress Website Creator
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-genie-review
AI Genie Review: Key Features
✅Creates Limitless Real-Time Unique Content, auto-publishing Posts, Pages & Images directly from Chat GPT & Open AI on WordPress in any Niche
✅First & Only Google Bard Approved Software That Publishes 100% Original, SEO Friendly Content using Open AI
✅Publish Automated Posts and Pages using AI Genie directly on Your website
✅50 DFY Websites Included Without Adding Any Images, Content Or Doing Anything Yourself
✅Integrated Chat GPT Bot gives Instant Answers on Your Website to Visitors
✅Just Enter the title, and your Content for Pages and Posts will be ready on your website
✅Automatically insert visually appealing images into posts based on keywords and titles.
✅Choose the temperature of the content and control its randomness.
✅Control the length of the content to be generated.
✅Never Worry About Paying Huge Money Monthly To Top Content Creation Platforms
✅100% Easy-to-Use, Newbie-Friendly Technology
✅30-Days Money-Back Guarantee
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIGenieApp #AIGenieBonus #AIGenieBonuses #AIGenieDemo #AIGenieDownload #AIGenieLegit #AIGenieLiveDemo #AIGenieOTO #AIGeniePreview #AIGenieReview #AIGenieReviewandBonus #AIGenieScamorLegit #AIGenieSoftware #AIGenieUpgrades #AIGenieUpsells #HowDoesAlGenie #HowtoBuyAIGenie #HowtoMakeMoneywithAIGenie #MakeMoneyOnline #MakeMoneywithAIGenie
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly.
The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppGoogle
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-fusion-buddy-review
AI Fusion Buddy Review: Key Features
✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini
✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique!
✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs!
✅Fully automated AI articles bulk generation!
✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more.
✅With one keyword or URL, generate complete websites, landing pages, and more…
✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7.
✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches.
✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all!
✅Save over $5000 per year and kick out dependency on third parties completely!
✅Brand New App: Not available anywhere else!
✅ Beginner-friendly!
✅ZERO upfront cost or any extra expenses
✅Risk-Free: 30-Day Money-Back Guarantee!
✅Commercial License included!
See My Other Reviews Article:
(1) AI Genie Review: https://sumonreview.com/ai-genie-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIFusionBuddyReview,
#AIFusionBuddyFeatures,
#AIFusionBuddyPricing,
#AIFusionBuddyProsandCons,
#AIFusionBuddyTutorial,
#AIFusionBuddyUserExperience
#AIFusionBuddyforBeginners,
#AIFusionBuddyBenefits,
#AIFusionBuddyComparison,
#AIFusionBuddyInstallation,
#AIFusionBuddyRefundPolicy,
#AIFusionBuddyDemo,
#AIFusionBuddyMaintenanceFees,
#AIFusionBuddyNewbieFriendly,
#WhatIsAIFusionBuddy?,
#HowDoesAIFusionBuddyWorks
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
E-commerce Application Development Company.pdfHornet Dynamics
Your business can reach new heights with our assistance as we design solutions that are specifically appropriate for your goals and vision. Our eCommerce application solutions can digitally coordinate all retail operations processes to meet the demands of the marketplace while maintaining business continuity.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
I’m a lead support engineer at DataStax
I’ve been at DataStax just over 3 years and this is my fourth summit
Overall troubleshooting process from a support engineer’s perspective
I’ll focus on the tools Cassandra gives you to do steps 1 to 4
Steps 5 and 6 are the hard parts; but luckily that’s what DataStax support, or the mailing list, or StackOverflow can help you with
Doing the legwork on the first part will make the second part happen much faster
It’s also helpful to keep in mind the various system resources that Cassandra consumes
CPU
Consider both single core and multi-core utilization
Some processes are single-threaded and bottlenecked by a single core
Memory
Heap space, typically limited to a subset of your total physical memory
Cassandra stores many objects off heap to avoid Garbage Collection issues
The OS will use whatever memory is left over for page cache; make sure you leave enough free memory for this
Disk
Available disk space
I/O bandwidth utilization
Network
Primarily concerned with bandwidth
Keep in mind firewalls; the path needs to be open
OS Resources/Limits
File handles, processes, etc.
Make sure you set high enough limits in limits.conf or ulimits
Linux Monitoring Commands: http://www.tecmint.com/command-line-tools-to-monitor-linux-performance/
Overview of some of the most useful nodetool commands and what they do
Nodetool documentation: http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsNodetool_r.html
Nodetool status gives an overview of the entire cluster’s status
The cluster is divided into datacenter
The first column shows whether a node is up or down
The second column shows the node’s state, which one of:
normal
leaving the cluster
joining the cluster
moving its token
The IP address of each node.
This will be the broadcast_address if specified; otherwise, the listen_address
The amount of data on each node
It is normal for this to differ slightly
Large discrepancies could indicate a problem (in terms percentage)
Wide rows/partitions
Uneven racks
Compaction issues
Number of tokens
Normally 256 for vnodes, 1 for non-vnodes
Percentage of ring each node owns
Note message at the top
Without a keyspace, assumes SimpleStrategy with RF=1
Can cause strange readout when using multiple DCs with a small offset between tokens (nothing to worry about)
With keyspace, shows ownership according to RF in that keyspace
With keyspace, should add up to RF times 100%
May differ slightly from node to node with vnodes due to random token distribution
UUID of the node. Used to uniquely identify the node when running removenode command.
The rack the node is on
Used to avoid single point of failure
Avoid if not using vnodes
If using vnodes, ensure the same number of nodes are in each rack
In this example, we have one node that is down, so that is where we’d focus our investigation
Nodetool ring is an old way of checking status
Shows the same information as nodetool status, except for the token
When using vnodes, it will show every token on each node and becomes difficult to read
Nodetool info shows information specific to the node where it is run
Some of the information is also shown by nodetool ring/status
Same information shown by nodetool status and ring
Not going to rehash this
Whether gossip, thrift, and native transport are enabled
Can be individually enabled/disabled with nodetool commands
Gossip generation; increases each time the node is restarted
How many seconds the node has been running
Heap and off heap memory usage
Heap memory shows amount currently in use and maximum
Amount currently in use may include garbage
The amount of garbage included varies depending on when GC was last run
Off heap memory is stored outside the heap but adds to the process’s overall resident size
If a process gets killed by the kernel OOM killer, make sure it isn’t using too much off heap memory
Number of exceptions that occurred since last restart
Not every error involves an exception, but it’s still a good indicator
Key cache caches the location of partition keys in memory
Avoids using bloom filters to determine which sstables to read
Capacity defaults to 100MB or 5% of the heap, whichever is less
Row cache stores entire rows in memory (entire partitions prior to 2.1)
Disabled by default
Only enable in these circumstances:
Small, very hot data set that will fit in memory
Data should be read much more than written because writes invalidate row in cache
Prior to 2.1, only useful on small partitions (no clustering keys!)
OK to use with clustering keys in 2.1 because individual rows are cached
Holds hot counter values in memory
2.1 introduced more accurate counters
Performance cost due to read-before-write
Counter cache mitigates performance cost
Capacity defaults to 50MB or 2.5% of heap, whichever is less
Entries, size and capacity
- Entries is number of keys/rows/counters in the cache
Size is amount actually in use (bytes)
Capacity is amount specified by key_cache_size_in_mb in cassandra.yaml
If size is consistently much less than capacity, you have this set too high
Cache hits, total requests, and hit rate
- If hit rate is low, try increasing cache capacity slowly until you see diminishing returns
Caches are periodically saved to disk and reloaded when a node is restarted
This is to avoid a cold cache after restarts
tpstats shows thread pool statistics
Cassandra has various thread pools that handle important foreground and background tasks
This information is also logged to system.log by StatusLogger.java periodically or when a message is dropped
Name of the thread pool
Number of tasks being actively serviced by a thread
For several stages, this can be configured in cassandra.yaml
For others, it is equal to the number of CPU cores
For a few others, it is a hardcoded limit, usually 1
Number tasks waiting to be serviced
Unless limited via yaml setting, limit is ~2 billion
You’ll run out of memory long before you ever hit this limit
High number of pending tasks indicates the stage is overloaded
Number of tasks completed since last restart
Number of tasks currently blocked for I/O
Should almost always be 0
Total number of tasks blocked since last restart.
Usually zero except for FlushWriter
At the bottom is a list of message types and the number of messages dropped
Load shedding drops requests that have been pending beyond timeout specified in cassandra.yaml
Dropped messages usually indicate overloaded cluster; check for other causes, then add more nodes
Handles local reads for which this node is a replica
Number of threads is controlled by concurrent_reads in cassandra.yaml
Various reads, all handled by ReadStage
READ - normal read on a single partition
RANGE_SLICE - A sequential or secondary index scan over multiple partitions
PAGED_RANGE - Used for automatic paging when result size exceeds row limit
Handles local writes for which this node is a replica
Number of threads is controlled by concurrent_writes in cassandra.yaml
Various writes, handled by MutationStage
MUTATION - normal write
COUNTER_MUTATION - incrementing a counter
Coordinator uses this to process responses from other nodes
Roughly indicates how often this node has been a coordinator
Roughly because unless using CL ONE, need to handle responses from multiple nodes
Request completed but timed out before coordinator could respond to it
Timeout controlled by request_timeout_in_ms in cassandra.yaml
May indicate that coordinator is overloaded
Ensure that client load balancing is set up correctly
Make sure use of batches is appropriate
Logged batches only when atomicity is required
Unlogged batches only when updating multiple rows with the same partition key
Otherwise use asynchronous execution to pipeline requests without overloading a single coordinator
Writes memtables to disk
Number of threads controlled by memtable_flush_writers, should equal number of data drives
Maximum pending tasks controlled by memtable_flush_queue_size in cassandra.yaml
Once queue is full writes are blocked until another flush writer is available
Large number of all-time-blocked tasks indicates a disk bottleneck; add more/faster disks or more nodes
Handles compactions
Number of threads controlled by concurrent_compactors in cassandra.yaml
Constantly pending compactions means compactions can’t keep up with writes; mitigation strategies:
Switch from LeveledCompactionStrategy to SizeTieredCompactionStrategy for write-heavy tables
Get faster disks (SSD) if needed
Increase compaction throughput
Get faster CPU cores
Asynchronous read repairs
Occur for a certain percentage of reads, configurable per table
Pending tasks indicate that you may have read repair chance set too high
READ_REPAIR - dropped write due to a read repair
Can sometimes show up as a TimedOutException on the read that triggered it
Handles hint delivery from the coordinator to a node that’s recently come back up
Large number of hints usually means an unhealthy cluster
Nodes gossip with each other once a second
Completed GossipStage tasks is an easy way to tell approximately how long a node has been up
Prior to 2.0, when using vnodes with a large cluster, gossip could become CPU-bound and get behind
Migrates schema changes to other nodes
If you see pending tasks here, you’re making too many schema changes too quickly
Repair related stages
AntiEntropyStage coordinates repairs
AntiEntropySesssions are active repairs in progress
ValidationExecutor builds merkle trees for repair
cfstats shows statistics for individual tables
Tables are grouped by keyspace
Values apply only to the node where cfstats is run, not cluster-wide
Keyspace and table names
Multiple tables grouped under each keyspace
Read and write counts
Per table table and at keyspace level
Read and write latency
Per table and averaged at keyspace level
Number of flushes pending against a table
Per table and summed at keyspace level
Space used by the table on this node
Must sum across different nodes to get total space
May include deleted or updated data that hasn’t been compacted yet
Live and total values are usually equal
If some tables have been discarded but not deleted yet, total may be larger
Space used by snapshots for the table; if you’re running out of space, look here
Space consumed by off-heap data structures
Total off heap memory
Broken down by data structure: bloom filter, index summary, compression metadata
Number of sstables comprising a table
Broken down by level when using LeveledCompactionStrategy
Bloom filter statistics
False positives
Too high, performance will suffer
Reduce false positive chance for table
Space used
Lower false positive chance requires more space
Bloom filter grows linearly with number of partitions
Increase false positive if bloom filter is too large
Percent of original size once data is compressed (lower is better)
Compression trades higher CPU usage for lower I/O usage
Usually this is a good tradeoff
But if compression ratio is high, you may want to turn it off for this table
Number of partition keys in the table on this node
Estimated to the nearest index_interval (128 by default)
Rows spread across multiple sstables will inflate this number
Memtable information
number of entries in the memtable
bytes of data in the memtable
bytes of data in the memtable stored off heap
the number of times the memtable has been switched (flushed to disk)
Statistics on partition size, calculated during compaction
Maximum, minimum, and average size
Helps identify tables containing large partitions
Tombstone statistics
Number of live cells versus tombstones encountered when scanning a partition
Rolling average for the last 5 minutes
cfhistograms provides deeper insight into a specific table
Must specify keyspace and table name when calling nodetool cfhistograms
Information shown by cfhistograms is local to the node where it was run
Keyspace and table for which histograms were requested
Percentiles (percent of X less than this value)
Minimum and maximum values
50% aka median
The number of sstables that had to be scanned for each read query
Scanning more sstables is more expensive and will lead to higher read latencies
Reports counts for reads within the last 5 minutes (approximately)
Write and Read latency in microseconds
Remember to divide by 1000 to get ms
Reports latencies for the last 5 minutes (approximately)
This is the latency for local reads so it doesn’t include round trip time for the coordinator
The size of each partition on the node in bytes
In this example:
the largest partition is 2.23MB
the smallest is 1.29KB
the median partition size is 28.8KB
99% of partitions are 444KB or smaller
Number of cells in a partition
Partition key is stored separately (not counted here)
Cells are name/value pairs used to store column data
There will be one cell per non-key column in each row, plus one sentinel cell per row
cfhistograms provides deeper insight into a specific table
Must specify keyspace and table name when calling nodetool cfhistograms
Information shown by cfhistograms is local to the node where it was run
cfhistograms provides deeper insight into a specific table
Must specify keyspace and table name when calling nodetool cfhistograms
Information shown by cfhistograms is local to the node where it was run
cfhistograms provides deeper insight into a specific table
Must specify keyspace and table name when calling nodetool cfhistograms
Information shown by cfhistograms is local to the node where it was run
cfhistograms provides deeper insight into a specific table
Must specify keyspace and table name when calling nodetool cfhistograms
Information shown by cfhistograms is local to the node where it was run
cfhistograms provides deeper insight into a specific table
Must specify keyspace and table name when calling nodetool cfhistograms
Information shown by cfhistograms is local to the node where it was run
Shows network activity
Mode, same as on status: NORMAL, JOINING, LEAVING, MOVING
Active streams
Read repair statistics
Commands sent and responses received while acting as coordinator
Repair session IDs
Nodes exchanging data
Data is streaming over listen address instead of broadcast address
Useful on EC2 because Amazon doesn’t charge for internal traffic
Total number of files and bytes of data to be received from specified node
Total number of files and bytes of data to be sent to specified node
Specific files currently being transferred
Progress for each file
Number of read repairs attempted
Number of mismatches resolved
foreground
background
Number of pending and completed commands sent to other nodes
Number of pending and completed responses received from other nodes
Status of current and pending compactions
Number of pending compactions
Compactions in progress
Compaction Type
Compaction - normal compaction
Validation - building merkle trees for repair
Keyspace and table
Bytes complete and total bytes for each compaction
Percent done
Estimated time remaining for the active tasks
Not useful, because:
estimates can be wrong
doesn’t account for pending tasks
History of recent compactions
Shows how much space a compaction reclaimed
Same information available in system.log
Unique ID
Keyspace and column family
Unix timestamp when compaction occurred
Seconds since Jan 1, 1970
Total bytes before compaction
Total bytes after compaction
Row merge counts
Actually the last column on each row
Moved to make output fit on slide
Count of sstables
Number of rows spread across that many sstables prior to compaction
Used to check for schema disagreements
This is not the information we’re interested in
We’re looking to see how many schema versions there are in the cluster.
No disagreement; only one version of the schema shared by all nodes
Schema disagreement; one node has a different version from all the others
Schema disagreements must be manually resolved
If only one node disagrees, run nodetool resetlocalschema on that node
If multiple nodes disagree
shut down nodes in the minority and delete system/schema_* sstables
start nodes back up one by one
system.log is the most important tool for troubleshooting Cassandra
Basic format
Level: INFO, WARN, ERROR by default; DEBUG only if configured
Thread: use the ID to correlate messages from the same thread
Date & Time: use to correlate messages across multiple nodes, time duration of events
Source File/Line No: code that logged the message, not necessarily where an error occurred; talk about stack traces later
system.log is the most important tool for troubleshooting Cassandra
Basic format
Level: INFO, WARN, ERROR by default; DEBUG only if configured
Thread: use the ID to correlate messages from the same thread
Date & Time: use to correlate messages across multiple nodes, time duration of events
Source File/Line No: code that logged the message, not necessarily where an error occurred; talk about stack traces later
Exception - What kind of error occurred?
Exception - What kind of error occurred?
Stack trace – where did the error happen?
Most local at top to most global at bottom
Wall of text — we’ll dissect it in the next slides
Organization names – whose fault is it?
Sub-packages usually group major application subsystems
Class Name – specific object
Will usually, but not always match the filename
Nested Classes - $ separates outer class from inner class(es)
Method belongs to the inner class
Method name – what was the class doing?
<init> indicates that the error occurred in a static initialization block or constructor
File name – where the source code is, should you want to look at it
Also pay attention to the package name so you can find the file within the nested directory structure
When source-diving, start with the most local method and work your way out
Line number – where to look in the code (available on github.com/apache/cassandra)
Be careful! Line numbers change between versions, so make sure you select the the right version in github
Pay attention to nested exceptions
Each exception has its own stack trace which may be completely different
The outer exception may be too general because it’s been rethrown from unrelated code
The innermost exception will be the actual root cause of the error
Best to use a combination of outer and inner exception as search terms
Exception will usually have an error message
Provides additional information about the circumstances of the exception
Usually good to search for the exception and message together
Look out for embedded numbers or strings; these may change from one message to the next, and including them will undesirably narrow your search
- Some additional examples of organizations and subsystems
- Some additional examples of organizations and subsystems
Use exception and several package+class+method names
Exception alone often isn’t sufficient because the same error can occur many different places
You’ll find the same exception in unrelated software if it’s a standard java exception.
Add several package/class/method combinations to narrow down the exception
Use at least the topmost method and the first org.apache.cassandra method
Use quotes around individual elements (especially if they contain spaces)
Line numbers shouldn’t be part of your search criteria because you may not find the same error in a different version
Likewise, exclude specific numbers and strings like names and counts from your search
Use Google’s site: feature to narrow search terms to apache JIRA, cassandra mailing list, or stackoverflow
Add or remove additional methods as needed to narrow or broaden search
These might be a good set of search terms for this exception
Include both exceptions and methods from both stack traces
Include exception and error message grouped together inside quotes
Know how to recognize a restart
Check versions of major components and JVM
Confirm settings are what you think they are
Make sure you have JNA installed
Know when node is ready to serve requests
Cassandra writes updates to the commit log on disk and the memtables in memory
The memtables are flushed to sstables on disk as required
First the flush is enqueued
There are a limited number of FlushWriter threads
Flushes wait in the queue until a FlushWriter is available
- When a FlushWriter becomes available, the flush begins
Eventually the flush completes.
This is not an instantaneous process because disks are slow.
This is the name of the table
You can use this to link the enqueuing and writing of the flush
Make note of the MemtableFlushWriter thread doing the flush
You can use this to link the beginning and end of the flush
The thread that enqueues the memtable provides clues about why it was flushed:
When the memory fills up, the flush is enqueued by SlabPoolCleaner (for on-heap) or NativePoolCleaner (for off-heap)
If the commit log space fills up, the flush is enqueued by OptionalTasks
Other threads may enqueue flushes for other reasons
If you have lots of tiny sstables, it may be from flushing too frequently so try to figure out why
This shows the number of bytes stored in the memtable both on-heap and off-heap
Also shows the percentage of total space devoted to memtables that this one consumed
serialized bytes, larger than in-memory bytes, due to serialization overhead
- This shows the name of the sstable that the memtable was written to on disk
Note the times on the messages
Time between first and second messages is how long the flush waited in the queue
Time between second and third messages is how long the flush took to complete
Sstables are immutable; updates go into new sstables
Reads scan over multiple sstables to stitch together a row
SSTables must eventually be compacted together to keep reads fast
Size-tiered compactions occur when a sufficient number of similarly sized sstables exist
In leveled compaction, sstables are written to Level 0 and moved to higher levels as they are compacted
Leveled compactions occur continuously as long as sstables exist in Level 0
Note the CompactionExecutor thread doing the compaction
The thread ID can be used to link together the messages
The compaction is beginning
The compaction is complete
The sstables that are going to be compacted
How many tables were compacted
The name of the new sstable created by the compaction
The number of bytes in the original files
The number of bytes in the new file
The percentage of the original size after:
Updates were merged
Tombstones and expired TTLs were removed
The time the compaction took and the rate in MB/sec
The sum of the number of rows in each sstable
The number of unique rows across all compacted sstables
X:Y where Y rows were split across X sstables
X:Y where Y rows were split across X sstables
X:Y where Y rows were split across X sstables
During compaction, you may see one or more messages about partitions being compacted incrementally
Logical CQL 3 rows sharing the same partition key form a physical row when stored in the cluster
Newer versions of Cassandra say partition instead of row
Large partitions can cause a number of problems for Cassandra
Uneven distribution of data between nodes
Large memory usage when large partitions are read all at once
Generating lots of garbage compacting a wide row
Slower compactions because they’re done incrementally on disk
These messages can help identify large rows
The keyspace, table, and partition key
The size of the partition
Garbage collections are a necessary evil
Some garbage collections run concurrently with Cassandra, but others stop the world
Cassandra logs any stop-the-world collections that last longer than 200ms
Stop the world collections cause nodes to stop responding to gossip and client requests
This shows the number of ms elapsed for the collection
Long collections increase latency of read and write requests
Very long collections will prevent the node from gossiping and other nodes will think it’s down
Note the time on each GCInspector message.
Even if the individual collections are fast, too many collections within a short timespan can hurt throughput
This shows the amount of data in each generation before and after the collection
Java divides the heap into different generations and uses different GC approaches on each
This is based on the observation that objects either tend to be short-lived or long-lived and different approaches work better in each scenario
The size of different spaces can be tweaked as well as the rules for moving objects between them
Java allocates new objects into the eden space
Eden objects that survive a single collection are promoted to the survivor space
Survivor objects that survive a specified number of collections get further promoted to the tenured generation
This shows the type of collection that occurred
ParNew collections occur when the young gen is collected. These are stop-the-world.
The young gen is usually small so ParNew is usually fast
If there’s not enough contiguous space in the old gen to promote an object, the old gen must be compacted, which takes a long time
ParNew collections occur when the young gen is collected. These are stop-the-world.
The young gen is usually small so ParNew is usually fast
If there’s not enough contiguous space in the old gen to promote an object, the old gen must be compacted, which takes a long time
ConcurrentMarkSweep normally runs concurrently with the application and does not stop the world
If the concurrent collection can’t keep up with the rate at which garbage is generated, a stop-the-world collection occurs
Stop-the-world CMS can take a very long time because the old gen is usually big
G1 is a newer garbage collector that can optionally be used with Cassandra
G1 divides the heap into multiple young and tenured regions and collects different regions independently
Recent tests have shown that G1 provides lower latency and throughput with less tuning than the older GC options
G1 will be the default garbage collector starting in Cassandra 3.0.
Flapping is often caused by garbage collections
The nodes go up-down-up-down, repeatedly
That’s why it’s called flapping
Notice the nodes that are flapping
If a single node is flapping, check the logs on that node during the same timeframe and see if GC is occurring
If multiple nodes are are reported up and down, the local node may be the problem
If a node is doing GC it won’t be able to receive gossip messages from another node and may think they’re down
Check for GC messages in the local log around the time that flapping occurs
Note the time that flapping occurs
If it happens infrequently, it may not be a problem
If it happens multiple times a minute, it is a problem
Check other node’s logs for GC events that occurred at the same time
If you don’t see anything in the other node’s log, it may be a network issue
You can reduce the failure detector’s sensitivity by increasing phi_convict_threshold in cassandra.yaml
Default value is 8; maximum recommended value is 12 (useful on high latency networks such as AWS)
If a write comes in for a down node, the coordinator will store hints for it
When the node comes back up, the nodes that have hints for it will send them
Hints are no longer stored after the node has been down over period of time specified in cassandra.yaml.
This is to prevent a node that comes back up from being inundated with more hints than it can handle
Any node that has been down longer than this period of time needs to run nodetool repair
Flapping can cause excessive hint buildup, which adds extra burden for both the coordinator and the node that is flapping
This can lead to cascading failures
Repairs are initiated by running nodetool repair
They do a full comparison of all the data for a particular token range with the other replicas for that range, then exchange any data that is out of sync
system.log shows the process from start to finish
Note the UUID for the repair session. This is your key to correlating the various messages.
When a repair session begins, you will see a “new session” message
It will report the nodes it’s going to sync with
The token ranges it’s going to sync
And the keyspace and column families it’s going to sync
The first step is for the repair leader to request merkle trees from all the other replicas
A message is logged reporting the receipt of each requested merkle tree
Make sure you see a message that the merkle tree was received from each node that it was requested from
After comparing the merkle trees, if the nodes are in sync, you’ll see a message like this
If not, you’ll see a message like this, reporting how many ranges are out of sync
The node will then begin a streaming repair with the out-of-sync replica
Another message reports when the streaming task has succeeded
A message will report when each table has been fully synced. This means either it was in sync to begin with, or all the streaming tasks necessary to sync it completed.
Once all tables are synced, a message will report that the overall repair session completed successfully.
If you see a “new session” message for a particular ID but not a “session completed successfully”, the repair is still running.
If a repair doesn’t complete successfully after some time, you should look more closely at the other messages for that session to see where it might be stuck.
Sometimes network issues can disrupt the streaming of data or a merkle tree, causing repair to hang
Other times, there is simply a lot of data, and building merkle trees can take a long time, as can streaming data
Increasing compaction throughput and streaming throughput will help speed the process, at the cost of using extra I/O and network bandwidth.
Check the other nodes involved in the repair for messages using the same session ID
Check for any errors that would have disrupted the repair
Before we end, I just want to go back to the troubleshooting process I discussed at the beginning
Next time you have a problem, think about the tools at your disposal and how they can help you with these steps