This document provides best practices for deploying Cassandra from tarball to production, covering topics like monitoring, configuration, backups, hardware selection, and more. It recommends using a Chef cookbook to automate the installation and configuration of Cassandra for consistency. While the cookbook covers many areas, there are additional steps discussed like setting the right snitch, adding racks, time synchronization, repair scheduling, authentication, and log consolidation.
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
AWS Webcast - Achieving consistent high performance with Postgres on Amazon W...Amazon Web Services
Postgres is a popular relational database and is the backend of a number of high traffic applications. Join AWS and PalominoDB, the company that helped Obama for America campaign optimize the database infrastructure on AWS, to learn about how you can run high throughput, I/O intensive Postgres clusters on the Amazon EBS storage platform. We will go over best practices including performance, durability and optimization related to deploying Postgres on AWS.
You hear about the best practices learned and applied for the Obama for America campaign.
In this webinar, you will learn about:
- Amazon Elastic Block Store (EBS)
- Why Provisioned IOPS volumes fit the needs of high I/O intensive applications
- Best practices for deploying Postgres on AWS
- How to leverage Provisioned IOPS volumes for Postgres
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.
Netflix tunes Amazon EC2 instances for maximum performance. In this session, you learn how Netflix configures the fastest possible EC2 instances, while reducing latency outliers. This session explores the various Xen modes (e.g., HVM, PV, etc.) and how they are optimized for different workloads. Hear how Netflix chooses Linux kernel versions based on desired performance characteristics and receive a firsthand look at how they set kernel tunables, including hugepages. You also hear about Netflix's use of SR-IOV to enable enhanced networking and their approach to observability, which can exonerate EC2 issues and direct attention back to application performance.
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per SecondAmazon Web Services
With the introduction of Amazon Elastic Block Store (EBS) GP2 and recent stability improvements, EBS has gained credibility in the Cassandra world for high performance workloads. By running Cassandra on Amazon EBS, you can run denser, cheaper Cassandra clusters with just as much availability as ephemeral storage instances. This talk walks through a highly detailed use case and configuration guide for a multi PetaByte, million write per second cluster that needs to be high performing and cost efficient. We explore the instance type choices, configuration, and low-level tuning that allowed us to hit 1.3 million writes per second with a replication factor of 3 on just 60 nodes.
Understanding Elastic Block Store Availability and PerformanceAmazon Web Services
Depending on your application needs, Elastic Block Store’s volumes can be configured for optimal performance and higher availability. In this session, we will present the different design characteristics of EBS Standard and Provisioned IOPS volumes, provide technical insights on how to think about EBS performance and availability, and share best practices to achieve higher availability and performance.
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
AWS Webcast - Achieving consistent high performance with Postgres on Amazon W...Amazon Web Services
Postgres is a popular relational database and is the backend of a number of high traffic applications. Join AWS and PalominoDB, the company that helped Obama for America campaign optimize the database infrastructure on AWS, to learn about how you can run high throughput, I/O intensive Postgres clusters on the Amazon EBS storage platform. We will go over best practices including performance, durability and optimization related to deploying Postgres on AWS.
You hear about the best practices learned and applied for the Obama for America campaign.
In this webinar, you will learn about:
- Amazon Elastic Block Store (EBS)
- Why Provisioned IOPS volumes fit the needs of high I/O intensive applications
- Best practices for deploying Postgres on AWS
- How to leverage Provisioned IOPS volumes for Postgres
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.
Netflix tunes Amazon EC2 instances for maximum performance. In this session, you learn how Netflix configures the fastest possible EC2 instances, while reducing latency outliers. This session explores the various Xen modes (e.g., HVM, PV, etc.) and how they are optimized for different workloads. Hear how Netflix chooses Linux kernel versions based on desired performance characteristics and receive a firsthand look at how they set kernel tunables, including hugepages. You also hear about Netflix's use of SR-IOV to enable enhanced networking and their approach to observability, which can exonerate EC2 issues and direct attention back to application performance.
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per SecondAmazon Web Services
With the introduction of Amazon Elastic Block Store (EBS) GP2 and recent stability improvements, EBS has gained credibility in the Cassandra world for high performance workloads. By running Cassandra on Amazon EBS, you can run denser, cheaper Cassandra clusters with just as much availability as ephemeral storage instances. This talk walks through a highly detailed use case and configuration guide for a multi PetaByte, million write per second cluster that needs to be high performing and cost efficient. We explore the instance type choices, configuration, and low-level tuning that allowed us to hit 1.3 million writes per second with a replication factor of 3 on just 60 nodes.
Understanding Elastic Block Store Availability and PerformanceAmazon Web Services
Depending on your application needs, Elastic Block Store’s volumes can be configured for optimal performance and higher availability. In this session, we will present the different design characteristics of EBS Standard and Provisioned IOPS volumes, provide technical insights on how to think about EBS performance and availability, and share best practices to achieve higher availability and performance.
Pulsar is used by a portfolio of products at Splunk for stream processing of different types of data, including metrics and logs. In this talk, Karthik Ramasamy will share how Splunk helped a flagship customer scale a Pulsar deployment to handle 10 PB/day in a single cluster. He will talk about the journey, the challenges faced, and the trade-offs made to scale Pulsar and operate it reliably and stably in Google Cloud Platform (GCP).
NoSQL data stores have extreme requirements for throughput, scalability, and management. Overview of the advantages of the new OSv cloud OS for NoSQL applications.
Presented to eRum (Budapest), May 2018
There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe the doAzureParallel package, a backend to the "foreach" package that automates the process of spawning a cluster of virtual machines in the Azure cloud to process iterations in parallel. This will include an example of optimizing hyperparameters for a predictive model using the "caret" package.
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss the performance implications of our new larger and faster SSD volumes (up to 16 TB with increased max throughput levels), as well as Amazon EBS encryption. Throughout, we share tips for success.
Amazon Aurora is a MySQL-compatible relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. Amazon Aurora is disruptive technology in the database space, bringing a new architectural model and distributed systems techniques to provide far higher performance, availability and durability than previously available using conventional monolithic database techniques. In this session, we will do a deep-dive into some of the key innovations behind Amazon Aurora, discuss best practices and configurations, and share early customer experience from the field.
This talk was given during DockerCon EU 2018.
It ain't just a whim - to be able to continue innovating, we’ve moved our good old static production to containers. We needed to be elastic, fast, reliable and production ready at any time - that's why we chose Docker. But like in most enterprises, lots of our apps run on the JVM and most JVMs’ ergonomics assume they “own” the server they are running on. So how do you containerize JVM apps? Should you really increase JVM heap if you have spare memory? What about OS caches? What are the differences between JDK 8, 9 and 10 when it comes to container awareness? Outages because of out of memory errors? Slowness because of long garbage collection and poor environment visibility? Long story short, in this session, we’ll look at the gotchas of running JVM apps in containers and teach you how to avoid costly mistakes.
Top 3 things attendees will learn:
1. Key differences between various JVM versions relevant for containerized Java apps.
2. Best practices for running JVM in containers.
3. Avoiding common pitfalls when running containerized JVM applications.
Tarantool: как сэкономить миллион долларов на базе данных на высоконагруженно...Ontico
Когда стоит дилемма, какое DBMS решение выбрать, то приходится принимать во внимание много факторов — latency, bandwidth, ACID-complience, наличие/отсутствие server-side-scripting, возможности репликации, удобство развертывания и администрирования, наличие известных багов или maintenance window и т.д.
Я хочу рассказать лишь об одном из факторов, который имеет особенное значение на проектах с многомиллионными аудиториями — это Total Cost of Ownership или, по-простому, цена. Чем больше аудитория у проекта, тем больше эта аудитория создает нагрузку на базы данных, тем больше должно быть серверов с базами данных, тем больше финансовых затрат это требует.
Можно экстенсивно наращивать количество серверов, но до определенного предела, когда становится понятным, что далее дешевле будет внедрить новое, более производительное решение, которое позволит радикально снизить цену и количество железа.
Мой рассказ будет посвящен тому, как мы в Почте@Mail.Ru перешли на Tarantool, и как его использование сэкономило нам миллион долларов.
OSv is a new, high-performance OS for virtual machines in the cloud. Designed to run one application per guest with minimal overhead, OSv eliminates important bottlenecks for NoSQL applications through improvements in memory management, network I/O, and scheduling. And many important bottlenecks for NoSQL applications are tunable on a conventional OS, but do not require tuning in the OSv environment.
OSv is fully stateless and can be configured at runtime with cloud-init or through a REST API, with zero configuration files. OSv offers unified tracing from the application layer through the JVM and the OS kernel. Attendees will learn how to boot Cassandra in one second, and create a simple cluster in a minute.
AWS - an introduction to bursting (GP2 - T2)Rasmus Ekman
An introduction to bursting on AWS. The presentation includes an introductory explanation of the new EC2 family, T2, and the new EBS volume type GP2.
Learn about the new services on AWS to get the most bang for your buck.
This talk was given during Lucene Revolution 2017 and has two goals: first, to discuss the tradeoffs for running Solr on Docker. For example, you get dynamic allocation of operating system caches, but you also get some CPU overhead. We'll keep in mind that Solr nodes tend to be different than your average container: Solr is usually long running, takes quite some RSS and a lot of virtual memory. This will imply, for example, that it makes more sense to use Docker on big physical boxes than on configurable-size VMs (like Amazon EC2).
The second goal is to discuss issues with deploying Solr on Docker and how to work around them. For example, many older (and some of the newer) combinations of Docker, Linux Kernel and JVM have memory leaks. We'll go over Docker operations best practices, such as using container limits to cap memory usage and prevent the host OOM killer from terminating a memory-consuming process - usually a Solr node. Or running Docker in Swarm mode over multiple smaller boxes to limit the spread of a single issue.
Pulsar is used by a portfolio of products at Splunk for stream processing of different types of data, including metrics and logs. In this talk, Karthik Ramasamy will share how Splunk helped a flagship customer scale a Pulsar deployment to handle 10 PB/day in a single cluster. He will talk about the journey, the challenges faced, and the trade-offs made to scale Pulsar and operate it reliably and stably in Google Cloud Platform (GCP).
NoSQL data stores have extreme requirements for throughput, scalability, and management. Overview of the advantages of the new OSv cloud OS for NoSQL applications.
Presented to eRum (Budapest), May 2018
There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe the doAzureParallel package, a backend to the "foreach" package that automates the process of spawning a cluster of virtual machines in the Azure cloud to process iterations in parallel. This will include an example of optimizing hyperparameters for a predictive model using the "caret" package.
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss the performance implications of our new larger and faster SSD volumes (up to 16 TB with increased max throughput levels), as well as Amazon EBS encryption. Throughout, we share tips for success.
Amazon Aurora is a MySQL-compatible relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. Amazon Aurora is disruptive technology in the database space, bringing a new architectural model and distributed systems techniques to provide far higher performance, availability and durability than previously available using conventional monolithic database techniques. In this session, we will do a deep-dive into some of the key innovations behind Amazon Aurora, discuss best practices and configurations, and share early customer experience from the field.
This talk was given during DockerCon EU 2018.
It ain't just a whim - to be able to continue innovating, we’ve moved our good old static production to containers. We needed to be elastic, fast, reliable and production ready at any time - that's why we chose Docker. But like in most enterprises, lots of our apps run on the JVM and most JVMs’ ergonomics assume they “own” the server they are running on. So how do you containerize JVM apps? Should you really increase JVM heap if you have spare memory? What about OS caches? What are the differences between JDK 8, 9 and 10 when it comes to container awareness? Outages because of out of memory errors? Slowness because of long garbage collection and poor environment visibility? Long story short, in this session, we’ll look at the gotchas of running JVM apps in containers and teach you how to avoid costly mistakes.
Top 3 things attendees will learn:
1. Key differences between various JVM versions relevant for containerized Java apps.
2. Best practices for running JVM in containers.
3. Avoiding common pitfalls when running containerized JVM applications.
Tarantool: как сэкономить миллион долларов на базе данных на высоконагруженно...Ontico
Когда стоит дилемма, какое DBMS решение выбрать, то приходится принимать во внимание много факторов — latency, bandwidth, ACID-complience, наличие/отсутствие server-side-scripting, возможности репликации, удобство развертывания и администрирования, наличие известных багов или maintenance window и т.д.
Я хочу рассказать лишь об одном из факторов, который имеет особенное значение на проектах с многомиллионными аудиториями — это Total Cost of Ownership или, по-простому, цена. Чем больше аудитория у проекта, тем больше эта аудитория создает нагрузку на базы данных, тем больше должно быть серверов с базами данных, тем больше финансовых затрат это требует.
Можно экстенсивно наращивать количество серверов, но до определенного предела, когда становится понятным, что далее дешевле будет внедрить новое, более производительное решение, которое позволит радикально снизить цену и количество железа.
Мой рассказ будет посвящен тому, как мы в Почте@Mail.Ru перешли на Tarantool, и как его использование сэкономило нам миллион долларов.
OSv is a new, high-performance OS for virtual machines in the cloud. Designed to run one application per guest with minimal overhead, OSv eliminates important bottlenecks for NoSQL applications through improvements in memory management, network I/O, and scheduling. And many important bottlenecks for NoSQL applications are tunable on a conventional OS, but do not require tuning in the OSv environment.
OSv is fully stateless and can be configured at runtime with cloud-init or through a REST API, with zero configuration files. OSv offers unified tracing from the application layer through the JVM and the OS kernel. Attendees will learn how to boot Cassandra in one second, and create a simple cluster in a minute.
AWS - an introduction to bursting (GP2 - T2)Rasmus Ekman
An introduction to bursting on AWS. The presentation includes an introductory explanation of the new EC2 family, T2, and the new EBS volume type GP2.
Learn about the new services on AWS to get the most bang for your buck.
This talk was given during Lucene Revolution 2017 and has two goals: first, to discuss the tradeoffs for running Solr on Docker. For example, you get dynamic allocation of operating system caches, but you also get some CPU overhead. We'll keep in mind that Solr nodes tend to be different than your average container: Solr is usually long running, takes quite some RSS and a lot of virtual memory. This will imply, for example, that it makes more sense to use Docker on big physical boxes than on configurable-size VMs (like Amazon EC2).
The second goal is to discuss issues with deploying Solr on Docker and how to work around them. For example, many older (and some of the newer) combinations of Docker, Linux Kernel and JVM have memory leaks. We'll go over Docker operations best practices, such as using container limits to cap memory usage and prevent the host OOM killer from terminating a memory-consuming process - usually a Solr node. Or running Docker in Swarm mode over multiple smaller boxes to limit the spread of a single issue.
Cassandra SF 2015 - Repeatable, Scalable, Reliable, Observable Cassandraaaronmorton
Slides from my talk at Cassandra Summit 2015
http://cassandrasummit-datastax.com/agenda/repeatable-scalable-reliable-observable-cassandra/
thelastpickle.com
Hardening cassandra for compliance or paranoiazznate
How to secure a cassandra cluster. Includes details on configuring SSL, setting up a certificate authority and creating certificates and trust chains for the JVM.
Case Study: Troubleshooting Cassandra performance issues as a developerCarlos Alonso Pérez
This talk will be a step by step walkthrough of a developer troubleshooting a real performance issue we had at MyDrive, from the very first steps diagnosing the symptoms, through looking at metric charts down to CQL queries, the Ruby CQL driver, and Ruby code profiling.
The Log-structured Merge-Tree storage engine in Apache Cassandra allows for fast write performance, but has some potential downsides when it comes to deleting data. Expired columns (from TTL's) and Tombstones can impact on read performance until they are purged from disk by Compaction. And while different Compaction Strategies are suited to different workloads, they must all ensure deleted data stays deleted.
Cassandra Summit 2015: Real World DTCS For OperatorsJeff Jirsa
Real World DTCS For Operators
The introduction of DateTieredCompactionStrategy in late 2014 was a significant step forward in providing a viable compaction strategy for time series data, especially time series data that will be TTL'd out. DateTieredCompactionStrategy's introduction was met with genuine excitement, and its rapid adoption is testament to developers' and operators' desire to have data compacted in a way that better matches their write patterns.
However, DateTieredCompactionStrategy's features come with significant limitations. This talk will review our real world benchmarking and use cases for DTCS as a vehicle to discuss the implications of DateTieredCompactionStrategy on operational tasks such as repair, read-repair, bootstrapping, and especially DR recovery scenarios, and it will also discuss how those various limitations lead us to proposing an operations-friendly alternative to DateTieredCompactionStrategy.
RDS for MySQL, No BS Operations and PatternsLaine Campbell
Amazon's RDS for MySQL is a wonderful tool with a significant value. It can also create a lot of havoc if you are not aware of it's limitations and changes before you make it a core part of your environment. In this deck, we discuss those issues.
Some vignettes and advice based on prior experience with Cassandra clusters in live environments. Includes some material from other operational slides.
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Виталий Стародубцев
##Что такое Storage Replica
##Архитектура и сценарии
##Синхронная и асинхронная репликация
##Междисковая, межсерверная, внутрикластерная и межкластерная репликация
##Дизайн и проектирование Storage Replica
##Нововведения в Windows Server 2016 TP5
##Графический интерфейс управления, и другие возможности - демонстрация и планы развития
##Интеграция Storage Replica с Storage Spaces Direct
Laine Campbell, CEO of Blackbird, will explain the options for running MySQL at high volumes at Amazon Web Services, exploring options around database as a service, hosted instances/storages and all appropriate availability, performance and provisioning considerations using real-world examples from Call of Duty, Obama for America and many more. Laine will show how to build highly available, manageable and performant MySQL environments that scale in AWS—how to maintain then, grow them and deal with failure. Some of the specific topics covered are:
* Overview of RDS and EC2 – pros, cons and usage patterns/antipatterns.
* Implementation choices in both offerings: instance sizing, ephemeral SSDs, EBS, provisioned IOPS and advanced techniques (RAID, mixed storage environments, etc…)
* Leveraging regions and availability zones for availability, business continuity and disaster recovery.
* Scaling patterns including read/write splitting, read distribution, functional dataset partitioning and horizontal dataset partitioning (aka sharding)
* Common failure modes – AZ and Region failures, EBS corruption, EBS performance inconsistencies and more.
* Managing and mitigating cost with various instance and storage options
SRV402 Deep Dive on Amazon EC2 Instances, Featuring Performance Optimization ...Amazon Web Services
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and Accelerated Computing (GPU and FPGA) instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
The DrupalCampLA 2011 presentation on backend performance. The slides go over optimizations that can be done through the LAMP (or now VAN LAMMP stack for even more performance) to get everything up and running.
Amazon Aurora is a cloud-optimized relational database that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. The recently announced PostgreSQL-compatibility, together with the original MySQL compatibility, are perfect for new application development and for migrations from overpriced, restrictive commercial databases. In this session, we’ll do a deep dive into the new architectural model and distributed systems techniques behind Amazon Aurora, discuss best practices and configurations, look at migration options and share customer experience from the field.
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...Amazon Web Services
Amazon Elastic Compute Cloud (Amazon EC2) provides a broad selection of instance types to accommodate a diverse mix of workloads. In this technical session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, and Memory Optimized families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Learning Objectives: • Understand the differences between instances • Learn best practices and tips for getting the most out of EC2 instances
Amazon EC2 provides a broad selection of instance types to deliver high performance for a diverse mix of applications. In this session, we overview the drivers of system performance and discuss in depth how Amazon EC2 instances deliver system performance while also providing elasticity and complete control over your infrastructure. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
SRV402 Deep Dive on Amazon EC2 Instances, Featuring Performance Optimization ...Amazon Web Services
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and Accelerated Computing (GPU and FPGA) instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Deep Dive on Amazon EC2 Instances - AWS Summit Cape Town 2017Amazon Web Services
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and Accelerated Computing (GPU and FPGA) instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
AWS Speaker: Ian Massingham, Sr Mgr, Technical Evangelist - Amazon Web Services
Customer Speaker: Andrew Mori, Konga, Technical Director
by Joyjeet Banerjee, Enterprise Solutions Architect, AWS
Amazon Aurora is a MySQL- and PostgreSQL-compatible database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. In this deep dive session, we’ll discuss best practices and explore new features in areas like high availability, security, performance management and database cloning. Level 300
Crimson: Ceph for the Age of NVMe and Persistent MemoryScyllaDB
Ceph is a mature open source software-defined storage solution that was created over a decade ago.
During that time new faster storage technologies have emerged including NVMe and Persistent memory.
The crimson project aim is to create a better Ceph OSD that is more well suited to those faster devices. The crimson OSD is built on the Seastar C++ framework and can leverage these devices by minimizing latency, cpu overhead, and cross-core communication. This talk will discuss the project design, our current status, and our future plans.
Those who out-compute can many times out-compete. The cloud gives you access to a massive amount of compute power when you need it. This talk will present an introduction to HPC in the cloud, including, the benefits of HPC in the cloud, how to get started, some tools to use, and how you can manage data. We will showcase several examples of HPC in the cloud by a number of public sector and commercial customers.
Created by: Dr. Jeff Layton, Principal, Solutions Architect
Amazon EC2 provides a broad selection of instance types to deliver high performance for a diverse mix of applications. In this session, we overview the drivers of system performance and discuss in depth how Amazon EC2 instances deliver system performance while also providing elasticity and complete control over your infrastructure. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Similar to Cassandra from tarball to production (20)
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
2. Why talk about this?
You are about to deploy Cassandra
You are looking for “best practices”
You don’t want:
... to scour through the documentation
... to do something known not to work well
... to forget to cover some important step
3. What we won’t cover
● Cassandra: how does
it work?
● How do I design my
schema?
● What’s new in
Cassandra X.Y?
4. So many things to do
Monitoring Snitch DC/Rack Settings Time Sync
Seeds/Autoscaling Full/Incremental
Backups
AWS Instance
Selection
Disk - SSD?
Disk Space - 2x? AWS AMI (Image)
Selection
Periodic Repairs Replication Strategy
Compaction
Strategy
SSL/VPC/VPN Authorization +
Authentication
OS Conf - Users
OS Conf - Limits OS Conf - Perms OS Conf - FSType OS Conf - Logs
C* Start/Stop OS Conf - Path Use case evaluation
5. Chef to the rescue?
Chef community cookbook available
https://github.com/michaelklishin/cassandra-chef-cookbook
Installs java Creates a “cassandra” user/group
Download/extract the tarball Fixes up ownership
Builds the C* configuration files
Sets the ulimits for filehandles, processes,
memory locking
Sets up an init script Sets up data directories
6. Chef Cookbook Coverage
Monitoring Snitch DC/Rack Settings Time Sync
Seeds/Autoscaling Full/Incremental
Backups
Disk - SSD? Disk - How much?
AWS Instance Type AWS AMI (Image)
Selection
Periodic Repairs Replication Strategy
Compaction
Strategy
SSL/VPC/VPN Authorization +
Authentication
OS Conf - Users
OS Conf - Limits OS Conf - Perms OS Conf - FSType OS Conf - Logs
C* Start/Stop OS Conf - Path Use case evaluation
7. Monitoring
Is every node answering queries?
Are nodes talking to each other?
Are any nodes running slowly?
Push UDP! (statsd)
http://hackers.lookout.com/2015/01/cassandra-monitoring/
https://github.com/lookout/cassandra-statsd-agent
8. Monitoring - Synthetic
Health checks, bad and good
● ‘nodetool status’ exit code
○ Might return 0 if the node is not accepting requests
○ Slow, cross node reads
● cqlsh -u sysmon -p password < /dev/null
● Verifies this node can read auth table
● https://github.com/lookout/cassandra-health-check
9. What about OpsCenter?
We chose not to use it
Want consistent interface for all monitoring
GUI vs Command Line argument
Didn’t see good auditing capabilities
Didn’t interface well with our chef solution
10. Snitch
Use the right snitch!
● AWS EC2MultiRegionSnitch
● Google? GoogleCloudSnitch
● GossipingPropertyFileSnitch
NOT
● SimpleSnitch (default)
Community cookbook: set it!
11. What is RF?
Replication Factor is how many copies of data
Value is hashed to determine primary host
Additional copies always next node
Hash here
12. What is CL?
Consistency Level -- It’s not RF!
Describes how many nodes must respond
before operation is considered COMPLETE
CL_ONE - only one node responds
CL_QUORUM - (RF/2)+1 nodes (round down)
CL_ALL - RF nodes respond
13. DC/Rack Settings
You might need to set these
Maybe you’re not in Amazon
Rack == Availability Zone?
Hard: Renaming DC or adding racks
14. Renaming DCs
Clients “remember” which DC they talk to
Renaming single DC causes all clients to fail
Better to spin up a new one than rename old
15. Adding a rack
Start with 6 node cluster, rack R1
Replication factor 3
Add 1 node in R2, and rebalance
ALL data in R2 node?
Good idea to keep racks balanced
16. I don’t have time for this
Clusters must have synchronized time
You will get lots of drift with: [0-3].amazon.pool.
ntp.org
Community cookbook doesn’t cover anything
here
17. Better make time for this
C* serializes write operations by time stamps
Clocks on virtual machines drift!
It’s the relative difference among clocks that matters
C* nodes should synchronize with each other
Solution: use a pair of peered NTP servers (level 2 or 3)
and a small set of known upstream providers
18. From a small seed…
Seeds are used for new nodes to find cluster
Every new node should use the same seeds
Seed nodes get topology changes faster
Each seed node must be in the config file
Multiple seeds per datacenter recommended
Tricky to configure on AWS
19. Backups - Full+Incremental
Nothing in the cookbooks for this
C* makes it “easy”: snapshot, then copy
Snapshots might require a lot more space
Remove the snapshot after copying it
20. Disk selection
SSD Rotational
Ephemeral
EBS
Low latency Any size instance Any size instance
Recommended Not cheap Less expensive
Great random r/w perf Good write performance No node rebuilds
No network use for disk No network use for disk
21. AWS Instance Selection
We moved to EC2
c3.2xlarge (15GiB mem, Disk 160GB)?
i2.xlarge (30GiB mem, 800GB disk)
Max recommended storage per node is 1TB
Use instance types that support HVM
Some previous generation instance types, such as T1, C1, M1, and M2 do not support Linux HVM AMIs. Some current generation instance
types, such as T2, I2, R3, G2, and C4 do not support PV AMIs.
22. How much can I use??
Snapshots take space (kind of)
Best practice: keep disks half full!
800GB disk becomes 400GB
Snapshots during repairs?
Lots of uses for snapshots!
23. Periodic Repairs
Buried in the docs:
“As a best practice, you should
schedule repairs weekly”
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html
● “-pr” (yes)
● “-par” (maybe)
● “--in-local-dc” (no)
24. Repair Tips
Raise gc_grace_seconds (tombstones?)
Run on one node at a time
Schedule for low usage hours
Use “par” if you have dead time (faster)
Tune with: nodetool setcompactionthroughput
25. I thought I deleted that
Compaction removes “old” tombstones
10 day default grace period (gc_grace_period)
After that, deletes will not be propagated!
Run ‘nodetool repair’ at least every 10 days
Once a week is perfect (3 day grace)
Node down >7 days? ‘nodetool remove’ it!
26. Changing RF within DC?
Easy to decrease RF
Impossible to increase RF without (usually)
Reads with CL_ONE might fail!
Hash here
27. Replication Strategy
How many replicas should we have?
What happens if some data is lost?
Are you write-heavy or read-heavy?
Quorum considerations: odd is better!
RF=1? RF=3? RF=5?
28. Magic JMX setting: reduce traffic to a node
Great when node is “behind” the 4 hour window
Used by gossiper to divert traffic during repairs
Writes: ok, read repair: ok, nodetool repair: ok
$ java -jar jmxterm.jar -l localhost:7199
$> set -b org.apache.cassandra.db:type=DynamicEndpointSnitch Severity
10000
Don’t be too severe!
29. Compaction Strategy
Solved by using a good C* design
SizeTiered or Leveled?
Leveled has better guarantees for read times
SizeTiered may require 10 (or more) reads!
Leveled uses less disk space
Leveled tombstone collection is slower
30. Auth*
Cookbooks default to OFF
Turn authenticator and authorizer on
‘cassandra’ user is super special
Requires QUORUM (cross-DC) for signon
LOCAL_ONE for all other users!
31. Users
OS users vs Cassandra users: 1 to 1?
Shared credentials for apps?
Nothing logs the user taking the action!
‘cassandra’ user is created by cookbook
All processes run as ‘cassandra’
34. Logs
To consolidate or not to consolidate?
Push or pull? Usually push!
FOSS: syslogd, syslog-ng, logstash/kibana,
heka, banana
Others: Splunk, SumoLogic, Loggly, Stackify
35. Shutdown
Nice init script with cookbook, steps are:
● nodetool disablethrift (no more clients)
● nodetool disablegossip (stop talking to
cluster)
● nodetool drain (flush all memtables)
● kill the jvm
36. Quick performance wins
● Disable assertions - cookbook property
● No swap space (or vm.swappiness=1)
● max_concurrent_reads
● max_concurrent_writes