opentsdb in a real enviroment
•
3 likes•11,597 views
This document discusses the architecture of OpenTSDB and provides instructions for setting up and using OpenTSDB to collect CPU metrics from a host. It describes: 1) The main components of OpenTSDB including the TSD, HBase, and Hadoop. 2) How to start and stop OpenTSDB by starting/stopping TSD, HBase, and Hadoop. 3) A sample scenario of collecting CPU load average metrics from a host and inserting them into OpenTSDB using a script to post metrics to the TSD socket.
Report
Share
Report
Share
Recommended
HBaseCon 2013: OpenTSDB at Box
This document discusses Box's use of OpenTSDB to store and query time series metrics data. It describes how OpenTSDB provides a scalable and easy way to collect, store, and query large amounts of metrics data compared to previous solutions. It includes examples of using OpenTSDB, such as a script to collect MySQL metrics and adding it as a cron job, and examples of querying the data through the OpenTSDB API and web interface. It also provides some statistics about Box's OpenTSDB deployment and next steps.
Monitoring MySQL with OpenTSDB
A monitoring system is arguably the most crucial system to have in place when administering and tweaking the performance of any database system. DBAs also find themselves with a variety of monitoring systems and plugins to use; ranging from small scripts in cron to complex data collection systems. In this talk, I’ll discuss how Box made a shift from the Cacti monitoring system and other various shell scripts to OpenTSDB and the changes made to our servers and daily interaction with monitoring to increase our agility in identifying and addressing changes in database behavior.
To Hire, or to train, that is the question (Percona Live 2014)
"We're hiring!"
How many times have you heard this phrase at a conference? Every database-driven company is hiring and that makes for pretty stiff competition when trying to get a new DBA. Instead of searching for the perfect database administrator from a conference or Linkedin, why not look internally at your organization for system administrators or engineers who may be an equally good fit given the right training.
In this talk, I'll explain how the DBAs at Box developed a knowledge-sharing culture around databases and disseminated important learnings to other members of the company. I'll also cover the mentorship process we established to train other members of our Operations team to become rock star DBAs and manage our MySQL and HBase infrastructure at Box.
OpenTSDB for monitoring @ Criteo
OpenTSDB is used at Criteo for monitoring their large Hadoop infrastructure which includes over 2500 servers running many different services like HDFS, YARN, HBase, Kafka, and Storm. OpenTSDB was chosen because it can handle the scale of metrics collected, store metrics for long periods of time with fine-grained resolution, and is easily extensible to add new metrics. It uses HBase for storage which is optimized for the time series data stored in OpenTSDB and can scale to meet Criteo's needs of storing billions of data points and handling high query loads.
OpenTSDB: HBaseCon2017
Another year, another talk about OpenTSDB running on HBase.
We'll discuss topics like:
Yahoo's append co-processor saving CPU resources by resolving atomic appends at compaction or query time.
The pros and cons of HBASE-15181, Date Tiered compaction for time series data.
Yahoo's experiments with unbounded secondary index on HBase.
OpenTSDB's 3.0 featuring a new query engine and API.
by Chris Larsen of Yahoo!
Gnocchi v3 brownbag
This document summarizes the author's experience optimizing Gnocchi, an open source time-series database, to store metrics for hundreds of thousands of resources over many months. The author describes improving performance by adding Ceph storage nodes, tuning Ceph configurations, minimizing I/O operations, and improving the storage format. Benchmark results show the new version achieves 50% higher write throughput, 40-60% faster computation times, 30-60% better overall performance, and 30-40% fewer operations. Usage hints are also provided to help optimize for different use cases.
Gnocchi v4 (preview)
early benchmarks on pre-release Gnocchi v4. includes benchmark comparison between all-ceph v3.x driver versus all-ceph v4 driver. also, shows benchmark using redis+ceph deployment.
Gnocchi Profiling 2.1.x
initial profile of Gnocchi 2.1.x, this shows performance using a basic configuration of Gnocchi with standard configuration of Ceph 10.2.1.
Recommended
HBaseCon 2013: OpenTSDB at Box
This document discusses Box's use of OpenTSDB to store and query time series metrics data. It describes how OpenTSDB provides a scalable and easy way to collect, store, and query large amounts of metrics data compared to previous solutions. It includes examples of using OpenTSDB, such as a script to collect MySQL metrics and adding it as a cron job, and examples of querying the data through the OpenTSDB API and web interface. It also provides some statistics about Box's OpenTSDB deployment and next steps.
Monitoring MySQL with OpenTSDB
A monitoring system is arguably the most crucial system to have in place when administering and tweaking the performance of any database system. DBAs also find themselves with a variety of monitoring systems and plugins to use; ranging from small scripts in cron to complex data collection systems. In this talk, I’ll discuss how Box made a shift from the Cacti monitoring system and other various shell scripts to OpenTSDB and the changes made to our servers and daily interaction with monitoring to increase our agility in identifying and addressing changes in database behavior.
To Hire, or to train, that is the question (Percona Live 2014)
"We're hiring!"
How many times have you heard this phrase at a conference? Every database-driven company is hiring and that makes for pretty stiff competition when trying to get a new DBA. Instead of searching for the perfect database administrator from a conference or Linkedin, why not look internally at your organization for system administrators or engineers who may be an equally good fit given the right training.
In this talk, I'll explain how the DBAs at Box developed a knowledge-sharing culture around databases and disseminated important learnings to other members of the company. I'll also cover the mentorship process we established to train other members of our Operations team to become rock star DBAs and manage our MySQL and HBase infrastructure at Box.
OpenTSDB for monitoring @ Criteo
OpenTSDB is used at Criteo for monitoring their large Hadoop infrastructure which includes over 2500 servers running many different services like HDFS, YARN, HBase, Kafka, and Storm. OpenTSDB was chosen because it can handle the scale of metrics collected, store metrics for long periods of time with fine-grained resolution, and is easily extensible to add new metrics. It uses HBase for storage which is optimized for the time series data stored in OpenTSDB and can scale to meet Criteo's needs of storing billions of data points and handling high query loads.
OpenTSDB: HBaseCon2017
Another year, another talk about OpenTSDB running on HBase.
We'll discuss topics like:
Yahoo's append co-processor saving CPU resources by resolving atomic appends at compaction or query time.
The pros and cons of HBASE-15181, Date Tiered compaction for time series data.
Yahoo's experiments with unbounded secondary index on HBase.
OpenTSDB's 3.0 featuring a new query engine and API.
by Chris Larsen of Yahoo!
Gnocchi v3 brownbag
This document summarizes the author's experience optimizing Gnocchi, an open source time-series database, to store metrics for hundreds of thousands of resources over many months. The author describes improving performance by adding Ceph storage nodes, tuning Ceph configurations, minimizing I/O operations, and improving the storage format. Benchmark results show the new version achieves 50% higher write throughput, 40-60% faster computation times, 30-60% better overall performance, and 30-40% fewer operations. Usage hints are also provided to help optimize for different use cases.
Gnocchi v4 (preview)
early benchmarks on pre-release Gnocchi v4. includes benchmark comparison between all-ceph v3.x driver versus all-ceph v4 driver. also, shows benchmark using redis+ceph deployment.
Gnocchi Profiling 2.1.x
initial profile of Gnocchi 2.1.x, this shows performance using a basic configuration of Gnocchi with standard configuration of Ceph 10.2.1.
Update on OpenTSDB and AsyncHBase
This document summarizes an update on OpenTSDB, an open source time series database. It discusses OpenTSDB's ability to store trillions of data points at scale using HBase, Cassandra, or Bigtable as backends. Use cases mentioned include systems monitoring, sensor data, and financial data. The document outlines writing and querying functionality and describes the data model and table schema. It also discusses new features in OpenTSDB 2.2 and 2.3 like downsampling, expressions, and data stores. Community projects using OpenTSDB are highlighted and the future of OpenTSDB is discussed.
OpenTSDB 2.0
Speakers: Chris Larsen (Limelight Networks) and Benoit Sigoure (Arista Networks)
The OpenTSDB community continues to grow and with users looking to store massive amounts of time-series data in a scalable manner. In this talk, we will discuss a number of use cases and best practices around naming schemas and HBase configuration. We will also review OpenTSDB 2.0's new features, including the HTTP API, plugins, annotations, millisecond support, and metadata, as well as what's next in the roadmap.
Gnocchi v3
The document summarizes optimisation opportunities and testing results for Gnocchi v3 compared to v2. It discusses improvements to coordination and scheduling to minimize contention, performance improvements for large datasets, a new storage format to reduce operations and disk size, and benchmark results showing processing time reductions of up to 23% and write throughput increases.
Gnocchi v4 - past and present
a dive into performance enhancements in Gnocchi v4.x (as of end of August) and how it compares to previous releases of Gnocchi.
"Metrics: Where and How", Vsevolod Polyakov
Abstract: Nowadays it’s only a lazy one who haven’t written his own metric storage and aggregation system. I am lazy, and that’s why I have to choose what to use and how to use. I don’t want you to do the same job, so I decided to share my considerations concerning architectures and test results.
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Запускаем сервер (БД, Web-сервер или что-то свое собственное) и не получаем желаемый RPS. Запускаем top и видим, что 100% выедается CPU. Что дальше, на что расходуется процессорное время? Можно ли подкрутить какие-то ручки, чтобы улучшить производительность? А если параметр CPU не высокий, то куда смотреть дальше?
Мы рассмотрим несколько сценариев проблем производительности, рассмотрим доступные инструменты анализа производительности и разберемся в методологии оптимизации производительности Linux, ответим на вопрос за какие ручки и как крутить.
Anatomy of an action
A key feature when monitoring and debugging any Cloud infrastructure is to provide the ability to trace, track, and collate all the individual, discrete steps that compose an event. A typical resource action in OpenStack is often a combination of smaller tasks -- which given the distributed nature of OpenStack -- can fail at unpredictable points in the workflow. By collecting the appropriate events, operators can view all events within Ceilometer, filter on a failed action and trace back the history of related events to spot anomalies or errors. In this talk, we provide an overview of the recent enhancements made in Ceilometer to support the collection of event notifications from OpenStack services. We will describe: how events are processed, transformed and stored in Ceilometer; how you can derive metrics from events; and how it’s possible to track the events of a resource and analyse where errors occur.
Tweaking performance on high-load projects
This document discusses optimizing performance for high-load projects. It summarizes the delivery loads and technologies used for several projects including mGage, mobclix and XXXX. It then discusses optimizations made to improve performance, including using Solr for search, Redis for real-time data, Hadoop for reporting, and various Java optimizations in moving to Java 7. Specific optimizations discussed include reducing garbage collection, improving random number generation, and minimizing I/O operations.
HBaseCon2017 gohbase: Pure Go HBase Client
gohbase is an implementation of an HBase client in pure Go: https://github.com/tsuna/gohbase. In this presentation we'll talk about its architecture and compare its performance against the native Java HBase client as well as AsyncHBase (http://opentsdb.github.io/asynchbase/) and some nice characteristics of golang that resulted in a simpler implementation.
Advanced Apache Cassandra Operations with JMX
Nodetool is a command line interface for managing a Cassandra node. It provides commands for node administration, cluster inspection, table operations and more. The nodetool info command displays node-specific information such as status, load, memory usage and cache details. The nodetool compactionstats command shows compaction status including active tasks and progress. The nodetool tablestats command displays statistics for a specific table including read/write counts, space usage, cache usage and latency.
Handling 20 billion requests a month
This document discusses the architecture and technical challenges of handling a large volume of requests for an online advertising platform. It summarizes three key projects handled by the platform that delivered 3 billion, 14 billion, and 20 billion requests per month respectively. It describes the technologies used, including Solr, Redis, MySQL, Hadoop and Amazon Web Services instances. It also outlines optimizations made to improve performance, such as data compression, query optimizations, and Java 7 improvements. The goal was to process over 11,000 requests per second on average while maintaining response times below 100ms.
10 things i wish i'd known before using spark in production
Vous avez récemment commencé à travailler sur Spark et vos jobs prennent une éternité pour se terminer ? Cette présentation est faite pour vous.
Himanshu Arora et Nitya Nand YADAV ont rassemblé de nombreuses bonnes pratiques, optimisations et ajustements qu'ils ont appliqué au fil des années en production pour rendre leurs jobs plus rapides et moins consommateurs de ressources.
Dans cette présentation, ils nous apprennent les techniques avancées d'optimisation de Spark, les formats de sérialisation des données, les formats de stockage, les optimisations hardware, contrôle sur la parallélisme, paramétrages de resource manager, meilleur data localité et l'optimisation du GC etc.
Ils nous font découvrir également l'utilisation appropriée de RDD, DataFrame et Dataset afin de bénéficier pleinement des optimisations internes apportées par Spark.
JVM performance options. How it works
This is presentation for Cogniance Java Evening 13.02.2013.
My related to topic article you can find here (russian) : http://habrahabr.ru/post/160049/
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
NoSQL key-value — популярное решение, но какие преимущества и какой ценой оно даёт?
Скорость? Возможно, но ценой урезанного, по сравнению с реляционными базами данных, функционала. Но данные и приложение всё еще разделены сетевым стеком, а иногда и десятками километров оптоволокна. В крупном проекте, работающем на десятках или сотнях серверов нельзя обеспечить высокую скорость доступа к данным с каждой машины. Если хранилище достаточно быстро, то время обработки запроса окажется значительно меньше затрат на работу с сетью, а производительность приложения будут определять сетевые задержки и частота запросов к БД.
В проекте Облако@Mail.Ru мы ушли от использования чистых key-value хранилищ в пользу микросервисов на Tarantool, что позволило свести общение с хранилищем данных к минимуму.
Да, Tarantool — это еще одна NoSQL база данных, но еще это полноценный сервер приложений. Приложений, расположенных рядом с данными!
Я расскажу, как мы пришли к использованию микросервисов на основе Tarantool. Приведу несколько сценариев использования, которые работают в Облаке и могут быть легко адаптированы для другого web-проекта. Вы узнаете о компонентах, которые разработаны и опубликованы нами уже сейчас, и о дальнейших планах развития.
ScyllaDB: NoSQL at Ludicrous Speed
ScyllaDB is a NoSQL database compatible with Apache Cassandra, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware.
Achieving such speed requires a great deal of diligent, deliberate mechanical sympathy: ScyllaDB employs a totally asynchronous, share-nothing programming model, relies on its own memory allocators, and meticulously schedules all its IO requests.
In this talk we will go over the low-level details of all the techniques involved - from a log-structured memory allocator to an advanced cache design -, covering how they are implemented and how they fully utilize the hardware resources they target.
Keynote: Apache HBase at Yahoo! Scale
Yahoo has long been involved in HBase and its community. In 2013, HBase was offered as a hosted service at Yahoo. Since then, adoption has grown rapidly., and today, HBase is used by numerous teams across the company, helping to enable a diverse set of use cases ranging from near real-time processing to data warehousing.
This was made possible thanks to HBase along with some enhancements to support multi-tenancy and scale. As our clusters continue to grow and use cases become more demanding we are working towards supporting a million regions in a single cluster.
In this keynote, we’ll paint a picture of where Yahoo! is today and the enhancements we have been working on to reach today’s scale as well as supporting a million regions and beyond.
Object Storage with Gluster
This document discusses using Gluster object storage with OpenStack Swift. Gluster-Swift mounts the Swift storage using FUSE and allows Swift to interface with Gluster backends. This avoids reimplementing the Swift object API. Gluster-Swift overrides Swift's distribution and replication to use the Gluster backend. The Swift API is implemented using FUSE operations on the Gluster volume. Future work includes upgrading Gluster-Swift, packaging, optimizations, and potentially developing a native Gluster object interface.
Bucket Your Partitions Wisely (Markus Höfer, codecentric AG) | Cassandra Summ...
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
About the Speaker
Markus Hofer IT Consultant, codecentric AG
Markus Hofer works as an IT Consultant for codecentric AG in Minster, Germany. He works on microservice architectures backed by DSE and/or Apache Cassandra. Markus supports and trains customers building cassandra based applications.
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
A solid backup strategy is a DBA's bread and butter. Cassandra's nodetool snapshot makes it easy to back up the SSTable files, but there remains the question of where to put them and how. Knewton's backup strategy uses Ansible for distributed backups and stores them in S3.
Unfortunately, it's all too easy to store backups that are essentially useless due to the absence of a coherent restoration strategy. This problem proved much more difficult and nuanced than taking the backups themselves. I will discuss Knewton's restoration strategy, which again leverages Ansible, yet I will focus on general principles and pitfalls to be avoided. In particular, restores necessitated modifying our backup strategy to generate cluster-wide metadata that is critical for a smooth automated restoration. Such pitfalls indicate that a restore-focused backup design leads to faster and more deterministic recovery.
About the Speaker
Joshua Wickman Database Engineer, Knewton
Dr. Joshua Wickman is currently part of the database team at Knewton, a NYC tech company focused on adaptive learning. He earned his PhD at the University of Delaware in 2012, where he studied particle physics models of the early universe. After a brief stint teaching college physics, he entered the New York tech industry in 2014 working with NoSQL, first with MongoDB and then Cassandra. He was certified in Cassandra at his first Cassandra Summit in 2015.
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
OpenTSDB was built on the belief that, through HBase, a new breed of monitoring systems could be created, one that can store and serve billions of data points forever without the need for destructive downsampling, one that could scale to millions of metrics, and where plotting real-time graphs is easy and fast. In this presentation we’ll review some of the key points of OpenTSDB’s design, some of the mistakes that were made, how they were or will be addressed, and what were some of the lessons learned while writing and running OpenTSDB as well as asynchbase, the asynchronous high-performance thread-safe client for HBase. Specific topics discussed will be around the schema, how it impacts performance and allows concurrent writes without need for coordination in a distributed cluster of OpenTSDB instances.
openTSDB - Metrics for a distributed world
These are the slides for my talk at the IPC13/WTC13 in Munich on openTSDB. openTSDB ist the software that we at gutefrage.net use to store about 200 million data points in several thousand time series per day.
I will talk about how openTSDB stores the data to efficiently query them afterwards. Some cultural issues and some myths are also covered.
More Related Content
What's hot
Update on OpenTSDB and AsyncHBase
This document summarizes an update on OpenTSDB, an open source time series database. It discusses OpenTSDB's ability to store trillions of data points at scale using HBase, Cassandra, or Bigtable as backends. Use cases mentioned include systems monitoring, sensor data, and financial data. The document outlines writing and querying functionality and describes the data model and table schema. It also discusses new features in OpenTSDB 2.2 and 2.3 like downsampling, expressions, and data stores. Community projects using OpenTSDB are highlighted and the future of OpenTSDB is discussed.
OpenTSDB 2.0
Speakers: Chris Larsen (Limelight Networks) and Benoit Sigoure (Arista Networks)
The OpenTSDB community continues to grow and with users looking to store massive amounts of time-series data in a scalable manner. In this talk, we will discuss a number of use cases and best practices around naming schemas and HBase configuration. We will also review OpenTSDB 2.0's new features, including the HTTP API, plugins, annotations, millisecond support, and metadata, as well as what's next in the roadmap.
Gnocchi v3
The document summarizes optimisation opportunities and testing results for Gnocchi v3 compared to v2. It discusses improvements to coordination and scheduling to minimize contention, performance improvements for large datasets, a new storage format to reduce operations and disk size, and benchmark results showing processing time reductions of up to 23% and write throughput increases.
Gnocchi v4 - past and present
a dive into performance enhancements in Gnocchi v4.x (as of end of August) and how it compares to previous releases of Gnocchi.
"Metrics: Where and How", Vsevolod Polyakov
Abstract: Nowadays it’s only a lazy one who haven’t written his own metric storage and aggregation system. I am lazy, and that’s why I have to choose what to use and how to use. I don’t want you to do the same job, so I decided to share my considerations concerning architectures and test results.
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Запускаем сервер (БД, Web-сервер или что-то свое собственное) и не получаем желаемый RPS. Запускаем top и видим, что 100% выедается CPU. Что дальше, на что расходуется процессорное время? Можно ли подкрутить какие-то ручки, чтобы улучшить производительность? А если параметр CPU не высокий, то куда смотреть дальше?
Мы рассмотрим несколько сценариев проблем производительности, рассмотрим доступные инструменты анализа производительности и разберемся в методологии оптимизации производительности Linux, ответим на вопрос за какие ручки и как крутить.
Anatomy of an action
A key feature when monitoring and debugging any Cloud infrastructure is to provide the ability to trace, track, and collate all the individual, discrete steps that compose an event. A typical resource action in OpenStack is often a combination of smaller tasks -- which given the distributed nature of OpenStack -- can fail at unpredictable points in the workflow. By collecting the appropriate events, operators can view all events within Ceilometer, filter on a failed action and trace back the history of related events to spot anomalies or errors. In this talk, we provide an overview of the recent enhancements made in Ceilometer to support the collection of event notifications from OpenStack services. We will describe: how events are processed, transformed and stored in Ceilometer; how you can derive metrics from events; and how it’s possible to track the events of a resource and analyse where errors occur.
Tweaking performance on high-load projects
This document discusses optimizing performance for high-load projects. It summarizes the delivery loads and technologies used for several projects including mGage, mobclix and XXXX. It then discusses optimizations made to improve performance, including using Solr for search, Redis for real-time data, Hadoop for reporting, and various Java optimizations in moving to Java 7. Specific optimizations discussed include reducing garbage collection, improving random number generation, and minimizing I/O operations.
HBaseCon2017 gohbase: Pure Go HBase Client
gohbase is an implementation of an HBase client in pure Go: https://github.com/tsuna/gohbase. In this presentation we'll talk about its architecture and compare its performance against the native Java HBase client as well as AsyncHBase (http://opentsdb.github.io/asynchbase/) and some nice characteristics of golang that resulted in a simpler implementation.
Advanced Apache Cassandra Operations with JMX
Nodetool is a command line interface for managing a Cassandra node. It provides commands for node administration, cluster inspection, table operations and more. The nodetool info command displays node-specific information such as status, load, memory usage and cache details. The nodetool compactionstats command shows compaction status including active tasks and progress. The nodetool tablestats command displays statistics for a specific table including read/write counts, space usage, cache usage and latency.
Handling 20 billion requests a month
This document discusses the architecture and technical challenges of handling a large volume of requests for an online advertising platform. It summarizes three key projects handled by the platform that delivered 3 billion, 14 billion, and 20 billion requests per month respectively. It describes the technologies used, including Solr, Redis, MySQL, Hadoop and Amazon Web Services instances. It also outlines optimizations made to improve performance, such as data compression, query optimizations, and Java 7 improvements. The goal was to process over 11,000 requests per second on average while maintaining response times below 100ms.
10 things i wish i'd known before using spark in production
Vous avez récemment commencé à travailler sur Spark et vos jobs prennent une éternité pour se terminer ? Cette présentation est faite pour vous.
Himanshu Arora et Nitya Nand YADAV ont rassemblé de nombreuses bonnes pratiques, optimisations et ajustements qu'ils ont appliqué au fil des années en production pour rendre leurs jobs plus rapides et moins consommateurs de ressources.
Dans cette présentation, ils nous apprennent les techniques avancées d'optimisation de Spark, les formats de sérialisation des données, les formats de stockage, les optimisations hardware, contrôle sur la parallélisme, paramétrages de resource manager, meilleur data localité et l'optimisation du GC etc.
Ils nous font découvrir également l'utilisation appropriée de RDD, DataFrame et Dataset afin de bénéficier pleinement des optimisations internes apportées par Spark.
JVM performance options. How it works
This is presentation for Cogniance Java Evening 13.02.2013.
My related to topic article you can find here (russian) : http://habrahabr.ru/post/160049/
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
NoSQL key-value — популярное решение, но какие преимущества и какой ценой оно даёт?
Скорость? Возможно, но ценой урезанного, по сравнению с реляционными базами данных, функционала. Но данные и приложение всё еще разделены сетевым стеком, а иногда и десятками километров оптоволокна. В крупном проекте, работающем на десятках или сотнях серверов нельзя обеспечить высокую скорость доступа к данным с каждой машины. Если хранилище достаточно быстро, то время обработки запроса окажется значительно меньше затрат на работу с сетью, а производительность приложения будут определять сетевые задержки и частота запросов к БД.
В проекте Облако@Mail.Ru мы ушли от использования чистых key-value хранилищ в пользу микросервисов на Tarantool, что позволило свести общение с хранилищем данных к минимуму.
Да, Tarantool — это еще одна NoSQL база данных, но еще это полноценный сервер приложений. Приложений, расположенных рядом с данными!
Я расскажу, как мы пришли к использованию микросервисов на основе Tarantool. Приведу несколько сценариев использования, которые работают в Облаке и могут быть легко адаптированы для другого web-проекта. Вы узнаете о компонентах, которые разработаны и опубликованы нами уже сейчас, и о дальнейших планах развития.
ScyllaDB: NoSQL at Ludicrous Speed
ScyllaDB is a NoSQL database compatible with Apache Cassandra, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware.
Achieving such speed requires a great deal of diligent, deliberate mechanical sympathy: ScyllaDB employs a totally asynchronous, share-nothing programming model, relies on its own memory allocators, and meticulously schedules all its IO requests.
In this talk we will go over the low-level details of all the techniques involved - from a log-structured memory allocator to an advanced cache design -, covering how they are implemented and how they fully utilize the hardware resources they target.
Keynote: Apache HBase at Yahoo! Scale
Yahoo has long been involved in HBase and its community. In 2013, HBase was offered as a hosted service at Yahoo. Since then, adoption has grown rapidly., and today, HBase is used by numerous teams across the company, helping to enable a diverse set of use cases ranging from near real-time processing to data warehousing.
This was made possible thanks to HBase along with some enhancements to support multi-tenancy and scale. As our clusters continue to grow and use cases become more demanding we are working towards supporting a million regions in a single cluster.
In this keynote, we’ll paint a picture of where Yahoo! is today and the enhancements we have been working on to reach today’s scale as well as supporting a million regions and beyond.
Object Storage with Gluster
This document discusses using Gluster object storage with OpenStack Swift. Gluster-Swift mounts the Swift storage using FUSE and allows Swift to interface with Gluster backends. This avoids reimplementing the Swift object API. Gluster-Swift overrides Swift's distribution and replication to use the Gluster backend. The Swift API is implemented using FUSE operations on the Gluster volume. Future work includes upgrading Gluster-Swift, packaging, optimizations, and potentially developing a native Gluster object interface.
Bucket Your Partitions Wisely (Markus Höfer, codecentric AG) | Cassandra Summ...
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
About the Speaker
Markus Hofer IT Consultant, codecentric AG
Markus Hofer works as an IT Consultant for codecentric AG in Minster, Germany. He works on microservice architectures backed by DSE and/or Apache Cassandra. Markus supports and trains customers building cassandra based applications.
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
A solid backup strategy is a DBA's bread and butter. Cassandra's nodetool snapshot makes it easy to back up the SSTable files, but there remains the question of where to put them and how. Knewton's backup strategy uses Ansible for distributed backups and stores them in S3.
Unfortunately, it's all too easy to store backups that are essentially useless due to the absence of a coherent restoration strategy. This problem proved much more difficult and nuanced than taking the backups themselves. I will discuss Knewton's restoration strategy, which again leverages Ansible, yet I will focus on general principles and pitfalls to be avoided. In particular, restores necessitated modifying our backup strategy to generate cluster-wide metadata that is critical for a smooth automated restoration. Such pitfalls indicate that a restore-focused backup design leads to faster and more deterministic recovery.
About the Speaker
Joshua Wickman Database Engineer, Knewton
Dr. Joshua Wickman is currently part of the database team at Knewton, a NYC tech company focused on adaptive learning. He earned his PhD at the University of Delaware in 2012, where he studied particle physics models of the early universe. After a brief stint teaching college physics, he entered the New York tech industry in 2014 working with NoSQL, first with MongoDB and then Cassandra. He was certified in Cassandra at his first Cassandra Summit in 2015.
What's hot (20)
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
10 things i wish i'd known before using spark in production
10 things i wish i'd known before using spark in production
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
Bucket Your Partitions Wisely (Markus Höfer, codecentric AG) | Cassandra Summ...
Bucket Your Partitions Wisely (Markus Höfer, codecentric AG) | Cassandra Summ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Viewers also liked
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
OpenTSDB was built on the belief that, through HBase, a new breed of monitoring systems could be created, one that can store and serve billions of data points forever without the need for destructive downsampling, one that could scale to millions of metrics, and where plotting real-time graphs is easy and fast. In this presentation we’ll review some of the key points of OpenTSDB’s design, some of the mistakes that were made, how they were or will be addressed, and what were some of the lessons learned while writing and running OpenTSDB as well as asynchbase, the asynchronous high-performance thread-safe client for HBase. Specific topics discussed will be around the schema, how it impacts performance and allows concurrent writes without need for coordination in a distributed cluster of OpenTSDB instances.
openTSDB - Metrics for a distributed world
These are the slides for my talk at the IPC13/WTC13 in Munich on openTSDB. openTSDB ist the software that we at gutefrage.net use to store about 200 million data points in several thousand time series per day.
I will talk about how openTSDB stores the data to efficiently query them afterwards. Some cultural issues and some myths are also covered.
HBaseCon 2015: OpenTSDB and AsyncHBase Update
OpenTSDB is an open source distributed time series database for storing large amounts of metrics data and performing fast queries. It is scalable and can store trillions of data points across multiple servers. Data is stored in HBase and queries are performed using a simple query language. OpenTSDB 2.x includes new features like salting to distribute writes across servers, compact storage formats using column appends, and downsampling to fill in missing data during aggregation.
000 proyecto tesis ccss jfcc ultimo12
El documento presenta un proyecto de tesis para optar el grado de Maestría en Ciencias Sociales con mención en Políticas Sociales: Gerencia Social. El proyecto examinará la estrategia de responsabilidad social que la Universidad Señor de Sipán implementa con su Proyecto Ciudad Educadora en la Urbanización Popular de Interés Social Pedro Pablo Atusparia. El objetivo es proponer mejoras para aumentar la participación de los pobladores en el desarrollo socio-cultural de su comunidad. El estudio realizará un diagnóst
Exposicion plataforma de monitoreo zabbix
Este documento describe la implementación de un sistema de monitoreo basado en Zabbix. Explica los componentes clave de Zabbix como el servidor, agentes y la interfaz web. También cubre temas como el protocolo SNMP, la integración de Zabbix con SNMP, la instalación de Zabbix y los costos asociados con el monitoreo de red.
4Linux - Monitoramento de Redes com o Zabbix
O documento discute o monitoramento com o software Zabbix, incluindo como ele pode monitorar ativos de rede, serviços e web cenários, além de apresentar gráficos, mapas e dashboards. Exemplos de casos de sucesso com o Zabbix são fornecidos, como o monitoramento de portais e agências em três estados brasileiros.
[FR] Timeseries appliqué aux couches de bébé
Il y a un nouveau type de bases de données à la mode : les timeseries. Je tenterais bien de vous expliquer comment ça fonctionne, mais je dois d'abord aller changer la couche de bébé... Par Vincent Casse, développeur chez OVH.
Monitoreo Atraves de GLpi
GLPI es una aplicación web de código abierto para la gestión de inventario informático y soporte técnico. Permite administrar recursos de hardware, software, usuarios e incidencias, así como gestionar licencias, consumibles, una base de conocimientos, reservas y soporte técnico. Su instalación requiere Apache, PHP y MySQL, y ofrece ventajas como reducción de costos y optimización de recursos.
Proyecto maestria
Este documento presenta un proyecto de investigación que propone utilizar el software educativo Edilim como una estrategia didáctica para mejorar la capacidad de resolución de problemas en matemáticas de los estudiantes del primer grado de secundaria de la Institución Educativa "Augusto Salazar Bondy" en Chiclayo, Perú. El proyecto se llevará a cabo durante un año bajo la supervisión de un asesor y será evaluado por un jurado. El objetivo es contribuir a mejorar el proceso de enseñanza-ap
Developing for Android (The movie)
Tras tres años programando en la plataforma Android esta es la película de mi vida como Android Developer, un conjunto de buenas practicas y conceptos necesarios que aún a día de hoy sigo viendo que no se cumplen en la mayoría de proyectos con los que me cruzo. Son la conclusiones sacadas de mi experiencia y de multitud de debates con compañeros. Se abordan temas como: fragmentación, uso de la clase Context, naming, maquetación en Android, memory leaks y S.O.L.I.D.
Criando Mapas de rede para monitoramento com Zabbix
O documento descreve como criar mapas de rede no Zabbix, incluindo configurar mapas, tipos de elementos em um mapa como hosts, grupos e triggers, e usar submapas. Ele também cobre fazer backup de mapas, ações em mapas, personalizar ícones e alternar ícones para diferentes situações.
Java Persistence Api (Jpa)
El documento describe la API de persistencia de Java (JPA), la cual proporciona un estándar para gestionar datos relacionales en aplicaciones Java. JPA simplifica el desarrollo de la persistencia de datos mediante el mapeo de objetos a tablas de base de datos a través de anotaciones. Define conceptos como entidades, asociaciones, consultas y transacciones.
instalacion e intregracion de OcsInventory-NG y GLPI
Este documento proporciona instrucciones para instalar y configurar OcsInventory y GLPI, dos herramientas de inventario de hardware y software. Explica cómo instalar los requisitos previos como Apache, PHP y MySQL, y luego instalar y configurar cada aplicación de forma independiente. También describe cómo integrar ambas aplicaciones utilizando un plugin de GLPI que permite importar datos de inventario de OcsInventory.
Work at GlobalLogic Latin America
El documento presenta información sobre la compañía GlobalLogic, una empresa de tecnología que opera en Latinoamérica. Ofrece programas para promover el bienestar de los empleados como flexibilidad horaria, home office y licencia por paternidad. También organiza eventos recreativos y de integración para los empleados como olimpiadas, fiestas y descuentos en gimnasios. La compañía brinda charlas y conferencias para compartir conocimientos con la comunidad IT, y trabaja con socios tecnológicos para ofrecer proyectos innov
Gerência de redes com Zabbix: conhecendo a ferramenta
O documento apresenta a ferramenta de gerenciamento de redes Zabbix. Discutiu porque é importante gerenciar redes, o que é um sistema de gerenciamento de rede e por que usar o Zabbix. Também resumiu as principais características, funcionalidades e demonstrações do Zabbix, além de discutir o futuro do Zabbix 2.0 e a comunidade brasileira Zabbix-BR.
Monitoring solutions comparison
This document provides a feature matrix comparing the monitoring solutions AdRem NetCrunch and AccelOps. Both solutions offer IP SLA reports, logical grouping, trending, auto discovery, SNMP support, syslog support, plugins, alerts, full viewing and acknowledgment capabilities, distributed monitoring, inventory tracking, and support applications and IPv6. The key differences are that AccelOps offers trend prediction, agent support, and PostgreSQL data storage while AdRem offers unknown platform, data storage in SQL, and unknown access control. Both solutions are commercial products.
Sistema de monitorización open nms
Este documento describe el sistema de gestión de red abierto OpenNMS, incluyendo sus funcionalidades principales como la gestión de eventos y alarmas, el rendimiento y la gestión de SLAs, y la integración y gestión de configuración. OpenNMS es una plataforma de código abierto escrita en Java que permite el monitoreo, la detección y la gestión de recursos de red.
Debian Jessie (8.0) + GLPI 9.0 + FusionInventory
Manual para la imlementación de un servidor en una máquina virtual con Debian Jessie (8.0), para utilizarlo con GLPI 9.0 para mesa de ayuda integrado con FusionInventory para manejo de inventario del parque informático de una organización.
Plantilla para realizar un manual de usuario de software
La siguiente presentación es la plantilla que se utiliza en el Departamento de Computación para realizar los manuales de usuario de los sistemas desarrollados tanto para ingeniería de software, sistemas de información y trabajos especiales de grado.
Implantación y monitorización con SNMP
Práctica realizada en el módulo de Servicios de Red e Internet del Grado Superior en Administración de Sistemas Informáticos en Red cursado en el IES Las Espeñetas de Orihuela (Alicante) en el Curso 2014/2015
Viewers also liked (20)
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
Criando Mapas de rede para monitoramento com Zabbix
Criando Mapas de rede para monitoramento com Zabbix
instalacion e intregracion de OcsInventory-NG y GLPI
instalacion e intregracion de OcsInventory-NG y GLPI
Gerência de redes com Zabbix: conhecendo a ferramenta
Gerência de redes com Zabbix: conhecendo a ferramenta
Plantilla para realizar un manual de usuario de software
Plantilla para realizar un manual de usuario de software
Similar to opentsdb in a real enviroment
Umleitung: a tiny mochiweb/CouchDB app
The slides from erlounge Wellington 02-2009. Umleintung is a tiny HTTP redirect daemon based on mochiweb and CouchDB.
It describes how to bootstrap a mochiweb application and integrate CouchDB.
Running Docker in Development & Production (#ndcoslo 2015)
The document discusses running Docker in development and production. It covers:
- Using Docker containers to run individual services like Elasticsearch or web applications
- Creating Dockerfiles to build custom images
- Linking containers together and using environment variables for service discovery
- Scaling with Docker Compose, load balancing with Nginx, and service discovery with Consul
- Clustering containers together using Docker Swarm for high availability
DPDK in Containers Hands-on Lab
This document provides an agenda and overview for a hands-on lab on using DPDK in containers. It introduces Linux containers and how they use fewer system resources than VMs. It discusses how containers still use the kernel network stack, which is not ideal for SDN/NFV usages, and how DPDK can be used in containers to address this. The hands-on lab section guides users through building DPDK and Open vSwitch, configuring them to work with containers, and running packet generation and forwarding using testpmd and pktgen Docker containers connected via Open vSwitch.
Wanting distributed volumes - Experiences with ceph-docker
Slides of a docker meetup presentation in Cologne (april 28,2016)
The presentation talks about how to run ceph in docker containers and how to use the ceph filesystems for volumes of docker containers in need of persistent storage.
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Cephalocon APAC 2018
March 22-23, 2018 - Beijing, China
David Byte, SUSE Senior Technology Strategist
Alex Lau, SUSE Storage Consultant
CEPH中的QOS技术
This document discusses ways to provide quality of service (QoS) without using traditional QoS mechanisms for modern storage systems like Ceph. It describes using techniques like limiting client I/O, traffic shaping at gateways, adjusting iSCSI queue depths, and using Linux traffic control (tc) tools to provide bandwidth caps or inject latency. While upstream Ceph efforts work on native QoS, these techniques can provide some control over performance for different clients in the interim. The document cautions that solutions like tc are complex to configure for many clients and may drop packets.
hacking with node.JS
Node.js is an exciting new platform developed by Ryan Dahl that allows JavaScript developers to create extremely high performance servers by leveraging Google's V8 JavaScript engine and asynchronous I/O. It uses a non-blocking I/O model that makes it lightweight and efficient, as well as features like CommonJS modules and the npm package manager. The document provides examples of building simple HTTP servers and modules in Node.js to demonstrate its capabilities.
Ceph Day Melbourne - Troubleshooting Ceph
This document provides troubleshooting guidance for issues with Ceph. It begins by suggesting identifying the problem domain as either performance, hang, crash, or unexpected behavior. For each problem, it recommends tools and techniques for further investigation such as debugging logs, profiling tools, and source code analysis. Debugging steps include establishing baselines, identifying implicated hosts or subsystems, increasing log verbosity, and tracing transactions through logs. The document emphasizes starting at the user end and working back towards Ceph to isolate issues.
Scale Apache with Nginx
The need to scale is in high demand in an age where everything is moving to the cloud. Though the standard Apache configuration could handle a website with moderate traffic, the minute it gets slash dotted or twitted multiple times could spell an embarrassing crash landing! If you are the administrator of such a website then good luck finding another job! On the other hand you value high availability in the midst of popularity then read on. On this one day workshop, we will show you how to scale your website and webapps to scale to handle thousands of simultaneous sessions the right way. The topics covered will include:
- Setting up Apache and NGiNXM
- Setting up a sample LAMP web app
- Benchmarking Apache performance
- Fine tuning Apache to improve performance
- Fine tuning NGiNX to improve performance
- Discussion about code level improvements when developing custom webapps using PHP
Acus08 Advanced Load Balancing Apache2.2
This document provides an overview of advanced load balancing capabilities in Apache HTTP Server 2.2 using the mod_proxy module. Key points include:
- Mod_proxy allows Apache to function as a reverse proxy or load balancer for backend servers.
- New in 2.2 are improvements like large file support, graceful stop, mod_dbd integration, and better debugging.
- Load balancing is implemented through balancer providers that can be customized. Default providers balance by requests, traffic, or server busyness.
- Features like connection pooling, sticky sessions, failover clusters, and an embedded admin interface provide robust load balancing functionality.
HAProxy scale out using open source
Scaling out on the cloud is easy. Especially, if you have a software provisioning system that helps you to deploy your environment wherever you want. This session will give you an overview of the fantastic new features of HAProxy V 1.5, and how you can integrate it into your environment to build a high available environment, using open source software. Starting with a single-webserver + mysql setup provisioned via chef, we will deploy an HA Proxy Cluster in front and scale out your nginx and mysql database backend.
0628阙宏宇
This document summarizes Lighttpd & Modcache, an event-driven web server and caching module. Lighttpd is lightweight and has a simple module structure. Modcache caches files locally and in memory to improve performance. It has advantages over Squid like being Lighttpd-based and keeping the code simple. The document provides configuration examples for Modcache and "cook books" for caching images, downloads, forums, and video sites.
Control your deployments with Capistrano
This document discusses Capistrano, a remote server automation and deployment tool. Some key points:
- Capistrano allows reliable deployment of web applications to multiple machines simultaneously, with features like rollback, adding tasks, and automating common tasks.
- It works by creating a new folder for each deployment on servers and symlinking the current version. Shared files are not overwritten on redeploys.
- Configuration involves setting stages, roles, branches, and other parameters in Capistrano files. Tasks can be added for custom actions.
- Deploying runs tasks sequentially like updating servers, publishing, finishing. Rollback has similar reversing tasks. Plugins add features like maintenance modes.
Performance measurement and tuning
Timo Schmidt
E-Mail: timo.schmidt@aoemedia.de
Phone: +49 (0)6122 70 70 7 - 123
Osol Pgsql
This document discusses PostgreSQL and Solaris as a low-cost platform for medium to large scale critical scenarios. It provides an overview of PostgreSQL, highlighting features like MVCC, PITR, and ACID compliance. It describes how Solaris and PostgreSQL integrate well, with benefits like DTrace support, scalability on multicore/multiprocessor systems, and Solaris Cluster support. Examples are given for installing PostgreSQL on Solaris using different methods, configuring zones for isolation, using ZFS for storage, and monitoring performance with DTrace scripts.
How to go the extra mile on monitoring
This document provides instructions for monitoring additional metrics from clusters and applications using Grafana, Prometheus, JMX, and PushGateway. It includes steps to export JMX metrics from Kafka and NiFi, setup and use PushGateway to collect and expose custom metrics, and create Grafana dashboards to visualize the metrics.
[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"
The document discusses using Ceph storage as a PaaS platform and service. It describes PASTA, an in-house PaaS platform that uses Ceph for persistent volumes for containers. Ceph provides strong consistency for block and filesystem volumes and is used for stateful containers in Docker Swarm farms for services like Jenkins, Elasticsearch, and DRUID storage. Operational issues with Ceph discussed include multi-mapped volumes, upgrades, network failures, scrub/deep-scrub performance impacts, recovering RBD images, and monitor failures/recovery. Configuration options and methods for addressing these issues are also provided.
Capistrano
This document provides instructions for deploying a Rails application using Capistrano. It includes steps to set up Capistrano, configure the deploy.rb file, generate SSH keys, add the deploy key to GitHub, run Capistrano tasks to deploy the application, and make subsequent deploys when code changes. The application is deployed to a server at 192.168.255.54 running Mongrel and uses Git for version control.
Start tracking your ruby infrastructure
This document discusses using Docker and Ansible together for infrastructure as code. It begins with an overview of problems with traditional deployment approaches and advantages of defining infrastructure programmatically. It then provides in-depth explanations of Docker concepts like images, containers, Dockerfiles and how Docker works. The remainder covers using Ansible for configuration management, explaining concepts like modules, inventory, playbooks and roles. It emphasizes that Docker and Ansible together provide an easy way to start automating infrastructure while removing dependencies on specific technologies.
Teflon - Anti Stick for the browser attack surface
The document discusses the browser as the new "desktop" and main attack surface for web applications. It outlines the browser architecture and analogizes it to the kernel/operating system model. It then discusses exploiting browsers through techniques like heap spraying and return-to-heap attacks using JavaScript payloads. The document also introduces Teflon, a browser extension designed to prevent such exploits by inspecting and blocking offensive JavaScript vectors. Teflon was tested against real exploits and obfuscated payloads with success. Further research directions are discussed to improve browser security.
Similar to opentsdb in a real enviroment (20)
Running Docker in Development & Production (#ndcoslo 2015)
Running Docker in Development & Production (#ndcoslo 2015)
Wanting distributed volumes - Experiences with ceph-docker
Wanting distributed volumes - Experiences with ceph-docker
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"
[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"
Teflon - Anti Stick for the browser attack surface
Teflon - Anti Stick for the browser attack surface
Recently uploaded
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Gursev Pirge, PhD
Senior Data Scientist - JohnSnowLabs
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Recently uploaded (20)
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
opentsdb in a real enviroment
- 2. Opentsdb architecture TSD TSD TSD Hbase hadoop
- 5. Opentsdb POC
- 6. How to Start & Stop opentsdb Start opentsdb: Start hadoop environment Start hbase (use internal zookeeper or outside zookeeper) Start TSD Stop Opentsdb: Stop tsd Stop hbase Stop hadoop
- 7. Sample scenario This scenario is about how to insert CPU metrics of a host to opentsdb: Add these metrics to hbase first: ./src/tsdbmkmetricproc.loadavg.1m proc.loadavg.5m 2. Collect on the client and post to the tsd socket: cat >loadavg-collector.sh <<OF #!/bin/bash set -e while true; do awk -v now=`date +%s` -v host=`hostname` '{ print "put proc.loadavg.1m " now " " $1 " host=" host; print "put proc.loadavg.5m " now " " $2 " host=" host }' /proc/loadavg sleep 15 done | nc -w 30 node1 8080 EOF chmod+x loadavg-collector.sh nohup./loadavg-collector.sh & 3. Put Metrics data like these: [root@node1 /]# ./vendor/stumbleupon-opentsdb-33dff14/src/loadavg-collector-result.sh put proc.loadavg.1m 1310023344 0.14 host=node1 put proc.loadavg.5m 1310023344 0.11 host=node2
- 8. Need to do: Collection metrics scripts: change from saving into RRD to save to Hbase; Graphs pages on how to extract data in opentsdb and better looking