The document discusses Memcached, an open-source high-performance distributed memory caching system. It provides an overview of Memcached and how companies like Facebook, Yahoo, and Amazon use it to speed up dynamic web applications by caching objects in memory to reduce database load. Examples of Memcached usage are shown for LiveJournal, Mixi, and Grazr. Details are also provided on Memcached servers, clients, data storage and replication, programming interfaces for Ruby, PHP, C, and more.
The document discusses locking and concurrency control in databases, demonstrating how table locks, row locks, and multi-version concurrency control work through examples of a database being backed up while concurrent changes are made. It shows how different locking strategies, like those used in MyISAM and InnoDB, allow for concurrent access to data while maintaining consistency and isolation. A live demo then highlights deadlocks and lock waits that can occur with concurrent access and how they are handled.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
Slides from a presentation I gave on SSH. Covers basics of ssh, password|keys|host-based authentication, agent/key forwarding, configuration files (global and user-specific), local/remote port forwarding, scp, rsync, and briefly mentions git's support.
NFS allows remote access to files on a server from client machines. It uses stateless servers so server disruptions don't affect clients, and clients can continue accessing files after a server reboot. The client parses file paths and looks up components individually to accommodate different file naming conventions. NFS adopted UNIX file semantics and operations like open, read, write, and close, along with basic file types and permissions.
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
This document discusses VMware performance troubleshooting. It covers topics like root cause analysis, performance characteristics of CPU, memory, disk and networking, and tools like ESXTop, vm-support and the service console. It provides guidelines on capacity planning, virtual machine optimization and design best practices.
The Linux Kernel Scheduler (For Beginners) - SFO17-421Linaro
Session ID: SFO17-421
Session Name: The Linux Kernel Scheduler (For Beginners) - SFO17-421
Speaker: Viresh Kumar
Track: Power Management
★ Session Summary ★
This talk will take you through the internals of the Linux Kernel scheduler.
---------------------------------------------------
★ Resources ★
Event Page: http://connect.linaro.org/resource/sfo17/sfo17-421/
Presentation:
Video: https://www.youtube.com/watch?v=q283Wm__QQ0
---------------------------------------------------
★ Event Details ★
Linaro Connect San Francisco 2017 (SFO17)
25-29 September 2017
Hyatt Regency San Francisco Airport
---------------------------------------------------
Keyword:
'http://www.linaro.org'
'http://connect.linaro.org'
---------------------------------------------------
Follow us on Social Media
https://www.facebook.com/LinaroOrg
https://twitter.com/linaroorg
https://www.youtube.com/user/linaroorg?sub_confirmation=1
https://www.linkedin.com/company/1026961
MySQL with DRBD/Pacemaker/Corosync on LinuxPawan Kumar
The document describes setting up a high availability MySQL cluster with DRBD, Corosync, and Pacemaker on Linux. DRBD is configured in active-passive mode to synchronize data between two nodes. Corosync and Pacemaker provide cluster management and failover capability. MySQL runs in active mode on one node, and the virtual IP and data are failed over to the other passive node if needed for high availability. The steps provided include installing and configuring DRBD, Corosync, Pacemaker, generating authentication keys, and configuring the DRBD resource and cluster.
The document discusses locking and concurrency control in databases, demonstrating how table locks, row locks, and multi-version concurrency control work through examples of a database being backed up while concurrent changes are made. It shows how different locking strategies, like those used in MyISAM and InnoDB, allow for concurrent access to data while maintaining consistency and isolation. A live demo then highlights deadlocks and lock waits that can occur with concurrent access and how they are handled.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
Slides from a presentation I gave on SSH. Covers basics of ssh, password|keys|host-based authentication, agent/key forwarding, configuration files (global and user-specific), local/remote port forwarding, scp, rsync, and briefly mentions git's support.
NFS allows remote access to files on a server from client machines. It uses stateless servers so server disruptions don't affect clients, and clients can continue accessing files after a server reboot. The client parses file paths and looks up components individually to accommodate different file naming conventions. NFS adopted UNIX file semantics and operations like open, read, write, and close, along with basic file types and permissions.
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
This document discusses VMware performance troubleshooting. It covers topics like root cause analysis, performance characteristics of CPU, memory, disk and networking, and tools like ESXTop, vm-support and the service console. It provides guidelines on capacity planning, virtual machine optimization and design best practices.
The Linux Kernel Scheduler (For Beginners) - SFO17-421Linaro
Session ID: SFO17-421
Session Name: The Linux Kernel Scheduler (For Beginners) - SFO17-421
Speaker: Viresh Kumar
Track: Power Management
★ Session Summary ★
This talk will take you through the internals of the Linux Kernel scheduler.
---------------------------------------------------
★ Resources ★
Event Page: http://connect.linaro.org/resource/sfo17/sfo17-421/
Presentation:
Video: https://www.youtube.com/watch?v=q283Wm__QQ0
---------------------------------------------------
★ Event Details ★
Linaro Connect San Francisco 2017 (SFO17)
25-29 September 2017
Hyatt Regency San Francisco Airport
---------------------------------------------------
Keyword:
'http://www.linaro.org'
'http://connect.linaro.org'
---------------------------------------------------
Follow us on Social Media
https://www.facebook.com/LinaroOrg
https://twitter.com/linaroorg
https://www.youtube.com/user/linaroorg?sub_confirmation=1
https://www.linkedin.com/company/1026961
MySQL with DRBD/Pacemaker/Corosync on LinuxPawan Kumar
The document describes setting up a high availability MySQL cluster with DRBD, Corosync, and Pacemaker on Linux. DRBD is configured in active-passive mode to synchronize data between two nodes. Corosync and Pacemaker provide cluster management and failover capability. MySQL runs in active mode on one node, and the virtual IP and data are failed over to the other passive node if needed for high availability. The steps provided include installing and configuring DRBD, Corosync, Pacemaker, generating authentication keys, and configuring the DRBD resource and cluster.
This document discusses Linux memory management. It outlines the buddy system, zone allocation, and slab allocator used by Linux to manage physical memory. It describes how pages are allocated and initialized at boot using the memory map. The slab allocator is used to optimize allocation of kernel objects and is implemented as caches of fixed-size slabs and objects. Per-CPU allocation improves performance by reducing locking and cache invalidations.
Java Performance Analysis on Linux with Flame GraphsBrendan Gregg
This document discusses using Linux perf_events (perf) profiling tools to analyze Java performance on Linux. It describes how perf can provide complete visibility into Java, JVM, GC and system code but that Java profilers have limitations. It presents the solution of using perf to collect mixed-mode flame graphs that include Java method names and symbols. It also discusses fixing issues with broken Java stacks and missing symbols on x86 architectures in perf profiles.
This document provides an overview of MySQL including its architecture, clients, connection layer, SQL layer, storage engines, and installation methods. Key points include:
- MySQL uses a connection thread pool, query cache, parser, optimizer, and various storage engines like InnoDB and MyISAM.
- Common MySQL clients allow executing queries, administering the server, checking tables, backing up data, and more.
- The connection layer handles authentication and the communication protocol.
- The SQL layer performs parsing, optimization, and determining the optimal execution plan.
- Storage engines like InnoDB and MyISAM support different features and have different performance characteristics for storage, indexing, and more.
Apache Kafka is a high-throughput distributed messaging system that allows for both streaming and offline log processing. It uses Apache Zookeeper for coordination and supports activity stream processing and real-time pub/sub messaging. Kafka bridges the gaps between pure offline log processing and traditional messaging systems by providing features like batching, transactions, persistence, and support for multiple consumers.
At Salesforce, we have deployed many thousands of HBase/HDFS servers, and learned a lot about tuning during this process. This talk will walk you through the many relevant HBase, HDFS, Apache ZooKeeper, Java/GC, and Operating System configuration options and provides guidelines about which options to use in what situation, and how they relate to each other.
Oracle ACFS High Availability NFS Services (HANFS) Part-IAnju Garg
To satisfy increasing demands for data storage from big data and IoT, Oracle DBAs will need to handle massive amounts of normal file system storage. While some organizations have adopted open stack storage, others are evaluating options like NAS, NFS or other file systems. Oracle Database 12c R1 introduces ACFS as a high availability NFS file system (HANFS) that allows files on ACFS clusters to be accessed outside the cluster using highly available NFS exports, providing continuous access even if the exporting node fails.
A brief overview of caching mechanisms in a web application. Taking a look at the different layers of caching and how to utilize them in a PHP code base. We also compare Redis and MemCached discussing their advantages and disadvantages.
Slides for presentation on ZooKeeper I gave at Near Infinity (www.nearinfinity.com) 2012 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/zookeeper-samples
"The benefit of real-time data can be measured by how frequently the data in question changes, nowhere is this more apparent than threat detection. Responding to an ever changing landscape of attacks and exploits requires a system that can not only handle the scale and dynamic nature of the data but also a dynamically changing set of detection rules. We developed Confluent SIGMA, an open source project built on Kafka Streams for the open SIGMA DSL, to handle real-time rule additions and modifications. In this talk we will cover:
* The architecture of our Kafka Streams layer that makes it possible to use external data feeds as rule input
* How we handle dynamic criteria for joins and filters
* Best practices for writing dynamic rule engines in Kafka Streams
* Upcoming improvements to Kafka Streams to support versioned rules
Although Confluent SIGMA focuses on cyber threat detection this same pattern can also be applied to any DSL (domain specific language) that would benefit from real-time stream processing. After attending you will have the framework to drive dynamic rules through Kafka Streams for any use case that might require it."
Fluentd is an open source data collector that allows flexible data collection, processing, and output. It supports streaming data from sources like logs and metrics to destinations like databases, search engines, and object stores. Fluentd's plugin-based architecture allows it to support a wide variety of use cases. Recent versions of Fluentd have added features like improved plugin APIs, nanosecond time resolution, and Windows support to make it more suitable for containerized environments and low-latency applications.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
※다운로드하시면 더 선명한 자료를 보실 수 있습니다.
동접 200만 명이 접속할 수백 대의 게임 서버가 최소한의 MySQL 서버만으로 서비스할 수 있는 구조를 설명합니다.
고성능/고효율의 MySQL 스케일링 기법을 공유합니다. 대규모 게임 서비스에서 이미 검증된 것은 안 비밀~
목차
1. 기본적인 아기텍처
2. ProxySQL을 이용한 더 나은 아키텍처
3. 최종 아키텍처
대상
- 대규모 게임 서비스에 MySQL을 사용한 경험에 관심 있는 분
- ProxySQL에 관심이 있는 서버 개발자 혹은 DBA
- 게임 서버 개발 과정에서 DB 쪽을 유연하게 구성하고 싶은 분
■관련 동영상: https://youtu.be/8Eb_n7JA1yA
Android uses cgroups to monitor system memory usage via the Low Memory Killer daemon and to group processes for effective CPU sharing. Cgroups are used to create mount points for memory and CPU control groups. The LMK daemon uses cgroups to receive memory pressure events and kill processes as needed. Init.rc uses cgroups to create groups for real-time and background tasks and assign CPU shares. Android further groups processes by scheduling policy for scheduling priorities.
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
This document discusses file system usage in HBase. It provides an overview of the three main file types in HBase: write-ahead logs (WALs), data files, and reference files. It describes durability semantics, IO fencing techniques for region server recovery, and how HBase leverages data locality through short circuit reads, checksums, and block placement hints. The document is intended help understand HBase's interactions with HDFS for tuning IO performance.
Stop the Guessing: Performance Methodologies for Production SystemsBrendan Gregg
Talk presented at Velocity 2013. Description: When faced with performance issues on complex production systems and distributed cloud environments, it can be difficult to know where to begin your analysis, or to spend much time on it when it isn’t your day job. This talk covers various methodologies, and anti-methodologies, for systems analysis, which serve as guidance for finding fruitful metrics from your current performance monitoring products. Such methodologies can help check all areas in an efficient manner, and find issues that can be easily overlooked, especially for virtualized environments which impose resource controls. Some of the tools and methodologies covered, including the USE Method, were developed by the speaker and have been used successfully in enterprise and cloud environments.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
Vold is the volume daemon in Android that manages storage volumes like external SD cards. It communicates with the Linux kernel via Netlink sockets to receive storage events and with the MountService via a local socket. When a new storage device is inserted, Vold receives the kernel event, mounts the volume if FAT format according to its configuration file, and notifies MountService to make the volume available to the user.
Open Device Programmability: Hands-on Intro to RESTCONF (and a bit of NETCONF)Cisco DevNet
In this small group, hands-on workshop session you'll learn how to write your first Python application that uses YANG, NETCONF and , RESTCONF to access operational and configuration data on a device.
Watch the DevNet 2044 replay from the Cisco Live On-Demand Library at: https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=92725&backBtn=true
Check out more and register for Cisco DevNet: http://ow.ly/jCNV3030OfS
Apache kafka performance(throughput) - without data loss and guaranteeing dat...SANG WON PARK
Apache Kafak의 성능이 특정환경(데이터 유실일 발생하지 않고, 데이터 전송순서를 반드시 보장)에서 어느정도 제공하는지 확인하기 위한 테스트 결과 공유
데이터 전송순서를 보장하기 위해서는 Apache Kafka cluster로 partition을 분산할 수 없게되므로, 성능향상을 위한 장점을 사용하지 못하게 된다.
이번 테스트에서는 Apache Kafka의 단위 성능, 즉 partition 1개에 대한 성능만을 측정하게 된다.
향후, partition을 증가할 경우 본 테스트의 1개 partition 단위 성능을 기준으로 예측이 가능할 것 같다.
This document provides an overview of memcached, a distributed memory caching system. It discusses memcached's architecture, applications, clients including PHP clients, operations, optimizations, and alternatives. The document is presented by Andrei Zmievski from Digg at OSCON 2009 and focuses on using memcached for distributed PHP applications.
The document discusses Varnish, an open-source HTTP accelerator. It provides information on Varnish's architecture, which uses mmap to map the cache store into memory. It discusses various Varnish configuration options and parameters like object_workspace and thread pools. It also covers Varnish's domain specific language VCL which is used to define caching rules and behaviors.
This document discusses Linux memory management. It outlines the buddy system, zone allocation, and slab allocator used by Linux to manage physical memory. It describes how pages are allocated and initialized at boot using the memory map. The slab allocator is used to optimize allocation of kernel objects and is implemented as caches of fixed-size slabs and objects. Per-CPU allocation improves performance by reducing locking and cache invalidations.
Java Performance Analysis on Linux with Flame GraphsBrendan Gregg
This document discusses using Linux perf_events (perf) profiling tools to analyze Java performance on Linux. It describes how perf can provide complete visibility into Java, JVM, GC and system code but that Java profilers have limitations. It presents the solution of using perf to collect mixed-mode flame graphs that include Java method names and symbols. It also discusses fixing issues with broken Java stacks and missing symbols on x86 architectures in perf profiles.
This document provides an overview of MySQL including its architecture, clients, connection layer, SQL layer, storage engines, and installation methods. Key points include:
- MySQL uses a connection thread pool, query cache, parser, optimizer, and various storage engines like InnoDB and MyISAM.
- Common MySQL clients allow executing queries, administering the server, checking tables, backing up data, and more.
- The connection layer handles authentication and the communication protocol.
- The SQL layer performs parsing, optimization, and determining the optimal execution plan.
- Storage engines like InnoDB and MyISAM support different features and have different performance characteristics for storage, indexing, and more.
Apache Kafka is a high-throughput distributed messaging system that allows for both streaming and offline log processing. It uses Apache Zookeeper for coordination and supports activity stream processing and real-time pub/sub messaging. Kafka bridges the gaps between pure offline log processing and traditional messaging systems by providing features like batching, transactions, persistence, and support for multiple consumers.
At Salesforce, we have deployed many thousands of HBase/HDFS servers, and learned a lot about tuning during this process. This talk will walk you through the many relevant HBase, HDFS, Apache ZooKeeper, Java/GC, and Operating System configuration options and provides guidelines about which options to use in what situation, and how they relate to each other.
Oracle ACFS High Availability NFS Services (HANFS) Part-IAnju Garg
To satisfy increasing demands for data storage from big data and IoT, Oracle DBAs will need to handle massive amounts of normal file system storage. While some organizations have adopted open stack storage, others are evaluating options like NAS, NFS or other file systems. Oracle Database 12c R1 introduces ACFS as a high availability NFS file system (HANFS) that allows files on ACFS clusters to be accessed outside the cluster using highly available NFS exports, providing continuous access even if the exporting node fails.
A brief overview of caching mechanisms in a web application. Taking a look at the different layers of caching and how to utilize them in a PHP code base. We also compare Redis and MemCached discussing their advantages and disadvantages.
Slides for presentation on ZooKeeper I gave at Near Infinity (www.nearinfinity.com) 2012 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/zookeeper-samples
"The benefit of real-time data can be measured by how frequently the data in question changes, nowhere is this more apparent than threat detection. Responding to an ever changing landscape of attacks and exploits requires a system that can not only handle the scale and dynamic nature of the data but also a dynamically changing set of detection rules. We developed Confluent SIGMA, an open source project built on Kafka Streams for the open SIGMA DSL, to handle real-time rule additions and modifications. In this talk we will cover:
* The architecture of our Kafka Streams layer that makes it possible to use external data feeds as rule input
* How we handle dynamic criteria for joins and filters
* Best practices for writing dynamic rule engines in Kafka Streams
* Upcoming improvements to Kafka Streams to support versioned rules
Although Confluent SIGMA focuses on cyber threat detection this same pattern can also be applied to any DSL (domain specific language) that would benefit from real-time stream processing. After attending you will have the framework to drive dynamic rules through Kafka Streams for any use case that might require it."
Fluentd is an open source data collector that allows flexible data collection, processing, and output. It supports streaming data from sources like logs and metrics to destinations like databases, search engines, and object stores. Fluentd's plugin-based architecture allows it to support a wide variety of use cases. Recent versions of Fluentd have added features like improved plugin APIs, nanosecond time resolution, and Windows support to make it more suitable for containerized environments and low-latency applications.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
※다운로드하시면 더 선명한 자료를 보실 수 있습니다.
동접 200만 명이 접속할 수백 대의 게임 서버가 최소한의 MySQL 서버만으로 서비스할 수 있는 구조를 설명합니다.
고성능/고효율의 MySQL 스케일링 기법을 공유합니다. 대규모 게임 서비스에서 이미 검증된 것은 안 비밀~
목차
1. 기본적인 아기텍처
2. ProxySQL을 이용한 더 나은 아키텍처
3. 최종 아키텍처
대상
- 대규모 게임 서비스에 MySQL을 사용한 경험에 관심 있는 분
- ProxySQL에 관심이 있는 서버 개발자 혹은 DBA
- 게임 서버 개발 과정에서 DB 쪽을 유연하게 구성하고 싶은 분
■관련 동영상: https://youtu.be/8Eb_n7JA1yA
Android uses cgroups to monitor system memory usage via the Low Memory Killer daemon and to group processes for effective CPU sharing. Cgroups are used to create mount points for memory and CPU control groups. The LMK daemon uses cgroups to receive memory pressure events and kill processes as needed. Init.rc uses cgroups to create groups for real-time and background tasks and assign CPU shares. Android further groups processes by scheduling policy for scheduling priorities.
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
This document discusses file system usage in HBase. It provides an overview of the three main file types in HBase: write-ahead logs (WALs), data files, and reference files. It describes durability semantics, IO fencing techniques for region server recovery, and how HBase leverages data locality through short circuit reads, checksums, and block placement hints. The document is intended help understand HBase's interactions with HDFS for tuning IO performance.
Stop the Guessing: Performance Methodologies for Production SystemsBrendan Gregg
Talk presented at Velocity 2013. Description: When faced with performance issues on complex production systems and distributed cloud environments, it can be difficult to know where to begin your analysis, or to spend much time on it when it isn’t your day job. This talk covers various methodologies, and anti-methodologies, for systems analysis, which serve as guidance for finding fruitful metrics from your current performance monitoring products. Such methodologies can help check all areas in an efficient manner, and find issues that can be easily overlooked, especially for virtualized environments which impose resource controls. Some of the tools and methodologies covered, including the USE Method, were developed by the speaker and have been used successfully in enterprise and cloud environments.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
Vold is the volume daemon in Android that manages storage volumes like external SD cards. It communicates with the Linux kernel via Netlink sockets to receive storage events and with the MountService via a local socket. When a new storage device is inserted, Vold receives the kernel event, mounts the volume if FAT format according to its configuration file, and notifies MountService to make the volume available to the user.
Open Device Programmability: Hands-on Intro to RESTCONF (and a bit of NETCONF)Cisco DevNet
In this small group, hands-on workshop session you'll learn how to write your first Python application that uses YANG, NETCONF and , RESTCONF to access operational and configuration data on a device.
Watch the DevNet 2044 replay from the Cisco Live On-Demand Library at: https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=92725&backBtn=true
Check out more and register for Cisco DevNet: http://ow.ly/jCNV3030OfS
Apache kafka performance(throughput) - without data loss and guaranteeing dat...SANG WON PARK
Apache Kafak의 성능이 특정환경(데이터 유실일 발생하지 않고, 데이터 전송순서를 반드시 보장)에서 어느정도 제공하는지 확인하기 위한 테스트 결과 공유
데이터 전송순서를 보장하기 위해서는 Apache Kafka cluster로 partition을 분산할 수 없게되므로, 성능향상을 위한 장점을 사용하지 못하게 된다.
이번 테스트에서는 Apache Kafka의 단위 성능, 즉 partition 1개에 대한 성능만을 측정하게 된다.
향후, partition을 증가할 경우 본 테스트의 1개 partition 단위 성능을 기준으로 예측이 가능할 것 같다.
This document provides an overview of memcached, a distributed memory caching system. It discusses memcached's architecture, applications, clients including PHP clients, operations, optimizations, and alternatives. The document is presented by Andrei Zmievski from Digg at OSCON 2009 and focuses on using memcached for distributed PHP applications.
The document discusses Varnish, an open-source HTTP accelerator. It provides information on Varnish's architecture, which uses mmap to map the cache store into memory. It discusses various Varnish configuration options and parameters like object_workspace and thread pools. It also covers Varnish's domain specific language VCL which is used to define caching rules and behaviors.
Fast and easy method of installing one or more MySQL servers in isolation.
Includes latest development, hints of Gearman remote install, and a lightning talk
Denish Patel, a database architect at OmniTi, gave a presentation on the RubyRep database replication tool. RubyRep is an asynchronous master-master and master-slave replication tool that is easy to install, platform independent, and supports large objects and tables without primary keys. It was developed as an alternative to Slony because Slony has limitations such as requiring tables to have primary keys and not supporting large objects or master-master replication. The presentation covered how to install, configure, scan, sync, and replicate databases using RubyRep.
The document discusses caching strategies for Rails applications to improve performance. It recommends caching late in the development process after optimizing code. Different caching mechanisms are described from page caching to object caching. Memcached is recommended for caching and supports features like expiration and versioning. Examples of large companies using caching and Memcached like Facebook are provided.
The document discusses the use of APC (Alternative PHP Cache) and Memcached for caching and improving performance in PHP applications. APC is useful for opcode caching and basic user data caching, while Memcached provides a more robust distributed caching system and additional features like data segmentation across multiple servers and atomic counters. The document provides examples of using the basic caching functions of APC and Memcached, as well as more advanced techniques.
This document discusses JRuby on Google App Engine. It provides an overview of Google App Engine, quotas and billing, current issues with JRuby on App Engine, required gems, the development app server, deployment process, available APIs, and sample code for using DataMapper and associations. It also lists resources for learning more about JRuby on App Engine.
Memcached: What is it and what does it do? (PHP Version)Brian Moon
Memcached has become the de facto standard for caching web applications. But, many users jump in feet first without understanding what it does or perhaps more importantly what it does not do. Once you understand memcached, you may come to realize that it is what it does not do that makes it so good. Memcached is a distributed memory based caching system. But, what does that mean for you? This session willcover the basics of memcached. What are all the components needed? Where is your data cached? What happens when there is a system failure? Is my data stored in more than one place? How do I know what isin my cache? All these questions and more will be answered.
The document discusses various techniques for optimizing a multisite WordPress architecture for a high-traffic music website including: using caching (APC, Memcached, Batcache), output buffering, CDNs, regionalized content/caching, shared content across sites, custom post types, web services, and configuration files. It also covers database scaling, PHP optimizations, and using Amazon services like EC2 and S3.
APC & Memcache the High Performance DuoAnis Berejeb
The document discusses Apc & Memcached as a high-performance caching duo for PHP applications. It provides an overview of APC for opcode caching and user data caching. It then discusses Memcached which allows distributed caching across multiple servers and provides more advanced features than APC. Examples are given for basic usage of storing, retrieving, and deleting data from the caches with both extensions.
This document provides tips and examples for creating shell scripts to automate database administration tasks. It recommends using shell scripts because shell is available everywhere and shell scripting is powerful and fast to write. It then provides several tips for writing robust shell scripts, such as using configuration files, running commands in parallel, and creating shortcuts. The document includes examples of scripts for installing MySQL replication across multiple servers and testing that replication is working.
The document discusses using APC (Alternative PHP Cache) and Memcached together for high performance caching. It provides an overview of APC for opcode and user caching and how it can accelerate PHP script performance. Memcached is presented as an alternative for user caching that offers a distributed caching system accessible from PHP through an object-oriented interface. Examples are given for basic usage of storing, retrieving, and deleting from the caches. Features like multi-server environments, data segmentation, and delayed data retrieval with Memcached are also covered.
The document discusses Google App Engine and options for building applications on it using Ruby and a new language called Duby. It provides an overview of App Engine architecture and billing models. It then covers using Ruby with App Engine via JRuby, frameworks like Rack and Sinatra, and the DataMapper ORM. Duby is introduced as a statically-typed language inspired by Ruby syntax that compiles to Java bytecode for the App Engine environment. Examples show a simple blog application built in both Ruby and Duby styles.
Spock is a testing framework for Java and Groovy applications that allows for highly expressive tests. It uses Groovy as its language and provides features like a JUnit runner, object mocking, and descriptive specifications. Spock tests can be written using fields, fixture methods, and feature methods with blocks to group test steps and assertions. Grails applications can also use Spock for integration and functional testing with the Geb plugin, which provides browser automation and page object modeling support.
Cook Up a Runtime with The New OSGi Resolver - Neil Bartlettmfrancis
OSGi DevCon 2013
OSGi applications are assembled from loosely coupled bundles communicating via services. While this model provides huge flexibility and the ability to reuse components, it creates a challenge for the assembler of the application since it is unclear which bundles are needed, which are optional and which are unnecessary.
For example some dependencies are implicit, such as the provider of a service or an extender. We do not want to prematurely lock down these dependencies at build time, but at deployment time a specific provider must be found otherwise the application will fail to behave as expected. Furthermore when third-party libraries are used they often contain static dependencies on additional libraries, which in turn contain additional dependencies, and so on. Much time is wasted in finding a set of bundles that will actually resolve and run in the OSGi environment, and once such a set is found, developers tend to fear making changes to it.
The generic requirements and capabilities model introduced in OSGi Release 4.3, along with the standard Repository and Resolver specifications introduced in OSGi Release 5, provide answers for both of these problems. Using capabilities, we can describe dependencies in an abstract way without prematurely binding to specific providers. Using the resolver, we can narrow our focus to the very small set of bundles that describe our application at the top level, and allow all other dependencies to be computed and managed for us.
This talk begins with a brief technical overview of the new 4.3 and 5.0 specifications and how they can be used to assemble applications with ease. We then demonstrate both development tooling and a runtime platform that can be used to put these ideas into practice.
In this session Stephan Hartmann presents Metamesh's OpenCms Cloud eXtensions (OCCX) implementation, which is the foundation to provide a cloud-ready, distributable and cluster-ready version of OpenCms that implements an integration with Open Data Grid Platforms. Arash Kaffamanesh will present OCCX in action running on Eucalyptus Private Cloud and Amazon EC2 and shows you the first cloudified version of OpenCms with Auto-Scaling and Multi-Workplace support for your OpenCms SaaS initiatives.
ClojureScript allows developers to use the Clojure programming language to build applications that compile to JavaScript. This enables Clojure code to run in environments where JavaScript is supported, like web browsers and mobile apps. ClojureScript leverages the Google Closure compiler and library to provide whole program optimization of Clojure code compiling to JavaScript.
Why you should be excited about ClojureScriptelliando dias
ClojureScript allows Clojure code to compile to JavaScript. Created by Rich Hickey and friends, it provides optimizations for performance while maintaining readability and abstraction. As a Lisp for JavaScript, ClojureScript controls complexity on the web and benefits from JavaScript's status as a compilation target for many languages.
Functional Programming with Immutable Data Structureselliando dias
1. The document discusses the advantages of functional programming with immutable data structures for multi-threaded environments. It argues that shared mutable data and variables are fundamentally flawed concepts that can lead to bugs, while immutable data avoids these issues.
2. It presents Clojure as a functional programming language that uses immutable persistent data structures and software transactional memory to allow for safe, lock-free concurrency. This approach allows readers and writers to operate concurrently without blocking each other.
3. The document makes the case that Lisp parentheses in function calls uniquely define the tree structure of computations and enable powerful macro systems, homoiconicity, and structural editing of code.
O documento lista e descreve as principais partes de um contêiner de carga seco, incluindo o painel frontal, laterais, traseira, teto, piso e estrutura inferior. Muitos componentes como painéis laterais, travessas do teto e fundo são numerados de acordo com sua localização. As portas traseiras contêm quadros, painéis, dobradiças e barras de fechamento.
O documento discute a história da geometria projetiva, desde Euclides até seu uso em computação gráfica. Aborda figuras-chave como Pascal, que foi pioneiro na área, e como a perspectiva foi aplicada nas artes ao longo dos séculos.
Polyglot and Poly-paradigm Programming for Better Agilityelliando dias
This document discusses the benefits of polyglot and poly-paradigm programming approaches for building more agile applications. It describes how using multiple languages and programming paradigms can optimize both performance and developer productivity. Specifically, it suggests that statically-typed compiled languages be used for core application components while dynamically-typed scripting languages connect and customize these components. This approach allows optimizing areas that require speed/efficiency separately from those requiring flexibility. The document also advocates aspects and functional programming to address cross-cutting concerns and concurrency challenges that arise in modern applications.
This document discusses JavaScript libraries and frameworks. It provides an overview of some popular options like jQuery, Prototype, Dojo, MooTools, and YUI. It explains why developers use libraries, such as for faster development, cross-browser compatibility, and animation capabilities. The document also discusses how libraries resemble CSS and use selector syntax. Basic examples are provided to demonstrate common tasks like hover effects and row striping. Factors for choosing a library are outlined like maturity, documentation, community, and licensing. The document concludes by explaining how to obtain library code from project websites or Google's AJAX Libraries API.
How to Make an Eight Bit Computer and Save the World!elliando dias
This document summarizes a talk given to introduce an open source 8-bit computer project called the Humane Reader. The talk outlines the goals of providing a cheap e-book reader and computing platform using open source tools. It describes the hardware design which uses an AVR microcontroller and interfaces like video output, SD card, and USB. The talk also covers using open source tools for development and sourcing low-cost fabrication and assembly. The overall goals are to create an inexpensive device that can provide educational resources in developing areas.
Ragel is a parser generator that compiles to various host languages including Ruby. It is useful for parsing protocols and data formats and provides faster parsing than regular expressions or full LALR parsers. Several Ruby projects like Mongrel and Hpricot use Ragel for tasks like HTTP request parsing and HTML parsing. When using Ragel with Ruby, it can be compiled to Ruby code directly, which is slow, or a C extension can be written for better performance. The C extension extracts the parsed data from Ragel and makes it available to Ruby.
A Practical Guide to Connecting Hardware to the Webelliando dias
This document provides an overview of connecting hardware devices to the web using the Arduino platform. It discusses trends in electronics and computing that make this easier, describes the Arduino hardware and software, and covers various connection methods including directly to a computer, via wireless modems, Ethernet shields, and services like Pachube that allow sharing sensor data over the internet. The document aims to demonstrate how Arduinos can communicate with other devices and be used to build interactive systems.
O documento introduz o Arduino, uma plataforma de desenvolvimento open-source. Discute as características e componentes do Arduino, incluindo microcontroladores, software e exemplos de código. Também fornece instruções básicas sobre como programar o Arduino usando linguagem C.
O documento apresenta um mini-curso introdutório sobre Arduino, abordando o que é a plataforma Arduino, como é estruturado seu hardware, como programá-lo, exemplos básicos de código e aplicações possíveis como controle residencial e robótica.
The document discusses various functions for working with datasets in the Incanter library for Clojure. It describes how to create, read, save, select rows and columns from, and sort datasets. Functions are presented for building datasets from sequences, reading datasets from files and URLs, saving datasets to files and databases, selecting single or multiple columns, and filtering rows based on conditions. The document also provides an overview of the Incanter library and its various namespaces for statistics, charts, and other functionality.
Rango is a lightweight Ruby web framework built on Rack that aims to be more robust than Sinatra but smaller than Rails or Merb. It is inspired by Django and Merb, uses Ruby 1.9, and supports features like code reloading, Bundler, routing, rendering, and HTTP error handling. The documentation provides examples and details on using Rango.
Fab.in.a.box - Fab Academy: Machine Designelliando dias
This document describes the design of a multifab machine called MTM. It includes descriptions of the XY stage and Z axis drive mechanisms, as well as the tool heads and network used to control the machine. Key aspects of the design addressed include the stepper motor selection, drive electronics, motion control firmware, and use of a virtual machine environment and circular buffer to enable distributed control of the machine. Strengths of the design include low inertia enabling high acceleration, while weaknesses include low basic resolution and stiffness unsuitable for heavy milling.
The Digital Revolution: Machines that makeselliando dias
The document discusses the results of a study on the effects of a new drug on memory and cognitive function in older adults. The double-blind study involved giving either the new drug or a placebo to 100 volunteers aged 65-80 over a 6 month period. Testing showed those receiving the drug experienced statistically significant improvements in short-term memory retention and processing speed compared to the placebo group.
The document discusses using Clojure for Hadoop programming. Clojure is a dynamic functional programming language that runs on the Java Virtual Machine. The document provides an overview of Clojure and how its features like immutability and concurrency make it well-suited for Hadoop. It then shows examples of implementing Hadoop MapReduce jobs using Clojure by defining mapper and reducer functions.
This document provides an overview of Hadoop, including:
1) Hadoop solves the problems of analyzing massively large datasets by distributing data storage and analysis across multiple machines to tolerate node failure.
2) Hadoop uses HDFS for distributed data storage, which shards massive files across data nodes with replication for fault tolerance, and MapReduce for distributed data analysis by sending code to the data.
3) The document demonstrates MapReduce concepts like map, reduce, and their composition with an example job.
Hadoop and Hive Development at Facebookelliando dias
Facebook generates large amounts of user data daily from activities like status updates, photo uploads, and shared content. This data is stored in Hadoop using Hive for analytics. Some key facts:
- Facebook adds 4TB of new compressed data daily to its Hadoop cluster.
- The cluster has 4800 cores and 5.5PB of storage across 12TB nodes.
- Hive is used for over 7500 jobs daily and by around 200 engineers/analysts monthly.
- Performance improvements to Hive include lazy deserialization, map-side aggregation, and joins.
Multi-core Parallelization in Clojure - a Case Studyelliando dias
The document describes a case study on using Clojure for multi-core parallelization of the K-means clustering algorithm. It provides background on parallel programming concepts, an introduction to Clojure, and details on how the authors implemented a parallel K-means algorithm in Clojure using agents and software transactional memory. They present results showing speedups from parallelization and accuracy comparable to R's implementation on both synthetic and real-world datasets.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Monitoring Java Application Security with JDK Tools and JFR Events
Memcached
1. Memcached
http://download.tangent.org/talks/Memcached%20Study.pdf
Thursday, April 23, 2009
2. memcached is a high-performance, distributed memory object
caching system, generic in nature, but intended for use in
speeding up dynamic web applications by alleviating
database load.
Thursday, April 23, 2009
6. LiveJournal
• Origin of memcached
• 30G of cache. Terabytes of data
• Writes to DB based on reads from the DB,
not cache
Thursday, April 23, 2009
7. Mixi
• Memcached on dedicated servers
• 300 servers in production (reuse old
MySQL Servers)
• 4/8 gigs of Memory
• 15,000 qps / 400Mbps throughput
Thursday, April 23, 2009
13. Server
• Slab Allocator
• Libevent based
• Simple Protocol (no xml)
• Server has Internal Hash Table
• Servers know nothing about each other
Thursday, April 23, 2009
14. Clients
• Client hashes Key to Server List (distribution)
• Serializes the Object (server just likes byte arrays)
• Compresses data
Thursday, April 23, 2009
16. So what should I ask?
• How do I dump data?
• How is it redundant? *
• How does it handle failover?*
• How does it authenticate?
Thursday, April 23, 2009
17. Details on the Server?
• set/get/replace/add
• append/prepend
• increment/decrement
• cas (compare and swap atomic!)
• stats (detail)
Thursday, April 23, 2009
18. Platforms
• FreeBSD and Linux are top tier
• Windows exists
• Solaris (as of 1.2.5)
• OSX Good Support
Thursday, April 23, 2009
21. # Set up client object
require 'memcache'
servers = ['127.0.0.1:43042', '127.0.0.1:43043']
CACHE = MemCache.new(servers, :namespace => 'my_app')
Thursday, April 23, 2009
22. # Get; fall through to Rails' MySQL load if missing
key = "recent_posts"
# Try to get; returns nil on failure
posts = CACHE.get(key)
unless posts
# Load from DB
posts = Post.find(:all, :limit => 10, :order => 'id DESC')
# Cache the new value, which is automatically serialized
CACHE.set(key), posts, 60
end
Thursday, April 23, 2009
23. # Ghetto locking implementation for memcache-client
# from http://fauna.rubyforge.org/svn/interlock/trunk/lib/interlock/lock.rb
def lock(key, lock_expiry = 30, retries = 5)
retries.times do |count|
# Try to acquire the lock
response = CACHE.add("lock:#{key}", "Locked by #{Process.pid}", lock_expiry)
if response == "STOREDrn"
# We got it
begin
# Yield the current value to the closure
value = yield(CACHE.get(key))
# Set the new value returned from the closure CACHE.set(key, value)
# We're done ('ensure' block will still run)
return value
ensure
# Release the lock
CACHE.delete("lock:#{key}")
end
else
# Exponentially back off requests if the lock can't be acquired
sleep((2**count) / 2.0)
end
end
# We waited and waited but our turn never came
raise MemCacheError, "Couldn't acquire lock for #{key}"
end
Thursday, April 23, 2009
25. /**
* Initalize Memcache object and add local server 127.0.0.1 using port 11211
*/
$cache = new Memcache;
$cache->addServer("127.0.0.1",11211);
Thursday, April 23, 2009
26. /** Look in cache, if not found, set object (misses null) **/
if (!$user = $cache->get($user_key)) {
/**
* Set object with expire of 1 hour and no compression
*/
$cache->set($user_key,$value,NULL, $expire);
$user = $value;
}
Thursday, April 23, 2009
27. /* Get user counter value (does not handle add fail) */
if (!$counter = $cache->get($counter_key)) {
/* No counter, set to 1 */
$cache->add($counter_key,1);
} else {
/* Increment by 1 */
$cache->increment($counter_key);
}
Thursday, April 23, 2009
28. /* Print out stats from memcached server */
print_r($cache->getStats());
/* Print out extended stats from memcached server */
print_r($cache->getExtendedStats());
/* Flush memcached (bad idea in production)*/
var_dump($cache->flush());
Thursday, April 23, 2009
37. lighttpd/mod_memcache
• Cache files from disk
• Specify mime/types
• Create Expire Times
Thursday, April 23, 2009
38. Apache (mod_memcached)
• CAS operations exposed
• GET/PUT/DELETE operations
• Still Alpha
• (pandoraport.com!)
Thursday, April 23, 2009
39. NGINX
• Support variable Time outs
• Based on Perl
• http://wiki.codemongers.com/
NginxHttpMemcachedModule
• http://www.igvita.com/2008/02/11/nginx-
and-memcached-a-400-boost/
Thursday, April 23, 2009
43. Installation
• CREATE FUNCTION memc_servers_add RETURNS INT SONAME
"libmemcached_functions_mysql.so";
• CREATE FUNCTION memc_set RETURNS INT SONAME
"libmemcached_functions_mysql.so";
• CREATE FUNCTION memc_get RETURNS STRING SONAME
"libmemcached_functions_mysql.so";
Thursday, April 23, 2009
44. Functions Available
• memc_servers_set();
• memc_servers_behavior_set()
• memc_set()
• memc_get()
• memc_append()
• memc_prepend()
Thursday, April 23, 2009
46. Trigger
DROP TRIGGER IF EXISTS feed_insert;
CREATE TRIGGER feed_insert BEFORE INSERT ON feeds FOR EACH ROW
BEGIN SET @mm= memc_set(concat('feeds:',md5(NEW.url)),
NEW.url);END |
insert into feeds (url) values ('http://grazr.com/feedlist.xml');
select memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml')));
+------------------------------------------------------------------+|
memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml'))) |
+------------------------------------------------------------------+|
http://grazr.com/feedlist.xml |
+------------------------------------------------------------------+
Thursday, April 23, 2009
47. Memcached Replication
via MySQL
INSERT INTO table_a VALUES (“key”, “value”, “values”);
INSERT INTO blackhole_table VALUES (memc_delete(“key”));
Thursday, April 23, 2009
49. Limits
• Key Size (250 bytes) (1.2 <)
• Data Size (under 1 megabyte)
• 32bit/64bit (maximum size of the process)
• Maxbytes (limits item cache, not everything!)
Thursday, April 23, 2009
50. LRU
• Least recently accessed items are up for
eviction
• One LRU exists per “slab class”
• LRU evictions don't need to be common
• Can be common if you set expiration to 0
• Monitor evictions via 'stats items' command
Thursday, April 23, 2009
51. Threads
• No single-threading in versions past 1.2
• Great for large instances (16G+)
• Also great for large multiget requests
• Improvements in 1.3+
• Don't set too many threads
• One per CPU
• No more than 8 total for 1.2
Thursday, April 23, 2009
52. Slab Allocator
• Memory permanently allocated from OS
• Classes created by chunk size
• Cannot (presently) reassign slab pages
• Carefully use 'stats sizes' to test efficiency
Thursday, April 23, 2009
53. Binary Protocol
24 HEADER Required
bytes
EXTRA FIELD
(Command Specific)
Sizes are
defined in KEY As needed
the header
VALUE
* Values must be in network byte order.
Thursday, April 23, 2009
54. Binary Protocol
MAGIC Opcode Key Length
(1 byte) (1 byte) (2 bytes)
Extra Length Data Type Reserved
(1 byte) (1 byte) (2 bytes)
Total Body Length
(4 bytes)
Opaque
(4 bytes)
CAS (Compare and Swap)
(8 bytes)
Thursday, April 23, 2009
55. Response Header
MAGIC Opcode Key Length
(1 byte) (1 byte) (2 bytes)
Extra Length Data Type Status
(1 byte) (1 byte) (2 bytes)
Total Body Length
(4 bytes)
Opaque
(4 bytes)
CAS (Compare and Swap)
(8 bytes)
Thursday, April 23, 2009
57. Protocol
• Telnet’able (see doc/protocol.txt)
• Store commands are binary (no escaping)
• WireShark support for Binary Protocol
Thursday, April 23, 2009
68. Connections
• Don't fear setting -c (max conns) too high!
• Watch 'listen_disabled_num' (1.2.8+)
• Audit any firewalls on the system (can run
out of firewall states!)
• Watch for connection overload. Probably an
application bug
Thursday, April 23, 2009
69. Memory
• Don't set -m (max bytes) too high
• -m only controls memory for stored values
• Other memory is used for connection
handling, LRU tracking, the hash table, etc
• Especially if running under 32bit (-m 2048
bad!)
• Careful of -k. Could force server to swap
• Monitor server well! If you are swapping,
memcached will be slow
Thursday, April 23, 2009
70. Things to remember!
• Memcached is great for scaling, but abuse
will slow down your page rendering time
• Fetching 10,000 separate keys to build a
page will not be fast
• Use persistent connections if possible
• Don't fetch more than you need. Takes time
to serialize/deserialize 100k+ of data
Thursday, April 23, 2009
71. Shrink data!
• Especially if you use multiget, try to use
shorter keys
• “fooooooooooooooooooooooo-$MD5”
will take more data packets than “foo1”
• Try using smaller keys anyway, saves
memory if your data is small too
Thursday, April 23, 2009
72. Future Solutions!
• Evaluate the binary protocol in 1.3
• New tricks will be possible, such as
smashing many commands into fewer
packets
• 'noreply' commands. Don't wait for the
response if you don't have to
• ('noreply' is in 1.2, but not recommended
for use)
Thursday, April 23, 2009
73. 1.2.8
• Bugfixes
• New stats
• listen_disabled_num
• cmd_flush (are you flushing your cache
and not realizing it? :) )
• evicted_time under 'stats items'
• Only bugfixes and minor features planned
for 1.2 series
Thursday, April 23, 2009
74. 1.3.* Beta
• Scalability improvements
• New stats (per-slab get/miss/etc counters)
• Binary protocol
• Memory optimizations (disable CAS
support)
Thursday, April 23, 2009
75. ASCII Protocol vs Binary Protocol (memcached-1.3.1 Development Branch)
task completion time (secs)
ASCII Protocol
Binary Protocol
concurrent connections
Binary Protocol
Thursday, April 23, 2009
76. Future
• 1.4.0 stable
• Multiple Engine Support
• Durable
• Highly Threaded
• New Slab Features
Thursday, April 23, 2009
77. More Resources...
• http://danga.com/memcached/
• #memcached on Freenode
• http://tangent.org/552/libmemcached.html
• http://groups.google.com/groups/
memcached
• http://dev.mysql.com/doc/refman/5.1/en/ha-
memcached.html
Thursday, April 23, 2009