This document provides an overview of Q4M, a message queue storage engine for MySQL. It describes the background and motivation for developing Q4M, how message queues work, basic usage of Q4M including creating tables and consuming messages, and several applications that use Q4M. The key functions for subscribing to messages - queue_wait(), queue_end(), and queue_abort() - are explained in detail.
Using Q4M - a message queue for MySQL #osdc.twKazuho Oku
This document provides an overview of Q4M, a message queue storage engine plugin for MySQL. It was created by Kazuho Oku and is used by several major Japanese companies for applications like asynchronous updates, delaying peak database demands, and connecting distant servers. The document discusses message queue concepts, applications of Q4M, installing and configuring Q4M, and basic usage like creating tables, inserting data, and selecting data from Q4M tables.
This document summarizes the steps to deploy a custom Linux operating system using PXE boot and discusses some key components:
1) It outlines the process of setting up PXE boot using a DHCP and TFTP server to serve boot files to clients over the network, allowing them to boot directly from the network.
2) It then describes building a customized initramfs image and kernel that contains drivers and packages needed to access storage and complete the OS installation.
3) Finally, it summarizes the process of installing the customized Linux distribution by downloading packages over the network, partitioning and formatting disks, and installing the target OS before rebooting.
Carriage of timed subtitles and graphics in MP4Cyril Concolato
The document discusses carrying timed text and graphics in ISOBMFF. It proposes making the carriage of these media more generic and future-proof. Specifically, it suggests allowing 'meta' boxes in movie fragments to carry timed data samples and using either timed metadata samples or 'meta' boxes as samples to store text/graphics documents and their resources. It also comments on issues in the current working draft.
Comments on carriage of timed text and visual overlays in MP4Cyril Concolato
This document discusses proposals for representing timed text streams in the ISO Base Media File Format (ISOBMFF). It addresses issues related to parsing WebVTT files, splitting cues that overlap, and sample entry types for different types of timed text tracks. It proposes adding a CueDurationBox to simplify cue splitting. It also discusses questions around sample entry design and placement of configuration strings. The overall aim is to define a conservative round-trip between WebVTT files and ISOBMFF that avoids forward compatibility issues.
NetApp commissioned Demartek to evaluate its FAS3240, one of the members of its full line of unified storage solutions, for its ability to handle a full load of mixed traffic types simultaneously.
The document discusses various techniques for global server load balancing (GSLB) including DNS-based methods, host route injection (HRI), and a new proposed technique called virtual block injection (VBI). It outlines the requirements for internet data centers (IDCs), describes existing GSLB methods and their pros and cons, and analyzes the potential benefits and limitations of VBI which aims to simplify routing changes compared to other methods. The document also considers factors that determine the applicability of different GSLB solutions depending on network topology and control.
GlusterFS is a clustered file system that aggregates multiple storage servers over InfiniBand RDMA into a large parallel network file system. It was designed for scalable capacity and I/O throughput beyond petabytes, reliability through its non-stop storage architecture with no single point of failure, and ease of manageability through self-healing capabilities. Benchmark tests using 16 storage servers and 64 clients achieved a peak aggregated read throughput of 13GBps over InfiniBand verbs transport.
Using Q4M - a message queue for MySQL #osdc.twKazuho Oku
This document provides an overview of Q4M, a message queue storage engine plugin for MySQL. It was created by Kazuho Oku and is used by several major Japanese companies for applications like asynchronous updates, delaying peak database demands, and connecting distant servers. The document discusses message queue concepts, applications of Q4M, installing and configuring Q4M, and basic usage like creating tables, inserting data, and selecting data from Q4M tables.
This document summarizes the steps to deploy a custom Linux operating system using PXE boot and discusses some key components:
1) It outlines the process of setting up PXE boot using a DHCP and TFTP server to serve boot files to clients over the network, allowing them to boot directly from the network.
2) It then describes building a customized initramfs image and kernel that contains drivers and packages needed to access storage and complete the OS installation.
3) Finally, it summarizes the process of installing the customized Linux distribution by downloading packages over the network, partitioning and formatting disks, and installing the target OS before rebooting.
Carriage of timed subtitles and graphics in MP4Cyril Concolato
The document discusses carrying timed text and graphics in ISOBMFF. It proposes making the carriage of these media more generic and future-proof. Specifically, it suggests allowing 'meta' boxes in movie fragments to carry timed data samples and using either timed metadata samples or 'meta' boxes as samples to store text/graphics documents and their resources. It also comments on issues in the current working draft.
Comments on carriage of timed text and visual overlays in MP4Cyril Concolato
This document discusses proposals for representing timed text streams in the ISO Base Media File Format (ISOBMFF). It addresses issues related to parsing WebVTT files, splitting cues that overlap, and sample entry types for different types of timed text tracks. It proposes adding a CueDurationBox to simplify cue splitting. It also discusses questions around sample entry design and placement of configuration strings. The overall aim is to define a conservative round-trip between WebVTT files and ISOBMFF that avoids forward compatibility issues.
NetApp commissioned Demartek to evaluate its FAS3240, one of the members of its full line of unified storage solutions, for its ability to handle a full load of mixed traffic types simultaneously.
The document discusses various techniques for global server load balancing (GSLB) including DNS-based methods, host route injection (HRI), and a new proposed technique called virtual block injection (VBI). It outlines the requirements for internet data centers (IDCs), describes existing GSLB methods and their pros and cons, and analyzes the potential benefits and limitations of VBI which aims to simplify routing changes compared to other methods. The document also considers factors that determine the applicability of different GSLB solutions depending on network topology and control.
GlusterFS is a clustered file system that aggregates multiple storage servers over InfiniBand RDMA into a large parallel network file system. It was designed for scalable capacity and I/O throughput beyond petabytes, reliability through its non-stop storage architecture with no single point of failure, and ease of manageability through self-healing capabilities. Benchmark tests using 16 storage servers and 64 clients achieved a peak aggregated read throughput of 13GBps over InfiniBand verbs transport.
The document discusses new features in Windows Server 2012 Hyper-V and System Center 2012 Virtual Machine Manager (VMM) SP1 for implementing private clouds, including increased scalability for VMs and hosts in a cluster, live storage migration capabilities, enhanced networking functionality through Hyper-V Network Virtualization, improved storage allocation and management, and expanded self-service user and delegated administration roles.
The document discusses using RDMA over Converged Ethernet (RoCE) for high performance data movement between KISTI and LBL. It describes how RDMA allows direct data placement through one-sided operations like RDMA write and read, avoiding CPU overhead. It also discusses challenges in using RDMA over wide area networks and for bulk data transfers, and experiments using GridFTP and a prototype FTP-like transfer application over RDMA.
Presentation on OSGi Cloud Ecosystems as presented during EclipseCon Europe 2012 (http://www.eclipsecon.org/europe2012/sessions/osgi-and-cloud-computing)
This document summarizes a presentation on improvements to RMF's Parallel Sysplex instrumentation over recent years. Some key points covered include:
1) Structure-level CPU reporting in SMF 74-4 allows for capacity planning at the individual structure level and examining CPU consumption of different structures.
2) Enhancements help match CPU data between SMF 70-1 and 74-4 to get a complete picture of Coupling Facility CPU usage.
3) Additional instrumentation provides useful information on topics like structure duplexing performance, XCF traffic patterns, and Coupling Facility link details.
Since 2003, the Atom format has been used as yet another feed format like RSS, but the Atom protocol opens the door to far more uses of Atom as both a means for distribution and publication. Ben Ramsey introduces the Atom format and protocol, explaining how these can form the foundation of any publishing service and serve as a platform for rich Internet applications.
High speed networks and Java (Ryan Sciampacone)Chris Bailey
The document discusses high-speed networks and their performance implications for Java development. It summarizes the results of tests comparing the throughput of C and Java programs when sending data over different network types, including InfiniBand and its protocols. The tests found that for small payloads, the C programs generally had higher throughput than the Java programs. But as payload sizes increased above around 128KB, the throughput of the Java programs approached that of the C programs. The document also discusses challenges for Java in exploiting features like remote direct memory access over high-speed networks.
This document summarizes a LINE Developer meetup in Kyoto on load balancing techniques. It discusses LINE's private cloud "Verda" and the load balancing architecture used including L3, L4, and L7 load balancers. It also covers programming load balancers using eBPF and XDP to improve performance by handling traffic directly on the NIC. Load testing is discussed using tools like TRex to generate and analyze traffic.
The document describes COSBench, a benchmark tool for evaluating the performance of cloud object storage services. It provides an overview of COSBench's key components, including its configurable workload definition file, controller for managing tests, and drivers for generating load. The document also shares sample results from using COSBench to measure the throughput and response times of OpenStack Swift in different configurations. It found that the proxy node's CPU was the bottleneck for larger workloads on one setup. The goal is to open source COSBench to help storage providers optimize performance.
Tungsten University: Setup and Operate Tungsten ReplicatorsContinuent
Do you have the background necessary to take full advantage of Tungsten Replicator in your environments? Tungsten offers enterprise-quality replication features in an open source package hosted on Google Code. This virtual course teaches you how to set up innovative topologies that solve complex replication problems. We start with single MySQL servers running MySQL replication and show a simple path migration path to Tungsten.
Course Topics
- Checking host and MySQL prerequisites
- Downloading code from http://code.google.com/p/tungsten-replicator/
- Installation using the tungsten-installer utility
- Transaction filtering using standard filters as well as customized filters you write yourself
- Enabling and managing parallel replication
- Configuring multi-master and fan-in using multiple replication services
- Backup and restore integration
- Troubleshooting replication problems
- Logging bugs and participating in the Tungsten Replicator community
Replication is a powerful technology that takes knowledge and planning to use effectively. We give you the background that makes replication easier to set up, and allows you to take full advantage of the Tungsten Replicator benefits. Learn how to configure and use it more effectively for your projects in the cloud as well as on-premises hardware.
Apache CloudStack is an open source cloud computing platform that provides infrastructure as a service. It was originally developed by Citrix and is now an Apache incubator project. It is hypervisor-agnostic and supports platforms like Xen, KVM, and VMware. It provides services for compute, storage, networking and a web-based user interface.
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, PythianAlex Gorbachev
Slides from MOW2010 presentation.
The presentation provides practical understanding of Oracle Clusterware/CRS and knowledge required for independent troubleshooting of Clusterware issues - why nodes are evicted, why resources don't start or fail for no reason. After the presentation, a DBA will know where to look for the answers instead of blindly running cluvfy.sh utility. The session includes demos of how to troubleshoot clusterware issues such as evictions. The presentation does goes into Oracle Clusterware internals but it's appropriate for all DBA's from beginners to experienced.
This document discusses front-end optimization techniques to improve site response time, including setting up content delivery networks (CDNs), image optimization, minifying and combining CSS and JavaScript files, gzip compression, browser caching, and preloading resources. It provides results from testing these techniques on a Joomla site and analyzing the impact on page load times and number of HTTP requests in different locations.
Enhanced introduction to CloudStack for the Geneva Java User Group. Includes Apache processes, DevCloud use cases, opportunities for Java developers and 4.0 release testing procedure.
The document discusses COSBench, a benchmark tool developed by Intel to evaluate the performance of cloud object storage services. It describes key components of COSBench including its configurable workload definition, drivers to generate load, and a web console to view results. The document also uses COSBench to analyze the performance of OpenStack Swift, finding that insufficient processing power can throttle overall performance.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.Takatoshi Matsuo
The document describes clustering PostgreSQL using Pacemaker for high availability. It discusses 3 main functions: 1) failover when the master node crashes, 2) switching between synchronous and asynchronous replication, and 3) managing old data. It provides examples of the basic actions for master failover and switching replication settings. It also discusses monitoring the timeline ID, recovery after failover, and transitioning between different PostgreSQL and Pacemaker states.
Optimize Performance of I/O-intensive Java applications Using Zero CopyIndicThreads
This session explains how you can improve performance of I/O-intensive Java™ applications through a technique called zero copy. Zero copy lets you to avoid redundant data copies between intermediate buffers and reduces number of context switches between user and kernel space.
Background: Many applications [Web Servers, FTP-like services] serve a significant amount of static content, which amounts to reading data off of a disk and writing the exact data back to the response socket. Each time data traverses the user-kernel boundary; it must be copied, which consumes CPU cycles and memory bandwidth. Fortunately, we can eliminate these copies through a technique called — zero copy.
The Java class libraries support zero copy on Linux and UNIX systems through the transferTo() method in java.nio.channels.FileChannel.
Session Agenda: The session will initially focus on “Zero Copy” concept and its relevance in Data transfer applications. The traditional approach of transferring data b/w processes using File and Socket I/O will be explained in detail. It will demonstrate the overhead incurred when using traditional copy semantics, and will show how transferTo() achieves better performance. transferTo() API brings down the time 65% compared to traditional approach.
Summary
The Session demonstrates performance advantages of using transferTo() compared to the traditional approach. Intermediate buffer copies — even those hidden in the kernel — can have a measurable cost. In applications that do a great deal of copying of data between channels, zero-copy technique can offer a significant performance improvement.
OSGi Cloud Ecosystems (OSGi Users Forum Germany)David Bosschaert
This document discusses OSGi Cloud Ecosystems, which provide a highly dynamic OSGi environment where cloud nodes/VMs, bundles, and services can dynamically come and go. In an OSGi Cloud Ecosystem, multiple OSGi frameworks run in separate VMs/cloud nodes and are bound together via an ecosystem-wide discovery mechanism. A provisioner orchestrates the ecosystem by remotely deploying bundles to frameworks based on application topology and runtime conditions. This allows building composite cloud applications where each entity runs on a different node and can scale dynamically.
Living the Easy Life with Rules-Based Autonomic Database ClustersLinas Virbalas
Our business at Continuent is development of database clusters with highly simplified management and the ability to be operated unattended for prolonged periods of time. As part of our Tungsten Clustering product we developed an extensible, low-latency, fault-tolerant management framework for database clusters built around a core of group communications and business rules. We have found that our system is easy to maintain and to extend. For example, an extension to switch virtual IP addresses in the event of a database node failure was implemented in an afternoon as a set of two rules and a single bash script. In our talk we will cover the following:
* Basic architecture of a rules-based management framework for databases
* Introduction to business rules, with code examples showing how they work to repair problems ranging from simple process failures to network partitions
* A quick demo of business rules in operation.
* Finally, some thoughts on the benefits of the approach and our experiences (good and bad) with autonomic management of database clusters.
This is an approach to management that we believe will be of interest to anyone who cares about keeping important data highly available as well anyone interested in learning about rules technology.
Rabbit MQ is a message broker software that implements the Advanced Message Queuing Protocol (AMQP) standard. It allows for push-style communication where the server initiates transactions by sending messages to clients. Rabbit MQ supports publish/subscribe and queuing of messages across platforms. It provides reliability, security, and scalability for message delivery. Common uses of Rabbit MQ include notifications, messaging, and communication between services.
17) 11 (may, 2003) squid master this proxy serverswarup1435
This document provides guidance on optimizing a Squid proxy server for high performance in a corporate environment. It recommends:
1. Using a server with a fast CPU, large RAM, and high-end SCSI hard drive. Installing the latest Linux kernel and using the ReiserFS file system.
2. Configuring Squid to use multiple threads, increasing the number of file descriptors, and compiling with async I/O support.
3. Optimizing the Squid configuration file by increasing cache_mem, tuning the cache_dir settings, disabling unnecessary logs, and adjusting other parameters.
4. Modifying the startup script to set high file descriptor limits and IP port ranges for maximum throughput
MessagePack(msgpack): Compact and Fast Serialization LibraryTakatoshi Kondo
This is a presentation on the lightning talks at C++Now2015.
http://cppnow.org/
I introduce a portable binary format named MessagePack and the library supports the format named msgpack-c. The msgpack-c library provides a zero-copy deserialization. I compare MessagePack and JSON, and the msgpack-c library and the Boost.Serialization library.
The document discusses new features in Windows Server 2012 Hyper-V and System Center 2012 Virtual Machine Manager (VMM) SP1 for implementing private clouds, including increased scalability for VMs and hosts in a cluster, live storage migration capabilities, enhanced networking functionality through Hyper-V Network Virtualization, improved storage allocation and management, and expanded self-service user and delegated administration roles.
The document discusses using RDMA over Converged Ethernet (RoCE) for high performance data movement between KISTI and LBL. It describes how RDMA allows direct data placement through one-sided operations like RDMA write and read, avoiding CPU overhead. It also discusses challenges in using RDMA over wide area networks and for bulk data transfers, and experiments using GridFTP and a prototype FTP-like transfer application over RDMA.
Presentation on OSGi Cloud Ecosystems as presented during EclipseCon Europe 2012 (http://www.eclipsecon.org/europe2012/sessions/osgi-and-cloud-computing)
This document summarizes a presentation on improvements to RMF's Parallel Sysplex instrumentation over recent years. Some key points covered include:
1) Structure-level CPU reporting in SMF 74-4 allows for capacity planning at the individual structure level and examining CPU consumption of different structures.
2) Enhancements help match CPU data between SMF 70-1 and 74-4 to get a complete picture of Coupling Facility CPU usage.
3) Additional instrumentation provides useful information on topics like structure duplexing performance, XCF traffic patterns, and Coupling Facility link details.
Since 2003, the Atom format has been used as yet another feed format like RSS, but the Atom protocol opens the door to far more uses of Atom as both a means for distribution and publication. Ben Ramsey introduces the Atom format and protocol, explaining how these can form the foundation of any publishing service and serve as a platform for rich Internet applications.
High speed networks and Java (Ryan Sciampacone)Chris Bailey
The document discusses high-speed networks and their performance implications for Java development. It summarizes the results of tests comparing the throughput of C and Java programs when sending data over different network types, including InfiniBand and its protocols. The tests found that for small payloads, the C programs generally had higher throughput than the Java programs. But as payload sizes increased above around 128KB, the throughput of the Java programs approached that of the C programs. The document also discusses challenges for Java in exploiting features like remote direct memory access over high-speed networks.
This document summarizes a LINE Developer meetup in Kyoto on load balancing techniques. It discusses LINE's private cloud "Verda" and the load balancing architecture used including L3, L4, and L7 load balancers. It also covers programming load balancers using eBPF and XDP to improve performance by handling traffic directly on the NIC. Load testing is discussed using tools like TRex to generate and analyze traffic.
The document describes COSBench, a benchmark tool for evaluating the performance of cloud object storage services. It provides an overview of COSBench's key components, including its configurable workload definition file, controller for managing tests, and drivers for generating load. The document also shares sample results from using COSBench to measure the throughput and response times of OpenStack Swift in different configurations. It found that the proxy node's CPU was the bottleneck for larger workloads on one setup. The goal is to open source COSBench to help storage providers optimize performance.
Tungsten University: Setup and Operate Tungsten ReplicatorsContinuent
Do you have the background necessary to take full advantage of Tungsten Replicator in your environments? Tungsten offers enterprise-quality replication features in an open source package hosted on Google Code. This virtual course teaches you how to set up innovative topologies that solve complex replication problems. We start with single MySQL servers running MySQL replication and show a simple path migration path to Tungsten.
Course Topics
- Checking host and MySQL prerequisites
- Downloading code from http://code.google.com/p/tungsten-replicator/
- Installation using the tungsten-installer utility
- Transaction filtering using standard filters as well as customized filters you write yourself
- Enabling and managing parallel replication
- Configuring multi-master and fan-in using multiple replication services
- Backup and restore integration
- Troubleshooting replication problems
- Logging bugs and participating in the Tungsten Replicator community
Replication is a powerful technology that takes knowledge and planning to use effectively. We give you the background that makes replication easier to set up, and allows you to take full advantage of the Tungsten Replicator benefits. Learn how to configure and use it more effectively for your projects in the cloud as well as on-premises hardware.
Apache CloudStack is an open source cloud computing platform that provides infrastructure as a service. It was originally developed by Citrix and is now an Apache incubator project. It is hypervisor-agnostic and supports platforms like Xen, KVM, and VMware. It provides services for compute, storage, networking and a web-based user interface.
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, PythianAlex Gorbachev
Slides from MOW2010 presentation.
The presentation provides practical understanding of Oracle Clusterware/CRS and knowledge required for independent troubleshooting of Clusterware issues - why nodes are evicted, why resources don't start or fail for no reason. After the presentation, a DBA will know where to look for the answers instead of blindly running cluvfy.sh utility. The session includes demos of how to troubleshoot clusterware issues such as evictions. The presentation does goes into Oracle Clusterware internals but it's appropriate for all DBA's from beginners to experienced.
This document discusses front-end optimization techniques to improve site response time, including setting up content delivery networks (CDNs), image optimization, minifying and combining CSS and JavaScript files, gzip compression, browser caching, and preloading resources. It provides results from testing these techniques on a Joomla site and analyzing the impact on page load times and number of HTTP requests in different locations.
Enhanced introduction to CloudStack for the Geneva Java User Group. Includes Apache processes, DevCloud use cases, opportunities for Java developers and 4.0 release testing procedure.
The document discusses COSBench, a benchmark tool developed by Intel to evaluate the performance of cloud object storage services. It describes key components of COSBench including its configurable workload definition, drivers to generate load, and a web console to view results. The document also uses COSBench to analyze the performance of OpenStack Swift, finding that insufficient processing power can throttle overall performance.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.Takatoshi Matsuo
The document describes clustering PostgreSQL using Pacemaker for high availability. It discusses 3 main functions: 1) failover when the master node crashes, 2) switching between synchronous and asynchronous replication, and 3) managing old data. It provides examples of the basic actions for master failover and switching replication settings. It also discusses monitoring the timeline ID, recovery after failover, and transitioning between different PostgreSQL and Pacemaker states.
Optimize Performance of I/O-intensive Java applications Using Zero CopyIndicThreads
This session explains how you can improve performance of I/O-intensive Java™ applications through a technique called zero copy. Zero copy lets you to avoid redundant data copies between intermediate buffers and reduces number of context switches between user and kernel space.
Background: Many applications [Web Servers, FTP-like services] serve a significant amount of static content, which amounts to reading data off of a disk and writing the exact data back to the response socket. Each time data traverses the user-kernel boundary; it must be copied, which consumes CPU cycles and memory bandwidth. Fortunately, we can eliminate these copies through a technique called — zero copy.
The Java class libraries support zero copy on Linux and UNIX systems through the transferTo() method in java.nio.channels.FileChannel.
Session Agenda: The session will initially focus on “Zero Copy” concept and its relevance in Data transfer applications. The traditional approach of transferring data b/w processes using File and Socket I/O will be explained in detail. It will demonstrate the overhead incurred when using traditional copy semantics, and will show how transferTo() achieves better performance. transferTo() API brings down the time 65% compared to traditional approach.
Summary
The Session demonstrates performance advantages of using transferTo() compared to the traditional approach. Intermediate buffer copies — even those hidden in the kernel — can have a measurable cost. In applications that do a great deal of copying of data between channels, zero-copy technique can offer a significant performance improvement.
OSGi Cloud Ecosystems (OSGi Users Forum Germany)David Bosschaert
This document discusses OSGi Cloud Ecosystems, which provide a highly dynamic OSGi environment where cloud nodes/VMs, bundles, and services can dynamically come and go. In an OSGi Cloud Ecosystem, multiple OSGi frameworks run in separate VMs/cloud nodes and are bound together via an ecosystem-wide discovery mechanism. A provisioner orchestrates the ecosystem by remotely deploying bundles to frameworks based on application topology and runtime conditions. This allows building composite cloud applications where each entity runs on a different node and can scale dynamically.
Living the Easy Life with Rules-Based Autonomic Database ClustersLinas Virbalas
Our business at Continuent is development of database clusters with highly simplified management and the ability to be operated unattended for prolonged periods of time. As part of our Tungsten Clustering product we developed an extensible, low-latency, fault-tolerant management framework for database clusters built around a core of group communications and business rules. We have found that our system is easy to maintain and to extend. For example, an extension to switch virtual IP addresses in the event of a database node failure was implemented in an afternoon as a set of two rules and a single bash script. In our talk we will cover the following:
* Basic architecture of a rules-based management framework for databases
* Introduction to business rules, with code examples showing how they work to repair problems ranging from simple process failures to network partitions
* A quick demo of business rules in operation.
* Finally, some thoughts on the benefits of the approach and our experiences (good and bad) with autonomic management of database clusters.
This is an approach to management that we believe will be of interest to anyone who cares about keeping important data highly available as well anyone interested in learning about rules technology.
Rabbit MQ is a message broker software that implements the Advanced Message Queuing Protocol (AMQP) standard. It allows for push-style communication where the server initiates transactions by sending messages to clients. Rabbit MQ supports publish/subscribe and queuing of messages across platforms. It provides reliability, security, and scalability for message delivery. Common uses of Rabbit MQ include notifications, messaging, and communication between services.
17) 11 (may, 2003) squid master this proxy serverswarup1435
This document provides guidance on optimizing a Squid proxy server for high performance in a corporate environment. It recommends:
1. Using a server with a fast CPU, large RAM, and high-end SCSI hard drive. Installing the latest Linux kernel and using the ReiserFS file system.
2. Configuring Squid to use multiple threads, increasing the number of file descriptors, and compiling with async I/O support.
3. Optimizing the Squid configuration file by increasing cache_mem, tuning the cache_dir settings, disabling unnecessary logs, and adjusting other parameters.
4. Modifying the startup script to set high file descriptor limits and IP port ranges for maximum throughput
MessagePack(msgpack): Compact and Fast Serialization LibraryTakatoshi Kondo
This is a presentation on the lightning talks at C++Now2015.
http://cppnow.org/
I introduce a portable binary format named MessagePack and the library supports the format named msgpack-c. The msgpack-c library provides a zero-copy deserialization. I compare MessagePack and JSON, and the msgpack-c library and the Boost.Serialization library.
Phoenix Energy Performance Contracting PresentationICF_HCD
Energy performance contracting (EPC) allows organizations to implement energy efficiency projects through guaranteed savings contracts. An energy services company (ESCO) conducts energy assessments and implements efficiency upgrades like lighting, HVAC, and building envelope improvements. The ESCO guarantees that the energy savings will cover the costs of the project without upfront capital from the client. EPC provides clients capital improvements, utility funding opportunities, guaranteed savings, and other benefits. The U.S. Department of Housing and Urban Development also offers incentives for public housing authorities to pursue EPC through programs like frozen utility baselines and additional operating subsidies. Case studies show EPC projects achieving substantial energy and cost savings.
In Croatia, several Christmas traditions are observed:
1) On St. Lucia's Day (December 13th), wheat is sown as a symbol of life for Catholics.
2) An Advent wreath is used as a symbol of eternity and faith in God, with a candle lit each Sunday for family prayers.
3) St. Nicholas brings gifts on his name day if children are good, while Krampus brings birch branches if they are bad.
The document discusses the benefits of exercise for mental health. Regular physical activity can help reduce anxiety and depression and improve mood and cognitive functioning. Exercise causes chemical changes in the brain that may help protect against mental illness and improve symptoms for those who already suffer from conditions like anxiety and depression.
The document outlines the 7 essential laws of marketing to help businesses get and keep loyal, profitable customers. The laws include: 1) the law of mindset which states we get what we think we will get, 2) the law of uniqueness which is to pick a competitive advantage, 3) the law of longing in that people buy wants not needs, 4) the law of likability that ideal customers are loyal and profitable, 5) the law of appreciation that customers buy offers with value, 6) the law of visceral triggers that people move on pain and stay for pleasure, and 7) the law of distinction where customers focused on price don't know why to choose you. The document encourages the reader to apply these laws to
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...Hendrik Drachsler
Presentation given at Workshop 22 Jahrestagung der Gesellschaft für Medizinische Ausbildung, 27.09.2013, GMA2013, Graz, Austria.
http://portal.ou.nl/documents/363049/fd32b9eb-df7b-4b18-bf5a-d9560425625e
http://creativecommons.org/licenses/by-nc-sa/3.0/
Sopka, S., Druener, S., Stieger, L., Hynes, H., Stoyanov, S., Orrego, C., Secanell, M., Maher, B., Henn, P., Drachsler, H. (2013). Standardized Medical handovers – How to Learn, teach and implement? Workshop at Jahrestagung der Gesellschaft für Medizinische Ausbildung (Annual Meeting of the Society for Medical Education), Graz, Austria.
Do Hierarchical Positions Influence Participant’s Network Behaviour within C...Martin Rehm
This document summarizes a study that examined how hierarchical positions influence participant behavior within online communities of learning. The study analyzed social network data from an online training program involving 249 participants from different hierarchical levels. It tested hypotheses that higher-level participants would have more connections (H1-H2) and interact more within their own level (H3), indicating homophily. The results found no differences in how much participants read posts across levels. However, higher-level participants had significantly more reply connections, contradicting H3 and suggesting learning occurred across levels rather than within. The findings imply hierarchical positions can impact engagement but not prevent cross-level collaboration.
Austria's wine industry suffered from scandals in 1985, 2006, and 2008 that negatively impacted its reputation abroad. A 1985 scandal involved the addition of diethylene glycol, a poisonous alcohol, to wines to make them appear more mature. This was devastating in the short term but in the long run led to more stringent regulations and higher quality wines as the industry aimed to rebuild trust. However, Austrian wines still remain relatively unknown outside the country.
The document outlines the steps taken by an organization to transition to Agile. It assesses the current practices, identifies areas for improvement, runs two pilot projects using Scrum-derived frameworks with training and coaching. Post-pilot surveys found improvements in requirements, planning, quality, and teamwork. Based on this success, eight more teams were selected for Agile transition.
Managing Communities of Learning: The Impact and Role of FacilitatorsMartin Rehm
Communities of Learning (CoL) have become a popular methodology for organizational training initiatives (Rehm, 2009). While such communities allow employees to collaboratively upgrade their knowledge and skills, they also enable participants to get in touch with colleagues from all over the world (de Bruyn, 2004). Nonetheless, in order to become an effective educational resource, they also need to be cherished and protected (Paloff & Pratt, 2003). Yet, previous research has largely neglected the impact of facilitators on CoL for working professionals. The present study addresses this lacuna by conducting an exploratory examination on the role of facilitators in CoL of a global training program for an international organization. Our results indicate that working professionals generally do not seem to require much stimulation from facilitators to participate in discussions. Moreover, we find evidence that participants consider the input of facilitators as an added-value to the discussions, rather than a replacement for their own contributions. By addressing these issues, our study can serve as a springboard for facilitators to design and implement an effective teaching strategy for similar CoL in the future, thereby contributing to the overall quality of participants’ learning experience.
Issues and Considerations regarding Sharable Data Sets for Recommender System...Hendrik Drachsler
Assistant: Assistant:
Assistant A Assistant B Assistant C
Evaluation: Evaluation: Evaluation:
Metrics A Metrics B Metrics C
Metrics D Metrics E Metrics D
17
The document is a calendar showing months from January 2010 through January 2011. Each page displays a month's calendar with dates, days of the week and includes the month and year. The calendar is formatted in a grid with weeks running across pages to display full months consecutively.
OpenMP and MPI are two common APIs for parallel programming. OpenMP uses a shared memory model where threads have access to shared memory and can synchronize access. It is best for multi-core processors. MPI uses a message passing model where separate processes communicate by exchanging messages. It provides portability and is useful for distributed memory systems. Both have advantages like performance and portability but also disadvantages like difficulty of debugging for MPI. Future work may include improvements to threading support and fault tolerance in MPI.
This document discusses design patterns for running Apache Camel applications on Kubernetes. It begins with an introduction of the presenter and an overview of trends driving cloud native application development. It then discusses what cloud native means and reviews popular container orchestration platforms like Kubernetes, Docker Swarm and Mesos. The remainder of the document focuses on deployment patterns for Camel applications on Kubernetes, including packaging, health checks, configuration, service discovery, circuit breakers and retries.
Life in a Queue - Using Message Queue with djangoTareque Hossain
Brief introduction on message queue and how its relevant in web applications
How to tell if your web application could benefit from message queue
Common example of tasks that could benefit from message queues
Choosing a broker/protocol
What broker/protocol PBS Education chose and why
Message queue solution architecture
Brief introduction on celery/carrot
Writing a message queue task using celery
How to invoke a message queue taks
What happens when you invoke a task (walk through architecture)
How to write tasks efficiently
What are the things that are good to know when writing tasks (things we experienced at PBS Education)
Integration and Batch Processing on Cloud FoundryJoshua Long
This talk explores the new possibilities for scale by using Spring Integration, Spring Batch and RabbitMQ on Cloud Foundry, the open source PaaS from VMWare.
Introduction to First Commercial Memcached Service for CloudGear6
Gear6 introduced the first commercial Memcached service for cloud platforms on Dec. 8, 2009. The deck provides an overview of the new offering. More info at http://www.gear6.com/memcached-product/cloud-aws.
This document discusses Freescale's verification of the QorIQ communication platform containing the CoreNet fabric using SystemVerilog. It describes the verification challenges, methodology used, and verification IP developed. Key aspects included developing a SystemVerilog testbench, CoreNet VIP, and hierarchical verification. This approach successfully verified the CoreNet platform and resulted in first silicon sampling to customers within 3 weeks with no major functional bugs found.
This document summarizes the experiences of Tarjei Romtveit and Enok K. Eskeland in using RabbitMQ to scale the staging component of their social media processing pipeline. They found that RabbitMQ provided robust clustering, easy maintenance through plugins, and guaranteed message delivery through features like publisher confirms. It allowed them to move from a custom solution to an open-source message broker and scale to handle millions of messages per month. They demonstrated how RabbitMQ supports common messaging patterns like work queues and how they leveraged clustering, publisher confirms, and client libraries. Overall RabbitMQ helped stabilize and scale their system with 50% less maintenance.
Five cool ways the JVM can run Apache Spark fasterTim Ellison
The IBM JVM runs Apache Spark fast! This talk explains some of the findings and optimizations from our experience of running Spark workloads.
The talk was originally presented at the SparkEU Summit 2015 in Amsterdam.
In this session attendees will get an overview of a Typical (logical and physical) deployment Architecture for AEM and flow of content from Author instances to Production.
The venerable Servlet Container still has some performance tricks up its sleeve - this talk will demonstrate Apache Tomcat's stability under high load, describe some do's (and some don'ts!), explain how to performance test a Servlet-based application, troubleshoot and tune the container and your application and compare the performance characteristics of the different Tomcat connectors. The presenters will share their combined experience supporting real Tomcat applications for over 20 years and show how a few small changes can make a big, big difference.
This document discusses using Amazon S3 and EC2 services in Ruby on Rails applications. It describes how S3 provides redundant data storage and EC2 offers on-demand virtual servers. Various tools are presented for backing up to S3, hosting assets on S3, and using EC2 for extra computing capacity and load balancing. Load balancers like Swiftiply are also covered for distributing requests across backend servers.
Memcached is an open-source, distributed memory object caching system that provides high performance data storage for dynamic web applications. It allows scaling out by accessing the same data from multiple machines. Data is stored in an in-memory hash table and is distributed across nodes using consistent hashing. Memcached improves performance by caching objects like database query results, HTML fragments, and the results of API calls to reduce the load on databases and web servers.
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...DVClub
This document discusses the verification of Freescale's QorIQ Communication Platform, which contains the new CoreNet fabric. It outlines the verification challenges, methodology used, and SystemVerilog verification IP (VIP) created. The methodology employed a transaction-based approach with hierarchical verification and extensive reuse. SystemVerilog VIP was developed for the CoreNet fabric and other components. This included object-oriented models, monitors, coverage, and stimulus. The verification was successful in validating the CoreNet platform early using SystemVerilog.
The document discusses using Apache Kafka for event detection pipelines. It describes how Kafka can be used to decouple data pipelines and ingest events from various source systems in real-time. It then provides an example use case of using Kafka, Hadoop, and machine learning for fraud detection in consumer banking, describing the online and offline workflows. Finally, it covers some of the challenges of building such a system and considerations for deploying Kafka.
This document summarizes a company's two year journey migrating their infrastructure to Kubernetes on AWS. It describes their stack including tools like Terraform, AWS, CoreOS, Kubernetes and Docker. It outlines their architecture with masters, workers and stateful/stateless nodes. It discusses their lifecycles for development, testing and production. It also covers some struggles they faced around node availability and networking issues. Finally, it provides lessons learned around costs, using Terraform with Kubernetes, separating concerns, and prioritizing automation and testing in their workflows.
The Next Frontier in Open Source Java Compilers: Just-In-Time Compilation as a Service
For Java developers, the Just-In-Time (JIT) compiler is key to improved performance. However, in a container world, the performance gains are often negated due to CPU and memory consumption constraints. To help solve this issue, the Eclipse OpenJ9 JVM provides JITServer technology, which separates the JIT compiler from the application.
JITServer allows the user to employ much smaller containers enabling a higher density of applications, resulting in cost savings for end-users and/or cloud providers. Because the CPU and memory surges due to JIT compilation are eliminated, the user has a much easier task of provisioning resources for his/her application. Additional advantages include: faster ramp-up time, better control over resources devoted to compilation, increased reliability (JIT compiler bugs no longer crash the application) and amortization of compilation costs across many application instances.
We will dig into JITServer technology, showing the challenges of implementation, detailing its strengths and weaknesses and illustrating its performance characteristics. For the cloud audience we will show how it can be deployed in containers, demonstrate its advantages compared to a traditional JIT compilation technique and offer practical recommendations about when to use this technology.
Liberty InstantOn is a technology from IBM that provides fast startup times for Java applications running in containers without compromising features or developer experience. It uses checkpoint and restore to save the state of a pre-initialized Java runtime so new instances can start up almost instantly. This makes Java more suitable for serverless and microservices workloads that require quick startups and the ability to scale to zero. Liberty InstantOn provides the benefits of fast startup times without limiting Java features or requiring changes to how developers build and test applications.
The document summarizes a benchmarking study conducted by Altoros Systems to compare the performance of Couchbase Server, MongoDB, and Cassandra. It outlines the benchmark goals of having a reproducible workload, using a realistic scenario, and comparing latency and throughput. It describes the benchmarking tools, scenario details involving data size, operations, and hardware configuration. Configuration details are provided for each database, including cluster specifications and parameter settings.
Similar to Using Q4M - a message queue storage engine for MySQL (20)
The document discusses the performance of HTTP/2 compared to HTTP/1.1 across different network conditions. It summarizes results from testing 8 real websites under 16 bandwidth and latency combinations with varying packet loss rates. Overall, HTTP/2 performs better for document complete time and speed index, especially on slower connections, though results vary depending on the specific site and metrics measured.
Reorganizing Website Architecture for HTTP/2 and BeyondKazuho Oku
This document discusses reorganizing website architecture for HTTP/2 and beyond. It summarizes some issues with HTTP/2 including errors in prioritization where some browsers fail to specify resource priority properly. It also discusses the problem of TCP head-of-line blocking where pending data in TCP buffers can delay higher priority resources. The document proposes solutions to these issues such as prioritizing resources on the server-side and writing only what can be sent immediately to avoid buffer blocking. It also examines the mixed success of HTTP/2 push and argues the server should not push already cached resources.
This document discusses programming TCP for responsiveness when sending HTTP/2 responses. It describes how to reduce head-of-line blocking by filling the TCP congestion window before sending data. The key points are reading TCP states via getsockopt to determine how much data can be sent immediately, and optimizing this only for high latency connections or small congestion windows to avoid additional response delays. Benchmarks show this approach can reduce response times from multiple round-trip times to a single RTT.
The document discusses optimizations to TCP and HTTP/2 to improve responsiveness on the web. It describes how TCP slow start works and the delays introduced in standard HTTP/2 usage from TCP/TLS handshakes. The author proposes adjusting the TCP send buffer polling threshold to allow switching between responses more quickly based on TCP congestion window state. Benchmark results show this can reduce response times by eliminating an extra round-trip delay.
Presentation material for TokyoRubyKaigi11.
Describes techniques used by H2O, including: techniques to optimize TCP for responsiveness, server-push and cache digests.
Cache aware-server-push in H2O version 1.5Kazuho Oku
This document discusses cache-aware server push in H2O version 1.5. It describes calculating a fingerprint of cached assets using a Golomb compressed set to identify what assets need to be pushed from the server. It also discusses implementing this fingerprint using a cookie or service worker. The hybrid approach stores responses in the service worker cache and updates the cookie fingerprint. H2O 1.5 implements cookie-based fingerprints to cancel push indications for cached assets, potentially improving page load speeds.
JSON SQL Injection and the Lessons LearnedKazuho Oku
This document discusses JSON SQL injection and lessons learned from vulnerabilities in SQL query builders. It describes how user-supplied JSON input containing operators instead of scalar values could manipulate queries by injecting conditions like id!='-1' instead of a specific id value. This allows accessing unintended data. The document examines how SQL::QueryMaker and a strict mode in SQL::Maker address this by restricting query parameters to special operator objects or raising errors on non-scalar values. While helpful, strict mode may break existing code, requiring changes to parameter handling. The vulnerability also applies to other languages' frameworks that similarly convert arrays to SQL IN clauses.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
What is an RPA CoE? Session 2 – CoE RolesDianaGray10
In this session, we will review the players involved in the CoE and how each role impacts opportunities.
Topics covered:
• What roles are essential?
• What place in the automation journey does each role play?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Session 1 - Intro to Robotic Process Automation.pdfUiPathCommunity
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyScyllaDB
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
AppSec PNW: Android and iOS Application Security with MobSFAjin Abraham
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
Essentials of Automations: Exploring Attributes & Automation ParametersSafe Software
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
High performance Serverless Java on AWS- GoTo Amsterdam 2024Vadym Kazulkin
Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless community. Java is known for its high cold start times and high memory footprint, comparing to other programming languages like Node.js and Python. In this talk I'll look at the general best practices and techniques we can use to decrease memory consumption, cold start times for Java Serverless development on AWS including GraalVM (Native Image) and AWS own offering SnapStart based on Firecracker microVM snapshot and restore and CRaC (Coordinated Restore at Checkpoint) runtime hooks. I'll also provide a lot of benchmarking on Lambda functions trying out various deployment package sizes, Lambda memory settings, Java compilation options and HTTP (a)synchronous clients and measure their impact on cold and warm start times.
Must Know Postgres Extension for DBA and Developer during MigrationMydbops
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
QA or the Highway - Component Testing: Bridging the gap between frontend appl...zjhamm304
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
ScyllaDB is making a major architecture shift. We’re moving from vNode replication to tablets – fragments of tables that are distributed independently, enabling dynamic data distribution and extreme elasticity. In this keynote, ScyllaDB co-founder and CTO Avi Kivity explains the reason for this shift, provides a look at the implementation and roadmap, and shares how this shift benefits ScyllaDB users.
3. Who am I?
Name: Kazuho Oku (奥一穂)
Original Developer of Palmscape / Xiino
The oldest web browser for Palm OS
Working at Cybozu Labs since 2005
Research subsidiary of Cybozu, Inc. in Japan
3
Apr 22 2009 Using Q4M
4. About Cybozu, Inc.
Japan’s largest groupware vendor
Mostly provides as software products, not as
services
Some of our apps bundle MySQL as storage
4
Apr 22 2009 Using Q4M
5. About Pathtraq
Started in Aug. 2007
Web ranking service
One of Japan’s largest
like Alexa, but semi-realtime, and per-page
running on MySQL
Need for a fast and reliable message
relay
for communication between the main server and
content analysis server(s)
5
Apr 22 2009 Using Q4M
6. Design Goals of Q4M
Robust
Do not lose data on OS crash or power failure
Fast
Transfer thousands of messages per second
Easy to Use
Use SQL for access / maintenance
Integration into MySQL
no more separate daemons to take care of
6
Apr 22 2009 Using Q4M
7. What is a Message Queue?
7
Apr 22 2009 Using Q4M
8. What is a Message Queue?
Middleware for persistent asynchronous
communication
communicate between fixed pairs (parties)
a.k.a. Message Oriented Middleware
MQ is intermediate storage
RDBMS is persistent storage
Senders / receivers may go down
8
Apr 22 2009 Using Q4M
9. Minimal Configuration of a MQ
Senders and receivers access a single
queue
Sender Receiver
Queue
9
Apr 22 2009 Using Q4M
10. MQ and Relays
Separate queue for sender and receiver
Messages relayed between queues
Relay
Sender Receiver
Queue Queue
10
Apr 22 2009 Using Q4M
11. Merits of Message Relays
Destination can be changed easily
Relays may transfer messages to different
locations depending on their headers
Robustness against network failure
no loss or duplicates when the relay fails
Logging and Multicasting, etc.
11
Apr 22 2009 Using Q4M
12. Message Brokers
Publish / subscribe model
Separation between components and their
integration
Components read / write to predefined queues
Integration is definition of routing rules between
the message queues
Messages are often transformed (filtered) within
the relay agent
12
Apr 22 2009 Using Q4M
13. What about Q4M?
Q4M itself is a message queue
Can connect Q4M instances to create a
message relay
Provides API for creating message relays
and brokers
13
Apr 22 2009 Using Q4M
14. Performance of Q4M
over 7,000 mess/sec.
message size: avg. 512 bytes
syncing to disk
Outperforming most needs
if you need more, just scale out
Can coexist with other storage engines without
sacrificing their performance
see http://labs.cybozu.co.jp/blog/kazuhoatwork/2008/06/q4m_06_release_and_benchmarks.php
14
Apr 22 2009 Using Q4M
16. Asynchronous Updates
Mixi (Japan's one of the largest SNS)
uses Q4M to buffer writes to DB, to
offload peak demands
from http://alpha.mixi.co.jp/blog/?p=272
16
Apr 22 2009 Using Q4M
17. Connecting Distant Servers
Pathtraq uses Q4M to create a relay
between its database and content
analysis processes
→ Contents to be analyzed →
Content
MySQL conn. Analysis
Pathtraq
over SSL,gzip Processes
DB
← Results of the analysis ←
17
Apr 22 2009 Using Q4M
18. To Prefetch Data
livedoor Reader (web-based feed
aggregator) uses Q4M to prefetch data
from database to memcached
uses Q4M for scheduling web crawlers
as well
from http://d.hatena.ne.jp/mala/20081212/1229074359
18
Apr 22 2009 Using Q4M
19. Scheduling Web Crawlers
Web crawlers with retry-on-error
Sample code included in Q4M dist.
If failed to fetch, store URL in retry queue
Store Result
Read URL
Spiders
URL
DB
Request Queue Retry Queue
Re-
scheduler
19
Apr 22 2009 Using Q4M
23. Installing Q4M
Compatible with MySQL 5.1
Download from q4m.31tools.com
Binary releases available for some platforms
Installing from source:
requires source code of MySQL
./configure && make && make install
run support-files/install.sql
23
Apr 22 2009 Using Q4M
24. Configuration Options of Q4M
--with-sync=no|fsync|fdatasync|fcntl
Controls synchronization to disk
default: fdatasync on linux
--enable-mmap
Mmap’ed reads lead to higher throughput
default: yes
--with-delete=pwrite|msync
msyncrecommended on linux>=2.6.20 if you
need really high performance
24
Apr 22 2009 Using Q4M
26. The Model
Various publishers write to queue
Set of subscribers consume the entries in queue
Publisher
Q4M table
Publisher
Subscribers
Publisher
26
Apr 22 2009 Using Q4M
27. Creating a Q4M Table
ENGINE=QUEUE creates
mysql> CREATE TABLE qt (
-> id int(10) unsigned NOT NULL,
a Q4M table
-> message varchar(255) NOT NULL
-> ) ENGINE=QUEUE;
Query OK, 0 rows affected (0.42 sec)
No primary keys or
indexes
Sorted by insertion
order (it’s a queue)
27
Apr 22 2009 Using Q4M
28. Modifying Data on a Q4M Table
No restrictions for
mysql> INSERT INTO qt (id,message)
-> VALUES
INSERT and DELETE
-> (1,'Hello'),
-> (2,'Bonjour'),
-> (3,'Hola');
No support for UPDATE Query OK, 3 rows affected (0.02 sec)
mysql> SELECT * FROM qt;
+----+---------+
| id | message |
+----+---------+
| 1 | Hello |
| 2 | Bonjour |
| 3 | Hola |
+----+---------+
3 rows in set (0.00 sec)
28
Apr 22 2009 Using Q4M
29. SELECT from a Q4M Table
Works the same as
mysql> SELECT * FROM qt;
+----+---------+
| id | message |
other storage +----+---------+
| 1 | Hello |
engines | 2 | Bonjour |
| 3 | Hola |
+----+---------+
SELECT COUNT(*) is 3 rows in set (0.00 sec)
cached mysql> SELECT COUNT(*) FROM qt;
+----------+
| COUNT(*) |
+----------+
| 3|
+----------+
1 row in set (0.00 sec)
How to subscribe to a queue?
29
Apr 22 2009 Using Q4M
30. Calling queue_wait()
After calling, only one
mysql> SELECT * FROM qt;
+----+---------+
| id | message |
row becomes visible +----+---------+
| 1 | Hello |
from the connection | 2 | Bonjour |
| 3 | Hola |
+----+---------+
3 rows in set (0.00 sec)
mysql> SELECT queue_wait('qt');
+------------------+
| queue_wait('qt') |
+------------------+
| 1|
+------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM qt;
+----+---------+
| id | message |
+----+---------+
| 1 | Hello |
+----+---------+
1 row in set (0.00 sec)
30
Apr 22 2009 Using Q4M
31. OWNER Mode and NON-OWNER Mode
In OWNER mode, only the OWNED row
is visible
OWNED row becomes invisible from other connections
rows of other storage engines are visible
NON-OWNER Mode OWNER Mode
queue_wait()
1,'Hello' 1,'Hello'
2,'Bonjour'
queue_end()
3,'Hola'
queue_abort()
31
Apr 22 2009 Using Q4M
32. Returning to NON-OWNER mode
By calling
mysql> SELECT QUEUE_ABORT();
+---------------+
queue_abort, the
| QUEUE_ABORT() |
+---------------+
connection returns to
| 1|
+---------------+
NON-OWNER mode
1 row in set (0.00 sec)
mysql> SELECT * FROM qt;
+----+---------+
| id | message |
+----+---------+
| 1 | Hello |
| 2 | Bonjour |
| 3 | Hola |
+----+---------+
3 rows in set (0.01 sec)
32
Apr 22 2009 Using Q4M
33. Consuming a Row
By calling
mysql> SELECT queue_wait('qt');
(snip)
mysql> SELECT * FROM qt;
queue_end, the OWNED +----+---------+
| id | message |
row is deleted, and +----+---------+
| 1 | Hello |
connection returns to
+----+---------+
1 row in set (0.01 sec)
NON-OWNER mode mysql> SELECT queue_end();
+-------------+
| queue_end() |
+-------------+
| 1|
+-------------+
1 row in set (0.01 sec)
mysql> SELECT * FROM qt;
+----+---------+
| id | message |
+----+---------+
| 2 | Bonjour |
| 3 | Hola |
+----+---------+
2 rows in set (0.00 sec)
33
Apr 22 2009 Using Q4M
34. Writing a Subscriber
Call two functions: queue_wait, queue_end
Multiple subscribers can be run concurrently
each row in the queue is consumed only once
while (true) {
SELECT queue_wait('qt'); # switch to owner mode
rows := SELECT * FROM qt; # obtain data
if (count(rows) != 0) # if we have any data, then
handle_row(rows[0]); # consume the row
SELECT queue_end(); # erase the row from queue
}
34
Apr 22 2009 Using Q4M
35. Writing a Subscriber (cont'd)
Or call queue_wait as a condition
Warning: conflicts with trigger-based insertions
while (true) {
rows := SELECT * FROM qt WHERE queue_wait('qt');
if (count(rows) != 0)
handle_row(rows[0]);
SELECT queue_end();
}
35
Apr 22 2009 Using Q4M
36. The Model – with code
INSERT INTO queue ...
Publisher
while (true) {
rows := SELECT * FROM qt
WHERE queue_wait('qt');
if (count(rows) != 0)
handle_row(rows[0]);
INSERT INTO queue ...
SELECT queue_end();
}
Q4M table
Publisher
Subscribers
INSERT INTO queue ...
Publisher
36
Apr 22 2009 Using Q4M
38. queue_wait(table)
Enters OWNER mode
0〜1 row becomes OWNED
Enters OWNER mode even if no rows were
available
Default timeout: 60 seconds
Returns 1 if a row is OWNED (0 on timeout)
If called within OWNER mode, the
owned row is deleted
38
Apr 22 2009 Using Q4M
39. Revisiting Subscriber Code
Calls to queue_end just before
queue_wait can be omitted
while (true) {
rows := SELECT * FROM qt WHERE queue_wait('qt');
if (count(rows) != 0)
handle_row(rows[0]);
SELECT queue_end();
}
39
Apr 22 2009 Using Q4M
40. Conditional queue_wait()
Consume rows of certain condition
Rows that do not match will be left untouched
Only numeric columns can be checked
Fast - condition tested once per each row
examples:
SELECT queue_wait('table:(col_a*3)+col_b<col_c');
SELECT queue_wait('table:retry_count<5');
40
Apr 22 2009 Using Q4M
41. queue_wait(tbl_cond,[tbl_cond…,timeout])
Accepts multiple tables and timeout
Data searched from leftmost table to
right
Returns table index (the leftmost table is
1) of the newly owned row
Returns zero if no rows are being owned
example:
SELECT queue_wait('table_A','table_B',60);
41
Apr 22 2009 Using Q4M
42. Functions for Exiting OWNER Mode
queue_end
Deletes the owned row and exits OWNER mode
queue_abort
Releases (instead of deleting) the owned row and
exits OWNER mode
Close of a MySQL connection does the same thing
42
Apr 22 2009 Using Q4M
44. The Problem
Relay (or router) consists of more than 3
processes, 2 conns
No losses, no duplicates on crash or
disconnection
Q4M Table Q4M Table
Relay Program
(source) (dest.)
44
Apr 22 2009 Using Q4M
45. Internal Row ID
Every row have a internal row ID
invisible from Q4M table definition
monotonically increasing 64-bit integer
Used for detecting duplicates
Use two functions to skip duplicates
Data loss prevented by using queue_wait /
queue_end
45
Apr 22 2009 Using Q4M
46. queue_rowid()
Returns row ID of the OWNED row (if
any)
Returns NULL if no row is OWNED
Call when retrieving data from source
46
Apr 22 2009 Using Q4M
47. queue_set_srcid(src_tbl_id, mode, src_row_id)
Call before inserting a row to destination
table
Checks if the row is already inserted into the
table, and ignores next INSERT if true
Parameters:
src_tbl_id - id to determine source table (0〜63)
mode - quot;aquot; to drop duplicates, quot;wquot; to reset
src_row_id - row ID obtained from source table
47
Apr 22 2009 Using Q4M
48. Pseudo Code
Relays data from src_tbl to dest_tbl
while (true) {
# wait for data
SELECT queue_wait(src_tbl) =>src_db;
# read row and rowid
row := (SELECT * FROM src_tbl =>src_db);
rowid := (SELECT queue_rowid() =>src_db);
# insert the row after setting srcid
SELECT queue_set_srcid(src_tbl_id, 'a', rowid) =>dest_db;
INSERT INTO dest_tbl (row) =>dest_db;
48
Apr 22 2009 Using Q4M
49. q4m-forward
Simple forwarder script
installed into mysql-dir/bin
usage: q4m-forward [options] src_addrdest_addr
example:
% support-files/q4m-forward
quot;dbi:mysql:database=db1;table=tbl1;user=foo;password=XXXquot;
quot;dbi:mysql:database=db2;table=tbl2;host=bar;user=fooquot;
options:
--reset reset duplicate check info.
--sender=idx slot no. used for checking duplicates (0..63, default: 0)
--help
49
Apr 22 2009 Using Q4M
51. Things that Need to be Fixed
Table compactions is a blocking
operation
runs when live data becomes <25% of log file
very bad, though not as bad as it seems
it's fast since it's a sequential write operation
Relays are slow
since transfer is done row-by-row
Binlog does not work
since MQ replication should be synchronous
51
Apr 22 2009 Using Q4M
52. Future of Q4M
2-phase commit with other storage
engines (maybe)
queue consumption and InnoDB updates can
become atomic operation
52
Apr 22 2009 Using Q4M
53. Thank you
http://q4m.31tools.com/
53
Apr 22 2009 Using Q4M