Druid beginner performance tips
•Download as PPTX, PDF•
1 like•236 views
some tips for druid performance. We did some benchmarks in our environment and these tips really helped us in getting better query execution times.
Report
Share
Report
Share
Recommended
Mp26 : Tachyon, sloppiness is bliss
This document describes Tachyon, an offensive tool for penetration testers to discover hidden files and folders on web servers through intelligent scanning. It utilizes a plugin architecture and multi-threading to quickly scan sites. The document outlines Tachyon's features like Tor support, path and file databases, plugin system, and false positive detection methods. It notes limitations and plans for future improvements like faster speeds, cleaner output, and a plugin documentation system.
Challenges with Gluster and Persistent Memory with Dan Lambright
This document discusses challenges in using persistent memory (SCM) with distributed storage systems like Gluster. It notes that SCM provides faster access than SSDs but must address latency throughout the storage stack, including network transfer times and CPU overhead. The document examines how Gluster's design amplifies lookup operations and proposes caching file metadata at clients to reduce overhead. It also suggests using SCM as a tiered cache layer and optimizing replication strategies to fully leverage the speed of SCM.
Lcna example-2012
This document summarizes a presentation about building a negative lookup caching translator for GlusterFS. The presentation demonstrates adding caching functionality to speed up lookups by caching previous misses. It shows the steps to hook the translator together, build it, configure it, debug it, and test its performance. Finally, it briefly introduces glupy, a new project for writing GlusterFS translators in Python, and demonstrates a Python implementation of the negative lookup cache.
Life as a GlusterFS Consultant with Ivan Rossi
This document describes the experiences of Ivan Rossi working as a Gluster consultant listed on gluster.org. It outlines the types of clients that contact him, including small businesses and those looking for help troubleshooting Gluster issues. It also shares some case studies, such as helping a company that had millions of unorganized files slowing down operations and advising businesses on using Gluster in private and public clouds. The document aims to convey what it is like to be a Gluster consultant and provide advice based on lessons learned from previous work.
Gluster intro-tdose
This document provides an introduction and overview of Gluster, an open source scale-out network-attached storage file system. It discusses what Gluster is, its architecture using distributed and replicated volumes, a quick start guide, use cases, features, and how to get involved in the community. The presentation aims to explain the benefits and capabilities of Gluster for scalable, high performance storage.
Gluster d thread_synchronization_using_urcu_lca2016
This document discusses using user space RCU (URCU) for thread synchronization in GlusterD, the management daemon for the Gluster distributed file system. It begins with an introduction to Gluster and GlusterD, noting that GlusterD was initially single-threaded but was changed to be multi-threaded. This introduced the need for thread synchronization, which was initially implemented with a "big lock" but had issues. The document then provides an overview of RCU and how it can provide advantages over read-write locks for thread synchronization. It covers the key mechanisms of RCU, various URCU flavors, URCU APIs, and examples of when URCU would be useful.
Performance bottlenecks for metadata workload in Gluster with Poornima Gurusi...
The document discusses metadata performance bottlenecks in Gluster. It analyzes the performance of various metadata operations like file creation, directory creation, listing files, deleting files, and reading/writing small files. For each operation, it identifies the performance bottlenecks and discusses some optimizations that could help improve performance, such as parallelizing operations, compounding fops, leveraging leases to reduce locks, and caching. The overall conclusion is that metadata performance in Gluster has not yet been saturated and there is potential for significant improvements without compromising consistency.
Tiering barcelona
Gluster tiering allows for the logical composition of diverse storage units like SSDs and HDDs. It uses fast storage like SSDs as a cache for slower storage like HDDs. Files are migrated between tiers based on usage patterns to optimize for access speeds. The tiering implementation in Gluster uses a metadata store and changetime recorder to track file access and make decisions about tier migrations. Integration with the Gluster distributed hash table and volume rebalancing process allows for dynamic attaching and detaching of tiers.
Recommended
Mp26 : Tachyon, sloppiness is bliss
This document describes Tachyon, an offensive tool for penetration testers to discover hidden files and folders on web servers through intelligent scanning. It utilizes a plugin architecture and multi-threading to quickly scan sites. The document outlines Tachyon's features like Tor support, path and file databases, plugin system, and false positive detection methods. It notes limitations and plans for future improvements like faster speeds, cleaner output, and a plugin documentation system.
Challenges with Gluster and Persistent Memory with Dan Lambright
This document discusses challenges in using persistent memory (SCM) with distributed storage systems like Gluster. It notes that SCM provides faster access than SSDs but must address latency throughout the storage stack, including network transfer times and CPU overhead. The document examines how Gluster's design amplifies lookup operations and proposes caching file metadata at clients to reduce overhead. It also suggests using SCM as a tiered cache layer and optimizing replication strategies to fully leverage the speed of SCM.
Lcna example-2012
This document summarizes a presentation about building a negative lookup caching translator for GlusterFS. The presentation demonstrates adding caching functionality to speed up lookups by caching previous misses. It shows the steps to hook the translator together, build it, configure it, debug it, and test its performance. Finally, it briefly introduces glupy, a new project for writing GlusterFS translators in Python, and demonstrates a Python implementation of the negative lookup cache.
Life as a GlusterFS Consultant with Ivan Rossi
This document describes the experiences of Ivan Rossi working as a Gluster consultant listed on gluster.org. It outlines the types of clients that contact him, including small businesses and those looking for help troubleshooting Gluster issues. It also shares some case studies, such as helping a company that had millions of unorganized files slowing down operations and advising businesses on using Gluster in private and public clouds. The document aims to convey what it is like to be a Gluster consultant and provide advice based on lessons learned from previous work.
Gluster intro-tdose
This document provides an introduction and overview of Gluster, an open source scale-out network-attached storage file system. It discusses what Gluster is, its architecture using distributed and replicated volumes, a quick start guide, use cases, features, and how to get involved in the community. The presentation aims to explain the benefits and capabilities of Gluster for scalable, high performance storage.
Gluster d thread_synchronization_using_urcu_lca2016
This document discusses using user space RCU (URCU) for thread synchronization in GlusterD, the management daemon for the Gluster distributed file system. It begins with an introduction to Gluster and GlusterD, noting that GlusterD was initially single-threaded but was changed to be multi-threaded. This introduced the need for thread synchronization, which was initially implemented with a "big lock" but had issues. The document then provides an overview of RCU and how it can provide advantages over read-write locks for thread synchronization. It covers the key mechanisms of RCU, various URCU flavors, URCU APIs, and examples of when URCU would be useful.
Performance bottlenecks for metadata workload in Gluster with Poornima Gurusi...
The document discusses metadata performance bottlenecks in Gluster. It analyzes the performance of various metadata operations like file creation, directory creation, listing files, deleting files, and reading/writing small files. For each operation, it identifies the performance bottlenecks and discusses some optimizations that could help improve performance, such as parallelizing operations, compounding fops, leveraging leases to reduce locks, and caching. The overall conclusion is that metadata performance in Gluster has not yet been saturated and there is potential for significant improvements without compromising consistency.
Tiering barcelona
Gluster tiering allows for the logical composition of diverse storage units like SSDs and HDDs. It uses fast storage like SSDs as a cache for slower storage like HDDs. Files are migrated between tiers based on usage patterns to optimize for access speeds. The tiering implementation in Gluster uses a metadata store and changetime recorder to track file access and make decisions about tier migrations. Integration with the Gluster distributed hash table and volume rebalancing process allows for dynamic attaching and detaching of tiers.
Time to rethink /proc
The current Linux kernel /proc/PID interface is great, time-proven and reliable way to get info about processes running on a system. Right? Well, yes and no. We found out (and you, too, might have noticed it) this is what makes ps and top slow when there are thousands of processes running. Besides the speed, there are a number of other problems with the current /proc/PID interface.
The talk describes all those in great details, then goes on to the alternative we are proposing for inclusion to the kernel, a new interface called task_diag. The new interface is slick, fast (5-10x speed improvement), and extendable.
Rhel cluster gfs_improveperformance
This document provides recommendations to improve performance of Red Hat Global File System (GFS) and GFS2 file systems. It discusses common problems that can cause performance issues or processes hanging, such as file/directory and resource group contention. The document also provides tips on how to determine the root cause of a problem, such as identifying contended inodes or tasks. Designing the file system layout and using appropriate mount options, block sizes, and I/O patterns can optimize performance.
Scale out backups-with_bareos_and_gluster
This document discusses integrating Bareos backups with the Gluster distributed file system for scalable backups. It begins with an agenda that covers the Gluster integration in Bareos, an introduction to GlusterFS, a quick start guide, an example configuration and demo, and future plans. It then provides more details on GlusterFS architecture including concepts like bricks, volumes, peers and site replication. The remainder of the document outlines quick start instructions for setting up Gluster and configuring Bareos to use the Gluster backend for scalable backups across multiple servers.
Lcna 2012-tutorial
GlusterFS uses "translators" to modify and route file requests between users and storage bricks. Translators can convert request types, modify request properties like paths or flags, intercept or block requests, and spawn new requests. This allows GlusterFS to provide features like replication, caching, and integration with other systems, but also enables custom file systems to be built by modifying the translators. The asynchronous programming model and shared context objects allow translators to cooperate complex workflows across multiple servers.
Sdc 2012-challenges
GlusterFS is a distributed file system that shards and replicates files across multiple servers without a central metadata server. It uses modular "translators" to handle functions like replication and distribution. Some challenges GlusterFS faces include multi-tenancy, distributed quota management, efficient data rebalancing, reducing replication latency, optimizing directory traversal, and handling many small files. The speaker argues these challenges are not unique to GlusterFS and that incremental, modular improvements are preferable to monolithic solutions.
State of the_gluster_-_lceu
The document summarizes the state of the Gluster community and GlusterFS distributed file system. It discusses that GlusterFS is a scale-out NAS platform that provides a unified, distributed storage system without single points of failure. It also outlines recent updates to GlusterFS 3.3 including improved granular locking, proactive self-healing, and easier rebalancing. The document concludes by previewing upcoming work including better support for virtual machine images, libgfapi client API improvements, and quorum enforcement to prevent split-brain issues.
Lcna tutorial-2012
GlusterFS uses "translators" to modify and route file requests between users and storage bricks. Translators can convert request types, modify request properties like paths or flags, intercept or block requests, and spawn new requests. This allows GlusterFS to provide features like replication, caching, and integration with other systems, but also enables custom file systems to be built by modifying the translators. The asynchronous programming model and shared context objects allow translators to cooperate complex workflows across multiple servers.
Gluster as Block Store in Containers
Gluster can provide block storage using LIO/TCMU. It was demonstrated providing an iSCSI block device from a Gluster volume, including block snapshots. Performance numbers were shown and it can integrate with containers by providing a persistent block device. Kubernetes was demonstrated using Gluster block storage by having nodes initiate iSCSI sessions to access a target device mounted in pods. Future work may include more testing, Heketi integration for provisioning, and hyper-convergence.
Gluster fs architecture_&_roadmap_atin_punemeetup_2015
Atin Mukherjee gave a presentation on GlusterFS at a February 2015 meetup. He discussed GlusterFS's current architecture and roadmap. For version 3.6, he highlighted new features like better SSL support, heterogeneous bricks, erasure coding, and volume snapshots. Version 3.7 plans to improve small file performance, add data classification, enable bit-rot detection, and integrate better with OpenStack. The long term 4.0 vision is for GlusterFS to be the best commodity distributed storage with unified data access across thousands of nodes.
Disperse xlator ramon_datalab
The document introduces the Disperse Translator, which allows for configurable fault tolerance in Gluster volumes using erasure codes. Key features include adjustable redundancy levels, minimized storage waste, and reduced bandwidth usage. It works by dispersing and encoding file chunks across bricks. The current implementation provides a functional disperse translator and healing processes, with future plans to add CLI support and optimize performance.
State of Gluster Performance
Gluster performance was analyzed for different workload classes and volume types. Replica volumes performed better than erasure coded volumes for low thread count sequential workloads, especially writes, while erasure coded volumes excelled for writes with higher thread counts. Kernal NFS generally outperformed Gluster for single-threaded sequential workloads but Gluster performed comparably or better for workloads with more concurrent threads. Random read performance was similar between volume types but replica volumes suffered on random writes due to the use of RAID-6. Small file performance is still being investigated. Erasure coding shows potential for lower cost video storage use cases.
Nats
NATS is a simple, secure, scalable, and high-performance messaging system for cloud native applications. It uses a text-based protocol and is written in Go, making it scalable and high-performing. The system supports pub/sub and request-reply messaging through subject-based routing and wildcards, allowing for flexible messaging patterns.
GlusterFS Containers
Setting up GlusterFS in Docker Environment and all the options briefly explained for GlusterFS container to run in a Docker Environment.
Filesystem Comparison: NFS vs GFS2 vs OCFS2
The document compares the performance of NFS, GFS2, and OCFS2 filesystems on a high-performance computing cluster with nodes split across two datacenters. Generic load testing showed that NFS performance declined significantly with more than 6 nodes, while GFS2 maintained higher throughput. Further testing of GFS2 and OCFS2 using workload simulations modeling researcher usage found that OCFS2 outperformed GFS2 on small file operations and maintained high performance across nodes, making it the best choice for the shared filesystem needs of the project.
Sharding: Past, Present and Future with Krutika Dhananjay
- Sharding is a client-side translator that splits files into equally sized chunks or shards to improve performance and utilization of storage resources. It sits above the distributed hash table (DHT) in Gluster.
- Sharding benefits virtual machine image storage by allowing data healing and replication at the shard level for better scalability. It also distributes load more evenly across bricks.
- For general purpose use, sharding aims to maximize parallelism during writes while maintaining consistency through atomic operations and locking frameworks. Key challenges include updating file metadata without locking and handling operations like truncates and appends correctly across shards.
Gluster overview & future directions vault 2015
The document provides an overview and future directions of Gluster distributed storage system. It discusses why Gluster is useful given increasing data volumes. It defines Gluster as a scale-out distributed storage system that aggregates storage over a network to provide a unified namespace. It outlines typical deployments and architecture, and describes various volume types like distributed, replicated, dispersed. It also covers access mechanisms, features, use cases and monitoring integration. Finally, it discusses recent releases and new features in development like data tiering, bitrot detection and sharding to improve performance and capabilities.
Gluster for sysadmins
GlusterFS is a POSIX-compliant distributed file system that aggregates various storage bricks across commodity servers into a single global namespace. It has no single point of failure or performance bottleneck. Red Hat Storage is an enterprise implementation of GlusterFS. It uses a elastic hashing algorithm to distribute files across bricks without a centralized metadata server. Various translators and volumes types provide features like replication, distribution, striping, and geo-replication. Administration involves adding peers, creating and managing distributed volumes, and manipulating bricks.
Gluster fs current_features_and_roadmap
The document discusses the current features and roadmap for GlusterFS. It summarizes the current stable releases, features added in recent versions, and plans for upcoming releases 3.7 and 4.0. The 3.7 release will focus on improvements to small file performance, tiering, rack awareness, trash support for undelete, and NFS Ganesha integration. The 4.0 release will aim to improve scalability and manageability with features like multiple networks support, new style replication, and REST APIs.
Gluster d2
The document discusses GlusterD 2.0, a redesign of the Gluster distributed file system management daemon. Some key points:
- GlusterD 1.0 had scalability and consistency issues that limited it to hundreds of nodes. GlusterD 2.0 was rewritten from scratch in Go for better performance.
- GlusterD 2.0 uses etcd for centralized management and configuration storage. It has REST APIs and plugins for modularity.
- Components include REST interfaces, etcd backend, RPC framework, transaction system, and a flexible volume generator.
- Upgrades from Gluster 3.x to 4.x will be disruptive but provide a migration path. Gluster
Gluster intro-tdose
This document provides an introduction and overview of Gluster, an open source scale-out network-attached storage file system. It discusses what Gluster is, its architecture using distributed and replicated volumes, and provides a quick start guide for setting up a basic Gluster volume. The document also outlines several use cases for Gluster and recently added features. It concludes by describing how readers can get involved in the Gluster community.
A talk on mysql & aurora
The document discusses various MySQL indexing concepts like primary key indexes, secondary indexes, clustered indexes and hash indexes. It explains how indexes are used based on the left prefix rule and selectivity. It also covers storage engines like InnoDB and MyISAM. The document discusses locking errors like lock wait timeouts and deadlocks. It explains isolation levels like repeatable read, read committed and serializable. It provides details about the Aurora undo log and how it differs from vanilla MySQL. It emphasizes monitoring MySQL using the error log, slow query log and metrics. It also briefly discusses Aurora parallel queries.
Introduction to Apache Kafka
- Publishers send messages to topics in Apache Kafka which are partitioned across brokers
- Brokers append messages to the ends of partitions and subscribers can request messages from specific offsets in partitions
- This allows subscribers to replay processing from any point in time as they request messages based on offset rather than relying on brokers to deliver messages
More Related Content
What's hot
Time to rethink /proc
The current Linux kernel /proc/PID interface is great, time-proven and reliable way to get info about processes running on a system. Right? Well, yes and no. We found out (and you, too, might have noticed it) this is what makes ps and top slow when there are thousands of processes running. Besides the speed, there are a number of other problems with the current /proc/PID interface.
The talk describes all those in great details, then goes on to the alternative we are proposing for inclusion to the kernel, a new interface called task_diag. The new interface is slick, fast (5-10x speed improvement), and extendable.
Rhel cluster gfs_improveperformance
This document provides recommendations to improve performance of Red Hat Global File System (GFS) and GFS2 file systems. It discusses common problems that can cause performance issues or processes hanging, such as file/directory and resource group contention. The document also provides tips on how to determine the root cause of a problem, such as identifying contended inodes or tasks. Designing the file system layout and using appropriate mount options, block sizes, and I/O patterns can optimize performance.
Scale out backups-with_bareos_and_gluster
This document discusses integrating Bareos backups with the Gluster distributed file system for scalable backups. It begins with an agenda that covers the Gluster integration in Bareos, an introduction to GlusterFS, a quick start guide, an example configuration and demo, and future plans. It then provides more details on GlusterFS architecture including concepts like bricks, volumes, peers and site replication. The remainder of the document outlines quick start instructions for setting up Gluster and configuring Bareos to use the Gluster backend for scalable backups across multiple servers.
Lcna 2012-tutorial
GlusterFS uses "translators" to modify and route file requests between users and storage bricks. Translators can convert request types, modify request properties like paths or flags, intercept or block requests, and spawn new requests. This allows GlusterFS to provide features like replication, caching, and integration with other systems, but also enables custom file systems to be built by modifying the translators. The asynchronous programming model and shared context objects allow translators to cooperate complex workflows across multiple servers.
Sdc 2012-challenges
GlusterFS is a distributed file system that shards and replicates files across multiple servers without a central metadata server. It uses modular "translators" to handle functions like replication and distribution. Some challenges GlusterFS faces include multi-tenancy, distributed quota management, efficient data rebalancing, reducing replication latency, optimizing directory traversal, and handling many small files. The speaker argues these challenges are not unique to GlusterFS and that incremental, modular improvements are preferable to monolithic solutions.
State of the_gluster_-_lceu
The document summarizes the state of the Gluster community and GlusterFS distributed file system. It discusses that GlusterFS is a scale-out NAS platform that provides a unified, distributed storage system without single points of failure. It also outlines recent updates to GlusterFS 3.3 including improved granular locking, proactive self-healing, and easier rebalancing. The document concludes by previewing upcoming work including better support for virtual machine images, libgfapi client API improvements, and quorum enforcement to prevent split-brain issues.
Lcna tutorial-2012
GlusterFS uses "translators" to modify and route file requests between users and storage bricks. Translators can convert request types, modify request properties like paths or flags, intercept or block requests, and spawn new requests. This allows GlusterFS to provide features like replication, caching, and integration with other systems, but also enables custom file systems to be built by modifying the translators. The asynchronous programming model and shared context objects allow translators to cooperate complex workflows across multiple servers.
Gluster as Block Store in Containers
Gluster can provide block storage using LIO/TCMU. It was demonstrated providing an iSCSI block device from a Gluster volume, including block snapshots. Performance numbers were shown and it can integrate with containers by providing a persistent block device. Kubernetes was demonstrated using Gluster block storage by having nodes initiate iSCSI sessions to access a target device mounted in pods. Future work may include more testing, Heketi integration for provisioning, and hyper-convergence.
Gluster fs architecture_&_roadmap_atin_punemeetup_2015
Atin Mukherjee gave a presentation on GlusterFS at a February 2015 meetup. He discussed GlusterFS's current architecture and roadmap. For version 3.6, he highlighted new features like better SSL support, heterogeneous bricks, erasure coding, and volume snapshots. Version 3.7 plans to improve small file performance, add data classification, enable bit-rot detection, and integrate better with OpenStack. The long term 4.0 vision is for GlusterFS to be the best commodity distributed storage with unified data access across thousands of nodes.
Disperse xlator ramon_datalab
The document introduces the Disperse Translator, which allows for configurable fault tolerance in Gluster volumes using erasure codes. Key features include adjustable redundancy levels, minimized storage waste, and reduced bandwidth usage. It works by dispersing and encoding file chunks across bricks. The current implementation provides a functional disperse translator and healing processes, with future plans to add CLI support and optimize performance.
State of Gluster Performance
Gluster performance was analyzed for different workload classes and volume types. Replica volumes performed better than erasure coded volumes for low thread count sequential workloads, especially writes, while erasure coded volumes excelled for writes with higher thread counts. Kernal NFS generally outperformed Gluster for single-threaded sequential workloads but Gluster performed comparably or better for workloads with more concurrent threads. Random read performance was similar between volume types but replica volumes suffered on random writes due to the use of RAID-6. Small file performance is still being investigated. Erasure coding shows potential for lower cost video storage use cases.
Nats
NATS is a simple, secure, scalable, and high-performance messaging system for cloud native applications. It uses a text-based protocol and is written in Go, making it scalable and high-performing. The system supports pub/sub and request-reply messaging through subject-based routing and wildcards, allowing for flexible messaging patterns.
GlusterFS Containers
Setting up GlusterFS in Docker Environment and all the options briefly explained for GlusterFS container to run in a Docker Environment.
Filesystem Comparison: NFS vs GFS2 vs OCFS2
The document compares the performance of NFS, GFS2, and OCFS2 filesystems on a high-performance computing cluster with nodes split across two datacenters. Generic load testing showed that NFS performance declined significantly with more than 6 nodes, while GFS2 maintained higher throughput. Further testing of GFS2 and OCFS2 using workload simulations modeling researcher usage found that OCFS2 outperformed GFS2 on small file operations and maintained high performance across nodes, making it the best choice for the shared filesystem needs of the project.
Sharding: Past, Present and Future with Krutika Dhananjay
- Sharding is a client-side translator that splits files into equally sized chunks or shards to improve performance and utilization of storage resources. It sits above the distributed hash table (DHT) in Gluster.
- Sharding benefits virtual machine image storage by allowing data healing and replication at the shard level for better scalability. It also distributes load more evenly across bricks.
- For general purpose use, sharding aims to maximize parallelism during writes while maintaining consistency through atomic operations and locking frameworks. Key challenges include updating file metadata without locking and handling operations like truncates and appends correctly across shards.
Gluster overview & future directions vault 2015
The document provides an overview and future directions of Gluster distributed storage system. It discusses why Gluster is useful given increasing data volumes. It defines Gluster as a scale-out distributed storage system that aggregates storage over a network to provide a unified namespace. It outlines typical deployments and architecture, and describes various volume types like distributed, replicated, dispersed. It also covers access mechanisms, features, use cases and monitoring integration. Finally, it discusses recent releases and new features in development like data tiering, bitrot detection and sharding to improve performance and capabilities.
Gluster for sysadmins
GlusterFS is a POSIX-compliant distributed file system that aggregates various storage bricks across commodity servers into a single global namespace. It has no single point of failure or performance bottleneck. Red Hat Storage is an enterprise implementation of GlusterFS. It uses a elastic hashing algorithm to distribute files across bricks without a centralized metadata server. Various translators and volumes types provide features like replication, distribution, striping, and geo-replication. Administration involves adding peers, creating and managing distributed volumes, and manipulating bricks.
Gluster fs current_features_and_roadmap
The document discusses the current features and roadmap for GlusterFS. It summarizes the current stable releases, features added in recent versions, and plans for upcoming releases 3.7 and 4.0. The 3.7 release will focus on improvements to small file performance, tiering, rack awareness, trash support for undelete, and NFS Ganesha integration. The 4.0 release will aim to improve scalability and manageability with features like multiple networks support, new style replication, and REST APIs.
Gluster d2
The document discusses GlusterD 2.0, a redesign of the Gluster distributed file system management daemon. Some key points:
- GlusterD 1.0 had scalability and consistency issues that limited it to hundreds of nodes. GlusterD 2.0 was rewritten from scratch in Go for better performance.
- GlusterD 2.0 uses etcd for centralized management and configuration storage. It has REST APIs and plugins for modularity.
- Components include REST interfaces, etcd backend, RPC framework, transaction system, and a flexible volume generator.
- Upgrades from Gluster 3.x to 4.x will be disruptive but provide a migration path. Gluster
Gluster intro-tdose
This document provides an introduction and overview of Gluster, an open source scale-out network-attached storage file system. It discusses what Gluster is, its architecture using distributed and replicated volumes, and provides a quick start guide for setting up a basic Gluster volume. The document also outlines several use cases for Gluster and recently added features. It concludes by describing how readers can get involved in the Gluster community.
What's hot (20)
Gluster fs architecture_&_roadmap_atin_punemeetup_2015
Gluster fs architecture_&_roadmap_atin_punemeetup_2015
Sharding: Past, Present and Future with Krutika Dhananjay
Sharding: Past, Present and Future with Krutika Dhananjay
More from vishnu rao
A talk on mysql & aurora
The document discusses various MySQL indexing concepts like primary key indexes, secondary indexes, clustered indexes and hash indexes. It explains how indexes are used based on the left prefix rule and selectivity. It also covers storage engines like InnoDB and MyISAM. The document discusses locking errors like lock wait timeouts and deadlocks. It explains isolation levels like repeatable read, read committed and serializable. It provides details about the Aurora undo log and how it differs from vanilla MySQL. It emphasizes monitoring MySQL using the error log, slow query log and metrics. It also briefly discusses Aurora parallel queries.
Introduction to Apache Kafka
- Publishers send messages to topics in Apache Kafka which are partitioned across brokers
- Brokers append messages to the ends of partitions and subscribers can request messages from specific offsets in partitions
- This allows subscribers to replay processing from any point in time as they request messages based on offset rather than relying on brokers to deliver messages
Mysql Relay log - the unsung hero
This document discusses using the relay log as a solution for failover in a multi-source replication scenario where the binary log positions and transaction IDs are different across slaves. The relay log contains the same position and transaction ID for a transaction on all slaves, allowing one to be used as a global transaction ID for failover. Specifically, a modified client was created to dump the relay log, and the MySQL server was updated to support relay log dumping, providing a hero - the unsung relay log - for high availability when GTIDs were not available.
simple introduction to hadoop
The document discusses data storage and processing. Data can be stored in memory or on disk using file systems like local XFS/ZFS or distributed systems like HDFS, S3, or Ceph. Distributed file systems allow for parallel processing of data by moving computation to the data locations. This map-reduce framework involves mapping functions to distributed data segments followed by reducing the results. Hadoop uses HDFS for storage and the MapReduce framework for distributed computation on large datasets across clusters.
Demystifying datastores
my attempt to demystify datastores.
how to choose a store that fits your needs what are the questions you need to ask ?
hbase hadoop mysql cassandra vertica etc
Visualising Basic Concepts of Docker
Docker allows for the delivery of applications using containers. Containers are lightweight and allow for multiple applications to run on the same host, unlike virtual machines which each require their own operating system. Docker images contain the contents and configuration needed to run an application. Images are built from manifests and layers of content and configuration are added. Running containers from images allows applications to be easily delivered and run. Containers can be connected to volumes to preserve data when the container is deleted. Docker networking allows containers to communicate and ports can be exposed to the host.
StormWars - when the data stream shrinks
Apache Storm is a stream processing framework that can be used to process real-time data from data streams like Apache Kafka or Amazon Kinesis. When data in Amazon Kinesis is repartitioned into new shards, the partition metadata used by Storm becomes invalid. To address this, a solution is to define a white list of shards for each Storm topology, so that individual topologies are not affected when shards are added or removed from the stream.
Punch clock for debugging apache storm
The document proposes a "Punch Clock" concept to help debug Apache Storm transactional topologies. A Punch Clock would record when batches of tuples enter and exit spouts and bolts. Each spout/bolt would have a Punch Card ID to track the batch. Punching in would add the ID to a data structure, punching out would remove it. This would help identify batches stuck in specific spouts/bolts on hosts. It could be exposed via JMX to aggregate data across worker JVMs running the spouts/bolts. The goal is to determine batch flow through the topology and find any that are stuck.
a wild Supposition: can MySQL be Kafka ?
Apache Kafka is a distributed publish-subscribe messaging system that uses a distributed commit log to store messages. It allows applications to publish and subscribe to streams of records. The document discusses some similarities and differences between using MySQL and Kafka for messaging, such as Kafka's ability to horizontally scale by adding more brokers with multiple partitions, versus the single partition in a MySQL instance. While unconventional, the document proposes that MySQL could potentially be used as a messaging system like Kafka.
Build your own Real Time Analytics and Visualization, Enable Complex Event Pr...
At the 5th Elephant BigData conference in bangalore, india , 27-july-2012.
https://fifthelephant.talkfunnel.com/2012/384-build-your-own-real-time-analytics-and-visualization-enable-complex-event-processing-event-patterns-and-aggregates
More from vishnu rao (10)
Build your own Real Time Analytics and Visualization, Enable Complex Event Pr...
Build your own Real Time Analytics and Visualization, Enable Complex Event Pr...
Recently uploaded
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
OpenID AuthZEN Interop Read Out - Authorization
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
Recently uploaded (20)
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Druid beginner performance tips
- 2. Playground ● Raw data 15 TB ● 1 Datasource ○ 8.5 TB ○ Each segment file 400 MB ○ 23 dimensions - 3 metrics (2sum & 1 hyperUniq) ● Hdfs Deep Storage ● 1 Broker - r3-2x-large
- 3. Some Tips ● DO-NOT enable Debug Log while running perf tests or in production. ● Set ChunkPeriod in QueryContext ● DO-NOT use pretty while making http request query. ○ curl -X POST "http://broker:8082/druid/v2/?pretty" -H 'content-type:application/json' -d @query ● Groupby version v2 seems to be better. ● Make Druid SMILE :) ○ Use Content-Type ‘application/x-jackson-smile’ while making http
- 4. Thank you :) I know this ppt is short! -Vishnu rao- mash213.wordpress.com Tweet @ sweetweet213 Jaihind213 @ gmail dot com