Apache tajo configuration
•
2 likes•898 views
This document discusses basic configurations in Apache Tajo 0.11, including cluster resources, concurrent disk access, and garbage collection. It recommends configuring the worker heap size, number of disks per node, minimum memory per task, number of tasks assigned per disk, and temporary directory locations. The document also notes that Tajo works well with default configurations and provides links for more information.
Report
Share
Report
Share
Download to read offline
Recommended
Performance evaluation of apache tajo
This document evaluates the performance of Apache Tajo compared to other systems like Hive and Spark SQL. It runs benchmarks using the TPC-DS benchmark on Google Cloud Platform. The evaluation shows that Tajo is generally the fastest system and scales well to larger datasets and clusters. Some queries that involve complex joins and aggregations see more room for improvement in Tajo. The document encourages contributors to get involved in optimizing and enhancing Tajo's performance.
Tajo case study bay area hug 20131105
This document summarizes the results of a performance test comparing Hive, Impala, and Tajo on queries against a 1.7TB dataset. Tajo outperformed Hive and Impala on scans with filters and joins. For queries with grouping, aggregation, and sorting, Tajo was faster than Hive and similar to or faster than Impala. The author concludes that even though Tajo materializes all results to HDFS, its performance is promising compared to Impala due to its dynamic task scheduling. Further performance enhancements are expected as the Tajo project continues.
[Paper Reading] Efficient Query Processing with Optimistically Compressed Has...
Modern query engines rely heavily on hash tables for query processing. Overall query performance and memory
footprint is often determined by how hash tables and the
tuples within them are represented. In this work, we propose three complementary techniques to improve this representation:
Domain-Guided Prefix Suppression bit-packs keys and values tightly to reduce hash table record width. Optimistic Splitting decomposes values (and operations on them) into (operations on) frequently-accessed and infrequently-accessed value slices.
By removing the infrequently-accessed value slices from the hash table record, it improves cache locality. The Unique Strings Selfaligned Region (USSR) accelerates handling frequently-occurring strings, which are very common in real-world data sets, by creating an on-the-fly dictionary of the most frequent strings. This allows executing many string operations with integer logic and reduces memory pressure.
We integrated these techniques into Vectorwise. On the TPC-H benchmark, our approach reduces peak memory consumption by 2–4× and improves performance by up to 1.5×. On a real-world BI workload, we measured a 2× improvement in performance and in micro-benchmarks we observed speedups of up to 25×.
Understanding blue store, Ceph's new storage backend - Tim Serong, SUSE
Audience Level
Intermediate
Synopsis
Ceph – the most popular storage solution for OpenStack – stores all data as a collection of objects. This object store was originally implemented on top of a POSIX filesystem, an approach that turned out to have a number of problems, notably with performance and complexity.
BlueStore, a new storage backend for Ceph, was created to solve these issues; the Ceph Jewel release included an early prototype. The code and on-disk format were declared stable (but experimental) for Ceph Kraken, and now in the upcoming Ceph Luminous release, BlueStore will be the recommended default storage backend.
With a 2-3x performance boost, you’ll want to look at migrating your Ceph clusters to BlueStore. This talk goes into detail about what BlueStore does, the problems it solves, and what you need to do to use it.
Speaker Bio:
Tim works for SUSE, hacking on Ceph and related technologies. He has spoken often about distributed storage and high availability at conferences such as linux.conf.au. In his spare time he wrangles pigs, chickens, sheep and ducks, and was declared by one colleague “teammate most likely to survive the zombie apocalypse”.
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
In this presentation, i have explained how Ceph Object Storage Performance can be improved drastically together with some object storage best practices, recommendations tips. I have also covered Ceph Shared Data Lake which is getting very popular.
Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys)
- Что такое SDS (общие места для (почти) всех решений — масштабирование, абстрагирование от аппаратных ресурсов, управление с помощью политик, кластерные ФС);
- Почему мы решили использовать SDS (нужно было объектное хранилище);
- Почему решили использовать именно Ceph, а не другие открытые (GlusterFS, Swift...) или проприетарные (IBM Elastic Storage, Huawei OceanStor) решения;
- Что еще умеет Ceph, кроме object storage (RBD, CephFS);
- Как работает Ceph (со стороны сервера);
- Что нового дает BlueStore по сравнению с классическим (поверх ФС);
- Сравнение производительности (метрики тестов);
- BlueStore — все еще tech preview;
- Заключение. Ссылки, литература.
Red Hat Gluster Storage
This document provides an overview and introduction to Red Hat Gluster Storage. It explains that Gluster Storage concatenates all disks from multiple servers into one large volume, allowing users to access much more storage. It also describes how Gluster Storage provides data protection through RAID 6 and replication between nodes, and can heal itself if a disk fails. Common use cases for Gluster Storage are also listed, such as network log analysis, HPC data, and big data analysis where large amounts of petabyte-scale storage are needed.
21st Athens Big Data Meetup - 1st Talk - Fast and simple data exploration wit...
Title: Fast and simple data exploration with ClickHouse
Speaker: Alexander Kuzmenkov (https://github.com/akuzm/)
Date: Thursday, March 5, 2020
Event: https://meetup.com/Athens-Big-Data/events/268379195/
Recommended
Performance evaluation of apache tajo
This document evaluates the performance of Apache Tajo compared to other systems like Hive and Spark SQL. It runs benchmarks using the TPC-DS benchmark on Google Cloud Platform. The evaluation shows that Tajo is generally the fastest system and scales well to larger datasets and clusters. Some queries that involve complex joins and aggregations see more room for improvement in Tajo. The document encourages contributors to get involved in optimizing and enhancing Tajo's performance.
Tajo case study bay area hug 20131105
This document summarizes the results of a performance test comparing Hive, Impala, and Tajo on queries against a 1.7TB dataset. Tajo outperformed Hive and Impala on scans with filters and joins. For queries with grouping, aggregation, and sorting, Tajo was faster than Hive and similar to or faster than Impala. The author concludes that even though Tajo materializes all results to HDFS, its performance is promising compared to Impala due to its dynamic task scheduling. Further performance enhancements are expected as the Tajo project continues.
[Paper Reading] Efficient Query Processing with Optimistically Compressed Has...
Modern query engines rely heavily on hash tables for query processing. Overall query performance and memory
footprint is often determined by how hash tables and the
tuples within them are represented. In this work, we propose three complementary techniques to improve this representation:
Domain-Guided Prefix Suppression bit-packs keys and values tightly to reduce hash table record width. Optimistic Splitting decomposes values (and operations on them) into (operations on) frequently-accessed and infrequently-accessed value slices.
By removing the infrequently-accessed value slices from the hash table record, it improves cache locality. The Unique Strings Selfaligned Region (USSR) accelerates handling frequently-occurring strings, which are very common in real-world data sets, by creating an on-the-fly dictionary of the most frequent strings. This allows executing many string operations with integer logic and reduces memory pressure.
We integrated these techniques into Vectorwise. On the TPC-H benchmark, our approach reduces peak memory consumption by 2–4× and improves performance by up to 1.5×. On a real-world BI workload, we measured a 2× improvement in performance and in micro-benchmarks we observed speedups of up to 25×.
Understanding blue store, Ceph's new storage backend - Tim Serong, SUSE
Audience Level
Intermediate
Synopsis
Ceph – the most popular storage solution for OpenStack – stores all data as a collection of objects. This object store was originally implemented on top of a POSIX filesystem, an approach that turned out to have a number of problems, notably with performance and complexity.
BlueStore, a new storage backend for Ceph, was created to solve these issues; the Ceph Jewel release included an early prototype. The code and on-disk format were declared stable (but experimental) for Ceph Kraken, and now in the upcoming Ceph Luminous release, BlueStore will be the recommended default storage backend.
With a 2-3x performance boost, you’ll want to look at migrating your Ceph clusters to BlueStore. This talk goes into detail about what BlueStore does, the problems it solves, and what you need to do to use it.
Speaker Bio:
Tim works for SUSE, hacking on Ceph and related technologies. He has spoken often about distributed storage and high availability at conferences such as linux.conf.au. In his spare time he wrangles pigs, chickens, sheep and ducks, and was declared by one colleague “teammate most likely to survive the zombie apocalypse”.
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
In this presentation, i have explained how Ceph Object Storage Performance can be improved drastically together with some object storage best practices, recommendations tips. I have also covered Ceph Shared Data Lake which is getting very popular.
Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys)
- Что такое SDS (общие места для (почти) всех решений — масштабирование, абстрагирование от аппаратных ресурсов, управление с помощью политик, кластерные ФС);
- Почему мы решили использовать SDS (нужно было объектное хранилище);
- Почему решили использовать именно Ceph, а не другие открытые (GlusterFS, Swift...) или проприетарные (IBM Elastic Storage, Huawei OceanStor) решения;
- Что еще умеет Ceph, кроме object storage (RBD, CephFS);
- Как работает Ceph (со стороны сервера);
- Что нового дает BlueStore по сравнению с классическим (поверх ФС);
- Сравнение производительности (метрики тестов);
- BlueStore — все еще tech preview;
- Заключение. Ссылки, литература.
Red Hat Gluster Storage
This document provides an overview and introduction to Red Hat Gluster Storage. It explains that Gluster Storage concatenates all disks from multiple servers into one large volume, allowing users to access much more storage. It also describes how Gluster Storage provides data protection through RAID 6 and replication between nodes, and can heal itself if a disk fails. Common use cases for Gluster Storage are also listed, such as network log analysis, HPC data, and big data analysis where large amounts of petabyte-scale storage are needed.
21st Athens Big Data Meetup - 1st Talk - Fast and simple data exploration wit...
Title: Fast and simple data exploration with ClickHouse
Speaker: Alexander Kuzmenkov (https://github.com/akuzm/)
Date: Thursday, March 5, 2020
Event: https://meetup.com/Athens-Big-Data/events/268379195/
BlueStore, A New Storage Backend for Ceph, One Year In
BlueStore is a new storage backend for Ceph OSDs that consumes block devices directly, bypassing the local XFS file system that is currently used today. It's design is motivated by everything we've learned about OSD workloads and interface requirements over the last decade, and everything that has worked well and not so well when storing objects as files in local files systems like XFS, btrfs, or ext4. BlueStore has been under development for a bit more than a year now, and has reached a state where it is becoming usable in production. This talk will cover the BlueStore design, how it has evolved over the last year, and what challenges remain before it can become the new default storage backend.
Intorduce to Ceph
This document introduces Ceph storage. It discusses the evolution of storage systems from individual disks to clustered storage. Ceph is introduced as a clustered, software-defined storage system that provides object, block, and file storage. Key Ceph components like RADOS, monitors, OSDs, and the CRUSH algorithm are explained. CRUSH provides a scalable and reliable way to determine object locations across multiple nodes. Python and command line interfaces for Ceph are also summarized. Finally, Yahoo's Ceph cluster architecture is briefly described.
Indexes don't mean slow inserts.
This document summarizes techniques for optimizing write performance in databases while maintaining indexes. It discusses how indexes can slow down writes due to overhead from index maintenance. It then covers various write optimization techniques used in databases like PostgreSQL, including insert buffers, cache-oblivious data structures, LSM trees, and covering indexes that can index multiple columns with a single index. The document argues that with the right techniques, databases can provide both fast writes and good read performance through indexing.
What is new in MariaDB 10.6?
On July 6, 2021, MariaDB 10.6 became generally available (production ready). This presentation focuses on the most important aspects of it as well as the influence it has. Improvements to InnoDB, SYS Schema Adoption, and deprecated variables and engines are all part of this presentation.
An introduction to Big-Data processing applying hadoop
I've prepared this presentation one year ago. It is about hadoop and big data processing using hadoop.
Bluestore
This document summarizes BlueStore, a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore manages metadata and data separately, with metadata stored in a key-value database (RocksDB) and data written directly to block devices. This avoids issues with POSIX filesystem transactions and enables more efficient features like checksumming, compression, and cloning. BlueStore addresses consistency and performance problems that arose with previous approaches like FileStore and NewStore.
Ceph Day KL - Bluestore
This document provides an overview and summary of BlueStore, a new storage backend for Ceph that provides faster performance compared to the previous FileStore backend. BlueStore uses a key/value database (RocksDB) to store metadata and writes data directly to block devices. It addresses issues with the transactional consistency of FileStore by avoiding double writes and using more efficient data structures. BlueStore aims to provide more natural transaction atomicity, efficient enumeration of objects, and optimal I/O patterns for different storage devices.
PostgreSQL performance archaeology
The document summarizes performance benchmarks comparing PostgreSQL versions from 7.4 to the latest version. The benchmarks show significant performance improvements across different workloads over time, including:
- Up to 6x faster transaction processing rates in pgbench
- Much better scalability to multiple cores and clients
- 6x faster query times for TPC-DS data warehouse benchmark
- Over 10x speedup for some full text search queries in latest version
The benchmarks demonstrate that PostgreSQL has become dramatically faster and more scalable over the past 10+ years, while also using less disk space and memory.
Availability and scalability in mongo
This document provides an overview of availability, scalability, and sharding in MongoDB. It discusses:
1) How replica sets provide availability and failover capabilities within a single data center.
2) The limitations of replica sets across multiple data centers in terms of write availability and consistency.
3) How sharding allows horizontal scaling across multiple servers and data centers to handle increased user and data growth beyond the capabilities of a single server.
4) The key components of a sharded cluster in MongoDB including config servers, mongos routers, shards, and how data is partitioned and distributed based on the shard key.
OSBConf 2015 | Scale out backups with bareos and gluster by niels de vos
During this talk, Niels will explain the basics of Gluster and show how Bareos integrates with it. Gluster provides a Software Defined Storage environment that can scale-out when the backup storage needs to grow. With a live demonstration Niels shows how simple it is to setup a small Gluster environment and configure Bareos to use the native Gluster protocol.
PostgreSQL on EXT4, XFS, BTRFS and ZFS
One of the most common question I'm asked by users and customer about configuring a Linux-based system for PostgreSQL is "What file fystem should I use to get the best performance?" Sadly, most of the recommendations is based on obsolete "common knowledge" passed from generation to generation. But in recent years the file systems improved a lot - we've seen both evolution of established file systems (EXT family, XFS, ...) and revolution in the form of BTRFS, ZFS or F2FS. And of course new kinds of hardware (SSDs) which certainly impacts the design of a file system.
Strategies for Backing Up MongoDB
For the first time this year, 10gen will be offering a track completely dedicated to Operations at MongoSV, 10gen's annual MongoDB user conference on December 4. Learn more at MongoSV.com
Come learn about the different ways to back up your single servers, replica sets, and sharded clusters
Gluster as Block Store in Containers
Gluster can provide block storage using LIO/TCMU. It was demonstrated providing an iSCSI block device from a Gluster volume, including block snapshots. Performance numbers were shown and it can integrate with containers by providing a persistent block device. Kubernetes was demonstrated using Gluster block storage by having nodes initiate iSCSI sessions to access a target device mounted in pods. Future work may include more testing, Heketi integration for provisioning, and hyper-convergence.
Setting up mongodb sharded cluster in 30 minutes
The document describes how to configure and deploy a MongoDB sharded cluster with 6 virtual machines in 30 minutes. It provides step-by-step instructions on installing MongoDB, setting up the config servers, adding shards, and enabling sharding for databases and collections. Key aspects include designating MongoDB instances as config servers, starting mongos processes connected to the config servers, adding shards by hostname and port, and enabling sharding on specific databases and collections with shard keys.
Java one2015 - Work With Hundreds of Hot Terabytes in JVMs
Presentation Summary: By leveraging on memory mapped files, the Chronicle Engine supports large maps that easily can exceed the size of your server’s RAM, thus allowing application developers to create huge JVM:s where data can be obtained quickly and with predictable latency. The Chronicle Engine can be synchronized with an underlying database using Speedment so that your in-memory maps will be “alive” and change whenever data changes in the underlying database. Speedment can also automatically derive domain models directly from the database so that you can start using the solution very quickly. Because the Java Maps are mapped onto files, the maps can also be shared instantly between several JVM:s and when you restart a JVM, it may start very quickly without having to reload data from the underlying database. The mapped files can be hundreds of terabytes which has been done in real world deployment cases.
MongoDB Backup & Disaster Recovery
Presentation on disaster recover for MongoDB Atlas. Different approaches to recover and restore MongoDb Data.
https://elang2.github.io/myblog/
A crash course in CRUSH
CRUSH is the powerful, highly configurable algorithm Red Hat Ceph Storage uses to determine how data is stored across the many servers in a cluster. A healthy Red Hat Ceph Storage deployment depends on a properly configured CRUSH map. In this session, we will review the Red Hat Ceph Storage architecture and explain the purpose of CRUSH. Using example CRUSH maps, we will show you what works and what does not, and explain why.
Presented at Red Hat Summit 2016-06-29.
Performance comparison of Distributed File Systems on 1Gbit networks
Compare the performance of a few Distributed File Systems on 1Gbit networks
- GlusterFS
- XtremeFS
- FhgFS
Vmfs
Fetch policy determines when pages are brought into main memory, with demand paging bringing pages in only when referenced while prepaging brings in more pages, often pages not referenced. The TLB caches page table entries to map virtual to physical addresses, with TLB misses similarly slow as instruction/data cache misses. For page replacement when no free frames exist, a page replacement algorithm selects a victim frame to swap out and free the frame.
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
This document discusses setting up and using Tajo, an Apache Hadoop-based data warehousing system, on AWS. It provides instructions on using Tajo Cloud to easily configure a Tajo cluster on AWS. Examples show how to connect external data from S3, perform queries, and analyze customer cohort data to understand purchase patterns over time. Tajo allows direct access to data in S3 and dynamic scaling of worker nodes, and its connector enables remote querying from SQL clients, Excel, and R.
Elasticsearch 101 - Cluster setup and tuning
Elasticsearch 101 provides an overview of setting up, configuring, and tuning an Elasticsearch cluster. It discusses hardware requirements including memory, avoiding high cardinality fields, indexing and querying data, and tooling. The document also covers potential issues like data loss during network partitions and exhausting available Java heap memory.
Backing up Wikipedia Databases
Slides presented at Percona Live Europe Open Source Database Conference 2019, Amsterdam, 2019-10-01.
Imagine a world where all Wikipedia articles disappear due to a human error or software bug. Sounds unreal? According to some estimations, it would take an excess of hundreds of million person-hours to be written again. To prevent that scenario from ever happening, our SRE team at Wikimedia recently refactored the relational database recovery system.
In this session, we will discuss how we backup 550TB of MariaDB data without impacting the 15 billion page views per month we get. We will cover what were our initial plans to replace the old infrastructure, how we achieved recovering 2TB databases in less than 30 minutes while maintaining per-table granularity, as well as the different types of backups we implemented. Lastly, we will talk about lessons learned, what went well, how our original plans changed and future work.
More Related Content
What's hot
BlueStore, A New Storage Backend for Ceph, One Year In
BlueStore is a new storage backend for Ceph OSDs that consumes block devices directly, bypassing the local XFS file system that is currently used today. It's design is motivated by everything we've learned about OSD workloads and interface requirements over the last decade, and everything that has worked well and not so well when storing objects as files in local files systems like XFS, btrfs, or ext4. BlueStore has been under development for a bit more than a year now, and has reached a state where it is becoming usable in production. This talk will cover the BlueStore design, how it has evolved over the last year, and what challenges remain before it can become the new default storage backend.
Intorduce to Ceph
This document introduces Ceph storage. It discusses the evolution of storage systems from individual disks to clustered storage. Ceph is introduced as a clustered, software-defined storage system that provides object, block, and file storage. Key Ceph components like RADOS, monitors, OSDs, and the CRUSH algorithm are explained. CRUSH provides a scalable and reliable way to determine object locations across multiple nodes. Python and command line interfaces for Ceph are also summarized. Finally, Yahoo's Ceph cluster architecture is briefly described.
Indexes don't mean slow inserts.
This document summarizes techniques for optimizing write performance in databases while maintaining indexes. It discusses how indexes can slow down writes due to overhead from index maintenance. It then covers various write optimization techniques used in databases like PostgreSQL, including insert buffers, cache-oblivious data structures, LSM trees, and covering indexes that can index multiple columns with a single index. The document argues that with the right techniques, databases can provide both fast writes and good read performance through indexing.
What is new in MariaDB 10.6?
On July 6, 2021, MariaDB 10.6 became generally available (production ready). This presentation focuses on the most important aspects of it as well as the influence it has. Improvements to InnoDB, SYS Schema Adoption, and deprecated variables and engines are all part of this presentation.
An introduction to Big-Data processing applying hadoop
I've prepared this presentation one year ago. It is about hadoop and big data processing using hadoop.
Bluestore
This document summarizes BlueStore, a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore manages metadata and data separately, with metadata stored in a key-value database (RocksDB) and data written directly to block devices. This avoids issues with POSIX filesystem transactions and enables more efficient features like checksumming, compression, and cloning. BlueStore addresses consistency and performance problems that arose with previous approaches like FileStore and NewStore.
Ceph Day KL - Bluestore
This document provides an overview and summary of BlueStore, a new storage backend for Ceph that provides faster performance compared to the previous FileStore backend. BlueStore uses a key/value database (RocksDB) to store metadata and writes data directly to block devices. It addresses issues with the transactional consistency of FileStore by avoiding double writes and using more efficient data structures. BlueStore aims to provide more natural transaction atomicity, efficient enumeration of objects, and optimal I/O patterns for different storage devices.
PostgreSQL performance archaeology
The document summarizes performance benchmarks comparing PostgreSQL versions from 7.4 to the latest version. The benchmarks show significant performance improvements across different workloads over time, including:
- Up to 6x faster transaction processing rates in pgbench
- Much better scalability to multiple cores and clients
- 6x faster query times for TPC-DS data warehouse benchmark
- Over 10x speedup for some full text search queries in latest version
The benchmarks demonstrate that PostgreSQL has become dramatically faster and more scalable over the past 10+ years, while also using less disk space and memory.
Availability and scalability in mongo
This document provides an overview of availability, scalability, and sharding in MongoDB. It discusses:
1) How replica sets provide availability and failover capabilities within a single data center.
2) The limitations of replica sets across multiple data centers in terms of write availability and consistency.
3) How sharding allows horizontal scaling across multiple servers and data centers to handle increased user and data growth beyond the capabilities of a single server.
4) The key components of a sharded cluster in MongoDB including config servers, mongos routers, shards, and how data is partitioned and distributed based on the shard key.
OSBConf 2015 | Scale out backups with bareos and gluster by niels de vos
During this talk, Niels will explain the basics of Gluster and show how Bareos integrates with it. Gluster provides a Software Defined Storage environment that can scale-out when the backup storage needs to grow. With a live demonstration Niels shows how simple it is to setup a small Gluster environment and configure Bareos to use the native Gluster protocol.
PostgreSQL on EXT4, XFS, BTRFS and ZFS
One of the most common question I'm asked by users and customer about configuring a Linux-based system for PostgreSQL is "What file fystem should I use to get the best performance?" Sadly, most of the recommendations is based on obsolete "common knowledge" passed from generation to generation. But in recent years the file systems improved a lot - we've seen both evolution of established file systems (EXT family, XFS, ...) and revolution in the form of BTRFS, ZFS or F2FS. And of course new kinds of hardware (SSDs) which certainly impacts the design of a file system.
Strategies for Backing Up MongoDB
For the first time this year, 10gen will be offering a track completely dedicated to Operations at MongoSV, 10gen's annual MongoDB user conference on December 4. Learn more at MongoSV.com
Come learn about the different ways to back up your single servers, replica sets, and sharded clusters
Gluster as Block Store in Containers
Gluster can provide block storage using LIO/TCMU. It was demonstrated providing an iSCSI block device from a Gluster volume, including block snapshots. Performance numbers were shown and it can integrate with containers by providing a persistent block device. Kubernetes was demonstrated using Gluster block storage by having nodes initiate iSCSI sessions to access a target device mounted in pods. Future work may include more testing, Heketi integration for provisioning, and hyper-convergence.
Setting up mongodb sharded cluster in 30 minutes
The document describes how to configure and deploy a MongoDB sharded cluster with 6 virtual machines in 30 minutes. It provides step-by-step instructions on installing MongoDB, setting up the config servers, adding shards, and enabling sharding for databases and collections. Key aspects include designating MongoDB instances as config servers, starting mongos processes connected to the config servers, adding shards by hostname and port, and enabling sharding on specific databases and collections with shard keys.
Java one2015 - Work With Hundreds of Hot Terabytes in JVMs
Presentation Summary: By leveraging on memory mapped files, the Chronicle Engine supports large maps that easily can exceed the size of your server’s RAM, thus allowing application developers to create huge JVM:s where data can be obtained quickly and with predictable latency. The Chronicle Engine can be synchronized with an underlying database using Speedment so that your in-memory maps will be “alive” and change whenever data changes in the underlying database. Speedment can also automatically derive domain models directly from the database so that you can start using the solution very quickly. Because the Java Maps are mapped onto files, the maps can also be shared instantly between several JVM:s and when you restart a JVM, it may start very quickly without having to reload data from the underlying database. The mapped files can be hundreds of terabytes which has been done in real world deployment cases.
MongoDB Backup & Disaster Recovery
Presentation on disaster recover for MongoDB Atlas. Different approaches to recover and restore MongoDb Data.
https://elang2.github.io/myblog/
A crash course in CRUSH
CRUSH is the powerful, highly configurable algorithm Red Hat Ceph Storage uses to determine how data is stored across the many servers in a cluster. A healthy Red Hat Ceph Storage deployment depends on a properly configured CRUSH map. In this session, we will review the Red Hat Ceph Storage architecture and explain the purpose of CRUSH. Using example CRUSH maps, we will show you what works and what does not, and explain why.
Presented at Red Hat Summit 2016-06-29.
Performance comparison of Distributed File Systems on 1Gbit networks
Compare the performance of a few Distributed File Systems on 1Gbit networks
- GlusterFS
- XtremeFS
- FhgFS
Vmfs
Fetch policy determines when pages are brought into main memory, with demand paging bringing pages in only when referenced while prepaging brings in more pages, often pages not referenced. The TLB caches page table entries to map virtual to physical addresses, with TLB misses similarly slow as instruction/data cache misses. For page replacement when no free frames exist, a page replacement algorithm selects a victim frame to swap out and free the frame.
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
This document discusses setting up and using Tajo, an Apache Hadoop-based data warehousing system, on AWS. It provides instructions on using Tajo Cloud to easily configure a Tajo cluster on AWS. Examples show how to connect external data from S3, perform queries, and analyze customer cohort data to understand purchase patterns over time. Tajo allows direct access to data in S3 and dynamic scaling of worker nodes, and its connector enables remote querying from SQL clients, Excel, and R.
What's hot (20)
BlueStore, A New Storage Backend for Ceph, One Year In
BlueStore, A New Storage Backend for Ceph, One Year In
An introduction to Big-Data processing applying hadoop
An introduction to Big-Data processing applying hadoop
OSBConf 2015 | Scale out backups with bareos and gluster by niels de vos
OSBConf 2015 | Scale out backups with bareos and gluster by niels de vos
Java one2015 - Work With Hundreds of Hot Terabytes in JVMs
Java one2015 - Work With Hundreds of Hot Terabytes in JVMs
Performance comparison of Distributed File Systems on 1Gbit networks
Performance comparison of Distributed File Systems on 1Gbit networks
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
Similar to Apache tajo configuration
Elasticsearch 101 - Cluster setup and tuning
Elasticsearch 101 provides an overview of setting up, configuring, and tuning an Elasticsearch cluster. It discusses hardware requirements including memory, avoiding high cardinality fields, indexing and querying data, and tooling. The document also covers potential issues like data loss during network partitions and exhausting available Java heap memory.
Backing up Wikipedia Databases
Slides presented at Percona Live Europe Open Source Database Conference 2019, Amsterdam, 2019-10-01.
Imagine a world where all Wikipedia articles disappear due to a human error or software bug. Sounds unreal? According to some estimations, it would take an excess of hundreds of million person-hours to be written again. To prevent that scenario from ever happening, our SRE team at Wikimedia recently refactored the relational database recovery system.
In this session, we will discuss how we backup 550TB of MariaDB data without impacting the 15 billion page views per month we get. We will cover what were our initial plans to replace the old infrastructure, how we achieved recovering 2TB databases in less than 30 minutes while maintaining per-table granularity, as well as the different types of backups we implemented. Lastly, we will talk about lessons learned, what went well, how our original plans changed and future work.
Tuning Linux for Databases.
This presentation discusses optimizing Linux systems for PostgreSQL databases. Linux is a good choice for databases due to its active development, features, stability, and community support. The presentation covers optimizing various system resources like CPU scheduling, memory, storage I/O, and power management to improve database performance. Specific topics include disabling transparent huge pages, tuning block I/O schedulers, and selecting appropriate scaling governors. The overall message is that Linux can be adapted for database workloads through testing and iterative changes.
Алексей Лесовский "Тюнинг Linux для баз данных. "
В любом проекте всегда есть данные которые важно не потерять. И если проект перешагнул размеры домашнего, то наверняка там уже используется что-то более надежное чем текстовые файлы на диске. Скорее всего это какая-то конкретная база данных. Учитывая важную роль баз данных к ним предъявляют жесткие требования производительности и стабильности. Отдельной особенностью является и операционная система на которой работает база данных. Опыт и практика подсказывает что это будет Linux - одна из наиболее популярных на сегодняшний день операционных систем. Прогрессивное развитие в сочетании с гибкостью и стабильностью делает Linux хорошим кандидатом в качестве платформы для баз данных (да и не только). Однако настройки по-умолчанию в большинстве дистрибутивов предназначены для широкого круга задач и потребностей. В этом докладе я расскажу о том, на какие настройки следует обратить внимание и как они влияют на работу СУБД. Как сделать работу СУБД более предсказуемой, как в простых условиях, так и в условиях непостоянной или растущей нагрузки.
MariaDB Server Performance Tuning & Optimization
This document discusses various techniques for optimizing MariaDB server performance, including:
- Tuning configuration settings like the buffer pool size, query cache size, and thread pool settings.
- Monitoring server metrics like CPU usage, memory usage, disk I/O, and MariaDB-specific metrics.
- Analyzing slow queries with the slow query log and EXPLAIN statements to identify optimization opportunities like adding indexes.
#GeodeSummit - Off-Heap Storage Current and Future Design
In this session we review the design of the newly released off heap storage feature in Apache Geode, and discuss use cases and potential direction for additional capabilities of this feature.
Oracle Performance On Linux X86 systems
On X86 systems, using an Unbreakable Enterprise Kernel (UEK) is recommended over other enterprise distributions as it provides better hardware support, security patches, and testing from the larger Linux community. Key configuration recommendations include enabling maximum CPU performance in BIOS, using memory types validated by Oracle, ensuring proper NUMA and CPU frequency settings, and installing only Oracle-validated packages to avoid issues. Monitoring tools like top, iostat, sar and ksar help identify any CPU, memory, disk or I/O bottlenecks.
Percona XtraBackup - New Features and Improvements
Percona XtraBackup is an open-source hot backup utility for MySQL - based servers that doesn't lock your database during the backup. In this talk, we will cover the latest development and new features introduced on Xtrabackup and its auxiliary tools: - Page Tracking - Azure Blob Storage Support - Exponential Backoff - Keyring Components - and more.
Optimizing Linux Servers
This document discusses optimizing Linux servers. It covers monitoring server performance using tools like top, vmstat and iostat to identify bottlenecks. It also discusses optimizing hardware, software, databases, disk I/O, memory, and network performance. The document provides tips for tuning Linux kernels, Apache, MySQL, OpenLDAP and other software. It emphasizes starting with monitoring to identify issues, making one change at a time, and using benchmark tools to test performance.
Zararfa SummerCamp 2012 - Performing fast backups in large scale environments...
This document discusses MySQL backup solutions and focuses on Percona's XtraBackup tool. It provides an overview of challenges with MySQL backups, available solutions, and key features of XtraBackup. The document demonstrates how to perform full and incremental backups with XtraBackup and provides examples of options like streaming, filtering tablespaces, and parallel copying. It also notes differences between the xtrabackup binary and innobackupex script and highlights new features in XtraBackup version 2.0.
PuppetConf 2016: An Introduction to Measuring and Tuning PE Performance – Cha...
This document provides an overview of measuring and tuning performance for the Puppet Enterprise (PE) platform. It discusses gathering data from PE services like Puppet Server and PuppetDB through JVM logging, metrics, and configurations. Important metrics for Puppet Server include JRuby usage and catalog compilation times. Tuning options involve adjusting JRuby capacity and rebalancing agent checkins. The document also covers monitoring PuppetDB for storage usage and command processing, as well as optimizing PostgreSQL query performance.
Apache Geode Offheap Storage
Motivation and goals for off-heap storage
Off-heap features and usage
Implementation overview
Preliminary benchmarks: off-heap vs. heap
Tips and best practices
MongoDb scalability and high availability with Replica-Set
One of the much awaited features in MongoDB 1.6 is replica sets, MongoDB replication solution providing automatic failover and recovery.
MongoDB High Availabiltity with Replica Sets
This talk will cover -
• What is Replica Set?
• Replication Process
• Advantaged of Replica Set vs master/slave
• How to set up replica set on production Demo
This video is tutorial for setting up the MongoDb replica-set ion production environment. In this i took 3 instances which have already mongo installed and running. This tutorial consists-:
1.Setup the each instance of replica set
2.modify the mongodb.conf to include replica set information
3.configure the servers to include in replica set
4.then cross checking if we kill one primary then secondary becomes primary or not.
SiteGround Tech TeamBuilding
The document describes Linux containerization and virtualization technologies including containers, control groups (cgroups), namespaces, and backups. It discusses:
1) How cgroups isolate and limit system resources for containers through mechanisms like cpuset, cpuacct, cpu, memory, blkio, and freezer.
2) How namespaces isolate processes by ID, mounting, networking, IPC, and other resources to separate environments for containers.
3) The new backup system which uses thin provisioning and snapshotting to efficiently backup container environments to backup servers and restore individual accounts or full servers as needed.
Loadays MySQL
This document discusses strategies for maintaining very large MySQL tables that have grown too big. It recommends creating a new database server with different configuration settings like InnoDB file per table to reduce size, using tools like MySQLTuner and tuning-primer to analyze settings, archiving old historical data with ptArchiver to reduce table sizes, and considering partitioning or changing the MySQL version. Monitoring tools like InnoDB status, global status, cacti and innotop are recommended to analyze server performance.
Backing up thousands of containers
This document discusses the challenges of building a backup system for thousands of containers. It describes the architecture of the backup systems the presenter has built over time, which use LVM and thin provisioning to back up each container to its own logical volume. Network speeds can be a bottleneck for large restores. The current system schedules daily incremental backups of 100,000 containers across multiple backup nodes. Issues encountered include corrupted metadata from too many logical volumes, sluggish performance from device scanning, and crashes from discarding data in older LVM versions. The system now backs up at the account level instead of whole containers to speed up restores.
linux monitoring and performance tunning
howto monitor linux server? what metrics are important when monitor server? what is related between metrics and monitoring tools? what are basic linux server optimization ? howto optimize ?
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
My session at AnDevCon Bostong, May 2013, Boston, MA.
This class introduces the concepts of AOSP and how to use it in order to configure and build one of the most popular Android devices available: The Android emulator, for an x86 target. You will then learn a reincarnation of the AOSP, intended to bring Android to as many x86 devices as possible. You will see its structure and compare it with the AOSP, and demonstrate how such a build works within Virtual Box, QEMU and more.
LEVEL: Intermediate
TOPIC AREA: Embedded Android
For Training/Consulting requests: info@thepscg.com
Bcache and Aerospike
This document discusses bcache, a Linux kernel feature that allows using solid-state drives (SSDs) as a cache for hard disk drives (HDDs). It creates a hybrid volume for improved performance. Bcache can cache an unlimited number of backing devices and features write-through caching, detection of sequential I/O, and throttling to prevent SSD congestion. Using bcache with Aerospike provides persistence with near SSD performance at lower cost than SSD-only or RAM-backed deployments.
Open Source Data Deduplication
Data deduplication is a hot topic in storage and saves significant disk space for many environments, with some trade offs. We’ll discuss what deduplication is and where the Open Source solutions are versus commercial offerings. Presentation will lean towards the practical – where attendees can use it in their real world projects (what works, what doesn’t, should you use in production, etcetera).
Similar to Apache tajo configuration (20)
#GeodeSummit - Off-Heap Storage Current and Future Design
#GeodeSummit - Off-Heap Storage Current and Future Design
Percona XtraBackup - New Features and Improvements
Percona XtraBackup - New Features and Improvements
Zararfa SummerCamp 2012 - Performing fast backups in large scale environments...
Zararfa SummerCamp 2012 - Performing fast backups in large scale environments...
PuppetConf 2016: An Introduction to Measuring and Tuning PE Performance – Cha...
PuppetConf 2016: An Introduction to Measuring and Tuning PE Performance – Cha...
MongoDb scalability and high availability with Replica-Set
MongoDb scalability and high availability with Replica-Set
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
More from Jihoon Son
Introduction to Apache Tajo: Future of Data Warehouse
Apache Tajo is a SQL-on-Hadoop system that provides both fast interactive analysis and stable long-running extract-transform-load (ETL) jobs. It supports various data formats and storage systems. Companies like SK Telecom and Bluehole Studio use Tajo for tasks such as data warehousing, game log analysis, and music streaming data discovery. Tajo is optimized for performance and supports features like cost-based query optimization and off-heap processing. Benchmark tests show it outperforms other SQL-on-Hadoop systems like Hive and Spark SQL.
Query optimization in Apache Tajo
This document discusses query optimization in Apache Tajo. It describes how Tajo generates logical, distributed, and local query plans and optimizes them using rule-based and cost-based techniques. Some key optimization techniques include pushing down filters, selecting join algorithms, and optimizing data partitioning progressively during query execution based on intermediate statistics. The document provides examples of query planning and optimization in Tajo.
Introduction to Apache Tajo: Data Warehouse for Big Data
Tajo can infer the schema of self-describing data formats like JSON, ORC, and Parquet at query execution time without needing to pre-define and store the schema separately. This allows Tajo to query nested, complex data without requiring tedious schema definition by the user. Tajo's support of self-describing formats simplifies the process of querying nested, hierarchical data from files like the JSON log example shown.
Apache Tajo on Swift: Bringing SQL to the OpenStack World
This slide was presented at the SK Telecom T Developer Forum. It contains the brief evaluation results of the query execution performance of Tajo on Swift.
I conducted two kinds of experiments; The first experiment was to compare the performance of Tajo with on another distributed storage, i.e., HDFS. And the second experiment was the scalability test of Swift.
Interestingly, the scan performance on Swift is slower more than two times than that on HDFS. In addition, the task scheduling time on Swift is much greater than that on HDFS, which means the query initialization cost is very high.
Apache Tajo on Swift
Apache Tajo supports OpenStack Swift as one of its data sources.
This slide is presented at OpenStack Day in Korea 2015.
Outline
● Introduction to OpenStack Swift
● Introduction to Apache Tajo
● Tajo on Swift
● Demo
● Our Roadmap
More from Jihoon Son (7)
Introduction to Apache Tajo: Future of Data Warehouse
Introduction to Apache Tajo: Future of Data Warehouse
Introduction to Apache Tajo: Data Warehouse for Big Data
Introduction to Apache Tajo: Data Warehouse for Big Data
Apache Tajo on Swift: Bringing SQL to the OpenStack World
Apache Tajo on Swift: Bringing SQL to the OpenStack World
Recently uploaded
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
Securing your Kubernetes cluster_ a step-by-step guide to success !
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Video Streaming: Then, Now, and in the Future
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Recently uploaded (20)
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Apache tajo configuration
- 1. Basic Configurations in Apache Tajo 0.11 Jihoon Son / Gruter Inc.
- 2. Preliminary ● This slides assume that your cluster is installed in fully distributed mode ○ For cluster setup, please see http://tajo.apache. org/docs/current/configuration/cluster_setup. html. 2
- 3. Configurations ● Cluster resource ● Concurrent disk access ● Garbage collection 3
- 4. Cluster Resource ● How efficiently utilize cluster resources ○ Ideally ... ■ In every machine ● Every core does some computation ● The whole memory is used ● Disk bandwidth is fully utilized 4
- 5. Cluster Resource ● Configurations ○ tajo-env.sh ■ TAJO_WORKER_HEAPSIZE ● Heap size allocated for each worker ● Default: 5000 MB (0.11.1 ~) ○ tajo-site.xml ■ tajo.worker.resource.disks ● # of disks per node ■ tajo.task.resource.min.memory-mb ● Minimum amount of allocatable memory per task ● Default: 1000 MB 5
- 6. http://www.cs.odu.edu/~cs471w/spring11/lectures/MassStorage_files/image002.jpg Concurrent Disk Access ● Frequent random access might degrade performance 6
- 7. Concurrent Disk Access ● Configurations ○ tajo-site.xml ■ tajo.worker.resource.disk.parallel-execution.num ● # of tasks assigned per disk ● Default: 2 ○ Increase for SSD ■ tajo.worker.tmpdir.locations ● Temporal directories which are used for query execution ● Recommended to use all available disks 7
- 8. Garbage Collection ● GC option in tajo-env.sh ○ -XX:+UseParallelOldGC ■ Parallel old GC ■ Default (0.11.1 ~) ○ -XX:+UseG1GC ■ G1 GC 8
- 9. Garbage Collection ● Migrate ParallelOldGC to G1GC when ○ Full GC durations are too long or too frequent ○ The rate of object allocation rate or promotion varies significantly ○ Undesired long garbage collection or compaction pauses (longer than 0.5 to 1 second) ● For further information, please see http://www. oracle.com/technetwork/tutorials/tutorials- 1876574.html. 9
- 10. Conclusion ● More configurations are found on http://tajo. apache.org/docs/current/configuration/tajo-site- xml.html ● But, Tajo works well with default configurations! 10
- 11. Get Involved! ● We are recruiting contributors! ● General ○ http://tajo.apache.org/ ● Getting Started ○ http://tajo.apache.org/docs/current/getting_started.html ● Downloads ○ http://tajo.apache.org/downloads.html ● Issue tracker ○ http://issues.apache.org/jira/browse/TAJO ● Join the mailing list ○ dev-subscribe@tajo.apache.org ○ issues-subscribe@tajo.apache.org 11
- 12. Q & A 12