This document provides a summary of key concepts related to data sharding and replication in YugabyteDB. It discusses three methods for sharding data across tablets: pre-splitting tablets at table creation, manual splitting of tablets at runtime, and automatic splitting of tablets as they grow beyond a size threshold. It also explains how YugabyteDB uses Raft consensus for replication, with tablets replicated across multiple nodes to provide fault tolerance. Transaction processing is achieved by writing provisional records to leader tablets and committing transactions once a majority is reached.
Apache Kudu - Updatable Analytical Storage #rakutentechCloudera Japan
This document provides an overview of Apache Kudu, an open source columnar storage system that enables fast analytics on fast changing data. It discusses Kudu's architecture including its use of tablets, replication using Raft consensus, and columnar storage with compression. The document also covers Kudu's write path involving memstores, delta memstores, and flushing to disk; its read path involving lookups without merging files; and compaction processes. Overall, the summary provides a high-level technical introduction to Kudu's capabilities and design.
cloudera Apache Kudu Updatable Analytical Storage for Modern Data PlatformRakuten Group, Inc.
Apache Kudu is an open source distributed storage for a real-time analytical workload. Since it supports Update and Inserts, Kudu can be used for both real-time operational database and analytic database. In this session, I will describe the detailed architecture of Kudu to reveal how it supports Update and Insert on columnar storage architecture.
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
This document provides a summary of best practices for high reliability data loading in ClickHouse. It discusses ClickHouse's ingestion pipeline and strategies for improving performance and reliability of inserts. Some key points include using larger block sizes for inserts, avoiding overly frequent or compressed inserts, optimizing partitioning and sharding, and techniques like buffer tables and compact parts. The document also covers ways to make inserts atomic and handle deduplication of records through block-level and logical approaches.
This document discusses strategies for maintaining very large MySQL tables that have grown too big. It recommends creating a new database server with different configuration settings like InnoDB file per table to reduce size, using tools like MySQLTuner and tuning-primer to analyze settings, archiving old historical data with ptArchiver to reduce table sizes, and considering partitioning or changing the MySQL version. Monitoring tools like InnoDB status, global status, cacti and innotop are recommended to analyze server performance.
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...Ceph Community
1) Arm-based microserver architecture can be used to build scalable and fault tolerant Ceph clusters with minimized failure domains. Using individual microservers for each OSD reduces the impact of single hardware failures.
2) Such an architecture provides benefits like lower power consumption, higher server density in racks, and ability to start HA Ceph clusters in smaller footprints compared to traditional server-based architectures.
3) Benchmarks show the Arm-based Ceph cluster provides over 2x higher performance and uses around half the power compared to an equivalent x86-based Ceph cluster.
Instaclustr has a diverse customer base including Ad Tech, IoT and messaging applications ranging from small start ups to large enterprises. In this presentation we share our experiences, common issues, diagnosis methods, and some tips and tricks for managing your Cassandra cluster.
About the Speaker
Brooke Jensen VP Technical Operations & Customer Services, Instaclustr
Instaclustr is the only provider of fully managed Cassandra as a Service in the world. Brooke Jensen manages our team of Engineers that maintain the operational performance of our diverse fleet clusters, as well as providing 24/7 advice and support to our customers. Brooke has over 10 years' experience as a Software Engineer, specializing in performance optimization of large systems and has extensive experience managing and resolving major system incidents.
Volume manager software provides logical volume management and virtualization of storage disks. It optimizes storage usage, increases filesystem limits, and provides flexibility, capacity, speed and resilience through features like mirroring and striping. Virtualization is performed by either the storage device itself or a software layer on the host system. Hot spares and snapshotting provide fault tolerance and online backups.
Fast Analytics (FA) uses an Enterprise Service Bus (ESB) to process high volumes of big data in real time, enabling decision makers to understand new trends and shifts as they occur. FA delivers analytics at decision-making speeds through technologies like Apache Kudu, which provides low latency random access and efficient analytical queries on columnar data. Kudu uses a log-structured storage approach and Raft consensus algorithm to replicate data across nodes for reliability and high availability.
Apache Kudu - Updatable Analytical Storage #rakutentechCloudera Japan
This document provides an overview of Apache Kudu, an open source columnar storage system that enables fast analytics on fast changing data. It discusses Kudu's architecture including its use of tablets, replication using Raft consensus, and columnar storage with compression. The document also covers Kudu's write path involving memstores, delta memstores, and flushing to disk; its read path involving lookups without merging files; and compaction processes. Overall, the summary provides a high-level technical introduction to Kudu's capabilities and design.
cloudera Apache Kudu Updatable Analytical Storage for Modern Data PlatformRakuten Group, Inc.
Apache Kudu is an open source distributed storage for a real-time analytical workload. Since it supports Update and Inserts, Kudu can be used for both real-time operational database and analytic database. In this session, I will describe the detailed architecture of Kudu to reveal how it supports Update and Insert on columnar storage architecture.
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
This document provides a summary of best practices for high reliability data loading in ClickHouse. It discusses ClickHouse's ingestion pipeline and strategies for improving performance and reliability of inserts. Some key points include using larger block sizes for inserts, avoiding overly frequent or compressed inserts, optimizing partitioning and sharding, and techniques like buffer tables and compact parts. The document also covers ways to make inserts atomic and handle deduplication of records through block-level and logical approaches.
This document discusses strategies for maintaining very large MySQL tables that have grown too big. It recommends creating a new database server with different configuration settings like InnoDB file per table to reduce size, using tools like MySQLTuner and tuning-primer to analyze settings, archiving old historical data with ptArchiver to reduce table sizes, and considering partitioning or changing the MySQL version. Monitoring tools like InnoDB status, global status, cacti and innotop are recommended to analyze server performance.
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...Ceph Community
1) Arm-based microserver architecture can be used to build scalable and fault tolerant Ceph clusters with minimized failure domains. Using individual microservers for each OSD reduces the impact of single hardware failures.
2) Such an architecture provides benefits like lower power consumption, higher server density in racks, and ability to start HA Ceph clusters in smaller footprints compared to traditional server-based architectures.
3) Benchmarks show the Arm-based Ceph cluster provides over 2x higher performance and uses around half the power compared to an equivalent x86-based Ceph cluster.
Instaclustr has a diverse customer base including Ad Tech, IoT and messaging applications ranging from small start ups to large enterprises. In this presentation we share our experiences, common issues, diagnosis methods, and some tips and tricks for managing your Cassandra cluster.
About the Speaker
Brooke Jensen VP Technical Operations & Customer Services, Instaclustr
Instaclustr is the only provider of fully managed Cassandra as a Service in the world. Brooke Jensen manages our team of Engineers that maintain the operational performance of our diverse fleet clusters, as well as providing 24/7 advice and support to our customers. Brooke has over 10 years' experience as a Software Engineer, specializing in performance optimization of large systems and has extensive experience managing and resolving major system incidents.
Volume manager software provides logical volume management and virtualization of storage disks. It optimizes storage usage, increases filesystem limits, and provides flexibility, capacity, speed and resilience through features like mirroring and striping. Virtualization is performed by either the storage device itself or a software layer on the host system. Hot spares and snapshotting provide fault tolerance and online backups.
Fast Analytics (FA) uses an Enterprise Service Bus (ESB) to process high volumes of big data in real time, enabling decision makers to understand new trends and shifts as they occur. FA delivers analytics at decision-making speeds through technologies like Apache Kudu, which provides low latency random access and efficient analytical queries on columnar data. Kudu uses a log-structured storage approach and Raft consensus algorithm to replicate data across nodes for reliability and high availability.
This document provides an overview of Google Bigtable, a distributed storage system for structured data. It discusses Bigtable's design including its use of column families, row keys, and versioning. It also describes Bigtable's basic implementation including its use of the Google File System (GFS) and how data is divided into tablets and distributed across tablet servers. The document then discusses related systems like HBase and how it compares to Bigtable. It provides examples of Bigtable's performance and real-world usage at Google. Finally, it poses some thoughts for discussion and provides useful references for further information.
This document discusses managing cluster parameters in Virtuozzo Storage. It describes:
- Cluster parameters control creating, locating, and managing replicas for data chunks. They include replication, encoding, and location parameters.
- Replication parameters define the normal and minimum number of replicas for data chunks. These are usually set to 3 and 2 respectively.
- Location parameters like failure domains determine where replicas are placed, such as by host, rack, or room, to avoid correlated failures bringing down all replicas.
- Encoding can provide redundancy through erasure coding instead of replication.
Understanding the architecture of MariaDB ColumnStoreMariaDB plc
MariaDB ColumnStore extends MariaDB Server, a relational database for transaction processing, with distributed columnar storage and parallel query processing for scalable, high-performance analytical processing. This session helps MariaDB users understand how MariaDB ColumnStore works and why it’s needed for more demanding analytical workloads, and covers:
Use cases
Query processing
Bulk data insertion
Distributed partitions
Query optimization
3 Things to Learn About:
-How Kudu is able to fill the analytic gap between HDFS and Apache HBase
-The trade-offs between real-time transactional access and fast analytic performance
-How Kudu provides an option to achieve fast scans and random access from a single API
M|18 Understanding the Architecture of MariaDB ColumnStoreMariaDB plc
The document provides an overview of MariaDB ColumnStore, including its history, components, disk storage architecture, writing and querying data processes. It was presented by Andrew Hutchings, the lead software engineer for MariaDB ColumnStore, who has previous experience with MySQL, HP, and other companies. The presentation covers the technical use cases for ColumnStore, differences from row-oriented databases, and optimizations for ColumnStore.
Bigdata netezza-ppt-apr2013-bhawani nandan prasadBhawani N Prasad
The document discusses various topics related to Netezza architecture and administration. It covers Netezza architecture including data stream processing. It also discusses connectivity, NZSQL, data types, metadata tables, joins, data loading and unloading, data distribution, transactions, groom/reclaim process, zone maps, and generate statistics. User management topics like creating and managing users and groups are also covered. The document provides details on permissions, backups, restores and performance tuning in Netezza.
XPDS13: VIRTUAL DISK INTEGRITY IN REAL TIME JP BLAKE, ASSURED INFORMATION SE...The Linux Foundation
This paper introduces the Virtual Disk Integrity in Real Time (vDIRT) monitor, a mechanism to measure virtual hard disks in real time from the Dom0 trusted computing base. vDIRT is an improvement over traditional methods for auditing file integrity which rely on a service in a potentially compromised host. It also overcomes the limitations of existing methods for assuring disk integrity that are coarse grained and do not scale to large disks. vDIRT is a capability to measure disk reads and writes in real time, allowing for fine grained tracking of sectors within files, as well as the overall disk. The vDIRT implementation and its impact on performance is discussed to show that disk operation monitoring from Dom0 is practical.
This document provides an overview of Oracle 12c Sharded Database Management. It defines what sharding is, how it works, and the benefits it provides such as extreme scalability, fault isolation, and cost reduction. It discusses Oracle's implementation of sharding using database partitioning and Global Data Services (GDS). Key concepts covered include shards, chunks, consistent hashing, and how Oracle supports operations across shards through GDS request routing.
PostgreSQL uses MVCC which creates multiple versions of rows during updates and deletes. This leads to bloat and fragmentation over time as unused row versions accumulate. The VACUUM command performs garbage collection to recover space from dead rows. HOT updates and pruning help reduce bloat by avoiding index bloat during certain updates. Future improvements include parallel and eager vacuuming as well as pluggable storage engines like zheap to further reduce bloat.
As part of NoSQL series, I presented Google Bigtable paper. In presentation I tried to give some plain introduction to Hadoop, MapReduce, HBase
www.scalability.rs
This document discusses Bronto's use of HBase for their marketing platform. Some key points:
- Bronto uses HBase for high volume scenarios, realtime data access, batch processing, and as a staging area for HDFS.
- HBase tables at Bronto are designed with the read/write patterns and necessary queries in mind. Row keys and column families are structured to optimize for these access patterns.
- Operations of HBase at scale require tuning of JVM settings, monitoring tools, and custom scripts to handle compactions and prevent cascading failures during high load. Table design also impacts operations and needs to account for expected workloads.
This document discusses ScyllaDB's process for sizing a Scylla cluster. It begins by outlining the importance of understanding business, application, and infrastructure requirements. Then it walks through building a sample system based on provided workload details. It shows how the sample system could be configured on different cloud platforms like AWS, Azure, and GCP. Finally, it highlights Scylla's sizing sheet tool for helping to determine hardware needs based on workload characteristics and performance goals.
The Google File System was designed by Google to store and manage large files across thousands of commodity servers. It uses a single master to manage metadata and track file locations across chunkservers. Chunks are replicated for reliability and placed across racks to improve bandwidth utilization. The system provides high throughput for concurrent reads and writes through leases to maintain consistency and pipelining of data flows. Logs and replication are used to provide fault tolerance against server failures.
A Comprehensive Introduction to Apache Cassandra.
Agenda:
- What is NoSQL?
- What is Cassandra?
- Architecture
- Data Model
- Key Features and Benefits
- Cassandra Tools
-- CQL
-- Nodetool
-- DataStax Opscenter
- Who’s using Cassandra?
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmAnne Nicolas
Building a full kernel takes time but is often necessary during development or when backporting patches. The nature of the kernel makes it easy to distribute its build on multiple cheap machines. This presentation will explain how to set up a build farm based on cost, size, and performance.
Willy Tarreau, HaProxy
In this day and age, data grows so fast it’s not uncommon for those of us using a relational database to reach the limits of its capacity. In this session, Kwangbock Lee explains how Samsung uses ClustrixDB to handle fast-growing data without manual database sharding. He highlights lessons learned, including a few hiccups along the way, and shares Samsung's experience migrating to ClustrixDB.
This document discusses Percona Fractal Tree (TokuDB) and compares it to B-Trees and LSM trees. It begins by explaining the limitations of B-Trees for write-heavy workloads and large datasets. It then introduces LSM trees and Fractal Trees as alternatives designed for better write performance. The bulk of the document describes the internals of Fractal Trees, including their use of messages to delay and combine writes. It provides recommendations for configuring Fractal Tree settings and discusses when Fractal Trees are most useful compared to other structures. In the end, it briefly mentions the history and applications of LSM trees.
Netezza uses a proprietary architecture called Asymmetric Massively Parallel Processing (AMPP). The AMPP architecture distributes data and query processing across multiple processing blades called S-Blades. Each S-Blade contains processors, memory, and is connected to disk arrays through a database accelerator card. This architecture allows Netezza to process large volumes of data in parallel across the S-Blades for high performance. Netezza also uses some unique tools and concepts compared to traditional databases, such as not enforcing constraints for improved load performance and using hidden columns to track transaction details instead of redo logs.
The document provides an overview of getting started with YugabyteDB, including installing it locally using binaries or Docker, deploying it on Kubernetes with Helm, using the managed YugabyteDB service, loading sample data and benchmarking workloads. It also describes next steps like the Yugabyte University training and certification.
This document discusses Concourse for Devops at Quoine. It provides background on the speaker and Quoine, describes some challenges with their previous CI/CD approach, reasons for adopting Concourse, current state of using Concourse and Kubernetes for Devops at Quoine, examples of Concourse pipelines, and a note about hiring.
This document provides an overview of Google Bigtable, a distributed storage system for structured data. It discusses Bigtable's design including its use of column families, row keys, and versioning. It also describes Bigtable's basic implementation including its use of the Google File System (GFS) and how data is divided into tablets and distributed across tablet servers. The document then discusses related systems like HBase and how it compares to Bigtable. It provides examples of Bigtable's performance and real-world usage at Google. Finally, it poses some thoughts for discussion and provides useful references for further information.
This document discusses managing cluster parameters in Virtuozzo Storage. It describes:
- Cluster parameters control creating, locating, and managing replicas for data chunks. They include replication, encoding, and location parameters.
- Replication parameters define the normal and minimum number of replicas for data chunks. These are usually set to 3 and 2 respectively.
- Location parameters like failure domains determine where replicas are placed, such as by host, rack, or room, to avoid correlated failures bringing down all replicas.
- Encoding can provide redundancy through erasure coding instead of replication.
Understanding the architecture of MariaDB ColumnStoreMariaDB plc
MariaDB ColumnStore extends MariaDB Server, a relational database for transaction processing, with distributed columnar storage and parallel query processing for scalable, high-performance analytical processing. This session helps MariaDB users understand how MariaDB ColumnStore works and why it’s needed for more demanding analytical workloads, and covers:
Use cases
Query processing
Bulk data insertion
Distributed partitions
Query optimization
3 Things to Learn About:
-How Kudu is able to fill the analytic gap between HDFS and Apache HBase
-The trade-offs between real-time transactional access and fast analytic performance
-How Kudu provides an option to achieve fast scans and random access from a single API
M|18 Understanding the Architecture of MariaDB ColumnStoreMariaDB plc
The document provides an overview of MariaDB ColumnStore, including its history, components, disk storage architecture, writing and querying data processes. It was presented by Andrew Hutchings, the lead software engineer for MariaDB ColumnStore, who has previous experience with MySQL, HP, and other companies. The presentation covers the technical use cases for ColumnStore, differences from row-oriented databases, and optimizations for ColumnStore.
Bigdata netezza-ppt-apr2013-bhawani nandan prasadBhawani N Prasad
The document discusses various topics related to Netezza architecture and administration. It covers Netezza architecture including data stream processing. It also discusses connectivity, NZSQL, data types, metadata tables, joins, data loading and unloading, data distribution, transactions, groom/reclaim process, zone maps, and generate statistics. User management topics like creating and managing users and groups are also covered. The document provides details on permissions, backups, restores and performance tuning in Netezza.
XPDS13: VIRTUAL DISK INTEGRITY IN REAL TIME JP BLAKE, ASSURED INFORMATION SE...The Linux Foundation
This paper introduces the Virtual Disk Integrity in Real Time (vDIRT) monitor, a mechanism to measure virtual hard disks in real time from the Dom0 trusted computing base. vDIRT is an improvement over traditional methods for auditing file integrity which rely on a service in a potentially compromised host. It also overcomes the limitations of existing methods for assuring disk integrity that are coarse grained and do not scale to large disks. vDIRT is a capability to measure disk reads and writes in real time, allowing for fine grained tracking of sectors within files, as well as the overall disk. The vDIRT implementation and its impact on performance is discussed to show that disk operation monitoring from Dom0 is practical.
This document provides an overview of Oracle 12c Sharded Database Management. It defines what sharding is, how it works, and the benefits it provides such as extreme scalability, fault isolation, and cost reduction. It discusses Oracle's implementation of sharding using database partitioning and Global Data Services (GDS). Key concepts covered include shards, chunks, consistent hashing, and how Oracle supports operations across shards through GDS request routing.
PostgreSQL uses MVCC which creates multiple versions of rows during updates and deletes. This leads to bloat and fragmentation over time as unused row versions accumulate. The VACUUM command performs garbage collection to recover space from dead rows. HOT updates and pruning help reduce bloat by avoiding index bloat during certain updates. Future improvements include parallel and eager vacuuming as well as pluggable storage engines like zheap to further reduce bloat.
As part of NoSQL series, I presented Google Bigtable paper. In presentation I tried to give some plain introduction to Hadoop, MapReduce, HBase
www.scalability.rs
This document discusses Bronto's use of HBase for their marketing platform. Some key points:
- Bronto uses HBase for high volume scenarios, realtime data access, batch processing, and as a staging area for HDFS.
- HBase tables at Bronto are designed with the read/write patterns and necessary queries in mind. Row keys and column families are structured to optimize for these access patterns.
- Operations of HBase at scale require tuning of JVM settings, monitoring tools, and custom scripts to handle compactions and prevent cascading failures during high load. Table design also impacts operations and needs to account for expected workloads.
This document discusses ScyllaDB's process for sizing a Scylla cluster. It begins by outlining the importance of understanding business, application, and infrastructure requirements. Then it walks through building a sample system based on provided workload details. It shows how the sample system could be configured on different cloud platforms like AWS, Azure, and GCP. Finally, it highlights Scylla's sizing sheet tool for helping to determine hardware needs based on workload characteristics and performance goals.
The Google File System was designed by Google to store and manage large files across thousands of commodity servers. It uses a single master to manage metadata and track file locations across chunkservers. Chunks are replicated for reliability and placed across racks to improve bandwidth utilization. The system provides high throughput for concurrent reads and writes through leases to maintain consistency and pipelining of data flows. Logs and replication are used to provide fault tolerance against server failures.
A Comprehensive Introduction to Apache Cassandra.
Agenda:
- What is NoSQL?
- What is Cassandra?
- Architecture
- Data Model
- Key Features and Benefits
- Cassandra Tools
-- CQL
-- Nodetool
-- DataStax Opscenter
- Who’s using Cassandra?
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmAnne Nicolas
Building a full kernel takes time but is often necessary during development or when backporting patches. The nature of the kernel makes it easy to distribute its build on multiple cheap machines. This presentation will explain how to set up a build farm based on cost, size, and performance.
Willy Tarreau, HaProxy
In this day and age, data grows so fast it’s not uncommon for those of us using a relational database to reach the limits of its capacity. In this session, Kwangbock Lee explains how Samsung uses ClustrixDB to handle fast-growing data without manual database sharding. He highlights lessons learned, including a few hiccups along the way, and shares Samsung's experience migrating to ClustrixDB.
This document discusses Percona Fractal Tree (TokuDB) and compares it to B-Trees and LSM trees. It begins by explaining the limitations of B-Trees for write-heavy workloads and large datasets. It then introduces LSM trees and Fractal Trees as alternatives designed for better write performance. The bulk of the document describes the internals of Fractal Trees, including their use of messages to delay and combine writes. It provides recommendations for configuring Fractal Tree settings and discusses when Fractal Trees are most useful compared to other structures. In the end, it briefly mentions the history and applications of LSM trees.
Netezza uses a proprietary architecture called Asymmetric Massively Parallel Processing (AMPP). The AMPP architecture distributes data and query processing across multiple processing blades called S-Blades. Each S-Blade contains processors, memory, and is connected to disk arrays through a database accelerator card. This architecture allows Netezza to process large volumes of data in parallel across the S-Blades for high performance. Netezza also uses some unique tools and concepts compared to traditional databases, such as not enforcing constraints for improved load performance and using hidden columns to track transaction details instead of redo logs.
Similar to Gwenn - Advanced level unlocked_.pdf (20)
The document provides an overview of getting started with YugabyteDB, including installing it locally using binaries or Docker, deploying it on Kubernetes with Helm, using the managed YugabyteDB service, loading sample data and benchmarking workloads. It also describes next steps like the Yugabyte University training and certification.
This document discusses Concourse for Devops at Quoine. It provides background on the speaker and Quoine, describes some challenges with their previous CI/CD approach, reasons for adopting Concourse, current state of using Concourse and Kubernetes for Devops at Quoine, examples of Concourse pipelines, and a note about hiring.
This document discusses a Cloud Foundry logging stack Bosh release that replaces the default syslog-based logging with Kafka. It notes that Kafka can ingest millions of logs per second, provides persistence, and is easier to consume than syslog. The release includes Kafka, a Firehose nozzle to send logs to Kafka, Kafka Manager, and a syslog forwarder. It will create Kafka topics for each event type and forward logs and app info to any syslog endpoint. The full release is available on GitHub.
This document discusses using webhooks in Concourse pipelines to reduce API calls. It begins with an introduction and background on the author. It then provides an overview of Concourse architecture and the roles of the ATC, workers, and resources/jobs. Next, it explains how webhooks can be used resource-agnostically in Concourse by defining a webhook token. It describes how webhooks allow external services like GitHub to notify Concourse of changes instead of Concourse constantly checking. Finally, it mentions a demo, online resources, documentation, Slack community, and invites questions.
- Concourse is a open source CI/CD tool that allows building and testing projects in containers. It uses a pluggable resource interface to check for changes and run builds in isolated containers.
- While Concourse natively supports running builds in containers, it does not support running external services like databases that may be needed to test against.
- One solution is to use "Docker in Docker in Garden" (DCINDG), which runs a Docker daemon inside a Docker container managed by Concourse's Garden container runtime. This allows testing tasks to use Docker Compose to stand up external services in isolated containers.
- The presentation demonstrated this approach with a sample project on GitHub and discussed Concourse concepts like jobs, resources
Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.
Monitor Cloud Foundry and Bosh with PrometheusGwenn Etourneau
This document discusses monitoring Cloud Foundry and BOSH with the open source monitoring system Prometheus. It introduces the firehose_exporter tool for exporting Cloud Foundry metrics to Prometheus and the prometheus-boshrelease for monitoring BOSH. The document ends with a demo of these monitoring solutions and a note about how Pivotal is transforming how software is built through products like Cloud Foundry and BOSH.
This document contains a summary of updates to the Concourse continuous integration and delivery (CI/CD) platform between versions 1.2.0 through 2.0.0. Key updates include adding the ability to pin builds to specific resource versions in 1.2.0, introducing build and test workflows in a single pipeline in 1.3.0, improving container retention and build log loading in 1.4.0 and 1.5.0, adding official Docker images and AWS ECR support in 1.6.0, and integrating multi-tenant team support with authentication in 2.0.0. The presentation also provides demonstrations of Concourse in action and links to documentation, tutorials, public pipelines, and Slack
This document summarizes a presentation about Pivotal's route service in Cloud Foundry. The route service allows requests to be forwarded to an external endpoint before being routed to applications. This can be used for authentication, rate limiting, inspection, or integrating with other systems. The presentation demonstrates configuring a route service in Cloud Foundry to block a hack attempt and provides examples of rate limiting, logging, and web application firewall route services on GitHub. It also reviews documentation on the Cloud Foundry route service.
This document summarizes a presentation about Bosh 2.0. It introduces Bosh as a tool for deploying software using deployment files. Bosh 2.0 features a new cloud-config file that defines availability zones, resource pools, and networks, separating this infrastructure configuration from deployment manifests. It also includes first-class support for availability zones and improved link functionality between jobs.
This document summarizes a presentation about managing Docker images with Concourse. It discusses building Docker images from a Dockerfile, triggering image builds when the Dockerfile or dependencies change. It also covers storing and pushing images, and using a Concourse pipeline to check for new images and code commits, rebuild images, tag images and the code repository with the new version.
- Concourse is a CI/CD tool that uses pipelines defined in YAML to automate workflows. It runs builds inside containers for isolation.
- It has three main concepts: resources that define inputs/outputs, tasks that define individual build steps, and jobs that define the actions in the pipeline.
- Concourse uses a pluggable resource model so many types of resources can be used as inputs or outputs like Git, Docker images, S3, etc. It can also integrate custom resource types.
- Tasks always behave the same way if inputs are the same. Jobs determine the order of tasks and resources in the pipeline.
- Concourse is installed either locally with Vagrant or on a cluster with Bosh
The document discusses Cloud Foundry route services, which allow requests to applications to be forwarded through an external service. A route service can be used for offloading authentication, rate limiting, inspecting requests, or integrating with other systems. It describes how route services work by intercepting requests from the router. The usage and configuration of route services is also covered, including creating user-provided and service broker types. Examples of rate limiting and reverse proxy route services are provided.
This document discusses Lattice, an open source Platform as a Service (PaaS) that was born from CloudFoundry. Lattice aims to make deploying and running containerized workloads easy through features like easy installation, clustering, scheduling, self-healing, load balancing, and log aggregation. The document provides an overview of Lattice's architecture and components like Diego for scheduling and X-Ray for visualization. It also demonstrates how to deploy Docker images and buildpacks, submit custom workloads, configure routing, and view logs using Lattice.
Securing BGP: Operational Strategies and Best Practices for Network Defenders...APNIC
Md. Zobair Khan,
Network Analyst and Technical Trainer at APNIC, presented 'Securing BGP: Operational Strategies and Best Practices for Network Defenders' at the Phoenix Summit held in Dhaka, Bangladesh from 23 to 24 May 2024.
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...APNIC
Adli Wahid, Senior Internet Security Specialist at APNIC, delivered a presentation titled 'Honeypots Unveiled: Proactive Defense Tactics for Cyber Security' at the Phoenix Summit held in Dhaka, Bangladesh from 23 to 24 May 2024.
HijackLoader Evolution: Interactive Process HollowingDonato Onofri
CrowdStrike researchers have identified a HijackLoader (aka IDAT Loader) sample that employs sophisticated evasion techniques to enhance the complexity of the threat. HijackLoader, an increasingly popular tool among adversaries for deploying additional payloads and tooling, continues to evolve as its developers experiment and enhance its capabilities.
In their analysis of a recent HijackLoader sample, CrowdStrike researchers discovered new techniques designed to increase the defense evasion capabilities of the loader. The malware developer used a standard process hollowing technique coupled with an additional trigger that was activated by the parent process writing to a pipe. This new approach, called "Interactive Process Hollowing", has the potential to make defense evasion stealthier.