DB vs. encryption
•
0 likes•111 views
The document discusses different approaches to storing sensitive data like credit card numbers in a database. It compares using full-disk encryption versus application-level encryption. Full-disk encryption protects against theft but prevents indexing and queries on encrypted fields. Application-level encryption requires decrypting and re-encrypting all data, moving processing to the application. The document proposes a solution of using a separate trusted component that can compare encrypted values to enable indexing and aggregation while keeping data encrypted in the database.
Report
Share
Report
Share
Download to read offline
Recommended
LDAP Injection
This document discusses LDAP injection, an attack where malicious code is inserted into a user input field to gain unauthorized access. It provides an overview of LDAP syntax and injection, demonstrating how an attacker can bypass authentication by closing parentheses in the username field. The document also notes LDAP injection can lead to privilege escalation and information disclosure. It recommends escaping special characters, using frameworks that encode LDAP queries, and applying least privilege to secure applications from LDAP injection.
App Security and Securing App
This document discusses securing apps and securing communication. It covers topics like secure network communication using TLS and SSL pinning, secure interprocess communication using URL schemes and universal links, securely storing data using the keychain and data protection APIs, and techniques for preventing data leakage. It provides an overview of the OWASP mobile security risks and recommends designing secure organizations through guidelines, training, testing, and secure development processes. Resources for mobile security tools and further reading are also included. The overall message is that security is an ongoing process rather than a single feature, and that no app is completely secure against a skilled attacker.
Fluentd and Docker - running fluentd within a docker container
Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.
Fun with Macros & Other Sneaky Tricks to Avoid Detection - SANS Manchester 2020
This document discusses techniques for bypassing endpoint detection and response (EDR) systems using living off the land binaries and macros. It describes what telemetry EDR systems collect and common evasion techniques like using bitsadmin, certutil, and installutil to download files, encode payloads, and execute code. The document warns that these system utilities could be abused to evade detection if used atypically and provides examples of abnormal usage patterns that security teams should watch out for.
Fluentd and Docker - running fluentd within a docker container
Fluentd is a data collection tool for unified logging that allows for extensible and reliable data collection. It uses a simple core with plugins to provide buffering, high availability, load balancing, and streaming data transfer based on JSON. Fluentd can collect log data from various sources and output to different destinations in a flexible way using its plugin architecture and configuration files. It is widely used in production for tasks like log aggregation, filtering, and forwarding.
Secure Code Review 101
Secure code review is probably the most effective technique to identify security bugs early in the system development lifecycle.
When used together with automated and manual penetration testing, code review can significantly increase the cost effectiveness of an application security verification effort. This presentation explain how can we start secure code review effectively.
Talking TUF: Securing Software Distribution
The Update Framework (TUF) secures new or existing software update systems by providing a specification and library that can be flexibly and universally integrated or natively implemented. The update procedure is notoriously susceptible to malicious attacks and TUF is designed to prevent these and other updater weaknesses.
Docker's Notary project integrates the Go implementation of TUF with Docker Content Trust to verify the publisher of Docker images.
https://github.com/theupdateframework/tuf
Shift Left Security
"Shift Lef Security" What the funk does that mean?
In the agile, lean, DevOps communities people talk about improving security by "shifting left". Patterns and tools are emerging, or re-emerging, that make security less of a pain in the development process while also making applications more secure.
Recommended
LDAP Injection
This document discusses LDAP injection, an attack where malicious code is inserted into a user input field to gain unauthorized access. It provides an overview of LDAP syntax and injection, demonstrating how an attacker can bypass authentication by closing parentheses in the username field. The document also notes LDAP injection can lead to privilege escalation and information disclosure. It recommends escaping special characters, using frameworks that encode LDAP queries, and applying least privilege to secure applications from LDAP injection.
App Security and Securing App
This document discusses securing apps and securing communication. It covers topics like secure network communication using TLS and SSL pinning, secure interprocess communication using URL schemes and universal links, securely storing data using the keychain and data protection APIs, and techniques for preventing data leakage. It provides an overview of the OWASP mobile security risks and recommends designing secure organizations through guidelines, training, testing, and secure development processes. Resources for mobile security tools and further reading are also included. The overall message is that security is an ongoing process rather than a single feature, and that no app is completely secure against a skilled attacker.
Fluentd and Docker - running fluentd within a docker container
Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.
Fun with Macros & Other Sneaky Tricks to Avoid Detection - SANS Manchester 2020
This document discusses techniques for bypassing endpoint detection and response (EDR) systems using living off the land binaries and macros. It describes what telemetry EDR systems collect and common evasion techniques like using bitsadmin, certutil, and installutil to download files, encode payloads, and execute code. The document warns that these system utilities could be abused to evade detection if used atypically and provides examples of abnormal usage patterns that security teams should watch out for.
Fluentd and Docker - running fluentd within a docker container
Fluentd is a data collection tool for unified logging that allows for extensible and reliable data collection. It uses a simple core with plugins to provide buffering, high availability, load balancing, and streaming data transfer based on JSON. Fluentd can collect log data from various sources and output to different destinations in a flexible way using its plugin architecture and configuration files. It is widely used in production for tasks like log aggregation, filtering, and forwarding.
Secure Code Review 101
Secure code review is probably the most effective technique to identify security bugs early in the system development lifecycle.
When used together with automated and manual penetration testing, code review can significantly increase the cost effectiveness of an application security verification effort. This presentation explain how can we start secure code review effectively.
Talking TUF: Securing Software Distribution
The Update Framework (TUF) secures new or existing software update systems by providing a specification and library that can be flexibly and universally integrated or natively implemented. The update procedure is notoriously susceptible to malicious attacks and TUF is designed to prevent these and other updater weaknesses.
Docker's Notary project integrates the Go implementation of TUF with Docker Content Trust to verify the publisher of Docker images.
https://github.com/theupdateframework/tuf
Shift Left Security
"Shift Lef Security" What the funk does that mean?
In the agile, lean, DevOps communities people talk about improving security by "shifting left". Patterns and tools are emerging, or re-emerging, that make security less of a pain in the development process while also making applications more secure.
Security in open source projects
The document discusses security in open source projects. It covers vulnerabilities in dependencies, detecting vulnerabilities in code bases, and improving security. Functionality is often prioritized over security. Various coding flaws are described like buffer overflows. Tools for analyzing dependencies, detecting vulnerabilities statically and dynamically are presented, like OWASP Dependency-Check, Snyk, SonarQube, ZAP. Best practices for open source security include following responsible disclosure policies and auditing code bases regularly.
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
This document summarizes a company's transition from a SQL database to a native graph database to power their identity resolution product. It describes the requirements of high read and write throughput and complex queries over billions of identities and linkages. It then outlines the evaluation of several graph databases, with JanusGraph on ScyllaDB performing the best. Key findings from prototyping include handling high query volume, managing supernodes, and tuning compaction strategies. The production implementation and architecture is also summarized.
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
Zeotap’s Connect product addresses the challenges of identity resolution and linking for AdTech and MarTech. Zeotap manages roughly 20 billion ID and growing. In their presentation, Zeotap engineers will delve into data access patterns, processing and storage requirements to make a case for a graph-based store. They will share the results of PoCs made on technologies such as D-graph, OrientDB, Aeropike and Scylla, present the reasoning for selecting JanusGraph backed by Scylla, and take a deep dive into their data model architecture from the point of ingestion. Learn what is required for the production setup, configuration and performance tuning to manage data at this scale.
WSO2Con USA 2015: An Introduction to the WSO2 Analytics Platform
In today’s connected world organizations have access to an enormous amount of data. We often don’t know what they mean or how we can use them, in terms of hindsight, oversight, insight and foresight, to gain competitive advantage in the market. Use cases ranging from simple system monitoring to complex fraud analysis demands this.
The WSO2 Data Analytics platform lets you collect data, allows you to explore it through batch, real-time, interactive and predictive processing technologies and allows you to communicate your results. In this talk, we will discuss the WSO2 Data Analytics platform and how it brings together all analytics technologies into a single platform and user experience.
Fluentd - Unified logging layer
Fluentd is an open source data collector that allows for flexible and extensible logging. It provides a unified way to collect logs, metrics, and events from various sources and send them to multiple destinations. It handles concerns like buffering, retries, and failover to provide reliable data transfer. Fluentd uses a plugin-based architecture so it can support many use cases like simple forwarding, lambda architectures, stream processing, and logging for Docker and Kubernetes.
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery
Every scientist who needs big data analytics to save millions of lives should have that power. Powering Interactive Data Analysis require massive architecture, and know-how to build a fast real-time computing system. You will learn how Google BigQuery solves this problem by enabling super-fast, SQL queries against petabytes of data using the processing power of Google’s infrastructure. After this session you will be able to work with BigQuery, do streaming inserts, write User Defined Functions in Javascript, and several use cases for everyday developer: funnel analytics, behavioral analytics, exploring unstructured data. You will be able to run arbitrary queries on open-data such as historical data about Github commits, Stackoverflow Q&A data, or analysing Reddit comments to find out books the community talks about.
G Data Retail 2011 English
G Data was founded in 1985 by Kai Figge. It is a security company headquartered in Germany with about 300 employees worldwide. They are known for releasing one of the first antivirus solutions in 1987. Some of G Data's unique technologies include DoubleScan Technology, fingerprinting of scanned files, whitelisting of system files, and cloud-based zero-hour virus and spam protection through inspection of over 50 million email inboxes. Their product lineup includes antivirus, internet security, and total security solutions for end users and companies.
Gartner Security & Risk Management Summit 2018
Slides from Paula Januszkiewicz's session "Crouching Admin, Hidden Hacker: Techniques for Attacking and Securing Your Infrastructure"
Game Analytics at London Apache Druid Meetup
This document summarizes a presentation about GameAnalytics' use of Apache Druid for analytics of user behavior data from video games. It discusses their data collection, annotation, and aggregation workflows. GameAnalytics ingests raw event data from SDKs and partners into S3, then an annotation system enriches the data before ingestion into Druid. They designed Druid schemas around dimensions, metrics and hyperloglog sketches to optimize queries for unique users and retention. Real-time and batch ingestion utilize Kinesis and EMR. Caching and tiering help optimize performance and costs. Future plans include A/B testing and funnel analysis using Druid.
Coding Security: Code Mania 101
The document discusses several coding security practices for developing secure software, including input validation, output handling, parameterizing queries, identity and authentication controls, and access controls. It provides examples and recommendations for implementing each practice to prevent common vulnerabilities like injection and data tampering. The goal is to integrate security at the code level from the beginning to reduce risks.
Secure Programming
In this presentation, we try to teach programmers how to avoid security flaws in the code.
The presentation is of the format of problem->solution->problem....
Given a piece of code the attendees have to identify the security bugs in it and the suggest a fix. Now, the attendees have to find security bugs in the fix. The exercise goes on and the attendees become secure code aware.
-- KnowBigData.com
XP Days 2019: First secret delivery for modern cloud-native applications
In this talk we’ll see how Authentication and Secrets delivery work in distributed containerized applications from the inside. We’ll start from the theory of security and will go through the topics like Container Auth Role, Static & Dynamic secrets, Env vars/volumes for secret delivery, Vault & K8S secrets. After this talk you’ll get an understanding how to securely deploy your containerized workloads.
Safeguarding artifact integrity in your Software Supply Chain
In questo talk descriveremo alcune best practice per migliorare la sicurezza della tua Software Supply Chain, capirai cos'è il framework SLSA e vedrai esempi di utilizzo degli strumenti di Google che aiutano a risolvere le vulnerabilità delle dipendenze sia nella fase di sviluppo che in quella di build, attestare l'integrità del processo di build, configurare gli ambienti di runtime per accettare solo codice attendibile e monitorare continuamente le applicazioni in esecuzione per nuove vulnerabilità.
OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...
The document discusses disaggregated data centers using OpenNebula. It describes how OpenNebula allows for scalability through elasticity and avoids issues from human/configuration errors. It discusses types of scalability like predictable, mixed/emergency, and unpredictable scalability. It also briefly discusses provisioning tools like Oneprovision and using provision templates in YAML format.
OpenNebulaConf 2019 - Crytek: A Video gaming Edge Implementation "on the shou...
Presentation covers various cyber security aspects that are stands behind the AAA-Level game projects. And what is most important it covers a practically proven way to provision own data (game services) in 22 geographical locations in 22 minutes, using opensource solution - OpenNebula and it's DDC features. During this 22 minutes you receive fully distributed mesh infrastructure, located in 22 different geo locations (datacenters) provisioned using only bare metal hardware servers, with preconfigured GNU/Linux OS and preconfigured VM on top of each server. Each server has own control server in own region with backconect to 'mother' server in central location with High Availability configured, own network segments in each datacenter, elastic IP's, Backend Transfer Facilities, Local BGP.
(In) Security graph database in real world
Graph databases are an "emerging" technology useful in the field of cybersecurity, especially in the detection of new threats based on the correlation of diverse sources of information. However, insufficient attention has been spent in terms of its security. In this talk, it will be reviewed the state of art of this kind of databases and its desing security problems, specially for Neo4J and OrientDB. We will release a hacking tool for testing and detecting graph databases and will show several examples of information leak in the real world.
Tool: https://github.com/grafscan/GraFScaN
Enhance system transparency and truthfulness with request tracing
Request tracing improves to the accessibility and discoverability of systems thus enhancing transparency.
This talk also shows a proof of concept real time rendering of a sequence diagram of a request using the tracing information.
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
The document compares various database companies and their products. It provides details on TmaxSoft, including its employee headcount, main products which include databases and middleware, and key technologies. It then discusses Tibero database specifically and how it compares to Oracle, including features, licensing model, compatibility, and security. Examples are also provided showing capabilities of Tibero compared to Oracle.
Dynamic Population Discovery for Lateral Movement (Using Machine Learning)
DEFCON XXIV - Wall of Sheep presentation. Dynamic population discovery for lateral movement (Using Machine Learning)
[PASS Summit 2016] Blazing Fast, Planet-Scale Customer Scenarios with Azure D...
Data analysts, data engineers, and application developers are supporting unprecedented rates of change, whether talking about latency requirements to the expanding arena of data usage scenarios. While the technology functionality must rapidly evolve to meet customer needs and respond to competitive pressures, how can we enhance the data platform to help manage this unpredictability?
To help address these realities, data practitioners from a diverse set of backgrounds are increasingly relying on schema-free, distributed, scalable, and high-performance data storage (also known as NoSQL databases). In this session, we will showcase a wide variety of customer scenarios, business goals, and technical challenges faced by real-world customers. More importantly, how adding Azure DocumentDB into a data practitioner's arsenal within the Microsoft/Azure data ecosystem will allow you to easily solve these complex design patterns at massive scale.
CREATE STATISTICS - What is it for? (PostgresLondon)
CREATE STATISTICS command was introduced in PostgreSQL 10, and improved in PostgreSQL 12. Let's see what queries it's meant to improve and how it works.
Data corruption
Let's discuss the various sources of data corruption in databases (and in PostgreSQL in general) - cosmic rays, storage, bugs in the OS, database or application. I'll share some general observations about those various sources, and also some recommendations how to detect those issues in PostgreSQL (data checksums, ...) and how to deal with them.
More Related Content
Similar to DB vs. encryption
Security in open source projects
The document discusses security in open source projects. It covers vulnerabilities in dependencies, detecting vulnerabilities in code bases, and improving security. Functionality is often prioritized over security. Various coding flaws are described like buffer overflows. Tools for analyzing dependencies, detecting vulnerabilities statically and dynamically are presented, like OWASP Dependency-Check, Snyk, SonarQube, ZAP. Best practices for open source security include following responsible disclosure policies and auditing code bases regularly.
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
This document summarizes a company's transition from a SQL database to a native graph database to power their identity resolution product. It describes the requirements of high read and write throughput and complex queries over billions of identities and linkages. It then outlines the evaluation of several graph databases, with JanusGraph on ScyllaDB performing the best. Key findings from prototyping include handling high query volume, managing supernodes, and tuning compaction strategies. The production implementation and architecture is also summarized.
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
Zeotap’s Connect product addresses the challenges of identity resolution and linking for AdTech and MarTech. Zeotap manages roughly 20 billion ID and growing. In their presentation, Zeotap engineers will delve into data access patterns, processing and storage requirements to make a case for a graph-based store. They will share the results of PoCs made on technologies such as D-graph, OrientDB, Aeropike and Scylla, present the reasoning for selecting JanusGraph backed by Scylla, and take a deep dive into their data model architecture from the point of ingestion. Learn what is required for the production setup, configuration and performance tuning to manage data at this scale.
WSO2Con USA 2015: An Introduction to the WSO2 Analytics Platform
In today’s connected world organizations have access to an enormous amount of data. We often don’t know what they mean or how we can use them, in terms of hindsight, oversight, insight and foresight, to gain competitive advantage in the market. Use cases ranging from simple system monitoring to complex fraud analysis demands this.
The WSO2 Data Analytics platform lets you collect data, allows you to explore it through batch, real-time, interactive and predictive processing technologies and allows you to communicate your results. In this talk, we will discuss the WSO2 Data Analytics platform and how it brings together all analytics technologies into a single platform and user experience.
Fluentd - Unified logging layer
Fluentd is an open source data collector that allows for flexible and extensible logging. It provides a unified way to collect logs, metrics, and events from various sources and send them to multiple destinations. It handles concerns like buffering, retries, and failover to provide reliable data transfer. Fluentd uses a plugin-based architecture so it can support many use cases like simple forwarding, lambda architectures, stream processing, and logging for Docker and Kubernetes.
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery
Every scientist who needs big data analytics to save millions of lives should have that power. Powering Interactive Data Analysis require massive architecture, and know-how to build a fast real-time computing system. You will learn how Google BigQuery solves this problem by enabling super-fast, SQL queries against petabytes of data using the processing power of Google’s infrastructure. After this session you will be able to work with BigQuery, do streaming inserts, write User Defined Functions in Javascript, and several use cases for everyday developer: funnel analytics, behavioral analytics, exploring unstructured data. You will be able to run arbitrary queries on open-data such as historical data about Github commits, Stackoverflow Q&A data, or analysing Reddit comments to find out books the community talks about.
G Data Retail 2011 English
G Data was founded in 1985 by Kai Figge. It is a security company headquartered in Germany with about 300 employees worldwide. They are known for releasing one of the first antivirus solutions in 1987. Some of G Data's unique technologies include DoubleScan Technology, fingerprinting of scanned files, whitelisting of system files, and cloud-based zero-hour virus and spam protection through inspection of over 50 million email inboxes. Their product lineup includes antivirus, internet security, and total security solutions for end users and companies.
Gartner Security & Risk Management Summit 2018
Slides from Paula Januszkiewicz's session "Crouching Admin, Hidden Hacker: Techniques for Attacking and Securing Your Infrastructure"
Game Analytics at London Apache Druid Meetup
This document summarizes a presentation about GameAnalytics' use of Apache Druid for analytics of user behavior data from video games. It discusses their data collection, annotation, and aggregation workflows. GameAnalytics ingests raw event data from SDKs and partners into S3, then an annotation system enriches the data before ingestion into Druid. They designed Druid schemas around dimensions, metrics and hyperloglog sketches to optimize queries for unique users and retention. Real-time and batch ingestion utilize Kinesis and EMR. Caching and tiering help optimize performance and costs. Future plans include A/B testing and funnel analysis using Druid.
Coding Security: Code Mania 101
The document discusses several coding security practices for developing secure software, including input validation, output handling, parameterizing queries, identity and authentication controls, and access controls. It provides examples and recommendations for implementing each practice to prevent common vulnerabilities like injection and data tampering. The goal is to integrate security at the code level from the beginning to reduce risks.
Secure Programming
In this presentation, we try to teach programmers how to avoid security flaws in the code.
The presentation is of the format of problem->solution->problem....
Given a piece of code the attendees have to identify the security bugs in it and the suggest a fix. Now, the attendees have to find security bugs in the fix. The exercise goes on and the attendees become secure code aware.
-- KnowBigData.com
XP Days 2019: First secret delivery for modern cloud-native applications
In this talk we’ll see how Authentication and Secrets delivery work in distributed containerized applications from the inside. We’ll start from the theory of security and will go through the topics like Container Auth Role, Static & Dynamic secrets, Env vars/volumes for secret delivery, Vault & K8S secrets. After this talk you’ll get an understanding how to securely deploy your containerized workloads.
Safeguarding artifact integrity in your Software Supply Chain
In questo talk descriveremo alcune best practice per migliorare la sicurezza della tua Software Supply Chain, capirai cos'è il framework SLSA e vedrai esempi di utilizzo degli strumenti di Google che aiutano a risolvere le vulnerabilità delle dipendenze sia nella fase di sviluppo che in quella di build, attestare l'integrità del processo di build, configurare gli ambienti di runtime per accettare solo codice attendibile e monitorare continuamente le applicazioni in esecuzione per nuove vulnerabilità.
OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...
The document discusses disaggregated data centers using OpenNebula. It describes how OpenNebula allows for scalability through elasticity and avoids issues from human/configuration errors. It discusses types of scalability like predictable, mixed/emergency, and unpredictable scalability. It also briefly discusses provisioning tools like Oneprovision and using provision templates in YAML format.
OpenNebulaConf 2019 - Crytek: A Video gaming Edge Implementation "on the shou...
Presentation covers various cyber security aspects that are stands behind the AAA-Level game projects. And what is most important it covers a practically proven way to provision own data (game services) in 22 geographical locations in 22 minutes, using opensource solution - OpenNebula and it's DDC features. During this 22 minutes you receive fully distributed mesh infrastructure, located in 22 different geo locations (datacenters) provisioned using only bare metal hardware servers, with preconfigured GNU/Linux OS and preconfigured VM on top of each server. Each server has own control server in own region with backconect to 'mother' server in central location with High Availability configured, own network segments in each datacenter, elastic IP's, Backend Transfer Facilities, Local BGP.
(In) Security graph database in real world
Graph databases are an "emerging" technology useful in the field of cybersecurity, especially in the detection of new threats based on the correlation of diverse sources of information. However, insufficient attention has been spent in terms of its security. In this talk, it will be reviewed the state of art of this kind of databases and its desing security problems, specially for Neo4J and OrientDB. We will release a hacking tool for testing and detecting graph databases and will show several examples of information leak in the real world.
Tool: https://github.com/grafscan/GraFScaN
Enhance system transparency and truthfulness with request tracing
Request tracing improves to the accessibility and discoverability of systems thus enhancing transparency.
This talk also shows a proof of concept real time rendering of a sequence diagram of a request using the tracing information.
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
The document compares various database companies and their products. It provides details on TmaxSoft, including its employee headcount, main products which include databases and middleware, and key technologies. It then discusses Tibero database specifically and how it compares to Oracle, including features, licensing model, compatibility, and security. Examples are also provided showing capabilities of Tibero compared to Oracle.
Dynamic Population Discovery for Lateral Movement (Using Machine Learning)
DEFCON XXIV - Wall of Sheep presentation. Dynamic population discovery for lateral movement (Using Machine Learning)
[PASS Summit 2016] Blazing Fast, Planet-Scale Customer Scenarios with Azure D...
Data analysts, data engineers, and application developers are supporting unprecedented rates of change, whether talking about latency requirements to the expanding arena of data usage scenarios. While the technology functionality must rapidly evolve to meet customer needs and respond to competitive pressures, how can we enhance the data platform to help manage this unpredictability?
To help address these realities, data practitioners from a diverse set of backgrounds are increasingly relying on schema-free, distributed, scalable, and high-performance data storage (also known as NoSQL databases). In this session, we will showcase a wide variety of customer scenarios, business goals, and technical challenges faced by real-world customers. More importantly, how adding Azure DocumentDB into a data practitioner's arsenal within the Microsoft/Azure data ecosystem will allow you to easily solve these complex design patterns at massive scale.
Similar to DB vs. encryption (20)
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
Zeotap: Moving to ScyllaDB - A Graph of Billions Scale
WSO2Con USA 2015: An Introduction to the WSO2 Analytics Platform
WSO2Con USA 2015: An Introduction to the WSO2 Analytics Platform
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery
XP Days 2019: First secret delivery for modern cloud-native applications
XP Days 2019: First secret delivery for modern cloud-native applications
Safeguarding artifact integrity in your Software Supply Chain
Safeguarding artifact integrity in your Software Supply Chain
OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...
OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...
OpenNebulaConf 2019 - Crytek: A Video gaming Edge Implementation "on the shou...
OpenNebulaConf 2019 - Crytek: A Video gaming Edge Implementation "on the shou...
Enhance system transparency and truthfulness with request tracing
Enhance system transparency and truthfulness with request tracing
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
Dynamic Population Discovery for Lateral Movement (Using Machine Learning)
Dynamic Population Discovery for Lateral Movement (Using Machine Learning)
[PASS Summit 2016] Blazing Fast, Planet-Scale Customer Scenarios with Azure D...
[PASS Summit 2016] Blazing Fast, Planet-Scale Customer Scenarios with Azure D...
More from Tomas Vondra
CREATE STATISTICS - What is it for? (PostgresLondon)
CREATE STATISTICS command was introduced in PostgreSQL 10, and improved in PostgreSQL 12. Let's see what queries it's meant to improve and how it works.
Data corruption
Let's discuss the various sources of data corruption in databases (and in PostgreSQL in general) - cosmic rays, storage, bugs in the OS, database or application. I'll share some general observations about those various sources, and also some recommendations how to detect those issues in PostgreSQL (data checksums, ...) and how to deal with them.
CREATE STATISTICS - what is it for?
Description of the CREATE STATISTICS command introduced in PostgreSQL 11 - when to use it, how it works, etc.
PostgreSQL performance improvements in 9.5 and 9.6
The document summarizes performance improvements in PostgreSQL versions 9.5 and 9.6. Some key improvements discussed include optimizations to sorting, hash joins, BRIN indexes, parallel query processing, aggregate functions, checkpoints, and freezing. Performance tests on sorting, hash joins, and parallel queries show significant speedups from these changes, such as faster sorting times and better scalability with parallel queries.
PostgreSQL na EXT4, XFS, BTRFS a ZFS / FOSDEM PgDay 2016
The document provides an overview of different file systems for PostgreSQL including EXT3/4, XFS, BTRFS and ZFS. It discusses the evolution and improvements made to EXT3/4 and XFS over time to address scalability, bugs and new storage technologies like SSDs. BTRFS and ZFS are described as designed for large data volumes and built-in features like snapshots and checksums but BTRFS is still considered experimental. Benchmark results show ZFS and optimized EXT4/XFS performing best and BTRFS performance significantly reduced due to copy-on-write. The conclusion recommends EXT4/XFS for traditional needs and ZFS for advanced features, avoiding BTRFS.
PostgreSQL na EXT4, XFS, BTRFS a ZFS / OpenAlt
Srovnání výkonu PostgreSQL na EXT3/4, XFS, BTRFS a ZFS (a obecně na současných Linuxových souborových systémech).
PostgreSQL on EXT4, XFS, BTRFS and ZFS
One of the most common question I'm asked by users and customer about configuring a Linux-based system for PostgreSQL is "What file fystem should I use to get the best performance?" Sadly, most of the recommendations is based on obsolete "common knowledge" passed from generation to generation. But in recent years the file systems improved a lot - we've seen both evolution of established file systems (EXT family, XFS, ...) and revolution in the form of BTRFS, ZFS or F2FS. And of course new kinds of hardware (SSDs) which certainly impacts the design of a file system.
Performance improvements in PostgreSQL 9.5 and beyond
This document discusses several performance improvements made in PostgreSQL versions 9.5 and beyond. Some key improvements discussed include:
- Faster sorting through allowing sorting by inlined functions, abbreviated keys for VARCHAR/TEXT/NUMERIC, and Sort Support benefits.
- Improved hash joins through reduced palloc overhead, smaller NTUP_PER_BUCKET, and dynamically resizing the hash table.
- Index improvements like avoiding index tuple copying, GiST and bitmap index scan optimizations, and block range tracking in BRIN indexes.
- Aggregate functions see speedups through using 128-bit integers for internal state instead of NUMERIC in some cases.
- Other optimizations affect PL/pgSQL performance,
Postgresql na EXT3/4, XFS, BTRFS a ZFS
Srovnání výkonu PostgreSQL na EXT3/4, XFS, BTRFS a ZFS (a obecně na současných Linuxových souborových systémech).
PostgreSQL on EXT4, XFS, BTRFS and ZFS
A comparison of how PostgreSQL performs on current Linux file systems - ext4, XFS, BTRFS and ZFS, with pgbench and (a subset of) TPC-DS.
PostgreSQL performance archaeology
The document summarizes performance benchmarks comparing PostgreSQL versions from 7.4 to the latest version. The benchmarks show significant performance improvements across different workloads over time, including:
- Up to 6x faster transaction processing rates in pgbench
- Much better scalability to multiple cores and clients
- 6x faster query times for TPC-DS data warehouse benchmark
- Over 10x speedup for some full text search queries in latest version
The benchmarks demonstrate that PostgreSQL has become dramatically faster and more scalable over the past 10+ years, while also using less disk space and memory.
Český fulltext a sdílené slovníky
Standardní implementace ispell slovníků v PostgreSQL má bohužel několik nevýhod co se CPU a paměti týká. Napsal jsem extension která umožňuje slovníky inicializovat jen jednou a sdílet je mezi spojeními.
SSD vs HDD / WAL, indexes and fsync
Jaký dopad na výkon má přesun transakčního logu nebo indexů na samostatný disk? Jak výsledek závisí na typu workloadu (OLTP vs. DWH) a na typu disku (HDD vs. SSD)?
Checkpoint (CSPUG 22.11.2011)
Vysvětlení pojmu checkpoint a interakce s write cache operačního systému. Ukázka jaké a proč vznikají problémy, možnosti ladění (OS, DB i hardware).
Čtení explain planu (CSPUG 21.6.2011)
Stručný úvod do čtení explain planu - seznámení s principem jak databáze vybírá postup vyhodnocení dotazu, na jaké základní operace v něm můžeme narazit (varianty čtení tabulek a joinů). Zejména se ale podíváme na to jak identifikovat problematická místa v exekučním plánu, v čem může být příčina a jak ji odstranit.
Replikace (CSPUG 19.4.2011)
Přehled variant databázové replikace (master-slave vs. master-master, physical vs. logical) a jejich pro a proti. Krátká historie replikace v PostgreSQL s přehled nástrojů které dnes máme k dispozici (slony-I, pgpool-II, Londiste a Bucardo). A nakonec stručné srovnání s replikací ve dvou populárních databázích (MySQL a Oracle).
PostgreSQL / Performance monitoring
Prezentace připravená pro konferenci Prague PostgreSQL Developers' Day 2010. Hlavním tématem je monitoring výkonu PostgreSQL databázového serveru, nástrojů které k tomu lze použít, atd. Na začátku jsou představeny základní pojmy (např. co je to snapshot), a poté je předvedeno několik užitečných nástrojů (jeden z nich vyvíjím já).
More from Tomas Vondra (19)
CREATE STATISTICS - What is it for? (PostgresLondon)
CREATE STATISTICS - What is it for? (PostgresLondon)
PostgreSQL performance improvements in 9.5 and 9.6
PostgreSQL performance improvements in 9.5 and 9.6
PostgreSQL na EXT4, XFS, BTRFS a ZFS / FOSDEM PgDay 2016
PostgreSQL na EXT4, XFS, BTRFS a ZFS / FOSDEM PgDay 2016
Performance improvements in PostgreSQL 9.5 and beyond
Performance improvements in PostgreSQL 9.5 and beyond
Recently uploaded
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Infrastructure Challenges in Scaling RAG with Custom AI models
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Recently uploaded (20)
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
DB vs. encryption
- 3. credit card numbers (or anything sensitive)
- 5. full-disk encryption ● data-at-rest protection (theft of device) ● SQL injection ● filesystem-level access ● evil DBA pgcrypto
- 6. full-disk encryption ● data-at-rest protection (theft of device) ● SQL injection ● filesystem-level access ● evil DBA pgcrypto ● data-in-flight protection ● easy to leak key into logs / monitoring systems
- 9. can't compare / hash values => no indexing, aggregation, ...
- 10. can't compare / hash values => no indexing, aggregation, ... (a lot of processing moves to app)
- 11. So what can we do about it?
- 16. application (encrypt + decrypt) database ● CREATE INDEX ● GROUP BY ● WHERE crypto (compare)
- 17. host B host C application (encrypt + decrypt) database ● CREATE INDEX ● GROUP BY ● WHERE crypto (compare) host A TCP
- 18. host B TrustZone / SGX HSM / usbarmory application (encrypt + decrypt) database ● CREATE INDEX ● GROUP BY ● WHERE crypto (compare) host A IPC
- 19. https://github.com/tvondra/ccnumber ● PoC / ugly prototype ● custom encrypted data type ● trusted component (comparator) ● communication over TCP/IP