Postgres continues to get more and more feature rich. But equally as impressive is the network of extensions that are growing around Postgres. With the rich extension APIs you can now add advanced functionality to Postgres without having to fork the codebase or wait for the main PostgreSQL release cycle. In this talk we'll cover some of the basics of what an extension is and then take a tour through a variety of Postgres extensions including:
pg_stat_statments
PostGIS
HyperLogLog and TopN
Timescale
pg_partman
Citus
Foreign data wrappers which are their own whole class
Obtaining the Perfect Smoke By Monitoring Your BBQ with InfluxDB and TelegrafInfluxData
Did you know you can use InfluxDB to monitor your BBQ and to ensure the tastiest results? Join this meetup to learn two different approaches to using a time series database to monitor a BBQ or a smoker. Learn how Will Cooke uses Python, MQTT, Telegraf and InfluxDB 2.0 to monitor his smoker and to gain insight into temperature changes, the stall, and other important stats about his brisket. Scott Anderson will demonstrate how he uses a FireBoard wireless thermometer, Telegraf and InfluxDB 2.0 to continuously work towards the perfect smoke.
Scaling infrastructure is tricky,
I will try to explain what methods I use when dealing with this issue, and demonstrate an approach which can be applied to almost any type of work load.
Obtaining the Perfect Smoke By Monitoring Your BBQ with InfluxDB and TelegrafInfluxData
Did you know you can use InfluxDB to monitor your BBQ and to ensure the tastiest results? Join this meetup to learn two different approaches to using a time series database to monitor a BBQ or a smoker. Learn how Will Cooke uses Python, MQTT, Telegraf and InfluxDB 2.0 to monitor his smoker and to gain insight into temperature changes, the stall, and other important stats about his brisket. Scott Anderson will demonstrate how he uses a FireBoard wireless thermometer, Telegraf and InfluxDB 2.0 to continuously work towards the perfect smoke.
Scaling infrastructure is tricky,
I will try to explain what methods I use when dealing with this issue, and demonstrate an approach which can be applied to almost any type of work load.
Обзорный рассказ про новые возможности в мире PostgreSQL для митапа Big Data Minsk User Group 29 апреля 2016 г.: https://www.facebook.com/events/120784531655479/
Nyc open data project ii -- predict where to get and return my citibikeVivian S. Zhang
NYC Data Science Academy, NYC Open Data Meetup, Big Data, Data Science, NYC, Vivian Zhang, SupStat Inc,NYC, GBM, Machine learning, Time Series, Citibike usage prodiction, advanced R
Extending Spark SQL API with Easier to Use Array Types Operations with Marek ...Databricks
Big companies typically integrate their data from various heterogeneous systems when building a data lake as single point for accessing data. To achieve this goal technical teams often deal with data defined by complex schemas and various data formats. Spark SQL Datasets are currently compatible with data formats such as XML, Avro and Parquet by providing primitive and complex data types such as structs and arrays.
Although Dataset API offers rich set of functions, general manipulation of array and deeply nested data structures is lacking. We will demonstrate this fact by providing examples of data which is currently very hard to process in Spark efficiently. We designed and developed an extension of Dataset API to allow developers to work with array and complex type elements in a more straightforward and consistent way. The extension should help users dealing with complex and structured big data to use Apache Spark as a truly generic processing framework.
Presentació a càrrec d'Ismael Fernández i Cristian
Gomollón (tècnics d'Aplicacions al CSUC) duta a terme a la "3a Jornada de formació sobre l'ús del servei de càlcul" celebrada el 29 d'octubre de 2020 en format virtual.
Intel Galileo acts as a chess client: read the chess figures position (NFC tags) and send the positions into the cloud. An Intel Edison reads the positions, calculates the best move (with Stockfish) and write the result back to the cloud. The Intel Galileo reads the bestmove result and shows it.
Apply Hammer Directly to Thumb; Avoiding Apache Spark and Cassandra AntiPatt...Databricks
Learn from someone who has made just about every basic Apache Spark mistake possible so you don’t have to! We’ll go over some of the most common things that users do that end up doing that cause unnecessary pain and actually explain how to avoid them.
Confused about serialization? Not sure what is meant by use a singleton to share connections? Together we will walk through concrete examples of how to handle these situation. Learn how to: do all your work remotely, not break your catalyst optimizations, use all your resources, and much more! Together lets learn how to make our Spark Applications better!
Molecular Shape Searching on GPUs: A Brave New WorldCan Ozdoruk
Shape is a fundamental three dimensional molecular property and a powerful descriptor for molecular comparison and similarity assessment; similarity in shape has proven to be a very effective method for predicting similarity in biology. As such shape-based virtual screening has become an integral part of computational drug discovery, due to both its speed and efficacy. OpenEye’s recent port of their shape similarity application, ROCS, to the GPU has resulted in a virtual screening tool of unprecedented power – FastROCS. FastROCS’ speed allows it to perform large-scale calculations of a kind inaccessible in the past and has accelerated more routine shape searching to the point that it has become competitive with more traditional, but less effective, two dimensional methods. Go through the slides to learn more. Try GPUs for free here: www.Nvidia.com/GPUTestDrive
Presentació a càrrec d'Adrián Macía, cap de Càlcul Científic del CSUC, duta a terme a la "3a Jornada de formació sobre l'ús del servei de càlcul" celebrada el 29 d'octubre de 2020 en format virtual.
early benchmarks on pre-release Gnocchi v4. includes benchmark comparison between all-ceph v3.x driver versus all-ceph v4 driver. also, shows benchmark using redis+ceph deployment.
ClickHouse and the Magic of Materialized Views, By Robert Hodges and Altinity...Altinity Ltd
Presented at the webinar, June 26, 2019
Materialized views are a killer feature of ClickHouse that can speed up queries 20X or more. Our webinar will teach you how to use this potent tool starting with how to create materialized views and load data. We'll then walk through cookbook examples to solve practical problems like deriving aggregates that outlive base data, answering last point queries, and using AggregateFunctions to handle problems like counting unique values, which is a special ClickHouse feature. There will be time for Q&A at the end. At that point you'll be a wizard of ClickHouse materialized views and able to cast spells of your own.
Como analisar planos de execução e estatísticas no PostgreSQL.
- Rastreamento de consultas lentas
- Uso do EXPLAIN
- Métodos de acesso
- Junções
- Parâmetros relevantes para o otimizador
Review this presentation to learn what it means to support a spatial database, and start to see the power of answering spatial questions inside a Postgres database.
Обзорный рассказ про новые возможности в мире PostgreSQL для митапа Big Data Minsk User Group 29 апреля 2016 г.: https://www.facebook.com/events/120784531655479/
Nyc open data project ii -- predict where to get and return my citibikeVivian S. Zhang
NYC Data Science Academy, NYC Open Data Meetup, Big Data, Data Science, NYC, Vivian Zhang, SupStat Inc,NYC, GBM, Machine learning, Time Series, Citibike usage prodiction, advanced R
Extending Spark SQL API with Easier to Use Array Types Operations with Marek ...Databricks
Big companies typically integrate their data from various heterogeneous systems when building a data lake as single point for accessing data. To achieve this goal technical teams often deal with data defined by complex schemas and various data formats. Spark SQL Datasets are currently compatible with data formats such as XML, Avro and Parquet by providing primitive and complex data types such as structs and arrays.
Although Dataset API offers rich set of functions, general manipulation of array and deeply nested data structures is lacking. We will demonstrate this fact by providing examples of data which is currently very hard to process in Spark efficiently. We designed and developed an extension of Dataset API to allow developers to work with array and complex type elements in a more straightforward and consistent way. The extension should help users dealing with complex and structured big data to use Apache Spark as a truly generic processing framework.
Presentació a càrrec d'Ismael Fernández i Cristian
Gomollón (tècnics d'Aplicacions al CSUC) duta a terme a la "3a Jornada de formació sobre l'ús del servei de càlcul" celebrada el 29 d'octubre de 2020 en format virtual.
Intel Galileo acts as a chess client: read the chess figures position (NFC tags) and send the positions into the cloud. An Intel Edison reads the positions, calculates the best move (with Stockfish) and write the result back to the cloud. The Intel Galileo reads the bestmove result and shows it.
Apply Hammer Directly to Thumb; Avoiding Apache Spark and Cassandra AntiPatt...Databricks
Learn from someone who has made just about every basic Apache Spark mistake possible so you don’t have to! We’ll go over some of the most common things that users do that end up doing that cause unnecessary pain and actually explain how to avoid them.
Confused about serialization? Not sure what is meant by use a singleton to share connections? Together we will walk through concrete examples of how to handle these situation. Learn how to: do all your work remotely, not break your catalyst optimizations, use all your resources, and much more! Together lets learn how to make our Spark Applications better!
Molecular Shape Searching on GPUs: A Brave New WorldCan Ozdoruk
Shape is a fundamental three dimensional molecular property and a powerful descriptor for molecular comparison and similarity assessment; similarity in shape has proven to be a very effective method for predicting similarity in biology. As such shape-based virtual screening has become an integral part of computational drug discovery, due to both its speed and efficacy. OpenEye’s recent port of their shape similarity application, ROCS, to the GPU has resulted in a virtual screening tool of unprecedented power – FastROCS. FastROCS’ speed allows it to perform large-scale calculations of a kind inaccessible in the past and has accelerated more routine shape searching to the point that it has become competitive with more traditional, but less effective, two dimensional methods. Go through the slides to learn more. Try GPUs for free here: www.Nvidia.com/GPUTestDrive
Presentació a càrrec d'Adrián Macía, cap de Càlcul Científic del CSUC, duta a terme a la "3a Jornada de formació sobre l'ús del servei de càlcul" celebrada el 29 d'octubre de 2020 en format virtual.
early benchmarks on pre-release Gnocchi v4. includes benchmark comparison between all-ceph v3.x driver versus all-ceph v4 driver. also, shows benchmark using redis+ceph deployment.
ClickHouse and the Magic of Materialized Views, By Robert Hodges and Altinity...Altinity Ltd
Presented at the webinar, June 26, 2019
Materialized views are a killer feature of ClickHouse that can speed up queries 20X or more. Our webinar will teach you how to use this potent tool starting with how to create materialized views and load data. We'll then walk through cookbook examples to solve practical problems like deriving aggregates that outlive base data, answering last point queries, and using AggregateFunctions to handle problems like counting unique values, which is a special ClickHouse feature. There will be time for Q&A at the end. At that point you'll be a wizard of ClickHouse materialized views and able to cast spells of your own.
Como analisar planos de execução e estatísticas no PostgreSQL.
- Rastreamento de consultas lentas
- Uso do EXPLAIN
- Métodos de acesso
- Junções
- Parâmetros relevantes para o otimizador
Review this presentation to learn what it means to support a spatial database, and start to see the power of answering spatial questions inside a Postgres database.
MySQL performance monitoring using Statsd and GraphiteDB-Art
This session will explain how you can leverage the MySQL-StatsD collector, StatsD and Graphite to monitor your database performance with metrics sent every second. In the past few years Graphite has become the de facto standard for monitoring large and scalable infrastructures.
This session will cover the architecture, functional basics and dashboard creation using Grafana. MySQL-StatsD is really easy to set up and configure. It will allow you to fetch your most important metrics from MySQL, run your own custom queries to parse your production data and if necessary transform this data into something different that can be used as a metric. Having this data with a fine granularity allows you to correlate your production data, system metrics with your MySQL performance metrics.
MySQL-StatsD is a daemon written in Python that was created during one of the hackdays at my previous employer (Spil Games) to solve the issue of fetching data from MySQL using a light weight client and send metrics to StatsD. I currently maintain this open source project on Github as it is my duty as creator of the project to look after it.
Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...Ontico
СУБД PostgreSQL — это огромный механизм, который состоит из множества подсистем, чья работа определяет производительность PostgreSQL. В процессе эксплуатации обеспечивается сбор статистики и информации о работе компонентов, что позволяет оценить эффективность PostgreSQL и принять меры для повышения производительности. Однако, этой информации очень много и представлена она в достаточно упрощенном виде. Обработка этой информации и ее интерпретация порой совсем нетривиальная задача, а зоопарк инструментов и утилит запросто поставит в тупик даже продвинутого DBA.
В докладе речь пойдет о подсистеме сбора статистики, о том какая информация доступна для оценки эффективности PostgreSQL, как её получить, не прибегая к зоопарку инструментов. Как интерпретировать и использовать полученную информацию, как найти узкие места, устранить их и повысить производительность PostgreSQL.
OSMC 2021 | pg_stat_monitor: A cool extension for better database (PostgreSQL...NETWAYS
The pg_stat_monitor is the statistics collection tool based on PostgreSQL’s contrib module pg_stat_statements. PostgreSQL’s pg_stat_statements provides only basic statistics, which is sometimes not enough. The major shortcoming in pg_stat_statements is that it accumulates all the queries and statistics, but does not provide aggregated statistics or histogram information. In this case, a user needs to calculate the aggregate, which is quite expensive. Pg_stat_monitor provides the pre-calculated aggregates. pg_stat_monitor collects and aggregates data on a bucket basis. The size and number of buckets should be configured using GUC (Grand Unified Configuration). The buckets are used to collect the statistics and aggregate them in a bucket. The talk will cover the usage of pg_stat_monitor and how it is better than pg_stat_statements.
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...Spark Summit
Scaling out doesn’t have to mean giving up transactions and efficient joins! Relational databases can scale horizontally, and using them as a store for Spark Streaming or batch computations can help cover areas in which Spark is typically weaker. Examples will be drawn from our experience using Citus (https://github.com/citusdata/citus), an open-source extension to Postgres, but lessons learned should be applicable to many databases.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries. These stored functions currently only work on Linux-based systems.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...Citus Data
A story about powering a 1.5 petabyte internal analytics application at Microsoft with 2816 cores and 18.7 TB of memory in the Citus cluster.
The internal RQV analytics dashboard at Microsoft helps the Windows team to assess the quality of upcoming Windows releases. The system tracks 20,000 diagnostic and quality metrics, digests data from 800 million Windows devices and currently supports over 6 million queries per day, with hundreds of concurrent users. The RQV analytics dashboard relies on Postgres—along with the Citus extension to Postgres to scale out horizontally—and is deployed on Microsoft Azure.
Data Modeling, Normalization, and De-Normalization | PostgresOpen 2019 | Dimi...Citus Data
As a developer using PostgreSQL one of the most important tasks you have to deal with is modeling the database schema for your application. In order to achieve a solid design, it’s important to understand how the schema is then going to be used as well as the trade-offs it involves.
As Fred Brooks said: “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.”
In this talk we're going to see practical normalisation examples and their benefits, and also review some anti-patterns and their typical PostgreSQL solutions, including Denormalization techniques thanks to advanced Data Types.
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...Citus Data
When do you use jsonb, and when don’t you? How do you make it fast? What operators are available, and what can they do? How will this change? These are all very good questions, but jsonb support in Postgres moves so fast that it’s hard to keep up.
In this talk, you will get details on these topics, complete with practical examples and real-world stories:
- When to use jsonb, what it’s good for, and when to not use it
- Operators and how to use them effectively
- Indexing, operator support for indexes, and the tradeoffs involved
- Postgres 12 improvements and new features
Tutorial: Implementing your first Postgres extension | PGConf EU 2019 | Burak...Citus Data
One of the strongest features of any database is its extensibility and PostgreSQL comes with a rich extension API. It allows you to define new functions, types, and operators. It even allows you to modify some of its core parts like planner, executor or storage engine. You read it right, you can even change the behavior of PostgreSQL planner. How cool is that?
Such freedom in extensibility created strong extension community around PostgreSQL and made way for a vast amount of extensions such as pg_stat_statements, citus, postgresql-hll and many more.
In this tutorial, we will look at how you can create your own PostgreSQL extension. We will start with more common stuff like defining new functions and types but gradually explore less known parts of the PostgreSQL's extension API like C level hooks which lets you change the behavior of planner, executor and other core parts of the PostgreSQL. We will see how to code, debug, compile and test our extension. After that, we will also look into how to package and distribute our extension for other people to use.
To get the best benefit from the tutorial, C and SQL knowledge would be beneficial. Some knowledge on PostgreSQL internals would also be useful but we will cover the necessary details, so it is not necessary.
Whats wrong with postgres | PGConf EU 2019 | Craig KerstiensCitus Data
Postgres is a powerful database, it continues to improve in terms of performance, extensibility, and more broadly in features. However it is not perfect.
Here I'll cover a highly opinionated view of all the areas Postgres falls flat, with some rough thought ideas on how we can make it better. Opinions are all informed by 10 years of interacting with customers running literally millions of databases for users.
When it all goes wrong | PGConf EU 2019 | Will LeinweberCitus Data
You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently…
Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice!
In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise GrandjoncCitus Data
SQL can seem like an obscure and complex but powerful language. Learning it can be intimidating. As a developer, we can easily be tempted using basic SQL provided by the ORM. But did you know that you can use window functions in some ORMs? Same goes for a lot of other fun SQL functionalities.
In this talk we will explore some advanced SQL features that you might find useful. We will discover the wonderful world of joins (lateral, cross…), subqueries, grouping sets, window functions, common table expressions.
But most importantly this talk is not only a talk to show you how great SQL is. This talk is here to show you how to use it in real life. What are the features supported by your ORM? And how can you use them if they don’t support them?
Wether you know SQL or not, whether you are a developer or a DBA working with developers, you might learn a lot about SQL, ORMs, and application development using Postgres.
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...Citus Data
Many people have asked us: “Why did Microsoft acquire Citus Data?” and “What do you plan to do with the Citus open source extension to Postgres?” Come join us to see the exciting work we are doing with Postgres and open source at Microsoft.
Deep Postgres Extensions in Rust | PGCon 2019 | Jeff DavisCitus Data
Postgres relies heavily on an extension ecosystem, but that is almost 100% dependent on C; which cuts out developers, libraries, and ideas from the world of Postgres. postgres-extension.rs changes that by supporting development of extensions in Rust. Rust is a memory-safe language that integrates nicely in any environment, has powerful libraries, a vibrant ecosystem, and a prolific developer community.
Rust is a unique language because it supports high-level features but all the magic happens at compile-time, and the resulting code is not dependent on an intrusive or bulky runtime. That makes it ideal for integrating with postgres, which has a lot of its own runtime, like memory contexts and signal handlers. postgres-extension.rs offers this integration, allowing the development of extensions in rust, even if deeply-integrated into the postgres internals, and helping handle tricky issues like error handling. This is done through a collection of Rust function declarations, macros, and utility functions that allow rust code to call into postgres, and safely handle resulting errors.
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...Citus Data
I spent the early part of my career working on developer tools, operating systems, high-speed file systems, and scale-out storage. Not databases. Frankly, I always thought that databases were a bit boring. So almost 2 years in to my new job at a Postgres company, I continue to be amazed at the enthusiasm of the PostgreSQL developer community and users. I mean, people’s eyes light up when you ask them why they love Postgres. Sure, a lot of us get animated when talking about our newest gadget, or Ronaldo’s phenomenal free-kick goal in the World Cup, or mint chip gelato from La Strega Nocciola—but most platform software simply doesn’t trigger this kind of passion. So why does Postgres? Why is this open source database having such a “moment”? Well, I’ve been trying to understand, looking at this “Postgres moment” from a few different angles. In this talk I’ll share what I’ve observed to be the top 10 business, technology, and community reasons so many of you have so much affection for PostgreSQL.
A story on Postgres index types | PostgresLondon 2019 | Louise GrandjoncCitus Data
Want to know everything about indexes in postgres? Here are the slides for a postgresql talk, and if you want to know more, you can read articles on www.louisemeta.com.
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...Citus Data
Many in today’s developer world look down on marketing. I mean, after all, the marketing team is usually “not technical.” And they’re not developers. It’s 2019 and while we try to promote inclusiveness of all types, inclusiveness doesn’t seem to apply to marketers. Why? Is that OK? Who does that hurt? I grew up in engineering and spent the first 15 years of my career as a developer or an engineering manager of some type. So now that I’m in marketing, it surprised me when one of my engineering colleagues blurted out “But it’s a technical conference!” when he learned one of my talks was accepted to a technical conference.
This keynote is about why developers really need marketing. About how good marketing managers can make it so visitors to your website don’t leave empty-handed, confused about what your technology actually does or why it matters. About how the ability to translate technology into what-users-actually-care-about can make your project be the one that takes off. About why Dormain Drewitz said at Monktoberfest: “I work in product marketing. My preferred programming language is English.” Finally, this talk explores how to be sensitive to the bias against marketing that pervades some of our teams—and how to instead embrace teamwork best practices employed by sailors, where everyone in the boat has an important role to play if you are to win the race.
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri FontaineCitus Data
PostgreSQL is the World’s Most Advanced Open Source Relational Database and by the end of this talk you will understand what that means for you, an application developer. What kind of problems PostgreSQL can solve for you, and how much you can rely on PostgreSQL in your daily activities, including unit-testing.
Optimizing your app by understanding your Postgres | RailsConf 2019 | Samay S...Citus Data
I’m a Postgres person. Period. After talking to many Rails developers about their application performance, I realized many performance issues can be solved by understanding your database a bit better. So I thought I’d share the statistics Postgres captures for you and how you can use them to find slow queries, un-used indexes, or tables which are not getting vacuumed correctly. This talk will cover Postgres tools and tips for the above, including pgstatstatements, useful catalog tables, and recently added Postgres features such as CREATE STATISTICS.
When it all goes wrong (with Postgres) | RailsConf 2019 | Will LeinweberCitus Data
You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently…
Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice!
In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri FontaineCitus Data
PostgreSQL is the World’s Most Advanced Open Source Relational Database and by the end of this talk you will understand what that means for you, an application developer. What kind of problems PostgreSQL can solve for you, and how much you can rely on PostgreSQL in your daily activities, including unit-testing.
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...Citus Data
Watch Sai Srirampur, Solutions Engineer at Citus Data (now part of the Microsoft family), give a live demo of how you can use Postgres and the Citus extension to Postgres to manage real-time analytics workloads.
View if you & your application need:
>> A relational database that scales for customer-facing analytics dashboards, with real-time data ingest and a large volume of queries
>> A way to scale out Postgres horizontally, to address the performance hiccups you’re experiencing as you run into the resource limits of single-node Postgres
>> A way to roll-up and pre-aggregate data to build fast data pipelines and enable sub-second response times.
>> A way to consolidate your database platforms, to avoid having separate stores for your transactional and analytics workloads
Using a 4-node Citus database cluster in the cloud, Sai will show you how Citus shards Postgres to give you lightning fast performance, at scale. Also featuring rollups.
How to write SQL queries | pgDay Paris 2019 | Dimitri FontaineCitus Data
Most of the time we see finished SQL queries, either in code repositories, blog posts of talk slides. This talk focus on the process of how to write an SQL query, from a problem statement expressed in English to code review and long term maintenance of SQL code.
When it all Goes Wrong |Nordic PGDay 2019 | Will LeinweberCitus Data
You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently…
Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice!
In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire GiordanoCitus Data
I spent the early part of my career working on developer tools, operating systems, high-speed file systems, and scale-out storage. Not databases. Frankly, I always thought that databases were a bit boring. So one year in to my new job at a Postgres company, I continue to be amazed at the enthusiasm of the PostgreSQL developer community and users. I mean, people’s eyes light up when you ask them why they love Postgres. Sure, a lot of us get animated when talking about our newest iPhone, or Ronaldo’s phenomenal free-kick goal in the World Cup, or mint chip gelato from La Strega Nociola—but most platform software simply doesn’t trigger this kind of passion. So why does Postgres? Why is this open source database having such a “moment”? Why now? Well, I’ve been trying to find out, looking at this “Postgres moment” from a few different angles. In this talk I’ll share what I’ve observed to be the top 10 business, technology, and community reasons so many of you have so much affection for PostgreSQL.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
2. About me
• Run Citus Cloud
• Previously ran Heroku Postgres
• Curate Postgres Weekly
3. What are extensions
• Low level hooks that allow you to change/extend
behavior of Postgres
• Can be written in C or higher level language
• Can be for new functionality, new data types, or
very dramatically change behavior
4. Extensions continued
• Postgres ships with some sort of native ones
(known as contrib)
• Other extensions have to be built and installed
• Hosting providers have some set of extensions they
support
9. What’s it do
• Records queries that were run and all sorts of stats
• I don’t understand those stats
• Think of it as parameterizing a query,
• How many times
• How long it ran
14. Citus stat statements
• Per tenant stats
• Persists tenant id
• With all the benefits of pg_stat_statements
• Can answer
• Which tenant is most noisy
• Which tenant consumes most resources
26. UPDATE helloworld
SET set = hll_add(set, hll_hash_integer(12345))
WHERE id = 1;
UPDATE helloworld
SET set = hll_add(set, hll_hash_text(‘hello
world’))
WHERE id = 1;
27. In the real world
CREATE TABLE daily_uniques (
date date unique,
users hll
);
28. Real world
INSERT INTO daily_uniques (date, users)
SELECT date,
hll_add_agg(hll_hash_integer(user_id))
FROM users
GROUP by 1;
29. Real world
SELECT users
FROM daily_uniques;
users |
x128b7f80c8b96a7321f6cec13b12ffb2242de7db3caddf
2d017737de20d6599165344df333908c162802510d314fd3
0926f3b50f16075cba0b38e94becbcd8bc8625e56066c5e4
cf119639775c69d027242eec7da2b22671431358
30. SELECT
EXTRACT (month from date) as month,
hll_cardinality(hll_union_agg(users))
FROM daily_uniques
WHERE date >= '2018-08-01'
AND date < '2018-11-01'
GROUP BY 1;
32. Tuning parameters
• log2m - log base 2 of registers
• Between 4 and 17
• An increase of 1 doubles storage
• regwidth - bits per register
• expthresh - threshold for explicit vs. sparse
• spareson - toggle sparse on/off
37. TopN
Top list of people that have done X/Y
Generally accurate on who is in the top
i.e.
Top 10 websites browsed
Top most frequent visitors
Top 10 search terms
38. TopN
CREATE TABLE aggregated_topns (day date, topn jsonb);
INSERT INTO
aggregated_topns select date_trunc('day', created_at),
topn_add_agg((repo::json)->> 'name') as topn
FROM github_events
GROUP BY 1;
43. Timescale
• Time-centric: Data records always have a timestamp.
• Append-only: Data is almost solely append-only
(INSERTs).
• Recent: New data is typically about recent time intervals,
and we more rarely make updates or backfill missing data
about old intervals.
44. Timescale
CREATE TABLE "rides"(
vendor_id TEXT,
pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
passenger_count NUMERIC,
trip_distance NUMERIC,
payment_type INTEGER,
fare_amount NUMERIC,
extra NUMERIC,
mta_tax NUMERIC,
tip_amount NUMERIC,
tolls_amount NUMERIC,
improvement_surcharge NUMERIC,
total_amount NUMERIC
);
52. List of relations
Schema | Name | Type | Owner
--------+-------------------------+----------+----------
public | events | table | citus
public | events_event_id_seq | sequence | citus
public | events_p2018_10_23_0900 | table | citus
public | events_p2018_10_23_0905 | table | citus
public | events_p2018_10_23_0910 | table | citus
public | events_p2018_10_23_0915 | table | citus
public | events_p2018_10_23_0920 | table | citus
public | events_p2018_10_23_0925 | table | citus
public | events_p2018_10_23_0930 | table | citus
public | events_p2018_10_23_0935 | table | citus
53. SELECT * from partman.part_config;
-[ RECORD 1 ]--------------+-------------------------------
parent_table | public.events
control | event_time
partition_type | native
partition_interval | 00:05:00
constraint_cols | #
premake | 4
optimize_trigger | 4
optimize_constraint | 30
epoch | none
inherit_fk | t
retention | #
retention_schema | #
retention_keep_table | t
retention_keep_index | t
infinite_time_partitions | t
datetime_string | YYYY_MM_DD_HH24MI
automatic_maintenance | on
jobmon | t
sub_partition_set_full | f
undo_in_progress | f
trigger_exception_handling | f
upsert |
trigger_return_null | t
template_table | partman.template_public_events
publications | #
54. Why not native partitioning
Partman is native, with extra bells and whistles
Automates creating/dropping partitions
56. Citus
Turns postgres into a distributed, horizontally
scalable database
You application still thinks it’s a single node, under
the covers, it is all sharded
58. More details
• Hash based on some id
• Postgres internal hash can work fine, or so can
your own
• Define your number of shards up front, make this
larger than you expect to grow to in terms of nodes
• (2 is bad)
• (2 million is also bad)
• Factors of 2 are nice, but not actually required
70. FDWs
• Foreign Data Wrapper
• Connect from within Postgres to something else
• (or to Postgres)
• Examples
• Redis
• Mongo
• cstore
•
71. Create foreign server
CREATE SERVER redis_server
FOREIGN DATA WRAPPER redis_fdw
OPTIONS (address '127.0.0.1', port '6379');
CREATE FOREIGN TABLE redis_db0 (key text, value text)
SERVER redis_server
OPTIONS (database '0');
CREATE USER MAPPING FOR PUBLIC
SERVER redis_server
OPTIONS (password 'secret');
72. # d
List of relations
Schema | Name | Type | Owner
--------+------------------+---------------+-------
public | products | table | craig
public | purchases | table | craig
public | redis_db0 | foreign table | craig
public | users | table | craig
(4 rows)