SQL can seem like an obscure and complex but powerful language. Learning it can be intimidating. As a developer, we can easily be tempted using basic SQL provided by the ORM. But did you know that you can use window functions in some ORMs? Same goes for a lot of other fun SQL functionalities.
In this talk we will explore some advanced SQL features that you might find useful. We will discover the wonderful world of joins (lateral, cross…), subqueries, grouping sets, window functions, common table expressions.
But most importantly this talk is not only a talk to show you how great SQL is. This talk is here to show you how to use it in real life. What are the features supported by your ORM? And how can you use them if they don’t support them?
Wether you know SQL or not, whether you are a developer or a DBA working with developers, you might learn a lot about SQL, ORMs, and application development using Postgres.
SQL can seem like an obscure but somehow useful language. In this talk we will look into things that SQL can do, sometimes more easily than using python, and how to get it in your ORM, running in your application. During this talk we will use an application analysing the lyrics of my favorite teenage band and show fun examples of these SQL statements, and how to integrate them in your code
Workshop on the Last.fm API given at Music Hack Day Stockholm (Jan 30, 2010). WIth contributions from Matthew Ogle, David Singleton, and Jonty Wareing.
Apologies for some text flowing off the slides, Slideshare doesn't seem to like the typeface we use. :-/
SQL can seem like an obscure but somehow useful language. In this talk we will look into things that SQL can do, sometimes more easily than using python, and how to get it in your ORM, running in your application. During this talk we will use an application analysing the lyrics of my favorite teenage band and show fun examples of these SQL statements, and how to integrate them in your code
Workshop on the Last.fm API given at Music Hack Day Stockholm (Jan 30, 2010). WIth contributions from Matthew Ogle, David Singleton, and Jonty Wareing.
Apologies for some text flowing off the slides, Slideshare doesn't seem to like the typeface we use. :-/
Program 02 Based on the previous problem you should impleme.pdfaddtechglobalmarketi
Program 02
Based on the previous problem, you should implement this second problem in the same file a05.c.
Find the places in the code marked with the ToDo label and complete the work. You should not
change any other part of the code. This includes do not add any global variables or any other
function. For this problem, we are reusing the same struct from problem 1, but adding a pointer to
the struct to be able to create a linked list. In this problem, you must use linked list to create a user
playlist (play_list).
The struct is holding the information of a song: ID, Name, Singer, Genre, Year, and the pointer
Next (the pointer to the struct).
There should be the following functions. In the similar way we have done in previous assignments,
complete the following functions:
Add a song to the play_list - always inserted at the end of the list. Search song by name - given
the name of the song, return the struct with the data of the song.
Edit a song - given the name of the song, find it in the play_list and modify any of the fields of the
song (ID cannot be edited, and all fields should have a valid value). Return -1 if the song was not
found.
Delete a song - given the name of the song, find it, delete the song from the play_list (i.e.,
disconnect the node from the list). Return -1 if the song was not found, 1 if it was deleted.
// Problem 2
// struct to hold information about a song
struct Song
{
int id;
char name[MAX_SONG_NAME_LENGTH];
char singer[MAX_SINGER_NAME_LENGTH];
genreType genre;
int year;
struct Song *next;
};
// Function declarations
struct Song *createSong(int id, char *name, char *singer, char *genre, int year);
void printPlaylist(struct Song *playlist);
void add_song(struct Song **playlist, struct Song *newSong);
struct Song *search_song(struct Song *playlist, char *name);
int edit_song(struct Song *playlist, char *name, char *singer, char *genre, int
year);
int delete_song(struct Song *playlist, char *name);
// function to create a new song
struct Song *createSong(int id, char *name, char *singer, char *genre, int year)
{
struct Song *newSong = (struct Song *)malloc(sizeof(struct Song));
newSong->id = id;
strcpy(newSong->name, name);
strcpy(newSong->singer, singer);
if (strcmp(genre, "Pop") == 0)
newSong->genre = 1;
else if (strcmp(genre, "Rock") == 0)
newSong->genre = 2;
else if (strcmp(genre, "Reggae") == 0)
newSong->genre = 3;
else if (strcmp(genre, "Country") == 0)
newSong->genre = 4;
else if (strcmp(genre, "Blues") == 0)
newSong->genre = 5;
else if (strcmp(genre, "Balad") == 0)
newSong->genre = 6;
else
newSong->genre = 0;
newSong->year = year;
newSong->next = NULL;
return newSong;
}
// function to print the playlist
void printPlaylist(struct Song *playlist)
{
struct Song *current = playlist;
while (current != NULL)
{
printf("%dt%st%st%dt%dn", current->id, current->name, current->singer,
current->genre, current->year);
current = current->next;
}
}
// function to add a song to the end of the playlist
void add_song(struct Song .
Over the last few years, there has been a stronger emphasis on functional programming languages and constructs in mainstream programming. But we are still far from programming nirvana. The real frontier is not the code we write, but the systems we build. Until we move towards "FP" thinking at a system level, we are just going to keep building similarly broken systems, albeit with a nicer GOTO syntax. The lifeblood of your system is data. The data that models the events, facts, and insights of your domain, and how they apply to your users. We have not yet mastered data modeling, and I would surmise that it is the root-cause of many project failures and costly refactors. This talk is an introduction to modeling business data in web applications. Together, we will explore how existing popular solutions model data, why they fail, and how we could do better with surprisingly little effort. The slides may include parentheses and square brackets, given my predilection for alien technologies, but the conclusions are broad and applicable to many.
Free The Enterprise With Ruby & Master Your Own DomainKen Collins
On the heals of Luis Lavena's RailsConf talk "Infiltrating Ruby Onto The Enterprise Death Star Using Guerilla Tactics" comes a local and frank talk about the current state of Open Source Software (OSS) participation from Windows developers. Learn what OSS is, what motivates its contributors, and how OSS can make you a stronger developer. Be prepared to fall in love with writing software again!
We will start off with a 101 introduction to both the Ruby programming language and the Ruby on Rails web application framework. You will learn about ActiveRecord, a powerful ORM that maps rich objects to your databases, and the latest components to use it with SQL Server. As a Rails core contributor and author of the SQL Server stack, I will give you a modern insight into both that will allow you to leverage your legacy data with Ruby.
Lastly, I will review the bleeding edge tools being actively created for Windows developers to ease the transition to Ruby, Rails and OSS from a POSIX driven world. Many things have changed. It is time to learn and perform some occupational maintenance.
Finding a lost song with Node.js and async iteratorsLuciano Mammino
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Querier – simple relational database access
Tue, July 14, 3:00pm – 3:15pm
Video: https://www.youtube.com/watch?v=c2yhd8BjFjU
First Name: Michal
Last Name: Balda
Type: Talk
Abstract: Querier is a library which simplifies the usage of relational
databases in Pharo. It directly exposes tables and rows, there is no ORM
layer. Table querying uses a collection-like interface, queries are
built transparently and executed lazily. Simple and effective access to
related tables is one of the key features of Querier.
Bio: Michal Balda is a web developer, coming from the Czech Technical
University in Prague. He's currently using Pharo and Seaside to build a
commercial web application in a university research project.
Analyze one year of radio station songs aired with Spark SQL, Spotify, and Da...Paul Leclercq
Paris Spark Meetup - May 2017
Video : https://www.youtube.com/watch?v=w5Zd-1wIJrU
AdHoc analysis of radio stations broadcasts stored in a parquet files with plain SQL, the dataframe API.
The aim was to notice radio stations habits, differences and if radio stations brainwashing is a thing
This talk's Databricks notebook can be found here : https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6937750999095841/3645330882010081/6197123402747553/latest.html
Finding a lost song with Node.js and async iteratorsLuciano Mammino
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Can someone please help me implement the addSong function .pdfakshpatil4
Can someone please help me implement the "addSong" function in the code below? Thank
you!
// You are given a partially completed program that creates a list of songs, like a music repository.
// Each song has this information: song's id, song's name, singer's name, genre of the song, and
its published year.
// The struct 'musicRepository' holds information of one song. Genre is enum type.
// An array of structs called 'list' is made to hold the list of songs.
// You should not modify any of the given code, the return types, or the parameters, you will risk of
getting compilation error.
// You are not allowed to modify main ().
// You can use string library functions.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#pragma warning(disable : 4996) // for Visual Studio Only
#define MAX_SONGS 20
#define MAX_SONG_NAME_LENGTH 40
#define MAX_SINGER_NAME_LENGTH 40
typedef enum
{
unclassified = 0,
Pop,
Rock,
Reggae,
Country,
Blues,
Balad,
} genreType; // enum type
struct musicRepository
{
// struct for song details
unsigned int ID;
char songName[MAX_SONG_NAME_LENGTH];
char singerName[MAX_SINGER_NAME_LENGTH];
genreType genre;
unsigned int year;
};
struct musicRepository list[MAX_SONGS]; // declare the list of songs
int numSongs = 0; // the number of songs currently stored in the list (initialized to 0)
// Given functions
void initializeRepository(struct musicRepository repo[], int size);
void printRepository(struct musicRepository repo[], int numSongs);
void flushSngIn();
// functions that need to be implemented by you
int addSong(struct musicRepository repo[], char* songName, char* singerName, char* genre,
unsigned int year, int numSongs);
struct musicRepository* searchSong(struct musicRepository repo[], char* songName, int
numSongs);
void printSong(struct musicRepository repo[], int ID);
int editSong(struct musicRepository* repo, int numSongs, char* songName);
int deleteSongByName(struct musicRepository* repo, char* songName, int numSongs);
// This function takes in an array of musicRepository structures and the size of the array as
parameters. It then loops through each index of the array and sets the values of id, year, genre,
name, and singer to their respective initial values.
// NOTE that strcpy is used to copy an empty string to name and singer, rather than setting them
equal to "". This is because arrays in C are not assignable, so the strcpy function must be used to
copy the empty string into the array.
void initializeRepository(struct musicRepository repo[], int size)
{
for (int i = 0; i < size; i++)
{
repo[i].ID = 0;
repo[i].year = 0;
repo[i].genre = 0;
strcpy(repo[i].songName, "");
strcpy(repo[i].singerName, "");
}
}
// This function takes in the musicRepository array and iterates through each element. If the ID of
the song is not 0, it prints the details of the song.
void printRepository(struct musicRepository repo[], int numSongs)
{
printf("n--- Song Repository ---n");
for (int i = 0; i < numSongs; i++)
{
if (repo[i].ID != 0)
.
Finding a lost song with Node.js and async iterators - NodeConf Remote 2021Luciano Mammino
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Apple's Swift has achieved the top place in Stack Overflow's "Most Loved" list of programming languages in its 2015 Developer Survey. Based on information gleaned from GitHub and Stack Overflow, analyst firm RedMonk has seen Swift's popularity ranking soar from 68 to 22 in an unprecedented 6 months.
The "Extreme Swift" event does not require advanced, or even any, knowledge of Swift. Learn about some of the more outrageous features of the language which help explain what the fuss is all about!
Never look at programming the same way again — even if you never end up writing a single line of Swift code in your life.
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...Citus Data
A story about powering a 1.5 petabyte internal analytics application at Microsoft with 2816 cores and 18.7 TB of memory in the Citus cluster.
The internal RQV analytics dashboard at Microsoft helps the Windows team to assess the quality of upcoming Windows releases. The system tracks 20,000 diagnostic and quality metrics, digests data from 800 million Windows devices and currently supports over 6 million queries per day, with hundreds of concurrent users. The RQV analytics dashboard relies on Postgres—along with the Citus extension to Postgres to scale out horizontally—and is deployed on Microsoft Azure.
Data Modeling, Normalization, and De-Normalization | PostgresOpen 2019 | Dimi...Citus Data
As a developer using PostgreSQL one of the most important tasks you have to deal with is modeling the database schema for your application. In order to achieve a solid design, it’s important to understand how the schema is then going to be used as well as the trade-offs it involves.
As Fred Brooks said: “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.”
In this talk we're going to see practical normalisation examples and their benefits, and also review some anti-patterns and their typical PostgreSQL solutions, including Denormalization techniques thanks to advanced Data Types.
More Related Content
Similar to Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc
Program 02 Based on the previous problem you should impleme.pdfaddtechglobalmarketi
Program 02
Based on the previous problem, you should implement this second problem in the same file a05.c.
Find the places in the code marked with the ToDo label and complete the work. You should not
change any other part of the code. This includes do not add any global variables or any other
function. For this problem, we are reusing the same struct from problem 1, but adding a pointer to
the struct to be able to create a linked list. In this problem, you must use linked list to create a user
playlist (play_list).
The struct is holding the information of a song: ID, Name, Singer, Genre, Year, and the pointer
Next (the pointer to the struct).
There should be the following functions. In the similar way we have done in previous assignments,
complete the following functions:
Add a song to the play_list - always inserted at the end of the list. Search song by name - given
the name of the song, return the struct with the data of the song.
Edit a song - given the name of the song, find it in the play_list and modify any of the fields of the
song (ID cannot be edited, and all fields should have a valid value). Return -1 if the song was not
found.
Delete a song - given the name of the song, find it, delete the song from the play_list (i.e.,
disconnect the node from the list). Return -1 if the song was not found, 1 if it was deleted.
// Problem 2
// struct to hold information about a song
struct Song
{
int id;
char name[MAX_SONG_NAME_LENGTH];
char singer[MAX_SINGER_NAME_LENGTH];
genreType genre;
int year;
struct Song *next;
};
// Function declarations
struct Song *createSong(int id, char *name, char *singer, char *genre, int year);
void printPlaylist(struct Song *playlist);
void add_song(struct Song **playlist, struct Song *newSong);
struct Song *search_song(struct Song *playlist, char *name);
int edit_song(struct Song *playlist, char *name, char *singer, char *genre, int
year);
int delete_song(struct Song *playlist, char *name);
// function to create a new song
struct Song *createSong(int id, char *name, char *singer, char *genre, int year)
{
struct Song *newSong = (struct Song *)malloc(sizeof(struct Song));
newSong->id = id;
strcpy(newSong->name, name);
strcpy(newSong->singer, singer);
if (strcmp(genre, "Pop") == 0)
newSong->genre = 1;
else if (strcmp(genre, "Rock") == 0)
newSong->genre = 2;
else if (strcmp(genre, "Reggae") == 0)
newSong->genre = 3;
else if (strcmp(genre, "Country") == 0)
newSong->genre = 4;
else if (strcmp(genre, "Blues") == 0)
newSong->genre = 5;
else if (strcmp(genre, "Balad") == 0)
newSong->genre = 6;
else
newSong->genre = 0;
newSong->year = year;
newSong->next = NULL;
return newSong;
}
// function to print the playlist
void printPlaylist(struct Song *playlist)
{
struct Song *current = playlist;
while (current != NULL)
{
printf("%dt%st%st%dt%dn", current->id, current->name, current->singer,
current->genre, current->year);
current = current->next;
}
}
// function to add a song to the end of the playlist
void add_song(struct Song .
Over the last few years, there has been a stronger emphasis on functional programming languages and constructs in mainstream programming. But we are still far from programming nirvana. The real frontier is not the code we write, but the systems we build. Until we move towards "FP" thinking at a system level, we are just going to keep building similarly broken systems, albeit with a nicer GOTO syntax. The lifeblood of your system is data. The data that models the events, facts, and insights of your domain, and how they apply to your users. We have not yet mastered data modeling, and I would surmise that it is the root-cause of many project failures and costly refactors. This talk is an introduction to modeling business data in web applications. Together, we will explore how existing popular solutions model data, why they fail, and how we could do better with surprisingly little effort. The slides may include parentheses and square brackets, given my predilection for alien technologies, but the conclusions are broad and applicable to many.
Free The Enterprise With Ruby & Master Your Own DomainKen Collins
On the heals of Luis Lavena's RailsConf talk "Infiltrating Ruby Onto The Enterprise Death Star Using Guerilla Tactics" comes a local and frank talk about the current state of Open Source Software (OSS) participation from Windows developers. Learn what OSS is, what motivates its contributors, and how OSS can make you a stronger developer. Be prepared to fall in love with writing software again!
We will start off with a 101 introduction to both the Ruby programming language and the Ruby on Rails web application framework. You will learn about ActiveRecord, a powerful ORM that maps rich objects to your databases, and the latest components to use it with SQL Server. As a Rails core contributor and author of the SQL Server stack, I will give you a modern insight into both that will allow you to leverage your legacy data with Ruby.
Lastly, I will review the bleeding edge tools being actively created for Windows developers to ease the transition to Ruby, Rails and OSS from a POSIX driven world. Many things have changed. It is time to learn and perform some occupational maintenance.
Finding a lost song with Node.js and async iteratorsLuciano Mammino
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Querier – simple relational database access
Tue, July 14, 3:00pm – 3:15pm
Video: https://www.youtube.com/watch?v=c2yhd8BjFjU
First Name: Michal
Last Name: Balda
Type: Talk
Abstract: Querier is a library which simplifies the usage of relational
databases in Pharo. It directly exposes tables and rows, there is no ORM
layer. Table querying uses a collection-like interface, queries are
built transparently and executed lazily. Simple and effective access to
related tables is one of the key features of Querier.
Bio: Michal Balda is a web developer, coming from the Czech Technical
University in Prague. He's currently using Pharo and Seaside to build a
commercial web application in a university research project.
Analyze one year of radio station songs aired with Spark SQL, Spotify, and Da...Paul Leclercq
Paris Spark Meetup - May 2017
Video : https://www.youtube.com/watch?v=w5Zd-1wIJrU
AdHoc analysis of radio stations broadcasts stored in a parquet files with plain SQL, the dataframe API.
The aim was to notice radio stations habits, differences and if radio stations brainwashing is a thing
This talk's Databricks notebook can be found here : https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6937750999095841/3645330882010081/6197123402747553/latest.html
Finding a lost song with Node.js and async iteratorsLuciano Mammino
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Can someone please help me implement the addSong function .pdfakshpatil4
Can someone please help me implement the "addSong" function in the code below? Thank
you!
// You are given a partially completed program that creates a list of songs, like a music repository.
// Each song has this information: song's id, song's name, singer's name, genre of the song, and
its published year.
// The struct 'musicRepository' holds information of one song. Genre is enum type.
// An array of structs called 'list' is made to hold the list of songs.
// You should not modify any of the given code, the return types, or the parameters, you will risk of
getting compilation error.
// You are not allowed to modify main ().
// You can use string library functions.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#pragma warning(disable : 4996) // for Visual Studio Only
#define MAX_SONGS 20
#define MAX_SONG_NAME_LENGTH 40
#define MAX_SINGER_NAME_LENGTH 40
typedef enum
{
unclassified = 0,
Pop,
Rock,
Reggae,
Country,
Blues,
Balad,
} genreType; // enum type
struct musicRepository
{
// struct for song details
unsigned int ID;
char songName[MAX_SONG_NAME_LENGTH];
char singerName[MAX_SINGER_NAME_LENGTH];
genreType genre;
unsigned int year;
};
struct musicRepository list[MAX_SONGS]; // declare the list of songs
int numSongs = 0; // the number of songs currently stored in the list (initialized to 0)
// Given functions
void initializeRepository(struct musicRepository repo[], int size);
void printRepository(struct musicRepository repo[], int numSongs);
void flushSngIn();
// functions that need to be implemented by you
int addSong(struct musicRepository repo[], char* songName, char* singerName, char* genre,
unsigned int year, int numSongs);
struct musicRepository* searchSong(struct musicRepository repo[], char* songName, int
numSongs);
void printSong(struct musicRepository repo[], int ID);
int editSong(struct musicRepository* repo, int numSongs, char* songName);
int deleteSongByName(struct musicRepository* repo, char* songName, int numSongs);
// This function takes in an array of musicRepository structures and the size of the array as
parameters. It then loops through each index of the array and sets the values of id, year, genre,
name, and singer to their respective initial values.
// NOTE that strcpy is used to copy an empty string to name and singer, rather than setting them
equal to "". This is because arrays in C are not assignable, so the strcpy function must be used to
copy the empty string into the array.
void initializeRepository(struct musicRepository repo[], int size)
{
for (int i = 0; i < size; i++)
{
repo[i].ID = 0;
repo[i].year = 0;
repo[i].genre = 0;
strcpy(repo[i].songName, "");
strcpy(repo[i].singerName, "");
}
}
// This function takes in the musicRepository array and iterates through each element. If the ID of
the song is not 0, it prints the details of the song.
void printRepository(struct musicRepository repo[], int numSongs)
{
printf("n--- Song Repository ---n");
for (int i = 0; i < numSongs; i++)
{
if (repo[i].ID != 0)
.
Finding a lost song with Node.js and async iterators - NodeConf Remote 2021Luciano Mammino
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Apple's Swift has achieved the top place in Stack Overflow's "Most Loved" list of programming languages in its 2015 Developer Survey. Based on information gleaned from GitHub and Stack Overflow, analyst firm RedMonk has seen Swift's popularity ranking soar from 68 to 22 in an unprecedented 6 months.
The "Extreme Swift" event does not require advanced, or even any, knowledge of Swift. Learn about some of the more outrageous features of the language which help explain what the fuss is all about!
Never look at programming the same way again — even if you never end up writing a single line of Swift code in your life.
Similar to Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc (20)
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...Citus Data
A story about powering a 1.5 petabyte internal analytics application at Microsoft with 2816 cores and 18.7 TB of memory in the Citus cluster.
The internal RQV analytics dashboard at Microsoft helps the Windows team to assess the quality of upcoming Windows releases. The system tracks 20,000 diagnostic and quality metrics, digests data from 800 million Windows devices and currently supports over 6 million queries per day, with hundreds of concurrent users. The RQV analytics dashboard relies on Postgres—along with the Citus extension to Postgres to scale out horizontally—and is deployed on Microsoft Azure.
Data Modeling, Normalization, and De-Normalization | PostgresOpen 2019 | Dimi...Citus Data
As a developer using PostgreSQL one of the most important tasks you have to deal with is modeling the database schema for your application. In order to achieve a solid design, it’s important to understand how the schema is then going to be used as well as the trade-offs it involves.
As Fred Brooks said: “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.”
In this talk we're going to see practical normalisation examples and their benefits, and also review some anti-patterns and their typical PostgreSQL solutions, including Denormalization techniques thanks to advanced Data Types.
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...Citus Data
When do you use jsonb, and when don’t you? How do you make it fast? What operators are available, and what can they do? How will this change? These are all very good questions, but jsonb support in Postgres moves so fast that it’s hard to keep up.
In this talk, you will get details on these topics, complete with practical examples and real-world stories:
- When to use jsonb, what it’s good for, and when to not use it
- Operators and how to use them effectively
- Indexing, operator support for indexes, and the tradeoffs involved
- Postgres 12 improvements and new features
Tutorial: Implementing your first Postgres extension | PGConf EU 2019 | Burak...Citus Data
One of the strongest features of any database is its extensibility and PostgreSQL comes with a rich extension API. It allows you to define new functions, types, and operators. It even allows you to modify some of its core parts like planner, executor or storage engine. You read it right, you can even change the behavior of PostgreSQL planner. How cool is that?
Such freedom in extensibility created strong extension community around PostgreSQL and made way for a vast amount of extensions such as pg_stat_statements, citus, postgresql-hll and many more.
In this tutorial, we will look at how you can create your own PostgreSQL extension. We will start with more common stuff like defining new functions and types but gradually explore less known parts of the PostgreSQL's extension API like C level hooks which lets you change the behavior of planner, executor and other core parts of the PostgreSQL. We will see how to code, debug, compile and test our extension. After that, we will also look into how to package and distribute our extension for other people to use.
To get the best benefit from the tutorial, C and SQL knowledge would be beneficial. Some knowledge on PostgreSQL internals would also be useful but we will cover the necessary details, so it is not necessary.
Whats wrong with postgres | PGConf EU 2019 | Craig KerstiensCitus Data
Postgres is a powerful database, it continues to improve in terms of performance, extensibility, and more broadly in features. However it is not perfect.
Here I'll cover a highly opinionated view of all the areas Postgres falls flat, with some rough thought ideas on how we can make it better. Opinions are all informed by 10 years of interacting with customers running literally millions of databases for users.
When it all goes wrong | PGConf EU 2019 | Will LeinweberCitus Data
You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently…
Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice!
In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...Citus Data
Many people have asked us: “Why did Microsoft acquire Citus Data?” and “What do you plan to do with the Citus open source extension to Postgres?” Come join us to see the exciting work we are doing with Postgres and open source at Microsoft.
Deep Postgres Extensions in Rust | PGCon 2019 | Jeff DavisCitus Data
Postgres relies heavily on an extension ecosystem, but that is almost 100% dependent on C; which cuts out developers, libraries, and ideas from the world of Postgres. postgres-extension.rs changes that by supporting development of extensions in Rust. Rust is a memory-safe language that integrates nicely in any environment, has powerful libraries, a vibrant ecosystem, and a prolific developer community.
Rust is a unique language because it supports high-level features but all the magic happens at compile-time, and the resulting code is not dependent on an intrusive or bulky runtime. That makes it ideal for integrating with postgres, which has a lot of its own runtime, like memory contexts and signal handlers. postgres-extension.rs offers this integration, allowing the development of extensions in rust, even if deeply-integrated into the postgres internals, and helping handle tricky issues like error handling. This is done through a collection of Rust function declarations, macros, and utility functions that allow rust code to call into postgres, and safely handle resulting errors.
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...Citus Data
I spent the early part of my career working on developer tools, operating systems, high-speed file systems, and scale-out storage. Not databases. Frankly, I always thought that databases were a bit boring. So almost 2 years in to my new job at a Postgres company, I continue to be amazed at the enthusiasm of the PostgreSQL developer community and users. I mean, people’s eyes light up when you ask them why they love Postgres. Sure, a lot of us get animated when talking about our newest gadget, or Ronaldo’s phenomenal free-kick goal in the World Cup, or mint chip gelato from La Strega Nocciola—but most platform software simply doesn’t trigger this kind of passion. So why does Postgres? Why is this open source database having such a “moment”? Well, I’ve been trying to understand, looking at this “Postgres moment” from a few different angles. In this talk I’ll share what I’ve observed to be the top 10 business, technology, and community reasons so many of you have so much affection for PostgreSQL.
A story on Postgres index types | PostgresLondon 2019 | Louise GrandjoncCitus Data
Want to know everything about indexes in postgres? Here are the slides for a postgresql talk, and if you want to know more, you can read articles on www.louisemeta.com.
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...Citus Data
Many in today’s developer world look down on marketing. I mean, after all, the marketing team is usually “not technical.” And they’re not developers. It’s 2019 and while we try to promote inclusiveness of all types, inclusiveness doesn’t seem to apply to marketers. Why? Is that OK? Who does that hurt? I grew up in engineering and spent the first 15 years of my career as a developer or an engineering manager of some type. So now that I’m in marketing, it surprised me when one of my engineering colleagues blurted out “But it’s a technical conference!” when he learned one of my talks was accepted to a technical conference.
This keynote is about why developers really need marketing. About how good marketing managers can make it so visitors to your website don’t leave empty-handed, confused about what your technology actually does or why it matters. About how the ability to translate technology into what-users-actually-care-about can make your project be the one that takes off. About why Dormain Drewitz said at Monktoberfest: “I work in product marketing. My preferred programming language is English.” Finally, this talk explores how to be sensitive to the bias against marketing that pervades some of our teams—and how to instead embrace teamwork best practices employed by sailors, where everyone in the boat has an important role to play if you are to win the race.
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri FontaineCitus Data
PostgreSQL is the World’s Most Advanced Open Source Relational Database and by the end of this talk you will understand what that means for you, an application developer. What kind of problems PostgreSQL can solve for you, and how much you can rely on PostgreSQL in your daily activities, including unit-testing.
Optimizing your app by understanding your Postgres | RailsConf 2019 | Samay S...Citus Data
I’m a Postgres person. Period. After talking to many Rails developers about their application performance, I realized many performance issues can be solved by understanding your database a bit better. So I thought I’d share the statistics Postgres captures for you and how you can use them to find slow queries, un-used indexes, or tables which are not getting vacuumed correctly. This talk will cover Postgres tools and tips for the above, including pgstatstatements, useful catalog tables, and recently added Postgres features such as CREATE STATISTICS.
When it all goes wrong (with Postgres) | RailsConf 2019 | Will LeinweberCitus Data
You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently…
Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice!
In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri FontaineCitus Data
PostgreSQL is the World’s Most Advanced Open Source Relational Database and by the end of this talk you will understand what that means for you, an application developer. What kind of problems PostgreSQL can solve for you, and how much you can rely on PostgreSQL in your daily activities, including unit-testing.
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...Citus Data
Watch Sai Srirampur, Solutions Engineer at Citus Data (now part of the Microsoft family), give a live demo of how you can use Postgres and the Citus extension to Postgres to manage real-time analytics workloads.
View if you & your application need:
>> A relational database that scales for customer-facing analytics dashboards, with real-time data ingest and a large volume of queries
>> A way to scale out Postgres horizontally, to address the performance hiccups you’re experiencing as you run into the resource limits of single-node Postgres
>> A way to roll-up and pre-aggregate data to build fast data pipelines and enable sub-second response times.
>> A way to consolidate your database platforms, to avoid having separate stores for your transactional and analytics workloads
Using a 4-node Citus database cluster in the cloud, Sai will show you how Citus shards Postgres to give you lightning fast performance, at scale. Also featuring rollups.
How to write SQL queries | pgDay Paris 2019 | Dimitri FontaineCitus Data
Most of the time we see finished SQL queries, either in code repositories, blog posts of talk slides. This talk focus on the process of how to write an SQL query, from a problem statement expressed in English to code review and long term maintenance of SQL code.
When it all Goes Wrong |Nordic PGDay 2019 | Will LeinweberCitus Data
You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently…
Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice!
In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire GiordanoCitus Data
I spent the early part of my career working on developer tools, operating systems, high-speed file systems, and scale-out storage. Not databases. Frankly, I always thought that databases were a bit boring. So one year in to my new job at a Postgres company, I continue to be amazed at the enthusiasm of the PostgreSQL developer community and users. I mean, people’s eyes light up when you ask them why they love Postgres. Sure, a lot of us get animated when talking about our newest iPhone, or Ronaldo’s phenomenal free-kick goal in the World Cup, or mint chip gelato from La Strega Nociola—but most platform software simply doesn’t trigger this kind of passion. So why does Postgres? Why is this open source database having such a “moment”? Why now? Well, I’ve been trying to find out, looking at this “Postgres moment” from a few different angles. In this talk I’ll share what I’ve observed to be the top 10 business, technology, and community reasons so many of you have so much affection for PostgreSQL.
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...Citus Data
There are a number of data architectures you could use when building a multi-tenant app. Some, such as using one database per customer or one schema per customer. These two options scale to an extent when you have say 10s of tenants. However as you start scaling to hundreds and thousands of tenants, you start running into challenges both from performance and maintenance of tenants perspective. You could solve the above problem by adding the notion of tenancy directly into the logic of your SaaS application. How to implement/automate this in Django-ORM is a challenge? We will talk about how to make the django app tenant aware and at a broader level explain how scale out applications that are built on top of Django ORM and follow a multi tenant data model. We'd take postgresql as our database of choice and the logic/implementation can be extended to any other relational databases as well.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
2. @louisemeta
About me
Software Engineer at Citus Data / Microsoft
Python developper
Postgres enthusiast
@louisemeta and @citusdata on twitter
www.louisemeta.com
louise.grandjonc@microsoft.com
3. @louisemeta
Why this talk
• I am a developper working with other developers
• I write code for applications using postgres
• I love both python and postgres
• I use ORMs
• I often attend postgres conferences
• A subject that I always enjoy is complex and modern SQL
• I want to use my database at its full potential
6. @louisemeta
Dataset
• 10 artists: Kyo, Blink-182, Maroon5, Jonas Brothers, Justin
Timberlake, Avril Lavigne, Backstreet Boys, Britney Spears, Justin
Bieber, Selena Gomez
• 72 albums
• 1012 songs
• 120,029 words: transformed the lyrics into tsvector and each
vector (> 3 letters) went into a kyo_word row with its position.
8. @louisemeta
A quick tour on basic SQL
SELECT columns
FROM table_a
(INNER, LEFT, RIGHT, OUTER) JOIN table_a ON table_a.x = table_b.y
WHERE filters
ORDER BY columns
LIMIT X
9. @louisemeta
A quick tour on basic SQL
Artist.objects.get(pk=10)
SELECT id, name
FROM kyo_artist
WHERE id=10;
Artist.find(10)
Artist[10]
Django ORM (python)
Activerecord (ruby)
Sequel (ruby)
session.query(Artist).get(10)
Sqlalchemy (python)
10. @louisemeta
A quick tour on basic SQL
Album.objects
.filter(year_gt=2009)
.order_by(‘year’)
SELECT id, name, year, popularity
FROM kyo_album
WHERE year > 2009
ORDER BY year;
Album.where('year > ?', 2009)
.order(:year)
Album.where(Sequel[:year] > 2009)
.order(:year)
Django ORM (python)
Activerecord (ruby) Sequel (ruby)
session.query(Album)
.filter(Album.year > 2009)
.order_by(Album.year)
Sqlalchemy (python)
11. @louisemeta
A quick tour on basic SQL
Album.objects
.filter(year_gt=2009)
.order_by(‘year’)
.select_related(‘artist’)
SELECT id, name, year, popularity
FROM kyo_album
INNER JOIN kyo_artist ON
kyo_artist.id=kyo_album.artist_id
WHERE year > 2009
ORDER BY year;
Album.where('year > ?', 2009)
.joins(:artist)
.order(:year)
Album.where(Sequel[:year] > 2009)
.join(:artist)
.order(:year)
Django ORM (python)
Activerecord (ruby) Sequel (ruby)
session.query(Album)
.join(‘artist')
.filter(Album.year > 2009)
.order_by(Album.year)
Sqlalchemy (python)
12. @louisemeta
A quick tour on basic SQL
Album.objects
.filter(artist_id=12)[5:10]
SELECT id, name, year, popularity
FROM kyo_album
WHERE artist_id = 12
ORDER BY id
OFFSET 5 LIMIT 10;
Album.where(artist_id: 12)
.limit(10).offset(5)
Album.where(artist_id: 12)
.limit(10).offset(5)
Django ORM (python)
Activerecord (ruby)
Sequel (ruby)
session.query(Album)
.filter(Album.artist_id == 12)
.offset(5).limit(10)
Sqlalchemy (python)
To go further in pagination:
https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
13. @louisemeta
Executing RAW SQL queries
Django
Word.objects.raw(query, args)
with connection.cursor() as cursor:
cursor.execute(query)
data = cursor.fetchall(cursor)
14. @louisemeta
Executing RAW SQL queries
SQLAlchemy
engine = create_engine(‘postgres://localhost:5432/kyo_game’)
with engine.connect() as con:
rs = con.execute(query, **{‘param1’: ‘value’})
rows = rs.fetchall()
15. @louisemeta
Executing RAW SQL queries
Activerecord
rows = ActiveRecord::Base.connection.execute(sql, params)
words = Word.find_by_sql ['SELECT * FROM words WHERE
artist_id=:artist_id', {:artist_id => 14}]
The functions select/where/group also can take raw SQL.
16. @louisemeta
Executing RAW SQL queries
Sequel
DB = Sequel.connect(‘postgres://localhost:5432/kyo_game_ruby')
DB['select * from albums where name = ?', name]
17. @louisemeta
Average popularity of Maroon5’s albums
SELECT AVG(popularity) FROM kyo_album WHERE artist_id=9;
# Django
popularity = Album.objects.filter(artist_id=9).aggregate(value=Avg(‘popularity’)
{'value': 68.16666666667}
# sqlalchemy
session.query(func.avg(Album.popularity).label(‘average'))
.filter(Album.artist_id == 9)
18. @louisemeta
Average popularity of Maroon5’s albums
Ruby - ActiveRecord/Sequel
SELECT AVG(popularity) FROM kyo_album WHERE artist_id=9;
#Activerecord
Album.where(artist_id: 9).average(:popularity)
# Sequel
Album.where(artist_id: 9).avg(:popularity)
19. @louisemeta
Words most used by Justin Timberlake
with the number of songs he used them in
Word Number of occurrences Number of song
love 503 56
know 432 82
like 415 68
girl 352 58
babi 277 59
come 227 58
caus 225 62
right 224 34
yeah 221 54
20. @louisemeta
Words most used by Justin Timberlake
SELECT value,
COUNT(id) AS total
FROM kyo_word
WHERE artist_id = 11
GROUP BY value
ORDER BY total DESC
LIMIT 10
SELECT COUNT(id) AS total,
FROM kyo_word
WHERE artist_id = 11;
17556
Word Number of
occurrenceslove 503
know 432
like 415
girl 352
babi 277
come 227
caus 225
right 224
yeah 221
21. @louisemeta
SELECT value,
COUNT(id) AS total,
COUNT(DISTINCT song_id) AS total_songs
FROM kyo_word
WHERE artist_id = 11
GROUP BY value ORDER BY total DESC
LIMIT 10
Word.objects.filter(artist_id=self.object.pk)
.values('value')
.annotate(total=Count(‘id'),
total_song=Count('song_id', distinct=True))
.order_by('-total')[:10])
Words most used by Justin Timberlake
Django
22. @louisemeta
SELECT value,
COUNT(id) AS total,
COUNT(DISTINCT song_id) AS total_songs
FROM kyo_word
WHERE artist_id = 11
GROUP BY value ORDER BY total DESC
LIMIT 10
session.query(
Word.value,
func.count(Word.value).label(‘total’),
func.count(distinct(Word.song_id)))
.group_by(Word.value)
.order_by(desc('total')).limit(10).all()
Words most used by Justin Timberlake
SQLAlchemy
23. @louisemeta
Words most used by Justin Timberlake
Activerecord
Word.where(artist_id: 11)
.group(:value)
.select('count(distinct song_id), count(id)’)
.order('count(id) DESC’)
.limit(10)
Word.where(artist_id: 11)
.group(:value)
.count(:id)
24. @louisemeta
Words most used by Justin Timberlake
Sequel
Word.group_and_count(:value)
.select_append{count(distinct song_id).as(total)}
.where(artist_id: 11)
.order(Sequel.desc(:count))
Word.where(artist_id: 11)
.group_and_count(:value)
26. @louisemeta
Support with ORMs
Django SQLAlchemy
Activerecord
(*)
Sequel
AVG Yes Yes Yes Yes
COUNT Yes Yes Yes Yes
Min Yes Yes Yes Yes
Max Yes Yes Yes Yes
Sum Yes Yes Yes Yes
* Activerecord: to cumulate operators, you will need to use select() with raw SQL
27. @louisemeta
Words that Avril Lavigne only used in the song
“Complicated”
Word
dress
drivin
foolin
pose
preppi
somethin
strike
unannounc
28. @louisemeta
Words that Avril Lavigne only used in the song
“Complicated” - Django
SELECT *
FROM kyo_word word
WHERE song_id=342
AND NOT EXISTS (
SELECT 1 FROM kyo_word word2 WHERE
word2.artist_id=word.artist_id
word2.song_id <> word.song_id
AND word2.value=word.value);
Filter the result if the
subquery returns no row
Subquery for the same
value for a word, but
different primary key
same_word_artist = (Word.objects
.filter(value=OuterRef(‘value’), artist=OuterRef('artist'))
.exclude(song_id=OuterRef(‘song_id’))
context['unique_words'] = Word.objects.annotate(is_unique=~Exists(same_word_artist))
.filter(is_unique=True, song=self.object)
29. @louisemeta
Words that Avril Lavigne only used in the song
“Complicated” - SQLAlchemy
word1 = Word
word2 = aliased(Word)
subquery = session.query(word2).filter(value == word1.value, artist_id ==
word1.artist_id, song_id != word1.song_id)
session.query(word1).filter(word1.song_id == 342, ~subquery.exists())
30. @louisemeta
Words that Avril Lavigne only used in the song
“Complicated” - Activerecord
Word.where(song_id: 342).where(
'NOT EXISTS (SELECT 1 FROM words word2 WHERE word2.artist_id=words.artist_id
AND word2.song_id <> words.song_id AND word2.value=words.value)’
)
There is an exists method:
Album.where(name: ‘Kyo').exists?
But in a subquery, we join the same table and they can’t have an alias
31. @louisemeta
An example where EXISTS performs better
I wanted to filter the songs that had no value in the table kyo_word yet.
A basic version could be
Song.objects.filter(words__isnull=True)
17-25ms
SELECT "kyo_song"."id", "kyo_song"."name",
"kyo_song"."album_id", "kyo_song"."language"
FROM "kyo_song"
LEFT OUTER JOIN "kyo_word" ON ("kyo_song"."id" =
"kyo_word"."song_id")
WHERE "kyo_word"."id" IS NULL
32. @louisemeta
An example where EXISTS performs better
And with an EXISTS
4-6ms
Song.objects.annotate(processed=Exists(Word.objects.filter(song_id=OuterRef('pk'))))
.filter(processed=False)
SELECT * ,
EXISTS(SELECT * FROM "kyo_word" U0 WHERE U0."song_id" = ("kyo_song"."id"))
AS "processed"
FROM "kyo_song"
WHERE EXISTS(SELECT * FROM "kyo_word" U0 WHERE U0."song_id" =
("kyo_song"."id")) = False
33. @louisemeta
To make it simple, we want to know what group of two words is
repeated more than twice.
“Say it ain’t so
I will not go
Turn the lights off
Carry me home”
Word Next word Occurences
turn light 4
light carri 4
carri home 4
Detecting the chorus of the song
“All the small things” - Blink 182
34. @louisemeta
Detecting the chorus of a song
Subquery
Step 1: Getting the words with their next word
SELECT value,
(
SELECT U0.value
FROM kyo_word U0
WHERE (U0.position > (kyo_word.position) AND U0.song_id = 441)
ORDER BY U0.position ASC
LIMIT 1
) AS "next_word",
FROM "kyo_word"
WHERE "kyo_word"."song_id" = 441
Subquery
35. @louisemeta
Detecting the chorus of a song
Subquery
Step 2: Getting the words with their next word, with counts
SELECT kyo_word.value,
(
SELECT U0.value
FROM kyo_word U0
WHERE (U0.position > (kyo_word.position) AND U0.song_id =
441)
ORDER BY U0.position ASC
LIMIT 1
) AS next_word,
COUNT(*) AS total
FROM kyo_word
WHERE kyo_word.song_id = 441
GROUP BY 1, 2
HAVING COUNT(*) > 2
We want thee count
grouped by the word
and its following word
A chorus should
appear more than
twice in a song
36. @louisemeta
Detecting the chorus of a song
Subquery - Django
Word Next word Occurences
turn light 4
light carri 4
carri home 4
next_word_qs = (Word.objects
.filter(song_id=self.object.pk,
position__gt=OuterRef('position'))
.order_by("position")
.values('value'))[:1]
context['words_in_chorus'] = (Word.objects
.annotate(next_word=Subquery(next_word_qs))
.values('value', 'next_word')
.annotate(total=Count('*'))
.filter(song=self.object, total__gt=2))
.order_by('-total')
37. @louisemeta
Detecting the chorus of a song
Subquery - SQLAlchemy
Word Next word Occurences
turn light 4
light carri 4
carri home 4
word1 = Word
word2 = aliased(Word)
next_word_qs = session.query(word2.value)
.filter(word2.song_id==word1.song_id,
word2.position > word1.position)
.order_by(word2.position).limit(1)
word_in_chorus = session.query(word1.value,
next_word_qs.label('next_word'),
func.count().label(‘total'))
.filter(word1.song_id == 441, func.count() > 2)
.group_by(word1.value, 'next_word')
38. @louisemeta
Detecting the chorus of a song
Subquery - ActiveRecord
Word.select('value,
(SELECT U0.value FROM words U0 WHERE U0.position >
(words.position) AND U0.song_id = words.song_id ORDER BY
U0.position ASC LIMIT 1) as next_word,
count(*) as total’)
.where(song_id: 441)
.having('count(*) > 2’)
.group('1, 2')
Word Next word Occurences
turn light 4
light carri 4
carri home 4
39. @louisemeta
Detecting the chorus of a song
Subquery - Sequel
Word Next word Occurences
turn light 4
light carri 4
carri home 4
Word.from(DB[:words].where(Sequel[:song_id] =~ 441).as(:word_2))
.select{[
Sequel[:word_2][:value],
Word.select(:value).where{(Sequel[:position] > Sequel[:word_2]
[:position]) & (Sequel[:song_id] =~ 441)}.
order(Sequel[:position]).limit(1).as(:following_word),
count(:id).as(:total)]}
.group(:value, :following_word)
.having(:total > 2)
40. @louisemeta
Support with ORMs
Django SQLAlchemy Activerecord Sequel
Subquery
into column
Yes Yes No Yes
FROM
subquery
No Yes Yes Yes
Subquery in
WHERE
clause
Yes Yes Yes-ish Yes
41. @louisemeta
LATERAL JOIN
We want all the artists with their last album.
Artist id Artist Name Album id Album name Year Popularity
6 Kyo 28 Dans a peau 2017 45
8 Blink-182 38 California 2016
9 Maroon5 44 Red Pill Blues 2017 82
10 Jonas Brothers 51 Happiness Begins 2019
11 Justin Timberlake 61 Man Of The Woods 2018
12 Avril Lavigne 69 Head Above Water 2019
13 Backstreet Boys 90 DNA 2019 68
14 Britney Spears 87 Glory 2016
15 Justin Bieber 104 Purpose 2015
16 Selena Gomez 98 Revival 2015
42. @louisemeta
LATERAL JOIN
SELECT artist.*,
(SELECT * FROM kyo_album album
WHERE album.artist_id = artist_id
ORDER BY year DESC
LIMIT 1) AS album
FROM kyo_artist artist;
ERROR: subquery must return only one column
Why can’t we do it with a subquery?
43. @louisemeta
LATERAL JOIN
SELECT artist.*,
album.*
FROM kyo_artist artist
INNER JOIN (
SELECT * FROM kyo_album
ORDER BY year DESC
LIMIT 1
) AS album ON artist.id = album.artist_id;
id | name | id | name | year | artist_id | popularity
----+----------------+----+------------------+------+-----------+------------
10 | Jonas Brothers | 51 | Happiness Begins | 2019 | 10 |
(1 row)
Why can’t we do it by joining subqueries?
44. @louisemeta
LATERAL JOIN
SELECT artist.*, album.*
FROM kyo_artist artist
INNER JOIN LATERAL (
SELECT * FROM kyo_album
WHERE kyo_album.artist_id = artist.id
ORDER BY year DESC LIMIT 1) album on true
Here is the solution
45. @louisemeta
LATERAL JOIN
So we get our wanted result
Artist id Artist Name Album id Album name Year Popularity
6 Kyo 28 Dans a peau 2017 45
8 Blink-182 38 California 2016
9 Maroon5 44 Red Pill Blues 2017 82
10 Jonas Brothers 51 Happiness Begins 2019
11 Justin Timberlake 61 Man Of The Woods 2018
12 Avril Lavigne 69 Head Above Water 2019
13 Backstreet Boys 90 DNA 2019 68
14 Britney Spears 87 Glory 2016
15 Justin Bieber 104 Purpose 2015
16 Selena Gomez 98 Revival 2015
46. @louisemeta
LATERAL JOIN
SQLAlchemy
It won’t return an object, as the result can’t really be matched to
one. So you will use the part of sqlalchemy that’s not ORM like
subquery = select([
album.c.id,
album.c.name,
album.c.year,
album.c.popularity])
.where(album.c.artist_id==artist.c.id)
.order_by(album.c.year)
.limit(1)
.lateral(‘album_subq')
query = select([artist, subquery.c.name, subquery.c.year,
subquery.c.popularity])
.select_from(artist.join(subquery, true()))
48. @louisemeta
For each album of the backstreet boys:
Words ranked by their frequency
Word Album Frequency Rank
roll Backstreet Boys 78 1
know Backstreet Boys 46 2
heart Backstreet Boys 43 3
babi Backstreet Boys 42 4
wanna Backstreet Boys 36 5
everybodi Backstreet Boys 33 6
parti Backstreet Boys 30 7
girl Backstreet Boys 28 8
nobodi Backstreet Boys 26 9
… … … …
crazi Backstreet Boys 8 24
wish Backstreet Boys 8 24
shake Backstreet Boys 7 25
… … … …
love Backstreet's Back 33 3
yeah Backstreet's Back 27 4
babi Backstreet's Back 23 5
… … … …
49. @louisemeta
For each album of the backstreet boys:
Words ranked by their frequency
SELECT value as word,
name as album_name,
COUNT(word.value) AS frequency,
DENSE_RANK() OVER (
PARTITION BY word.album_id
ORDER BY COUNT(word.value) DESC
) AS ranking
FROM kyo_word word
INNER JOIN kyo_album album
ON (word.album_id = album.id)
WHERE word.artist_id = 13
GROUP BY word.value,
album.name,
word.album_id
ORDER BY word.album_id ASC
Dense rank is the
function we use for the
window function
We indicate the
partition by album_id,
because we want a rank
per album
We indicate the order
to use to define the
rank
50. @louisemeta
For each album of the backstreet boys:
Window Functions - Django
from django.db.models import Count, Window, F
from django.db.models.functions import DenseRank
dense_rank_by_album = Window(
expression=DenseRank(),
partition_by=F("album_id"),
order_by=F("frequency").desc())
ranked_words = (Word.objects
.filter(artist_id=self.object)
.values('value', 'album__name')
.annotate(frequency=Count('value'),
ranking=dense_rank_by_album)
.order_by('album_id'))
51. @louisemeta
Ranking backstreet boys vocabulary by frequency
The result being very long, we want to limit it to the words ranked
5 or less for each album.
Problem: This won’t work
SELECT value as word,
name as album_name,
COUNT(word.value) AS frequency,
DENSE_RANK() OVER (
PARTITION BY word.album_id
ORDER BY COUNT(word.value) DESC
) AS ranking
FROM kyo_word word
INNER JOIN kyo_album album
ON (word.album_id = album.id)
WHERE word.artist_id = 13 AND ranking < 6
GROUP BY word.value,
album.name,
word.album_id
ORDER BY word.album_id ASC
52. @louisemeta
Ranking backstreet boys vocabulary by frequency
SELECT * FROM (
SELECT value,
name as album_name,
COUNT(word.value) AS frequency,
DENSE_RANK() OVER (
PARTITION BY word.album_id
ORDER BY COUNT(word.value) DESC
) AS ranking
FROM kyo_word word
INNER JOIN kyo_album album
ON (word.album_id = album.id)
WHERE word.artist_id = 13
GROUP BY word.value,
album.name,
word.album_id
ORDER BY word.album_id ASC) a
WHERE a.ranking < 6 AND a.frequency > 5;
This is our previous
query wrapped into a
subquery
Instead of selecting
FROM a table, we
select from the query
We can now filter on
ranking
53. @louisemeta
Ranking backstreet boys vocabulary by frequency
Window Functions + Subquery - Django
In Django, you can’t do a SELECT … FROM (subquery)
query = “””
SELECT * FROM (…)
WHERE a.ranking < %s AND a.frequency > %s;"""
“””
queryset = Word.objects.raw(query, [13, 6, 5])
for word in queryset:
print(word.value, word.ranking)
55. @louisemeta
For each album of the backstreet boys:
Window Functions + subquery - Activerecord
my_subquery = Word
.where(artist_id: 13)
.joins(:album)
.group(:value, :name, :album_id)
.select(‘DENSE_RANK() OVER(PARTITION BY album_id ORDER BY
COUNT(value) DESC) AS ranking’)
Word.select(‘*’).from(my_subquery, :subquery).where('ranking < 6')
56. @louisemeta
For each album of the backstreet boys:
Window Functions + subquery - Sequel
Word
.where(artist_id: 13)
.group(:value, :album_id)
.select{dense_rank.function.over(partition: :album_id,
:order=>Sequel.desc(count(:value))).as(:ranking)}
.select_append(:album_id)
.from_self(:alias => ‘subquery')
.where(Sequel[:ranking] > 5)
57. @louisemeta
To go further
Window Functions: performs a calculation across a set of
rows. Comparable to aggregate functions though each row
remains in the result of the query.
AVG
RANK / DENSE_RANK
SUM
COUNT
58. @louisemeta
Support with ORMs
Django SQLAlchemy Activerecord Sequel
Avg Yes Yes No Yes
Rank Yes Yes No Yes
Dense Rank Yes Yes No Yes
Sum Yes Yes No Yes
Count Yes Yes No Yes
59. @louisemeta
GROUPING SETS
The goal of grouping sets is to have sub result in a query with different
group by.
Here we want for all bands, the number of songs:
- Per album
- Per artist
- Per year, for all artist
- Total
60. @louisemeta
GROUPING SETS
Result expected:
Artist Album Year Number of songs
Maroon5 Songs About Jane 2002 12
Maroon5 It Won't Be Soon Before Long 2007 17
Maroon5 Hands All Over 2010 17
Maroon5 Overexposed 2012 16
Maroon5 V 2014 14
Maroon5 All albums 76
Selena Gomez Kiss & Tell 2009 14
Selena Gomez A Year Without Rain 2010 11
Selena Gomez When The Sun Goes Down 2011 13
Selena Gomez Stars Dance 2013 15
Selena Gomez For You 2014 15
Selena Gomez Revival 2015 16
Selena Gomez All albums 84
All artists All albums 1992 8
All artists All albums 1994 30
All artists All albums 1995 13
All artists All albums 1997 26
All artists All albums 1999 38
All artists All albums 2000 39
All artists All albums 2001 34
All artists All albums 2002 40
… … … …
All artists All albums 2019 43
All artists All albums 1012
66. @louisemeta
Common Table Expression (CTE)
• Defined by a WITH clause
• You can see it as a temporary table, private to a query
• Helps break down big queries in a more readable way
• A CTE can reference other CTEs within the same WITH clause
(Nest). A subquery cannot reference other subqueries
• A CTE can be referenced multiple times from a calling query.
A subquery cannot be referenced.
67. @louisemeta
WITH last_album AS (
SELECT album.id FROM kyo_album album
WHERE artist_id = 15
ORDER BY year DESC LIMIT 1),
older_songs AS (
SELECT song.id FROM kyo_song song
INNER JOIN kyo_album album ON (album.id = song.album_id)
WHERE album_id NOT IN (SELECT id FROM last_album)
AND album.artist_id=15
)
SELECT value, COUNT(*) FROM kyo_word
INNER JOIN last_album ON kyo_word.album_id=last_album.id
WHERE value NOT IN (
SELECT value
FROM kyo_word
INNER JOIN older_songs ON kyo_word.song_id=older_songs.id)
GROUP BY value ORDER BY 2 DESC;
Common Table Expression (CTE)
68. @louisemeta
Common Table Expression (CTE)
value count
sorri 21
journey 9
mark 8
blame 6
direct 4
wash 4
children 4
serious 4
human 3
delusion 3
disappoint 2
confus 2
70. @louisemeta
Things I haven’t talked about
• Indexes
• Constraints
• Fulltext search (fully supported in the Django ORM)
• ´Recursive CTE
• INSERT / UPDATE / DELETE
• INSERT … ON CONFLICT
• …
71. @louisemeta
Conclusion
Here are the features we saw today and their compatibility
with Django ORM
Django SQLAlchemy Activerecord Sequel
Aggregations
(count, avg, sum)
Yes Yes Yes Yes
DISTINCT Yes Yes Yes Yes
GROUP BY Yes Yes Yes Yes
(NOT) EXISTS Yes Yes Yes Yes
Subqueries in SELECT Yes Yes No Yes
Subqueries in FROM No Yes Yes Yes
Subqueries in WHERE Yes Yes Kind of Yes
LATERAL JOIN No Yes No Yes
Window functions Yes Yes No Yes
GROUPINGS SETS No Yes No Yes
CTE No Yes No Yes
72. @louisemeta
Thank you for your attention!
louisemeta.com
citusdata.com/newsletter
@louisemeta. @citusdata