Love Your Database Railsconf 2017

Guyren G Howe
guyren@relevantlogic.com
@unclouded
Brought to you by
https://github.com/gisborne/railsconf_lydb

Information Technology -
Database Language SQL
(Proposed revised text of DIS 9075)
July 1992

Postgres’ Little-Known Secret
• Postgres is a great programming platform

Novel Postgres Features
• PostGIS
• Background workers
• Full-text search
• Custom dictionaries
• Stemming
• Stop words
• Normalization
• Synonyms
• Multiple languages
• Phrase search
• Proximity search

• Proper type system
• Custom types
• Functions
• Automatic casts
• Tables are types
• Functions take/act as
relations
• Arrays
• JSON
• HStore
• XML

• Foreign Data Wrappers
• SQL Databases
• LDAP
• Elastic Search
• Redis
• MongoDB
• Cassandra, CouchDB, …
• CSV/JSON/XML/…
• git/file system/imap/twitter
• google spreadsheet
• Roll your own (Python)
• …

• Program in civilized languages
• Python
• Perl
• Javascript
• R
• Scheme
• C
• Java
• PHP
• sh
• …

Check out public.all_join view

CREATE OR REPLACE VIEW public.all_join AS
SELECT c.customerid,
c.firstname,
c.lastname,
c.address1,
c.address2,
c.city,
c.state,
c.zip,
c.country,
c.email,
c.phone,
c.creditcardtype,
c.creditcard,
c.creditcardexpiration,
c.username,
c.password,
c.age,
c.income,
c.gender,
o.orderid,
o.orderdate,
o.netamount,
o.tax,
o.totalamount,
ol.quantity AS qty,
p.prod_id,
p.title,
p.price,
inv.quan_in_stock,
inv.sales,
cat.categoryname AS category
FROM customers c
FULL JOIN orders o USING (customerid)
FULL JOIN orderlines ol USING (orderid)
FULL JOIN products p USING (prod_id)
FULL JOIN categories cat ON p.category_id = cat.category
FULL JOIN inventory inv USING (prod_id)

Create a Schema
CREATE SCHEMA reporting;

Create a Schema
class AddReportingSchema < ActiveRecord::Migration[5.0]
def up
execute 'CREATE SCHEMA reporting'
end
def down
execute 'DROP SCHEMA reporting'
end
end

Create a Reports View 1
CREATE VIEW
all_views AS
SELECT
c.relname AS name,
obj_description(c.oid) AS description,
ns.nspname AS namespace
FROM
pg_class c JOIN
pg_namespace ns ON c.relnamespace = ns.oid
WHERE
c.relkind = 'v'::"char"

Create a Reports View 2
CREATE VIEW
public.reports AS
SELECT
name,
description
FROM
all_views
WHERE
namespace = 'reporting'::name

Create reporting.customers ViewCREATE VIEW
reporting.customers
AS
SELECT
customerid,
lastname,
firstname,
zip,
ARRAY_AGG(prod_id ORDER BY orderdate, prod_id) AS prod_ids,
ARRAY_AGG(title ORDER BY orderdate, prod_id) AS titles,
ARRAY_AGG(category ORDER BY orderdate, prod_id) AS categories,
ARRAY_AGG(price ORDER BY orderdate, prod_id) AS prices,
SUM(totalamount) AS spent
FROM
all_join
GROUP BY
customerid, lastname, firstname, zip

Add Comment to reporting.customers
COMMENT ON
VIEW
reporting.customers
IS
'Customers with products sold to them';

Create Reports Controller, Model, Route

Create another reportWITH
scifi_viewers AS (
SELECT customerid
FROM all_join
WHERE ((category)::text = 'Sci-Fi'::text)
),
californians AS (
SELECT customerid
FROM all_join
WHERE ((state)::text = 'CA'::text)
),
targets AS (
SELECT customerid,
firstname,
lastname,
address1,
address2,
city,
state,
zip
FROM
all_join
WHERE
customerid IN (
SELECT customerid FROM scifi_viewers
)
OR customerid IN
(SELECT customerid FROM californians)
)
…
These are
Common Table Expressions

Create another reportSELECT prod_id,
title,
price,
quan_in_stock,
sales,
sum(totalamount) AS total_amt,
count(DISTINCT orderid) AS order_ct,
array_agg(lastname ORDER BY lastname, firstname, customerid) AS lastnames,
array_agg(firstname ORDER BY lastname, firstname, customerid) AS firstnames,
array_agg(state ORDER BY lastname, firstname, customerid) AS states,
array_agg(zip ORDER BY lastname, firstname, customerid) AS zips
FROM
targets
GROUP BY
prod_id,
title,
price,
quan_in_stock,
sales

Create reporting.customers_by_state View
…
SELECT
state,
CASE
WHEN LAG(state) OVER (ORDER BY state, lastname) IS NULL
OR
LAG(state) OVER (ORDER BY state, lastname)::text <> state::text
THEN
(('{"break": [1, "'::text || state::text) || '"]}'::text)::jsonb
ELSE
NULL::jsonb
END AS _meta),
…
Copy customers and add this

WHERE (length((state)::text) > 0)
GROUP BY
customerid,
lastname,
firstname,
state,
zip
ORDER BY
state,
lastname
… also add this

What’s this?
LAG(state) OVER (ORDER BY state, lastname)

From Postgres Docs
• A window function performs a calculation across a set of table
rows that are somehow related to the current row…
• …unlike regular aggregate functions, use of a window function does
not cause rows to become grouped

Three parts
• The function
• Which rows (optional)
• Which order (optional)

LAG(
state
)
OVER (
ORDER BY
state,
lastname
)
The function:
previous value of field (state)
SQL is sets.
“Previous” is meaningless
without an order.
OVER indicates window function

SELECT
depname,
empno,
salary,
rank() OVER (
PARTITION BY
depname
ORDER BY
salary DESC
)
FROM
empsalary;
Which rows
Optional.
Without it, applies to all rows

Order required
Function Use
rank Rank within partition
row_number Number of row within partition
lag Value from preceding row
lead Value from later row
…

Order Optional
• Makes function cumulative
• Applies to rows up to or equal on ORDER BY

Without ORDER BY
SELECT
salary,
SUM(salary)
OVER ()
FROM
empsalary;
salary | sum
--------+-------
5200 | 47100
5000 | 47100
3500 | 47100
4800 | 47100
3900 | 47100
4200 | 47100
4500 | 47100
4800 | 47100
6000 | 47100
5200 | 47100
(10 rows)

With ORDER BY
SELECT
salary,
SUM(salary)
OVER (
ORDER BY
salary)
FROM
empsalary;
salary | sum
--------+-------
3500 | 3500
3900 | 7400
4200 | 11600
4500 | 16100
4800 | 25700
4800 | 25700
5000 | 30700
5200 | 41100
5200 | 41100
6000 | 47100
(10 rows)

Create another schema
CREATE SCHEMA reporting_helpers;

Create a view
CREATE VIEW
reporting_helpers.customers_by_state_ranked
AS
SELECT
*,
rank() OVER (
PARTITION BY
c.state
ORDER BY
c.spent) AS rank
FROM
reporting.customers_by_state c

Add Javascript
CREATE EXTENSION plv8;

Create a Javascript FunctionCREATE FUNCTION
reporting.formatted_cust_by_state_bold(
c reporting_helpers.customers_by_state_ranked
)
RETURNS
LANGUAGE
plv8
STRICT COST 1
AS $function$
if (c.rank == 1) {
c['_meta'] = c['_meta'] || {}
c['_meta']['raw'] = c['_meta']['raw'] || {}
c['_meta']['raw']['spent'] = c['_meta']['raw']['spent'] || {}
c['_meta']['raw']['spent'] = "<td><b>" + c.spent + "</b></td>"
}
return c
$function$
A table/view defines a type
STRICT means “functional on a given db state”

Create a Javascript Function
CREATE FUNCTION
reporting.formatted_cust_by_state_bold(
c reporting_helpers.customers_by_state_ranked
)
RETURNS
LANGUAGE
plv8
STRICT COST 1
AS $function$

LANGUAGE
plv8
STRICT COST 1
AS $function$
if (c.rank == 1) {
c['_meta'] = c['meta'] || {}
c['_meta']['raw'] = c['_meta']['raw'] || {}
c['_meta']['raw']['spent'] = c['_meta']['raw']['spent'] || {}
c['_meta']['raw']['spent'] = "<td><b>" + c.spent + "</b></td>"
}
return c
$function$
Create a Javascript Function

Create View
CREATE VIEW
reporting.customers_by_state_ranked
AS
SELECT
(reporting.formatted_cust_by_state_bold(c)).*
FROM
reporting_helpers.customers_by_state_ranked c
ORDER BY
state,
lastname
SQL sucks

This is different
• Fast
• Language Neutral
• Functional
• Relational

“Web Services Architecture”

“Web Services Architecture”
Every arrow requires:
• Route
• Controller Actions
• Tests
• Model Methods
• Tests

“Services Architecture”
Every arrow requires:
• Model Methods
• Tests

• Proper type system
• Custom types
• Functions
• Automatic casts
• Tables are types
• Arrays
• JSON
• XML

In Console (on a Mac)
ActiveRecord::Base.connection_pool.with_connection do |connection|
c = connection.instance_variable_get(:@connection)
c.async_exec "LISTEN speak"
c.wait_for_notify do |channel, pid, payload|
p "got #{channel} #{pid} #{payload}"
`say #{payload}`
end
c.async_exec "UNLISTEN *"
end

In psql
SELECT pg_notify('speak', 'Hello from Postgres');
or another console:
ActiveRecord::Base.connection.execute("
SELECT pg_notify('speak', 'Hello from Postgres');")

We’re hiring
California
Connecticut
Illinois
Minnesota
New Jersey
North Carolina
Ohio
Pennsylvania
Tennessee
Texas
Washington

Guyren G Howe
guyren@relevantlogic.com
@unclouded
Brought to you by
http://scriptscribe.org
https://medium.com/@gisborne

Love Your Database Railsconf 2017

More Related Content

What's hot

Similar to Love Your Database Railsconf 2017

Recently uploaded

Love Your Database Railsconf 2017