Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Love Your Database Railsconf 2017

301 views

Published on

Love Your Database. An interesting example of how to exploit Postgres' advanced types and of running code in the database. Take a step beyond stored procedures!

Published in: Software
  • Be the first to comment

  • Be the first to like this

Love Your Database Railsconf 2017

  1. 1. Guyren G Howe guyren@relevantlogic.com @unclouded Brought to you by https://github.com/gisborne/railsconf_lydb
  2. 2. Information Technology - Database Language SQL (Proposed revised text of DIS 9075) July 1992
  3. 3. Postgres’ Little-Known Secret • Postgres is a great programming platform
  4. 4. Novel Postgres Features • PostGIS • Background workers • Full-text search • Custom dictionaries • Stemming • Stop words • Normalization • Synonyms • Multiple languages • Phrase search • Proximity search
  5. 5. Novel Postgres Features • Proper type system • Custom types • Functions • Automatic casts • Tables are types • Functions take/act as relations • Arrays • JSON • HStore • XML
  6. 6. Novel Postgres Features • Foreign Data Wrappers • SQL Databases • LDAP • Elastic Search • Redis • MongoDB • Cassandra, CouchDB, … • CSV/JSON/XML/… • git/file system/imap/twitter • google spreadsheet • Roll your own (Python) • …
  7. 7. Novel Postgres Features • Program in civilized languages • Python • Perl • Javascript • R • Scheme • C • Java • PHP • sh • …
  8. 8. Model All the Things
  9. 9. Reporting
  10. 10. Check out public.all_join view
  11. 11. CREATE OR REPLACE VIEW public.all_join AS SELECT c.customerid, c.firstname, c.lastname, c.address1, c.address2, c.city, c.state, c.zip, c.country, c.email, c.phone, c.creditcardtype, c.creditcard, c.creditcardexpiration, c.username, c.password, c.age, c.income, c.gender, o.orderid, o.orderdate, o.netamount, o.tax, o.totalamount, ol.quantity AS qty, p.prod_id, p.title, p.price, inv.quan_in_stock, inv.sales, cat.categoryname AS category FROM customers c FULL JOIN orders o USING (customerid) FULL JOIN orderlines ol USING (orderid) FULL JOIN products p USING (prod_id) FULL JOIN categories cat ON p.category_id = cat.category FULL JOIN inventory inv USING (prod_id)
  12. 12. Create a Schema CREATE SCHEMA reporting;
  13. 13. Create a Schema class AddReportingSchema < ActiveRecord::Migration[5.0] def up execute 'CREATE SCHEMA reporting' end def down execute 'DROP SCHEMA reporting' end end
  14. 14. Create a Reports View 1 CREATE VIEW all_views AS SELECT c.relname AS name, obj_description(c.oid) AS description, ns.nspname AS namespace FROM pg_class c JOIN pg_namespace ns ON c.relnamespace = ns.oid WHERE c.relkind = 'v'::"char"
  15. 15. Create a Reports View 2 CREATE VIEW public.reports AS SELECT name, description FROM all_views WHERE namespace = 'reporting'::name
  16. 16. Create reporting.customers ViewCREATE VIEW reporting.customers AS SELECT customerid, lastname, firstname, zip, ARRAY_AGG(prod_id ORDER BY orderdate, prod_id) AS prod_ids, ARRAY_AGG(title ORDER BY orderdate, prod_id) AS titles, ARRAY_AGG(category ORDER BY orderdate, prod_id) AS categories, ARRAY_AGG(price ORDER BY orderdate, prod_id) AS prices, SUM(totalamount) AS spent FROM all_join GROUP BY customerid, lastname, firstname, zip
  17. 17. View that
  18. 18. Add Comment to reporting.customers COMMENT ON VIEW reporting.customers IS 'Customers with products sold to them';
  19. 19. Create Reports Controller, Model, Route
  20. 20. Rails Report Index View
  21. 21. _report partial
  22. 22. Start Rails; hit /reports
  23. 23. Add detail View
  24. 24. Hit /reports/customers
  25. 25. Cool
  26. 26. Create another reportWITH scifi_viewers AS ( SELECT customerid FROM all_join WHERE ((category)::text = 'Sci-Fi'::text) ), californians AS ( SELECT customerid FROM all_join WHERE ((state)::text = 'CA'::text) ), targets AS ( SELECT customerid, firstname, lastname, address1, address2, city, state, zip FROM all_join WHERE customerid IN ( SELECT customerid FROM scifi_viewers ) OR customerid IN (SELECT customerid FROM californians) ) … These are Common Table Expressions
  27. 27. Create another reportSELECT prod_id, title, price, quan_in_stock, sales, sum(totalamount) AS total_amt, count(DISTINCT orderid) AS order_ct, array_agg(lastname ORDER BY lastname, firstname, customerid) AS lastnames, array_agg(firstname ORDER BY lastname, firstname, customerid) AS firstnames, array_agg(state ORDER BY lastname, firstname, customerid) AS states, array_agg(zip ORDER BY lastname, firstname, customerid) AS zips FROM targets GROUP BY prod_id, title, price, quan_in_stock, sales
  28. 28. Get Jiggy Wid It
  29. 29. Create reporting.customers_by_state View … SELECT state, CASE WHEN LAG(state) OVER (ORDER BY state, lastname) IS NULL OR LAG(state) OVER (ORDER BY state, lastname)::text <> state::text THEN (('{"break": [1, "'::text || state::text) || '"]}'::text)::jsonb ELSE NULL::jsonb END AS _meta), … Copy customers and add this
  30. 30. WHERE (length((state)::text) > 0) GROUP BY customerid, lastname, firstname, state, zip ORDER BY state, lastname … also add this
  31. 31. What’s this? LAG(state) OVER (ORDER BY state, lastname)
  32. 32. From Postgres Docs • A window function performs a calculation across a set of table rows that are somehow related to the current row… • …unlike regular aggregate functions, use of a window function does not cause rows to become grouped
  33. 33. Three parts • The function • Which rows (optional) • Which order (optional)
  34. 34. LAG( state ) OVER ( ORDER BY state, lastname ) The function: previous value of field (state) SQL is sets. “Previous” is meaningless without an order. OVER indicates window function
  35. 35. SELECT depname, empno, salary, rank() OVER ( PARTITION BY depname ORDER BY salary DESC ) FROM empsalary; Which rows Optional. Without it, applies to all rows
  36. 36. Order required Function Use rank Rank within partition row_number Number of row within partition lag Value from preceding row lead Value from later row …
  37. 37. Order Optional • Makes function cumulative • Applies to rows up to or equal on ORDER BY
  38. 38. Without ORDER BY SELECT salary, SUM(salary) OVER () FROM empsalary; salary | sum --------+------- 5200 | 47100 5000 | 47100 3500 | 47100 4800 | 47100 3900 | 47100 4200 | 47100 4500 | 47100 4800 | 47100 6000 | 47100 5200 | 47100 (10 rows)
  39. 39. With ORDER BY SELECT salary, SUM(salary) OVER ( ORDER BY salary) FROM empsalary; salary | sum --------+------- 3500 | 3500 3900 | 7400 4200 | 11600 4500 | 16100 4800 | 25700 4800 | 25700 5000 | 30700 5200 | 41100 5200 | 41100 6000 | 47100 (10 rows)
  40. 40. _break partial
  41. 41. View that
  42. 42. Get More Jiggy Wid It
  43. 43. Create another schema CREATE SCHEMA reporting_helpers;
  44. 44. Create a view CREATE VIEW reporting_helpers.customers_by_state_ranked AS SELECT *, rank() OVER ( PARTITION BY c.state ORDER BY c.spent) AS rank FROM reporting.customers_by_state c
  45. 45. Add Javascript CREATE EXTENSION plv8;
  46. 46. Create a Javascript FunctionCREATE FUNCTION reporting.formatted_cust_by_state_bold( c reporting_helpers.customers_by_state_ranked ) RETURNS reporting_helpers.customers_by_state_ranked LANGUAGE plv8 STRICT COST 1 AS $function$ if (c.rank == 1) { c['_meta'] = c['_meta'] || {} c['_meta']['raw'] = c['_meta']['raw'] || {} c['_meta']['raw']['spent'] = c['_meta']['raw']['spent'] || {} c['_meta']['raw']['spent'] = "<td><b>" + c.spent + "</b></td>" } return c $function$ A table/view defines a type STRICT means “functional on a given db state”
  47. 47. Create a Javascript Function CREATE FUNCTION reporting.formatted_cust_by_state_bold( c reporting_helpers.customers_by_state_ranked ) RETURNS reporting_helpers.customers_by_state_ranked LANGUAGE plv8 STRICT COST 1 AS $function$
  48. 48. LANGUAGE plv8 STRICT COST 1 AS $function$ if (c.rank == 1) { c['_meta'] = c['meta'] || {} c['_meta']['raw'] = c['_meta']['raw'] || {} c['_meta']['raw']['spent'] = c['_meta']['raw']['spent'] || {} c['_meta']['raw']['spent'] = "<td><b>" + c.spent + "</b></td>" } return c $function$ Create a Javascript Function
  49. 49. Create View CREATE VIEW reporting.customers_by_state_ranked AS SELECT (reporting.formatted_cust_by_state_bold(c)).* FROM reporting_helpers.customers_by_state_ranked c ORDER BY state, lastname SQL sucks
  50. 50. View that …
  51. 51. This is different • Fast • Language Neutral • Functional • Relational
  52. 52. “Web Services Architecture”
  53. 53. “Web Services Architecture” Every arrow requires: • Route • Controller Actions • Tests • Model Methods • Tests
  54. 54. “Services Architecture”
  55. 55. “Services Architecture” Every arrow requires: • Model Methods • Tests
  56. 56. Novel Postgres Features • Proper type system • Custom types • Functions • Automatic casts • Tables are types • Arrays • JSON • XML
  57. 57. Novel Postgres Features • Foreign Data Wrappers • SQL Databases • LDAP • Elastic Search • Redis • MongoDB • Cassandra, CouchDB, … • CSV/JSON/XML/… • git/file system/imap/twitter • google spreadsheet • Roll your own (Python) • …
  58. 58. In Console (on a Mac) ActiveRecord::Base.connection_pool.with_connection do |connection| c = connection.instance_variable_get(:@connection) c.async_exec "LISTEN speak" c.wait_for_notify do |channel, pid, payload| p "got #{channel} #{pid} #{payload}" `say #{payload}` end c.async_exec "UNLISTEN *" end
  59. 59. In psql SELECT pg_notify('speak', 'Hello from Postgres'); or another console: ActiveRecord::Base.connection.execute(" SELECT pg_notify('speak', 'Hello from Postgres');")
  60. 60. Discussion Go.
  61. 61. We’re hiring California Connecticut Illinois Minnesota New Jersey North Carolina Ohio Pennsylvania Tennessee Texas Washington
  62. 62. Guyren G Howe guyren@relevantlogic.com @unclouded Brought to you by http://scriptscribe.org https://medium.com/@gisborne

×