Successfully reported this slideshow.
Your SlideShare is downloading. ×

Scrap your query boilerplate with specql

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 29 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Scrap your query boilerplate with specql (20)

Advertisement

Recently uploaded (20)

Scrap your query boilerplate with specql

  1. 1. SCRAP YOUR QUERY BOILERPLATE WITH SPECQL 2.9.2017 ClojuTRE Tatu Tarvainen (@tatut)
  2. 2. WHAT IS SPECQL? › The combination of clojure.spec, PostgreSQL and the power of macros clojure.spec PostgreSQL λ
  3. 3. SOME BACKGROUND › Yesql/Jeesql/Hugsql are all fine and specql does not try to ”solve” SQL so that you don’t have to use it anymore › When you have multiple slightly different queries, there’s duplication or fetching too much › Adding dynamic WHERE clauses leads to ugly NULL checking of bound parameters
  4. 4. EXAMPLE › SELECT it.foo, it.bar, it.baz › FROM interesting_table it › WHERE it.archived = FALSE › AND it.organization = :user_org_id › AND it.category IN (:categories)
  5. 5. EXAMPLE › SELECT it.foo, it.bar, it.baz › FROM interesting_table it › WHERE it.archived = FALSE › AND it.organization = :user_org_id › AND it.category IN (:categories) How to make variants without duplication or selecting too much?
  6. 6. EXAMPLE › SELECT it.foo, it.bar, it.baz › FROM interesting_table it › WHERE it.archived = FALSE › AND it.organization = :user_org_id › AND it.category IN (:categories) How to add/remove clauses dynamically?
  7. 7. EXAMPLE › SELECT it.foo, it.bar, it.baz, › o.name AS organization_name › FROM interesting_table it › JOIN organization o ON it.organization_id = o.id › WHERE it.archived = FALSE › AND it.organization = :user_org_id › AND it.category IN (:categories)
  8. 8. EXAMPLE › SELECT it.foo, it.bar, it.baz, › o.name AS organization_name › FROM interesting_table it › JOIN organization o ON it.organization_id = o.id › WHERE it.archived = FALSE › AND it.organization = :user_org_id › AND it.category IN (:categories) How to cleanly add these without code duplication?
  9. 9. THE SPECQL WAY: DEFINE › (define-tables db › [”interesting_table” ::it/interesting › {::it/organization (rel/has-one › ::it/organization_id › ::org/organization › ::org/id)}])
  10. 10. THE SPECQL WAY: DEFINE › (define-tables db › [”interesting_table” ::it/interesting › {::it/organization (rel/has-one › ::it/organization_id › ::org/organization › ::org/id)}]) The main macro for defining things
  11. 11. THE SPECQL WAY: DEFINE › (define-tables db › [”interesting_table” ::it/interesting › {::it/organization (rel/has-one › ::it/organization_id › ::org/organization › ::org/id)}]) The name of the table in the database
  12. 12. THE SPECQL WAY: DEFINE › (define-tables db › [”interesting_table” ::it/interesting › {::it/organization (rel/has-one › ::it/organization_id › ::org/organization › ::org/id)}]) The namespaced keyword for the table
  13. 13. THE SPECQL WAY: DEFINE › (define-tables db › [”interesting_table” ::it/interesting › {::it/organization (rel/has-one › ::it/organization_id › ::org/organization › ::org/id)}]) Additional definitions for columns
  14. 14. THE SPECQL WAY: FETCH › (defn fetch-interesting [db org-id categories] › (fetch db ::it/interesting › #{::it/foo ::it/bar ::it/baz} › {::it/archived false › ::it/organization_id org-id › ::it/category (op/in categories)}))
  15. 15. THE SPECQL WAY: FETCH › (defn fetch-interesting [db org-id categories] › (fetch db ::it/interesting › #{::it/foo ::it/bar ::it/baz} › {::it/archived false › ::it/organization_id org-id › ::it/category (op/in categories)})) The table to fetch from
  16. 16. THE SPECQL WAY: FETCH › (defn fetch-interesting [db org-id categories] › (fetch db ::it/interesting › #{::it/foo ::it/bar ::it/baz} › {::it/archived false › ::it/organization_id org-id › ::it/category (op/in categories)})) Set of columns to retrieve
  17. 17. THE SPECQL WAY: FETCH › (defn fetch-interesting [db org-id categories] › (fetch db ::it/interesting › #{::it/foo ::it/bar ::it/baz} › {::it/archived false › ::it/organization_id org-id › ::it/category (op/in categories)})) The WHERE clause map
  18. 18. THE SPECQL WAY: RESULT › (fetch-interesting db 1 #{”stuff” ”things”}) › ;; => ({::it/foo 1 › ::it/bar ”example” › ::it/baz true} …) ›
  19. 19. THE SPECQL WAY: JOIN › (fetch db ::it/interesting › #{::it/foo ::it/bar ::it/baz › [::it/organization #{::org/id ::org/name}]} › {::it/archived false › ::it/organization_id org-id › ::it/category (op/in categories)}))
  20. 20. THE SPECQL WAY: JOIN › (fetch db ::it/interesting › #{::it/foo ::it/bar ::it/baz › [::it/organization #{::org/id ::org/name}]} › {::it/archived false › ::it/organization_id org-id › ::it/category (op/in categories)})) Nested column definition for a JOINed tables
  21. 21. THE SPECQL WAY: JOIN › ;; => ({::it/foo 1 › ::it/bar ”example” › ::it/baz true › ::it/organization {::org/id 1 › ::org/name ”Acme Inc”} › …)
  22. 22. THE SPECQL WAY: JOIN › ;; => ({::it/foo 1 › ::it/bar ”example” › ::it/baz true › ::it/organization {::org/id 1 › ::org/name ”Acme Inc”} › …) Joined entity is available in a nested map
  23. 23. SPECQL BENEFITS › Clojure data! › The column set is just data and can be easily manipulated • Even given as parameters from the frontend › Where clauses can be easily added and are (mostly) just data as well › Every table and column has a single ns keyword definition • No more ”order” vs ”order-id” vs ”ord” differences in returned query results • Namespaced keys are the way of the future
  24. 24. SPECQL BENEFITS › Clojure data! › The column set is just data and can be easily manipulated • Even given as parameters from the frontend › Where clauses can be easily added and are (mostly) just data as well › Every table and column has a single ns keyword definition • No more ”order” vs ”order-id” vs ”ord” differences in returned query results • Namespaced keys are the way of the future HERE TO STAY
  25. 25. SPECQL BENEFITS › Works with ClojureScript • The spec generation part, put your specs in .cljc files and enjoy the same specs and have your db definitions drive your frontend as well › Works well with ”typed document storage” pattern • A column can be a user defined type or array › Provides generic upsert! • No need to branch code, let the db handle it › Works with database VIEWs
  26. 26. STATUS › The github project page (https://github.com/tatut/specql) still says EXPERIMENTAL • Current 0.7 alpha stage, should be ready this year • We are already using it in production in Harja › No concrete promises before the experimental flag is gone, but the API should only grow • The test suite is comprehensive and should not break
  27. 27. STATUS › 0.7 is coming and adds support for stored procedures • Macro for defining a stored procedure as a function • Better JOIN handling • Currently fetching multiple ”has many” collections at once doesn’t work properly › How to help • Please try it out if you are using PostgreSQL • Report rough edges, I try to provide good error messages
  28. 28. › EXAMPLE
  29. 29. THANK YOU github.com/tatut/specql twitter.com/tatut

Editor's Notes

  • Specql is a new library for using PostgreSQL from a Clojure application. It introspects your database at compile time (with the help of some macro magic) and provides clojure.spec specs for your tables and their columns.
    In addition to creating specs, Specql also retains runtime information so that it can provide generic query and update operations. More on those later.
  • We have found that slightly different queries are painful to maintain.

    Hand written SQL also suffers from the problem of having different keys for different table columns. One query might use :order and another might use :orderid. There is no single truth of what a keyword means.
  • Let's compare by an example. Consider some table called interesting_table that we want to query from.
  • How can we easily vary the set of columns we fetch without having multiple almost identical queries or fetching everything and selecting the wanted keys on the caller.
  • These two problems, in our experience, lead to making similar queries or make the queries you write awkwardly complicated.
    And we haven’t even talked about JOINs yet. What if I have two similar queries, but the other one needs to join a table?
  • Adding columns from a JOINed table further changes how our SQL query looks. Making it difficult to even see that they are conceptually the same.
  • First we define our tables so specql can generate the specs and record information about it.
  • First we define our tables so specql can generate the specs and record information about it.
  • First we define our tables so specql can generate the specs and record information about it.
  • The spec for the table (a keys spec) is generated for the keyword specified here. All column specs will be generated in the same namespace.
  • The column options can be used to do transformations for the columns (like converting an enum to keywords on the fly).
    In this example we are defining a JOIN specification. The organization virtual column is a one to one JOIN to the organization table.

  • Specql provides a function called fetch that can be used to make queries against any previously defined table.
  • The table is simply the keyword we registered the table under in a previous call to define-tables
  • The columns is just a Clojure set of the namespaced keywords of the table columns
  • The where map is interesting. The keys are columns of the table being queried the values can either be values (which are simple equality checks) or operators. The basic SQL operators are provided (equality, less than, greater than, in, like) and there is a protocol for you to define more.

    Multiple maps of where clauses can be combined with op/and and op/or operators.
  • Finally the result is a sequence of maps which contain the namespaced keys.
  • How do we add the JOIN?
  • We simply add a vector to the column set. The first element is the join definition we defined in define-tables and the second element is a set of columns to retrieve from the join table.
  • We simply add a vector to the column set. The first element is the join definition we defined in define-tables and the second element is a set of columns to retrieve from the join table.
  • We simply add a vector to the column set. The first element is the join definition we defined in define-tables and the second element is a set of columns to retrieve from the join table.
  • ClojureScript: speql understands some of the constraints on fields: NOT NULL and string lengths.

    PostgreSQL columns can be arbitrarily complex objects. You can have an array of a user defined type as a column and specql will work great with that.

    Upsert:
    Upsert, which came in pg 9.5, is fully supported.
    Upsert takes the db, the table keyword, a record map to insert and a where clause for the update
    Optionally the unique column set to upsert on can also be specified, and defaults to the primary key
    Views: you can write your complicated queries as views that and spec those like any table.
  • Although it is a very young library, we are already using it in production in Harja, which is a large public sector application for road infastructure maintenance.
    We started adding specql to our new features and it has been helpful.

×