Simpycity and Exceptable


Published on

Aurynn Shaw

Stored Procedures Made Easier

In this talk, we will be covering the Simpycity and Exceptable libraries, a Python-based database abstraction which provides an ORM-equivalent stack around stored procedure development.

We will be covering why stored procedures are an important tool in database application development, how to design business logic abstractions for your database, and how Exceptable aids in providing for cleaner client code.

Finally, we will cover example model code implementing the Simpycity abstractions, and cover how these abstractions isolate the underlying data model without damaging functionality or data access.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Simpycity and Exceptable

  1. 1. Simpycity and Exceptable A Q u e ry - F o c u s s e d D a t a b a s e A b s t r a c t i o n A u ry n n S h a w P o s t g r e S Q L E a s t , 2 0 1 0 Saturday, March 27, 2010 1
  2. 2. What is Simpycity? Query Mapping library Saturday, March 27, 2010 2
  3. 3. Whys and Wherefores Lowest level of our data design The humble table Saturday, March 27, 2010 3
  4. 4. Whys and Wherefores This is our ideal - normalized, good natural keys, good constraints, minimal data duplication. The humble table Best possible relational representation Saturday, March 27, 2010 4
  5. 5. Whys and Wherefores Are tables the best representation to model? The humble table Best possible relational representation Generally, ORMs model tables Saturday, March 27, 2010 5
  6. 6. But wait... Saturday, March 27, 2010 6
  7. 7. But wait... Objects aren’t like relations Saturday, March 27, 2010 7
  8. 8. But wait... Objects encapsulate attributes, properties, methods, all the stuff that constitutes a single entity Objects aren’t like relations A single, coherent concept Saturday, March 27, 2010 8
  9. 9. But wait... Proper normalization, users are a good example. Objects aren’t like relations A single, coherent concept Spanning multiple tables Saturday, March 27, 2010 9
  10. 10. So... Why are we using ORMs to represent individual rows in our tables? Our objects aren’t relations. Saturday, March 27, 2010 10
  11. 11. Data Modesty They are our physical data representation - the lowest abstraction we work with. Physical data Saturday, March 27, 2010 11
  12. 12. Data Modesty Exposing your underlying tables means that you end up being locked to that representation - even if you don’t want to be. Physical data We argue this is an important reason for object abstractions, why Exposure is Commitment is it different for relational abstractions? Saturday, March 27, 2010 12
  13. 13. Data Modesty What we should be doing is aiming for a logical representation of our data, as opposed to matching the Physical data physical representation. We use methods and Exposure is Commitment attributes on our objects to manipulate data, we Tables aren’t the logical interface don’t directly fiddle the private attributes. Saturday, March 27, 2010 13
  14. 14. If we’re not modelling tables... What are we modelling? What should we be modelling? Saturday, March 27, 2010 14
  15. 15. What is Simpycity? Why not queries? Queries are the mechanism by which we interact with the physical data layer, and queries bind multiple tables Query Mapping library together via JOIN, so, are they not a good representation for our business layer? Saturday, March 27, 2010 15
  16. 16. What is Simpycity? The core is arbitrary methods invoking arbitrary queries, seamlessly and deliciously. Query Mapping library Methods invoke Queries Saturday, March 27, 2010 16
  17. 17. How to work with Simpycity systems firstly, setting up the data connection And now, some Code Saturday, March 27, 2010 17
  18. 18. And now, some Code >>> from simpycity.context import Context >>> ctx = Context(dsn=“dbname=‘foo’ username=‘bar’”) a Context is a single point from which all data constructs are derived - the DB connection can be committed and closed from this point. Saturday, March 27, 2010 18
  19. 19. Basic Queries >>> i = ctx.Raw(“““SELECT * ... FROM users.user ... WHERE id = 1”””) ... >>> user = i() >>> user[‘id’] 1 Most absolutely basic usage of Simpycity’s Raw >>> datatype. Saturday, March 27, 2010 19
  20. 20. Parameterized Queries >>> i = ctx.Raw(“““SELECT * ... FROM users.user ... WHERE id = %s”””, ... [‘id’]) ... >>> user = i(1) >>> user[‘id’] Parameterization - Methods suddenly look 1 like actual Python functions! >>> Saturday, March 27, 2010 20
  21. 21. Parameterized Queries >>> i = ctx.Raw(“““SELECT * ... FROM users.user ... WHERE id = %s”””, ... [‘id’]) ... >>> user = i(1) >>> user[‘id’] Simpycity callables even 1 work with standard keyword >>> user = i(id=1) parameterization. >>> user[‘id’] 1 >>> Saturday, March 27, 2010 21
  22. 22. Procedural Logic >>> inst = ctx.Function(“““get_user”””, [‘id’]) >>> user_positional = inst(1) >>> user_positional[‘id’] 1 >>> user_keyword = inst(id=1) >>> user_keyword[‘id’] 1 Functions work exactly the same - Same >>> arguments, syntax, and parameterization semantics. Saturday, March 27, 2010 22
  23. 23. Positional and Keyword >>> i = ctx.Function(“““complex_get”””, [‘id’,‘table’,‘schemaname’]) >>> rs = i(1, ‘some_table’, ‘public’) - OR - >>> rs = i(id=1, table=‘some_table’, schemaname=‘public’); - OR - >>> rs = i(schemaname=‘public’, id=1, table=‘some_table’); A huge advantage of Simpycity is treating the argument chain as positional or keyword arguments - Allowing your APIs to be indistinguishable from normal Python. Saturday, March 27, 2010 23
  24. 24. Upshots Saturday, March 27, 2010 24
  25. 25. Upshots You’re dealing with constructs that act and respond like a Python method should act and respond - It allows for a very consistent interface Running queries is calling a method. Saturday, March 27, 2010 25
  26. 26. Upshots Instead of writing our queries using a half- assed badly- implemented subset of SQL, we write queries directly in SQL, and never have to worry about the query generator writing ridiculous queries. Instead, we write our Running queries is calling a method. own ridiculous queries. ;) Underlying DB is abstracted away Saturday, March 27, 2010 26
  27. 27. Downsides Due to Simpycity’s current architecture, insert/update/delete statements aren’t really supported - it definitely expects to get something *Back* from the DB. Insert/Update/Delete requires a procedure.. Saturday, March 27, 2010 27
  28. 28. Downsides Simpycity’s queries also don’t have a concept of defaults, as yet - all arguments declared by the definition *must* be present in the call. Insert/Update/Delete requires a procedure.. ..All arguments must be accounted for.. Saturday, March 27, 2010 28
  29. 29. This won’t work. >>> inst = ctx.Function(“““complex_get”””, [‘id’,‘table’,‘schemaname’]) >>> item = inst(1, ‘some_table’) Traceback (most recent call last): .. <SNIP> .. Exception: Insufficient arguments: Expected 3, got 2 So this isn’t going to work. Saturday, March 27, 2010 29
  30. 30. Or this. >>> inst = ctx.Function(“““complex_get”””, [‘id’,‘table’,‘schemaname’]) >>> item = inst(id=1, table=‘some_table’) Traceback (most recent call last): .. <SNIP> .. Exception: Insufficient arguments: Expected 3, got 2 Saturday, March 27, 2010 30
  31. 31. Downsides Insert/Update/Delete requires a procedure.. ..All arguments must be accounted for.. ..Big resultsets will be entirely pulled into memory.. Another disadvantage is running a query that has a lot of results will pull them *all* into memory, by default. This is a limitation of the underlying result set representation, for reasons I’ll get into in a moment. Saturday, March 27, 2010 31
  32. 32. Downsides Simpycity doesn’t do *any* actual query generation - the most it will do is the select * from function. None of the more advanced query generation exists here. Insert/Update/Delete requires a procedure.. All arguments must be accounted for Big resultsets will be entirely pulled into memory .. And, you’ll be writing a lot of SQL... Saturday, March 27, 2010 32
  33. 33. This is all it does >>> inst = ctx.Function(“““complex_get”””, [‘id’,‘table’,‘schemaname’]) Becomes SELECT * FROM complex_get(%s, %s, %s) Saturday, March 27, 2010 33
  34. 34. SQL is programming - Upshots code. It’s as important as the rest of your application, and you’re better at writing it than a computer is. Even more, computers cannot extract semantic meaning from your relational design, and cannot Running queries is calling a method. build appropriate representations - only the programmer can. Underlying DB is abstracted away ...but you should be anyway. Saturday, March 27, 2010 34
  35. 35. While the queries are useful on their own, they don’t really provide an easy way to manage data at an application level. For that, Applications! Saturday, March 27, 2010 35
  36. 36. Applications! The reasoning behind that Need Logical Representations is that applications require logical abstractions that make sense from the application perspective - Saturday, March 27, 2010 36
  37. 37. Applications! Need Logical Representations Business models! Divorcing us from the underlying table representations. Instead of modelling tables, we should be modelling *objects*, concepts that are complete unto themselves. Saturday, March 27, 2010 37
  38. 38. Models in Simpycity Don’t model tables Models in Simpycity follow this logical chain - we don’t model the tables. Instead, we work to find what the best representation of a given object is, Saturday, March 27, 2010 38
  39. 39. Models in Simpycity Don’t model tables Aligned towards application requirements the one that most clearly fits what the application itself requires, in terms of internal architecture and logical consistency. Instead of fighting to make relational concepts fit into objects, we should be making our objects accurately represent the concepts we need. Saturday, March 27, 2010 39
  40. 40. Models in Simpycity Don’t model tables Aligned towards application requirements Still allow for Active Record-style manipulation At the same time, the Active Record pattern has a lot of useful concepts, like direct instancing and .save() on dirty objects. For these reasons, the model pattern that Simpycity uses is less Active Record, as we’re not modelling result sets, but more Saturday, March 27, 2010 40
  41. 41. Active Object. Let’s have a look at how Simpycity handles Active Object. Active Object Saturday, March 27, 2010 41
  42. 42. Basic Models >>> base = ctx.Model() - OR - >>> class base(ctx.Model()): ... pass First, we create a base class that all our models derive from - This allows us to add additional functionality on a global level to our application models. For instance, Vertically Challenged creates a base model with authentication tokens baked in. Saturday, March 27, 2010 42
  43. 43. Basic Models >>> base = ctx.Model() >>> class ourUser(base): ... table = [“id”, “username”] Declaring the basic model - our instance, and the table declares what our internal attributes are. Note how we don’t really enforce data types - this just declares what the business object looks like. Saturday, March 27, 2010 43
  44. 44. Basic Models base = ctx.Model() class ourUser(base): table = [“id”, “username”] __load__ = ctx.Function(“get_user”,[‘id’]) __load__ is the basic instancing mechanism in a Simpycity - under the covers, any arguments passed to the instancing of a new object will be passed to this function, and then mapped to the object’s attributes. Saturday, March 27, 2010 44
  45. 45. Basic Models >>> base = ctx.Model() >>> class ourUser(base): ... table = [“id”, “username”] ... __load__ = ctx.Function(“get_user”, [‘id’]) From load, we instance our >>> u = ourUser(1) models just as if we were running the query directly - >>> only now, we have the model attributes available, that we 1 didn’t have before. >>> u.username “Test User” >>> Saturday, March 27, 2010 45
  46. 46. Basic Models For instance, we can update the model with new values. But, they’re just a part of that particular model instance. They’re not persisted out to the database. >>> u = ourUser(1) Fortunately, Simpycity can deal with this, too: >>> 1 >>> u.username “Test User” >>> u.username = “PGEast Demo” >>> u.username “PGEast Demo” >>> Saturday, March 27, 2010 46
  47. 47. Basic Models class ourUser(base): table = [“id”, “username”] __load__ = ctx.Function(“get_user”,[‘id’]) __save__ = ctx.Function(“save_user”, [‘id’,‘username’]) >>> u.username = “PG Demo” Coming back to our model, we’ve added >>> u.username another property - __save__. By declaring __save__ on our model, we “PG Demo” get access to the dynamic .save() method. >>> This method will take the current model’s state, test for any dirty values, and use >>> ctx.commit() the provided callable to save those values. The callable is allowed to be anything you like - it doesn’t have to be a Simpycity function. Saturday, March 27, 2010 47
  48. 48. Basic Models class ourUser(base): table = [“id”, “username”, “password”] __load__ = ctx.Function(“get_user”,[‘id’]) __save__ = ctx.Function(“save_user”, [‘id’,‘username’]) delete = ctx.Function(“delete_user”, [‘id’]) Additionally, models need custom methods, and Simpycity covers that too - assigning a Simpycity function to an attribute will be automatically converted into a bound callable, and will map any arguments named in the table as that value. In this case, we’ve provided a delete method on the model, which Simpycity does not currently support, and anything else you can conceive works in this way. Saturday, March 27, 2010 48
  49. 49. Other Instance Patterns So, at this point, we have a model that can be instanced, manipulated, and deleted, similar to a normal ORM. All we know about the underlying database is there’s a couple of functions, and what their arguments are. Unfortunately, we only have a single instancing mechanism. This works for limited cases, but is really lacking in flexibility. What we need is multiple instancing patterns, a variety of ways to pull data from the database. Simpycity can handle that, too - with a nifty little pattern. Saturday, March 27, 2010 49
  50. 50. Other Instance Patterns class ourUser(base): ... # Remember the Code. by_id = ctx.Function(“get_user”,[‘id’], returns_type=base) by_username = ctx.Function(“get_user”, [‘username’], return_type=ourUser) Here, we’re stepping back to the core Raw and Function mechanisms, and passing a new argument, return_type. Return type takes the provided class and will map the results of the query to that object. If the query returned multiple rows, then you’ll get a normal list of objects, all correctly mapped. This functionality even allows for the easy creation of new generators, such as Saturday, March 27, 2010 50
  51. 51. Newly Minted Objects class ourUser(base): ... # Code was here. new = ctx.Function(“new_user”,[‘username’, ‘password’], return_type=ourUser) with this new method. By declaring the database function to return the new user object, we both insert our new record and get our working model in a single call. And, since it’s already persisted to our database, any methods hanging on the model that require database backing will still work. Any single aspect goes wrong, and it all rolls back, just as it should. Saturday, March 27, 2010 51
  52. 52. How about Search? >>> search = ctx.Function(“user_search”, [‘content’], return_type=ourUser) >>> u = search(“content in their profile”) >>> A search method would even work great in this format - any function that can generate the target object type is a great fit. Saturday, March 27, 2010 52
  53. 53. It’s all about good abstractions So far, everything that we’ve talked about has been discussing building good abstractions at the business logic level of our app - abstracting the underlying able design into views and stored procedures, as necessary. But this does overlook a single critical aspect of building good APIs, and that is Exceptions. Saturday, March 27, 2010 53
  54. 54. Exceptions By default, exceptions in plpgsql are very generic - a single exception type, with a text argument. While functional, this does not provide an abundance of manipulatable error types. Stored Procedures get RAISE EXCEPTION Saturday, March 27, 2010 54
  55. 55. Exceptions When this single exception type reaches the Python layer, psycopg2 will convert it into an InternalError - the string is preserved. This is somewhat useful, but, In stored procedures, RAISE EXCEPTION In Python, this becomes an InternalError Saturday, March 27, 2010 55
  56. 56. Current Exceptions CREATE OR REPLACE FUNCTION except_test() RETURNS VOID AS $$ but, this is what we BEGIN currently have to work with from the DB. RAISE EXCEPTION 'Test!'; Not all that useful, END; right? $$ LANGUAGE PLPGSQL; >>> c.Function("except_test")() Traceback (most recent call last): ... <SNIP> ... psycopg2.InternalError: Test! Saturday, March 27, 2010 56
  57. 57. Exceptions In stored procedures, RAISE EXCEPTION This becomes an InternalError But what was it *really*? Unless you’re deeply adhering to a consistent style in your exception text - not always easily done - you’re going to end up with inconsistencies. Because of this, you’ll end up in a position where you’re parsing error strings, looking for specific errors. Not the best design practise. Saturday, March 27, 2010 57
  58. 58. Exceptable Consistency of exception text The first advantage that Exceptable in Simpycity brings is significantly more consistent exception text - allowing for Exceptable to consistently parse and re- raise exceptions. Saturday, March 27, 2010 58
  59. 59. Exceptable Consistency of exception text Simple DB API Working with Exceptable at the DB level is also incredibly easy: there’s really only 2 queries that need to be remembered. Saturday, March 27, 2010 59
  60. 60. Easy! CREATE OR REPLACE FUNCTION except_test() RETURNS VOID AS $$ SELECT exceptable.raise( Firstly, raising exceptions ‘YourException’, - easily handled, though slightly more verbose than a standard ‘This is the Error Text’); exception. $$ LANGUAGE SQL; Saturday, March 27, 2010 60
  61. 61. and in Python >>> c.Function("except_test")() Traceback (most recent call last): ... <SNIP> ... psycopg2.InternalError: YourException:: This is the Error Text This is what the error string looks like, now! Saturday, March 27, 2010 61
  62. 62. Adding new Exceptions your_database=> SELECT exceptable.register (‘YourException’,‘This is our custom exception!’, NULL); register ---------- t The second aspect is adding new exceptions to the Exceptable tables - without this, attempting to use an exception will throw an error. This is done solely so that, even though it’s not yet implemented, Exceptable can introspect the exceptions table and automatically generate exceptions. It also allows for enforced consistency - Typos happen to even the best of us. Saturday, March 27, 2010 62
  63. 63. Exceptable Consistency of exception text All of this is trying to work to bringing *good* exceptions to the application fabric - An Simple DB API easily parsed exception from the database, and the Exceptable integration in Good Application Exceptions Simpycity means, we can have first-class Python exceptions from our stored procedures. Here’s how: Saturday, March 27, 2010 63
  64. 64. Register the Exception.. >>> from simpycity.exceptions import base >>> class YourException(Exception): ... pass ... >>> base.register(“YourException”, YourException) The syntax for registering a new exception at the Python layer is similar to the DB layer, with the first argument providing the same text value as the exception type in the DB. The second argument is the class definition, and instances will be used any time your exception is raised from the database. Saturday, March 27, 2010 64
  65. 65. And then use it. >>> c.Function(“except_test”)() Traceback (most recent call last): ... SNIP ... YourException: This is the Error text And using it is identical to any other Python exception. Simple and easy. Saturday, March 27, 2010 65
  66. 66. PG Error Codes But wait! 8.4 supports custom error codes! A cool feature of 8.4 and higher is the ability to raise an exception inside a given error code. There’s a large list of error codes, and a sub-feature of that is the ability to raise custom error codes. Saturday, March 27, 2010 66
  67. 67. PG Error Codes But wait! 8.4 supports custom error codes But Exceptable doesn’t! (yet) This is one of those features that Exceptable *will* be supporting, eventually. Instead of using a regex to map exceptions, exceptable will just compare a list of pg error codes - much simpler to implement. Saturday, March 27, 2010 67
  68. 68. See! Custom! CREATE OR REPLACE FUNCTION ex_test() RETURNS void AS $body$ BEGIN RAISE EXCEPTION 'testing' USING ERRCODE= 'EX111'; END; $body$ LANGUAGE PLPGSQL; Saturday, March 27, 2010 68
  69. 69. And the Python >>> try: ... c.Function(“ex_test”)() ... except Exception, e: ... print e.pgcode ... EX111 And our ex111 exception gets properly propagated upwards, and Exceptable would be able to catch it and map it correctly. Saturday, March 27, 2010 69
  70. 70. And, thus A n y Q u e s t i o n s ? Saturday, March 27, 2010 70
  71. 71. Get it! h t t p s : / / p r o j e c t s . c o m m a n d p r o m p t . c o m / p u b l i c / s i m p y c i t y and h t t p s : / / p r o j e c t s . c o m m a n d p r o m p t . c o m / p u b l i c / e x c e p t a b l e Saturday, March 27, 2010 71
  72. 72. And finally, Thank you. Saturday, March 27, 2010 72