Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A3 from sql to orm

916 views

Published on

SQL NoSQL ORM ODM Delphi MongoDB OpenSource mORMot

Published in: Software
  • Be the first to comment

  • Be the first to like this

A3 from sql to orm

  1. 1. Software Architecture & Design  Architecture  From n-Tier to SOA  From SOAP to REST  Technical Debt  Design  From SQL to ORM, NoSQL and ODM  From RAD to MVC  SOLID principles  Domain Driven Design (DDD) Applying patterns on Delphi code using mORMot Software Architecture & Design
  2. 2. From SQL to ORM Arnaud Bouchez
  3. 3. From SQL to ORM  SQL  NoSQL  ORM  ODM  practice From SQL to ORM
  4. 4. SQL  De-Facto standard for data manipulation  Schema-based  Relational-based  ACID by transactions  Time proven and efficient  Almost standard From SQL to ORM
  5. 5. SQL  De-Facto standard for data manipulation  Schema-based  Relational-based From SQL to ORM
  6. 6. Not Only SQL  Two main families of NoSQL databases:  Aggregate-oriented databases  Graph-oriented databases Martin Fowler From SQL to ORM
  7. 7. Not Only SQL  Graph Database  Store data by their relations / associations From SQL to ORM
  8. 8. Not Only SQL  Aggregate  is a collection of data that we interact with as a unit  forms the boundaries for ACID operations in a given model (Domain-Driven Design modeling) From SQL to ORM
  9. 9.  Aggregate Database families  Document-based e.g. MongoDB, CouchDB  Key/Value e.g. Redis  Column family e.g. Cassandra From SQL to ORM Not Only SQL
  10. 10. Not Only SQL  Are designed to scale for the web  Examples: Google farms, Amazon  Should not be used as scaling RDBMS  Can be schema-less  For document-based and key/value or column-driven  Can be BLOB storage From SQL to ORM
  11. 11. Not Only SQL  RBMS stores data per table  JOIN the references to get the aggregate From SQL to ORM
  12. 12. Not Only SQL  NoSQL stores aggregate as documents  Whole data is embedded From SQL to ORM
  13. 13. Not Only SQL  NoSQL stores aggregate as documents  Whole data is embedded (boundaries for ACID behavior) From SQL to ORM
  14. 14. Not Only SQL  Data modeling From SQL to ORM
  15. 15. Not Only SQL  Data modeling SQL  Normalization  Consistency  Transactions  Vertical scaling NoSQL  Denormalization  Duplicated data  Document ACID  Horizontal scaling From SQL to ORM
  16. 16. Not Only SQL  SQL > NoSQL  Ubiquitous SQL  Easy vertical scaling  Data size (avoid duplicates and with no schema)  Data is stored once, therefore consistent  Complex ACID statements  Aggregation functions (depends) From SQL to ORM
  17. 17. Not Only SQL From SQL to ORM
  18. 18. Not Only SQL From SQL to ORM SELECT SUM(price) AS total FROM orders db.orders.aggregate( [ { $group: { _id: null, total: { $sum: "$price" } } } ] )
  19. 19. Not Only SQL  NoSQL > SQL  Uncoupled data: horizontal scaling  Schema-less: cleaner evolution  Straight-To-ODM  Version management (e.g. CouchDB)  Graph storage (e.g. Redis) From SQL to ORM
  20. 20. Object Relational Mapping (ORM)  ORM  gives a set of methods (CRUD)  to ease high-level objects persistence  into an RDBMS  via mapping From SQL to ORM
  21. 21. Object Relational Mapping (ORM)  Since decades  Classes are the root of our OOP model  RDBMS is (the) proven way of storage  Some kind of "glue" is needed to let class properties be saved into one or several tables From SQL to ORM
  22. 22. Object Relational Mapping (ORM)  In fact  Classes are accessed via Delphi/Java/C# code  RDBMS is accessed via SQL  SQL by itself is a full programming language  With diverse flavors (e.g. data types)  Difficult to switch the logic  Error prone, and difficult to maintain From SQL to ORM
  23. 23. Object Relational Mapping (ORM)  Sometimes,  there will be nothing better than a tuned SQL statement  But most of the time,  You need just to perform some basic CRUD operations (Create Retrieve Update Delete) From SQL to ORM object instance SQL RDBMS object instance ORM CRUD operations SQL mapping RDBMS
  24. 24. Object Relational Mapping (ORM) myObject.Value := 10; myContext.Update(myObject); UPDATE OBJTABLE SET … 01010101001110011111 From SQL to ORM
  25. 25. Object Relational Mapping (ORM)  Benefits  Stay at OOP level  Manage SQL flavors  Persistence Ignorance  Optimize SQL  Cache From SQL to ORM  No silver bullet  Hard task  Hidden process  Legacy (tuned) SQL  Performance cost
  26. 26. Object Relational Mapping (ORM)  Benefits  Stay at OOP level  Manage SQL flavors  Persistence Ignorance  Optimize SQL  Cache From SQL to ORM  No silver bullet  Hard task  Hidden process  Legacy (tuned) SQL  Performance cost (or not)
  27. 27. Object Relational Mapping (ORM)  Mapping may be created:  Code-first  Objects are defined, then persisted (dehydrated)  Usually the best for a new project  Model-first  From an (existing) database model  Sometimes difficult to work with legacy structures From SQL to ORM ORM class type data model object instance RDBMS
  28. 28. Object Relational Mapping (ORM)  Mapping may be defined:  by configuration  via code (attributes or fluent interface)  via external files (XML/JSON)  by convention  from object layout  from database layout From SQL to ORM
  29. 29. Object Relational Mapping (ORM)  Objects may be:  Any business class (POJO/POCO/PODO)  Eases integration with existing code  But may pollute the class (attributes)  Inherit from a common class  Get a shared behavior  Tend to enforce DDD’s persistence agnosticity From SQL to ORM
  30. 30. Object Relational Mapping (ORM)  ORM advantages  Compile time naming and types check (strong types and names)  One language to rule all your logic (no mix with SQL nor LINQ syntax)  Database abstraction (the ORM knows all dialects, and can switch)  Able to cache the statements and the values  You still can write hand-tuned SQL if needed From SQL to ORM
  31. 31. Object Relational Mapping (ORM)  Don't think about…  tables with data types (varchar/number...), but objects with high level types  Master/Detail, but logical units (your aggregates)  writing SQL, but writing your business code  "How will I store it?", but "Which data do I need?". From SQL to ORM
  32. 32. Object Document Mapping (ODM)  Aggregate = all data in a given context Such documents do map our objects! From SQL to ORM
  33. 33. Object Document Mapping (ODM)  CRUD operation on Aggregates documents mORMot’s ODM is able to share code with ORM From SQL to ORM
  34. 34. mORMot’s RESTful ORM/ODM From SQL to ORM
  35. 35. mORMot’s RESTful ORM/ODM  mORMot’s client-server RESTful ORM/ODM  code-first or model-first ORM/ODM  ORM for RDBMS: generates SQL  ODM for NoSQL: handles documents and queries  convention (TSQLRecord) over configuration  can be consumed via REST  over HTTP or WebSockets, via JSON transmission  or locally, on the server side, e.g. in SOA services From SQL to ORM
  36. 36. mORMot’s RESTful ORM/ODM From SQL to ORM UI Components DataBase RAD UI Delphi classes code mapping DataBase Handwritten SQL UI ORM MVC binding Database Generated SQL UI 1 Client 1 MVC/MVVM binding Server Secure protocol (REST) UI 2 (web) Client 2 MVC binding ORM Persistence layer Database Generated SQL
  37. 37. mORMot’s RESTful ORM  Defining the objects TSQLRecord  ID: Int64 / TID primary key  A lot of useful methods  Define a data model gathering objects TSQLModel  Defining the REST instance TSQLRestServer TSQLRestClient …  CRUD (and SOA) methods From SQL to ORM
  38. 38. mORMot’s RESTful ORM  Defining the object From SQL to ORM /// some enumeration // - will be written as 'Female' or 'Male' in our UI Grid // - will be stored as its ordinal value, i.e. 0 for sFemale, 1 for sMale // - as you can see, ladies come first, here TSex = (sFemale, sMale); /// table used for the Babies queries TSQLBaby = class(TSQLRecord) private fName: string; fAddress: string; fBirthDate: TDateTime; fSex: TSex; published property Name: string read fName write fName; property Address: string read fAddress write fAddress; property BirthDate: TDateTime read fBirthDate write fBirthDate; property Sex: TSex read fSex write fSex; end;
  39. 39. mORMot’s RESTful ORM  Defining the object From SQL to ORM /// some enumeration // - will be written as 'Female' or 'Male' in our UI Grid // - will be stored as its ordinal value, i.e. 0 for sFemale, 1 for sMale // - as you can see, ladies come first, here TSex = (sFemale, sMale); /// table used for the Babies queries TSQLBaby = class(TSQLRecord) private fName: RawUTF8; fAddress: RawUTF8; fBirthDate: TDateTime; fSex: TSex; published property Name: RawUTF8 read fName write fName; property Address: RawUTF8 read fAddress write fAddress; property BirthDate: TDateTime read fBirthDate write fBirthDate; property Sex: TSex read fSex write fSex; end;
  40. 40. mORMot’s RESTful ORM  Defining the Model and the Server: Model := TSQLModel.Create([TSQLBaby],'rootURI'); ServerDB := TSQLRestServerDB.Create(Model,'data.db'),true); ServerDB.CreateMissingTables; Server := TSQLHttpServer.Create('8080',ServerDB);  Defining the Model and the Client: Model := TSQLModel.Create([TSQLBaby],'rootURI'); Client := TSQLHttpClient.Create('127.0.0.1','8080', Model);  Both will now communicate e.g. over http://server:8080/rootURI/Baby From SQL to ORM
  41. 41. mORMot’s RESTful ORM  Use CRUD methods  TSQLRest.Add()  TSQLRest.Retrieve()  TSQLRest.Update()  TSQLRest.UpdateField()  TSQLRest.Delete()  TSQLRecord.Create*()  and some dedicated methods for BLOB process From SQL to ORM
  42. 42. mORMot’s RESTful ORM From SQL to ORM var Baby: TSQLBaby; ID: integer; begin // create a new record, since Smith, Jr was just born Baby := TSQLBaby.Create; try Baby.Name := 'Smith'; Baby.Address := 'New York City'; Baby.BirthDate := Now; Baby.Sex := sMale; ID := Client.Add(Baby); finally Baby.Free; end; // update record data Baby := TSQLBaby.Create(Client,ID); // retrieve record try assert(Baby.Name='Smith'); Baby.Name := 'Smeeth'; Client.Update(Baby); finally Baby.Free; end; // retrieve record data Baby := TSQLBaby.Create; try Client.Retrieve(ID,Baby); // we may have written: Baby := TSQLBaby.Create(Client,ID); assert(Baby.Name='Smeeth'); finally Baby.Free; end; // delete the created record Client.Delete(TSQLBaby,ID); end;
  43. 43. mORMot’s RESTful ORM  Returning several objects  Only one object instance is allocated and filled  See also function TSQLRest.RetrieveList(): TObjectList  Results are transmitted as a JSON array  Results can be cached at client or server side  Full WHERE clause of the SELECT is at hand From SQL to ORM aMale := TSQLBaby.CreateAndFillPrepare(Client, 'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]); try while aMale.FillOne do DoSomethingWith(aMale); finally aMale.Free; end;
  44. 44. mORMot’s RESTful ORM  JSON per-representation transmission layout  Expanded / standard (AJAX) JSON array of JSON objects [{"ID":1},{"ID":2},{"ID":3},{"ID":4},{"ID":5},{"ID":6},{"ID":7}]  Non expanded / faster (Delphi) JSON object of JSON array of values, first row being idents {"fieldCount":1,"values":["ID",1,2,3,4,5,6,7]} In mORMot, all JSON content is parsed and processed in-place, then directly mapped to UTF-8 structures. From SQL to ORM
  45. 45. mORMot’s RESTful ORM  « One to one » / « One to many » cardinality From SQL to ORM TSQLMyFileInfo = class(TSQLRecord) private FMyFileDate: TDateTime; FMyFileSize: Int64; published property MyFileDate: TDateTime read FMyFileDate write FMyFileDate; property MyFileSize: Int64 read FMyFileSize write FMyFileSize; end; TSQLMyFile = class(TSQLRecord) private FSecondOne: TSQLMyFileInfo; FFirstOne: TSQLMyFileInfo; FMyFileName: RawUTF8; published property MyFileName: RawUTF8 read FMyFileName write FMyFileName; property FirstOne: TSQLMyFileInfo read FFirstOne write FFirstOne; property SecondOne: TSQLMyFileInfo read FSecondOne write FSecondOne; end;
  46. 46. mORMot’s RESTful ORM  Automatic JOINed query  At constructor level  With nested instances memory management  See also CreateAndFillPrepareJoined() From SQL to ORM var MyFile: TSQLMyFile; begin MyFile := TSQLMyFile.CreateJoined(Client,aMyFileID); try // here MyFile.FirstOne and MyFile.SecondOne are true instances // and have already retrieved from the database by the constructor // so you can safely access MyFile.FirstOne.MyFileDate or MyFile.SecondOne.MyFileSize here! finally MyFile.Free; // will release also MyFile.FirstOne and MyFile.SecondOne end; end;
  47. 47. mORMot’s RESTful ORM  « One to one » / « One to many » cardinality From SQL to ORM TSQLMyFileInfo = class(TSQLRecord) private FMyFileDate: TDateTime; FMyFileSize: Int64; published property MyFileDate: TDateTime read FMyFileDate write FMyFileDate; property MyFileSize: Int64 read FMyFileSize write FMyFileSize; end; TSQLMyFileInfoID = type TID; TSQLMyFile = class(TSQLRecord) private FSecondOne: TSQLMyFileInfoID; FFirstOne: TSQLMyFileInfoID; FMyFileName: RawUTF8; published property MyFileName: RawUTF8 read FMyFileName write FMyFileName; property FirstOne: TSQLMyFileInfoID read FFirstOne write FFirstOne; property SecondOne: TSQLMyFileInfoID read FSecondOne write FSecondOne; end;
  48. 48. mORMot’s RESTful ORM  « One to one » / « One to many » cardinality  Properties store true Int64 values, not “fake pointers”  This is the preferred way when working with Aggregates From SQL to ORM TSQLMyFileInfoID = type TID; TSQLMyFile = class(TSQLRecord) private FSecondOne: TSQLMyFileInfoID; FFirstOne: TSQLMyFileInfoID; FMyFileName: RawUTF8; published property MyFileName: RawUTF8 read FMyFileName write FMyFileName; property FirstOne: TSQLMyFileInfoID read FFirstOne write FFirstOne; property SecondOne: TSQLMyFileInfoID read FSecondOne write FSecondOne; end;
  49. 49. mORMot’s RESTful ORM  Server or Client Cache  Disabled by default  May be activated  For a given table  For a given set of IDs  On a given TSQLRest  In conjunction with Server DB cache From SQL to ORM
  50. 50. mORMot’s RESTful ORM  Object Time Machine ServerDB.TrackChanges([TSQLInvoice]);  Every TSQLInvoice modification will be tracked and stored in a separated TSQLRecordHistory table.  i.e. Add / Update / Delete ORM commands  Stored in an optimized JSON/binary format  History depth, and storage details are customizable  Previous TSQLInvoice state could then be retrieved from this TSQLRecordHistory table From SQL to ORM
  51. 51. mORMot’s RESTful ORM  Object Time Machine ServerDB.TrackChanges([TSQLInvoice]); aInvoice := TSQLInvoice.Create; aHist := TSQLRecordHistory.CreateHistory(ServerDB,TSQLInvoice,400); try writeln('History Count: ',aHist.HistoryCount); for i := 0 to aHist.HistoryCount-1 do begin aHist.HistoryGet(i,aEvent,aTimeStamp,aInvoice); writeln('Event: ',ord(aEvent))^); writeln('TimeStamp: ',TTimeLogBits(aTimeStamp).ToText); writeln('Identifier: ',aInvoice.Number); end; ... From SQL to ORM
  52. 52. mORMot’s RESTful ORM  Object Time Machine ServerDB.TrackChanges([TSQLInvoice]); aInvoice := TSQLInvoice.Create; aHist := TSQLRecordHistory.CreateHistory(Client,TSQLInvoice,400); try writeln('History Count: ',aHist.HistoryCount); for i := 0 to aHist.HistoryCount-1 do begin aHist.HistoryGet(i,aEvent,aTimeStamp,aInvoice); writeln('Event: ',ord(aEvent))^); writeln('TimeStamp: ',TTimeLogBits(aTimeStamp).ToText); writeln('Identifier: ',aInvoice.Number); end; ... From SQL to ORM
  53. 53. mORMot’s RESTful ORM  Master/Slave Replication  Hidden monotonic version number will be maintained to monitor Add/Update/Delete ORM commands  Deletions stored in a separated table From SQL to ORM TSQLRecordPeopleVersioned = class(TSQLRecordPeople) protected fFirstName: RawUTF8; fLastName: RawUTF8; fVersion: TRecordVersion; published property FirstName: RawUTF8 read fFirstName write fFirstName; property LastName: RawUTF8 read fLastName write fLastName; property Version: TRecordVersion read fVersion write fVersion; end;
  54. 54. mORMot’s RESTful ORM  Master/Slave Replication MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3'); HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer]); MasterClient := TSQLHttpClientHTTP.Create( '127.0.0.1',HTTP_DEFAULTPORT,MasterModel); SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3'); SlaveServer.RecordVersionSynchronizeSlave( TSQLRecordPeopleVersioned,MasterClient); From SQL to ORM
  55. 55. mORMot’s RESTful ORM  Master/Slave Replication MasterServer := TSQLRestServerDB.Create( MasterModel,'master.db3'); HttpMasterServer := TSQLHttpServer.Create( '8888',[MasterServer]); MasterClient := TSQLHttpClientHTTP.Create( '127.0.0.1',HTTP_DEFAULTPORT,MasterModel); SlaveServer := TSQLRestServerDB.Create( SlaveModel,'slave.db3'); SlaveServer.RecordVersionSynchronizeSlave( TSQLRecordPeopleVersioned,MasterClient); From SQL to ORM Master Slave MasterServer (MasterModel) master.db3 MasterClient (MasterModel) HTTP SlaveServer (SlaveModel) On Demand Replication slave.db3
  56. 56. mORMot’s RESTful ORM  Master/Slave Real-Time Replication  Push ORM modifications over WebSockets From SQL to ORM
  57. 57. mORMot’s RESTful ORM  Master/Slave Real-Time Replication MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3'); HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer],'+',useBidirSocket); HttpMasterServer.WebSocketsEnable(Server,'PrivateAESEncryptionKey'); MasterServer.RecordVersionSynchronizeMasterStart; MasterClient := TSQLHttpClientWebSockets.Create( '127.0.0.1',HTTP_DEFAULTPORT,MasterModel); MasterClient.WebSocketsUpgrade('PrivateAESEncryptionKey'); SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3'); SlaveServer.RecordVersionSynchronizeSlaveStart( TSQLRecordPeopleVersioned,MasterClient); From SQL to ORM
  58. 58. mORMot’s RESTful ORM  Master/Slave Real-Time Replication SlaveServer := TSQLRestServerDB.Create( SlaveModel,'slave.db3'); SlaveServer.RecordVersionSynchronizeSlaveStart( TSQLRecordPeopleVersioned,MasterClient); From SQL to ORM Master Slave MasterServer (MasterModel) master.db3 MasterClient (MasterModel) WebSockets TCP/IP SlaveServer (SlaveModel) Replication slave.db3
  59. 59. mORMot’s RESTful ORM  Master/Slave Replication – Multi Offices Synch From SQL to ORM Main Office Office A Office B Main Server External DB Local Server A HTTP Local Server B HTTP Client 1 Client 2 Client 3 local network Client 1 Client 2 Client 3 Client 4 local network
  60. 60. mORMot’s RESTful ORM  Master/Slave Replication – Multi Offices Synch From SQL to ORM Main Office Office A Office B Main Server Local Data A Reference Read Only Local Data B Reference Read Only Local Data A Business Read/Write Replication Local Data B Business Read Only Local Data A Business Read Only Replication Local Data B Business Read/Write
  61. 61. mORMot’s RESTful ORM  Automated Data Sharding  On production servers, databases may store:  Roaming / user input data with mostly updates resulting in a reasonable growing size of storage  Events which are mostly added resulting in an always increasing database  You may reach out of disk space and/or encounter performances issues  Purge via periodic deletion is possible, but may slow down the process From SQL to ORM
  62. 62. mORMot’s RESTful ORM  Automated Data Sharding  On production servers, you may define a table as “sharded”  A TSQLRecord will be stored in its own set of databases  Each database will contain up to a given number of items (e.g. 500,000 entries)  Only up to a given number of databases will be active  And older databases may be archived/deleted instantly just by packing or deleting the corresponding file From SQL to ORM
  63. 63. mORMot’s RESTful ORM  Automated Data Sharding const SHARD_RANGE = 500000; function TTestMemoryBased.CreateShardDB( maxshard: Integer): TSQLRestServer; begin result := TSQLRestServer.CreateWithOwnModel( [TSQLRecordTest],false,'shardtest'); Check(result.StaticDataAdd(TSQLRestStorageShardDB.Create( TSQLRecordTest,result,SHARD_RANGE,[],'',maxshard))); result.CreateMissingTables; … Will setup sharding for the TSQLRecordTest class using local SQlite3 databases. From SQL to ORM
  64. 64. mORMot’s RESTful ORM  Automated Data Sharding R := TSQLRecordTest.Create; try for i := 1 to 50 do begin R.FillWith(i); Check(db.AddWithBlobs(R)=i); R.CheckWith(self,i); end; finally R.Free; end; Data will be sharded on disk, using Test####.dbs files. By convention, TSQLRecordTest →Test####.dbs From SQL to ORM
  65. 65. mORMot’s RESTful ORM  ORM² for DDD persistence service  ORM for plain Delphi class storage  Create TSQLRecord from any domain class  Fields mapping (name, data)  Aggregate flat / un-normalized orientation (no join) From SQL to ORM
  66. 66. mORMot’s RESTful ORM  ORM² for DDD persistence service  ORM for plain Delphi class storage  Create TSQLRecord from any domain class  Fields mapping (name, data)  Aggregate flat / un-normalized orientation (no join)  Isolate persistence from business logic  Built-in CQRS dual-phase commit service  Object-level tuning from domain logic to storage logic From SQL to ORM
  67. 67. mORMot’s RESTful ORM  Persistence over External RDMS  SynDB optimized data access layer From SQL to ORM SynDB ZDBC ODBC OleDB Oracle SQLite3 DB.pas TDataset NexusDB BDE DBExpress FireDAC AnyDAC UniDAC
  68. 68. mORMot’s RESTful ORM  SynDB classes  By-pass the DB.pas unit avoids TDataSet bottleneck, works with Starter Edition … but can still use it (e.g. FireDAC)  Simple KISS designed API  Less data types, interface-based, no legacy  Statement cache, ORM aware  Array binding (bulk insert / batch mode)  Direct JSON generation  Optional remote access via HTTP From SQL to ORM
  69. 69. mORMot’s RESTful ORM  SynDB integration with ORM From SQL to ORM TSQLRestServerDB.Add TSQLRestServerDB.EngineAdd internal table TSQLRestServerStaticExternal.EngineAdd external table REST TSQLRequest INSERT INTO... SQlite3 engine internal engine SQLite3 file ISQLDBStatement INSERT INTO... External DB client ODBC/ZDBC/OleDB... External DB server
  70. 70. mORMot’s RESTful ORM  SQLite3 Virtual Tables From SQL to ORM
  71. 71. mORMot’s RESTful ORM  SQLite3 Virtual Tables  Mapping of external field names  JOIN several external databases  Very slight performance penalty  Access TObjectList instances  Access NoSQL databases  By-passed if possible  Full SQL-92 engine at hand From SQL to ORM
  72. 72. mORMot’s RESTful ORM  SQLite3 Virtual Tables From SQL to ORM mORMot ORM SQLite3 direct TObjectList direct External DB direct virtual Oracle SQLite3 ODBC OleDB ZDBC direct FireDAC AnyDAC UniDAC BDE DBExpress NexusDB DB.pas TDataSet
  73. 73. mORMot’s RESTful ORM From SQL to ORM
  74. 74. mORMot’s RESTful ORM From SQL to ORM
  75. 75. mORMot’s RESTful ORM  Defining the object From SQL to ORM type TSQLRecordPeopleExt = class(TSQLRecord) private fData: TSQLRawBlob; fFirstName: RawUTF8; fLastName: RawUTF8; fYearOfBirth: integer; fYearOfDeath: word; fLastChange: TModTime; fCreatedAt: TCreateTime; published property FirstName: RawUTF8 index 40 read fFirstName write fFirstName; property LastName: RawUTF8 index 40 read fLastName write fLastName; property Data: TSQLRawBlob read fData write fData; property YearOfBirth: integer read fYearOfBirth write fYearOfBirth; property YearOfDeath: word read fYearOfDeath write fYearOfDeath; property LastChange: TModTime read fLastChange write fLastChange; property CreatedAt: TCreateTime read fCreatedAt write fCreatedAt; end; ID : integer Data : TSQLRawBlob FirstName : RawUTF8 LastName : RawUTF8 YearOfBirth : integer YearOfDeath : word ID : INTEGER Data : BLOB FirstName : NVARCHAR(40) LastName : NVARCHAR(40) YearOfBirth : INTEGER YearOfDeath : INTEGER
  76. 76. mORMot’s RESTful ORM  Defining the Model and the Server: Props := TOleDBMSSQLConnectionProperties.Create( '.SQLEXPRESS','AdventureWorks2008R2','',''); Model := TSQLModel.Create([TSQLRecordPeopleExt],'root'); VirtualTableExternalRegister(Model,TSQLRecordPeopleExt,Props,'Test.People'); ServerDB := TSQLRestServerDB.Create(Model,'application.db'),true); ServerDB.CreateMissingTables; HttpServer := TSQLHttpServer.Create('8080',ServerDB);  Defining the Model and the Client: Model := TSQLModel.Create([TSQLRecordPeopleExt],'root'); Client := TSQLHttpClient.Create('localhost','8080', Model);  Both will now communicate e.g. over http://server:8080/root/PeopleExt From SQL to ORM
  77. 77. mORMot’s RESTful ORM  Refining the mapping Model on the Server: Props := TOleDBMSSQLConnectionProperties.Create( '.SQLEXPRESS','AdventureWorks2008R2','',''); Model := TSQLModel.Create([TSQLRecordPeopleExt],'root'); VirtualTableExternalRegister(Model,TSQLRecordPeopleExt,Props,'Test.People'); Model.Props[TSQLRecordPeopleExt].ExternalDB. MapField('ID','Key'). MapField('YearOfDeath','YOD'); ServerDB := TSQLRestServerDB.Create(Model,'application.db'),true); ServerDB.CreateMissingTables; Server := TSQLHttpServer.Create('8080',ServerDB); From SQL to ORM ID : integer Data : TSQLRawBlob FirstName : RawUTF8 LastName : RawUTF8 YearOfBirth : integer YearOfDeath : word Key : INTEGER Data : BLOB FirstName : NVARCHAR(40) LastName : NVARCHAR(40) YearOfBirth : INTEGER YOD : INTEGER
  78. 78. mORMot’s RESTful ORM/ODM  Data Sharding / Denormalization pattern  Store the whole aggregate as once  Without JOIN  From ORM to ODM  Object Relational Mapping (ORM)  Object Document Mapping (ODM) From SQL to ORM
  79. 79. mORMot’s RESTful ODM  Object Document Mapping  ODM over regular RDBMS  NoSQL storage via TDocVariant property  Stored as JSON, handled as a variant  ODM over dedicated NoSQL engine  TObjectList  MongoDB direct access From SQL to ORM
  80. 80. mORMot’s RESTful ODM  NoSQL storage via TDocVariant property  Document is indexed:  by TSQLRecord.ID: integer  by Name: RawUTF8 From SQL to ORM TSQLRecordData = class(TSQLRecord) private fName: RawUTF8; fData: variant; public published property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE; property Data: variant read fData write fData; end;
  81. 81. mORMot’s RESTful ODM  NoSQL storage via TDocVariant custom type  Data: variant will store any document  As JSON in the RDBMS  Accessed via late-binding in Delphi code From SQL to ORM TSQLRecordData = class(TSQLRecord) private fName: RawUTF8; fData: variant; public published property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE; property Data: variant read fData write fData; end;
  82. 82. mORMot’s RESTful ODM  Data: variant will store a TDocVariant  Schema-less data { name : "Joe", x : 3.3, y : [1,2,3] } { name : "Kate", x : "abc" } { q : 456 }  Real-world evolving data { name : "Joe", age : 30, interests : "football" } { name : "Kate", age : 25 } From SQL to ORM
  83. 83. mORMot’s RESTful ODM From SQL to ORM var aRec: TSQLRecordData; aID: integer; begin // initialization of one record aRec := TSQLRecordData.Create; aRec.Name := 'Joe'; // one unique key aRec.data := _JSONFast('{name:"Joe",age:30}'); // create a TDocVariant // or we can use this overloaded constructor for simple fields aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]); // now we can play with the data, e.g. via late-binding: writeln(aRec.Name); // will write 'Joe' writeln(aRec.Data); // will write '{"name":"Joe","age":30}' (auto-converted to JSON string) aRec.Data.age := aRec.Data.age+1; // one year older aRec.Data.interests := 'football'; // add a property to the schema aID := aClient.Add(aRec); // will store {"name":"Joe","age":31,"interests":"footbal"} aRec.Free; // now we can retrieve the data either via the aID created integer, or via Name='Joe' end;
  84. 84. mORMot’s RESTful ODM From SQL to ORM var aRec: TSQLRecordData; aID: integer; begin // initialization of one record aRec := TSQLRecordData.Create; aRec.Name := 'Joe'; // one unique key aRec.data := _JSONFast('{name:"Joe",age:30}'); // create a TDocVariant // or we can use this overloaded constructor for simple fields aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]); // now we can play with the data, e.g. via late-binding: writeln(aRec.Name); // will write 'Joe' writeln(aRec.Data); // will write '{"name":"Joe","age":30}' (auto-converted to JSON string) aRec.Data.age := aRec.Data.age+1; // one year older aRec.Data.interests := 'football'; // add a property to the schema aID := aClient.Add(aRec); // will store {"name":"Joe","age":31,"interests":"footbal"} aRec.Free; // now we can retrieve the data either via the aID created integer, or via Name='Joe' end;  We just mutated a classical RDBMS into a NoSQL schema-less storage engine!
  85. 85. mORMot’s RESTful ODM  NoSQL Engines  TObjectList - TSQLRestStorageInMemory  Fast in-memory storage  Supports hash indexes on (unique) properties  Binary or JSON persistence as file  Could embed some unpersisted live process  MongoDB  SynMongoDB.pas for direct access  mORMotMongoDB.pas for ODM integration From SQL to ORM
  86. 86. mORMot’s RESTful ODM  SynMongoDB.pas  BSON types, including BLOB, ObjectID, date/time…  TBSONVariant with late-binding  (Extended) JSON as input or output  Document-level access  Bulk document insertion  Latest 3.2 compatibility (WiredTiger, scram-sha1)  Could be used standalone, without the ODM From SQL to ORM
  87. 87. mORMot’s RESTful ODM  mORMotMongoDB.pas  Full integration with mORMot’s REST  Share the same logic code with RDBMS  Just one line to change on the server side to switch from an ORM to an ODM  Same exact RESTful business code From SQL to ORM
  88. 88. mORMot’s RESTful ODM uses SynMongoDB, mORMotMongoDB; Model := TSQLModel.Create([TSQLORM]); ServerDB := TSQLRestServerDB.Create(Model,nil,':memory:'); MongoClient := TMongoClient.Create('localhost',27017); MongoDatabase := MongoClient.Database[DB_NAME]; StaticMongoDBRegister(TSQLORM,ServerDB,MongoDatabase); ServerDB.CreateMissingTables; … usual ORM code From SQL to ORM
  89. 89. mORMot’s RESTful ODM  mORMotMongoDB.pas  On-the-fly REST GET query translation  SQL WHERE clause translated to MongoDB query pipelines  Including aggregate functions, limits, offset aClient.OneFieldValue(TSQLOrders,'sum(price)','',[],aTotal); SELECT SUM(price) AS total FROM orders db.orders.aggregate([{$group:{_id: null,total:{$sum:"$price"}}}])  Returned BSON is used to fill the TSQLRecord From SQL to ORM
  90. 90. mORMot’s RESTful ODM  mORMotMongoDB.pas  On-the-fly REST GET query translation  SQL WHERE clause translated to MongoDB query pipelines  Including aggregate functions, limits, offset  Returned BSON is used to fill the TSQLRecord  BATCH support  As bulk document insertion From SQL to ORM
  91. 91. mORMot’s RESTful ORM/ODM  BATCH process From SQL to ORM // start the BATCH sequence Check(ClientDist.BatchStart(TSQLRecordPeople,5000)); // delete some elements for i := 0 to n-1 do Check(ClientDist.BatchDelete(IntArray[i])=i); // update some elements nupd := 0; for i := 0 to aStatic.Count-1 do if i and 7<>0 then begin // not yet deleted in BATCH mode Check(ClientDist.Retrieve(aStatic.ID[i],V)); V.YearOfBirth := 1800+nupd; Check(ClientDist.BatchUpdate(V)=nupd+n); inc(nupd); end; // add some elements V.LastName := 'New'; for i := 0 to 1000 do begin V.FirstName := RandomUTF8(10); V.YearOfBirth := i+1000; Check(ClientDist.BatchAdd(V,true)=n+nupd+i); end; // send the BATCH sequences to the server Check(ClientDist.BatchSend(Results)=200); ORM CRUD operation ORM HTTP Client In-process no latency ORM HTTP Server Internet 100 ms latency ORM database core In-process no latency
  92. 92. mORMot’s RESTful ORM/ODM  BATCH process  Truly ACID, even with several RDBMS backends  Automatic transaction handling  Avoids slow Client-Server roundtrips  Unit-Of-Work pattern, even on Server side From SQL to ORM
  93. 93. mORMot’s RESTful ORM/ODM  BATCH process  Optimized SQL  Parameter Array Binding  e.g. Oracle, ZDBC, FireDAC  Multiple INSERT statement  Depending on the database dialect  Re-use of prepared statements  NoSQL (MongoDB) bulk insertion  Sent as array of BSON documents  With ID pre-computation on client side From SQL to ORM
  94. 94. mORMot’s RESTful ORM/ODM  Asynchronous BATCH process  Add pending writes into a list using: TSQLRest.AsynchBatchAdd TSQLRest.AsynchBatchUpdate TSQLRest.AsynchBatchDelete  A background thread will perform the BATCH  when the internal list reaches a limit or after some ms period From SQL to ORM
  95. 95. mORMot’s RESTful ORM/ODM  Asynchronous BATCH process  Add pending writes into a list using: TSQLRest.AsynchBatchAdd TSQLRest.AsynchBatchUpdate TSQLRest.AsynchBatchDelete  A background thread will perform the BATCH  when the internal list reaches a limit or after some ms period  Perfect for transparent event store  May be associated with local Automated Data Sharding or a remote MongoDB Big-Data store From SQL to ORM
  96. 96. mORMot’s RESTful ORM/ODM  Security  Rooted on framework’s authentication(s)  Secured by SHA-256 challenge to avoid MIM / replay  Alternatives: basic, digest, SSPI/Kerberos, custom class  Users belong to Groups  Per-table CRUD access right for each group  To be used as firewall, above application-level rights  Optional JSON compression and encryption  WebSockets bi-directional binary transmission e.g. From SQL to ORM
  97. 97. mORMot’s RESTful ORM/ODM  RESTful Client-Server  In-process  Stand-alone client, fast server-side access  Named pipes or GDI messages  Stand-alone client, fast server-side access  HTTP/1.1 via kernel-mode http.sys API  Part of the OS since Windows XP SP2, used by IIS and WCF  Kernel-mode execution, IOCP driven  System-wide URI registration: share root and port  Socket-based server is also available, e.g. under Linux  Optional query/answer REST emulation over WebSockets From SQL to ORM
  98. 98. mORMot’s RESTful ORM/ODM  Cross-platform Clients  Generated client code using Mustache templates  Delphi: Windows, MacOS, Android, iOS  FPC: Windows, MacOS, Android, iOS, Linux, …  (Cross)Kylix: Linux  SmartMobileStudio, EWB: Ajax / HTML5  Featuring almost all framework abilities  JSON, security, TSQLRest, TSQLRestBatch From SQL to ORM
  99. 99. From SQL to ORM

×