Data access layer and schema definitions

15,635 views

Published on

Published in: Technology
  • Be the first to comment

Data access layer and schema definitions

  1. 1. Data Access Layerand Schema DefinitionsLuciano Resendelresende@shutterfly.comlresende@apache.orghttp://lresende.blogspot.com/
  2. 2. About Luciano Resende •  Staff Software Engineer – Shutterfly, Inc. •  Member – Apache Software Foundation •  Committer: Apache Tuscany, Wink, Nuvem, PhotoArkDecember 8, 2011 MongoSV 2011 2
  3. 3. MongoDB Characteristics •  Scalability •  Simplicity •  Flexible Schema •  Low TCODecember 8, 2011 MongoSV 2011 3
  4. 4. MongoDB initial POCs .NET   Component       firstName   first_name   MongoDB   firstname   Java   Java   Component   Component          Floor  plan  image  from  :  h3p://benchmarkoffices.com/floorplans.htm  December 8, 2011 MongoSV 2011 4
  5. 5. The Platform •  Applications will interact with MongoDB persistence via services •  Enforce a consistent schema across multiple applications and cross-platform Services   •  Data Access Layer •  Data Access Layer that in Data  Access  Layer   conjunction with UDDL abstracts the persistence layer allowing for more flexible and UDDL   clean code •  A cross platform data schema definition (text DSL) •  Simple human (& machine) readable language to describe our domain model entitiesDecember 8, 2011 MongoSV 2011 5
  6. 6. Schema Definition
  7. 7. Schema Definition – Why using Text DSL •  Platform independent data definition. •  (Annotated) POJO (JAXB, JPA, Morphia) has too much noises and is abuse-prone. We want to enforce the patterns. •  XSD is too complicated and it doesn’t fit all for web 2.0 •  We need a simple human (& machine) readable language to describe our domain model to provide/promote: •  Service-friendly DTO, Persistence (RDB and NoSQL), Governance, Reuse, Validation, MVC, Best Practice, Documentation •  We call the DSL as Universal Data Definition Language (UDDL). •  Unify Data Definition across different platform application domainsDecember 8, 2011 MongoSV 2011 7
  8. 8. Our decision: Textual DSL •  Derived from an Apache licensed open source project Sculptor •  http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29 •  Easy to learn, intuitive syntax of the textual DSL, based on the concepts from DDD. •  Textual DSL has a lot of productivity benefits over graphical tools •  Quick development round trip, short feedback loop •  Generation of complete application from a single model, not only fragments that are hard to fit in to the overall design •  Supports JPA (oracle/mysql/postgresql)/MongoDB/JAXB/Spring/SpringMVC •  Great extensibility and customization •  Based on Eclipse Xtext/Xtend/Xpand code generation framework •  Can be used with text editor or any IDE, but DSL editor with error highlight, code completion, and outline is provided for Eclipse users •  Documentation w/ diagrams for domain model •  No runtime magic is built into the tools (generated code can be used without the tooling)December 8, 2011 MongoSV 2011 8
  9. 9. Schema Definition - The idea 4 metadata   POJO   XSD   Protocol   Domain   1.  Persistence   Buffer/ThriT   2.  DTO   Model   JSON   Textual  DSL   3.  Mapping   Schema   (such  as   DDL   4.  IntrospecFon   Sculptor)   5.  ValidaFon   6.  MVC/UI   2 1 CRUD   DTO   DTO   DTO   REST   Root   Locate   3 RDB   REST   EnFty   Resource   Sub-­‐ CRUD   EnFty   Resource   EnFty   REST   Sub-­‐ CRUD   MongoDB   Resource   Generated  arFfacts   Java  code   Code   uddl   generaFon   ConfiguraFon  files   templates   Documents   (UML  diagrams,  HTML  December 8, 2011 docs)   MongoSV 2011 9
  10. 10. Schema Definition - Sample Application AddressBook { basePackage=com.shutterfly.model Persistence  in   Module addressbook { hint="store=mongodb” use…   Entity Contact { dto mappedName="contact" String id key; String userId ! nullable indexed; EnFty   String prefix; String firstName; String middleName; String lastName; Required   String company; Indexes   - @ContactType type; - @Gender gender; List<String> groups; //group names - List<@ContactEmail> emails; Embedded  and   - List<@ContactAddress> addresses; References   @javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) Timestamp created mappedName="updtime"; @javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) Timestamp lastUpdated mappedName="instime"; } BasicType ContactEmail { String label; Boolean isPrimary; String emailAddress ! nullable indexed; Embedded  and   } } References  December 8, 2011 MongoSV 2011 10
  11. 11. Schema Definition – Generated Java Code @Generated("uddl:AddressBook/addressbook/Contact") @XmlRootElement @XmlAccessorType(javax.xml.bind.annotation.XmlAccessType.FIELD) @Entity(value = "contact", noClassnameStored = true) public class Contact implements Serializable, Cloneable { @Id private String id; @XmlElement() private String userId; @XmlElement() private String prefix; @XmlElement() private String firstName; @XmlElement() private String middleName; @XmlElement() private String lastName; @XmlElement() private String company; @XmlElement() private List<String> groups = new ArrayList<String>(); @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "updtime") @Property("updtime") private Date created; @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "instime") @Property("instime") private Date lastUpdated; @XmlElement() private ContactType type; @XmlElement() private Gender gender; @XmlElementWrapper(name = "emails") @XmlElement(name = "email") @Embedded private List<ContactEmail> emails = new ArrayList<ContactEmail>(); @XmlElementWrapper(name = "addresses") @XmlElement(name = "address") @Embedded private List<ContactAddress> addresses = new ArrayList<ContactAddress>(); … }December 8, 2011 MongoSV 2011 11
  12. 12. Schema Definition - DocumentationDecember 8, 2011 MongoSV 2011 12
  13. 13. DSL language development SemanFc   model   Generated  arFfacts     Grammar   Language   Eclipse  Model   Load/save   (DSL)   Workflow  Engine   Java  code     (MWE)  Parse/validate   Eclipse   Xtext   UI   ConfiguraFon   files   Xpand   Documents   Parse  tree   Grammar   model   model   (UML   diagrams,   HTML  docs)   EMF  Ecore  model   Xtend/ Check   EMF  Ecore  model     Other  DSLs   (such  as  XSD,   GPB,  ThriT  or   Extend/ Avro)     transform/ validate  December 8, 2011 MongoSV 2011 13
  14. 14. Schema Definition - Open source tools Sculptor: •  http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29 Eclipse Xtext: •  http://www.eclipse.org/Xtext/December 8, 2011 MongoSV 2011 14
  15. 15. Data Access Layer
  16. 16. Data Access Layer •  Provides an Abstraction layer to Persistence API and storage •  Also facilitating migration from one Persistence storage to another >  Or even Y adaptor pattern to write to multiple Persistence storages •  Promote rapidly adoption of new technologies •  Provides different concrete APIs •  Query based APIs •  Entity based APIs •  Binary APIs (GridFS) •  Cache •  Query Abstraction •  Ability to plug key generators •  Promotes best practices across all applicationsDecember 8, 2011 MongoSV 2011 16
  17. 17. Data Access Layer Services   Data  Access  Layer   Query   Persistence   ORM   Cache   Builder   Driver   UDDL  December 8, 2011 MongoSV 2011 17
  18. 18. Data Access Layer – Query Builder •  Abstract how application developers define queries •  Agnostic of Persistence Storage •  Support the following Operators •  EQUAL, NOTEQUAL, HASTHISONE, HASALLOF, HASANYOF, HASNONEOF, HASTHISELEMENT,GREATERTHAN,GREATERHANOREQ,LESSTHAN,LESST HANOREQ,WHERE,EXISTS,SIZEEQ,MOD,PATTERN •  Support AND and OR compound operators •  Syntax originally based on the Morphia ‘Query’  QueryBuilder  query  =                                  dal.createQueryBuilder().filter(Group.USERID,  QueryOperator.EQUAL,  userId)                                          .filter(Group.ID,  QueryOperator.HASANYOF,  groupNames);      QueryBuilder  query  =                                  dal.createQueryBuilder().orFilter("_id",  QueryOperator.EQUAL,  contactId)                                          .orFilter(Contact.LEGACYID,  QueryOperator.EQUAL,  contactId);  December 8, 2011 MongoSV 2011 18
  19. 19. Data Access Layer - Cache •  Provides out-of-the-box support for cache annotations for Queries and entities @Cacheable(cacheName  =  ”contacts")   public  final  List<Contact>  getContacts(QueryBuilder  query)  throws  ExcepFon  December 8, 2011 MongoSV 2011 19
  20. 20. Data Access Layer - ORM •  Morphia is a lightweight type-safe library for mapping Java objects to/from MongoDB •  Morphia provides easy support for a strongly-typed programming modelDecember 8, 2011 MongoSV 2011 20
  21. 21. Data Access Layer – ORM & Partial Updates •  Scenario : Allow client applications to send partial documents and merge these changes on current document. •  Regular behavior : update({"_id":ObjectId(”…")},   {lastName:"Updated"},  false)   MongoDB   {        "_id"  :  ObjectId ("4ee1b6ac5c31a483c62d2b15"),        "firstName"  :  "Luciano",        "lastName"  :  "Resende",        "email"  :  {    "isPrimary"  :  true,    "emailAddress"  :    “lresende@...”        }   }   See  :  git://gist.github.com/1450628.git  December 8, 2011 MongoSV 2011 21
  22. 22. Data Access Layer – ORM & Partial Updates •  Scenario : Allow client applications to send partial documents and merge these changes on current document. •  Regular behavior : update({"_id":ObjectId(”…")},   {lastName:"Updated"},  false)   MongoDB   {        "_id"  :  ObjectId ("4ee1b6ac5c31a483c62d2b15"),        "firstName"  :  "Luciano",   }   See  :  git://gist.github.com/1450628.git  December 8, 2011 MongoSV 2011 22
  23. 23. Data Access Layer – ORM & Partial Updates •  Supporting Partial Updates •  Provides specific operations for explicitly applying partial updates •  Internally, flatten the update structure into various $set operations MongoDB   {  update({"_id":ObjectId(”…")},          "_id"  :  ObjectId{"$set":  {"lastName":  "Updated","email.isPrimary":   ("4ee1b6ac5c31a483c62d2b15"),        "firstName"  :  "Luciano",  false}},  false)        "lastName"  :  ”Resende",        "email"  :  {    "isPrimary"  :  false,    "emailAddress"  :    “lresende@...”        }   }   See  :  git://gist.github.com/1450628.git   December 8, 2011 MongoSV 2011 23
  24. 24. Data Access Layer – ORM & Partial Updates •  Supporting Partial Updates •  Provides specific operations for explicitly applying partial updates •  Internally, flatten the update structure into various $set operations MongoDB   {  update({"_id":ObjectId(”…")},          "_id"  :  ObjectId{"$set":  {"lastName":  "Updated","email.isPrimary":   ("4ee1b6ac5c31a483c62d2b15"),        "firstName"  :  "Luciano",  false}},  false)        "lastName"  :  ”Updated",        "email"  :  {    "isPrimary"  :  false,    "emailAddress"  :    “lresende@...”        }   }   See  :  git://gist.github.com/1450628.git   December 9, 2011 MongoSV 2011 24
  25. 25. Data Access Layer - Open source tools •  Spring Data - MongoDB: •  http://www.springsource.org/spring-data/mongodb •  Morphia •  http://code.google.com/p/morphia/December 9, 2011 MongoSV 2011 25
  26. 26. Summary
  27. 27. Summary •  Schema Definition •  Cross Platform, Simple human (& machine) readable language to describe our domain model entities •  Data Access Layer •  Data Access Layer that in conjunction with UDDL abstracts the persistence layer allowing for more flexible and clean code •  Promote rapidly adoption of new technologiesDecember 8, 2011 MongoSV 2011 27
  28. 28. Q&A

×