Agile NoSQL with XRX                     Dan McCreary                     President   M                 Dan McCreary & Ass...
Session Description    What if all members of your software development team from    Project Managers, Business Analysts, ...
During this session viewers will:    •   Understand the challenges associated with traditional four-        translation we...
Executive Summary    • Schema-free, zero translation NoSQL      systems have the ability to have a big      impact on over...
Background for Dan McCreary           •   Circuit Designer at Bell Labs           •   UNIX/Supercomputers           •   Ne...
Origins: The Humble Data DictionaryM                                                              6    D            Copyri...
Electronic Certificate of Real Estate                                                         Summer 2006                 ...
250 Data Elements                                                      XForms                                             ...
Four Translations                          T1                        T2                          T4                       ...
Kurts Suggestion           Use a        A Native XML         Database!                                                    ...
Zero Translation                XForms             Web Browser                                    XML database        •   ...
Key Question: Impact on Agility                    • What impact do zero translation                      NoSQL           ...
No-Shredding!                          My Form                           Data    • Relational databases take a single hier...
Is Shredding Really Necessary?                         • Every time you take                           hierarchical data a...
Many Processes Today Are Driven By…     The constraints of yesterday…    Challenge:       Ask ourselves the question…     ...
"Schema Free"           • Systems that automatically determine             how to index data as the data is loaded        ...
Monoculture and Mono-architectureM                                                            Image Source: Wikipedia Mono...
Storage Architectural Patterns              gTables                             Trees    Triples    Ti l                  ...
Finding the Right Match        Schema-Free                                 Standards Compliant                            ...
Architectural Summary          Four Translation                                Zero Translation              T            ...
Origins: The XML Data DictionaryM    D                                                     21               Copyright 2010...
Electronic Certificate of Real Estate                                                       Summer 2006                   ...
250 Data Elements                                                    XForms                                               ...
The NO-SQL Universe    Key-Value Stores                           Document Stores                                         ...
Three Core Processes    1. Add new data to the database        1. Use XML example but JSON and other           formats cou...
It is Easy to Import Data  SQL                                          XQuery  1. Analyze data for all parent child      ...
The XML File system          • XML File system – a way of            storing information in XML that            can be qui...
Functional Programming                       y = f(x)    •   Computer programs are like mathematical functions    •   Deve...
Its Easy to Query XML DataSELECT COL1, Col2                        for $r in doc(‘t.xml’)//rowFROM TABLE                  ...
It is Easy to Create A Web Service         The Java/JDBC/SQL Way                               The XRX Way                ...
Insert/Select/Publish Comparison                                                          SQL    SQL                      ...
Steps for Adding a New Element                       • Add the element to the "new                         instance       ...
Six Translation – Web Service                                                     Web Service                             ...
XML Stored in XForms Model           Browser                                        Database        model                 ...
XRX Core Process               Browser                                          Database        model                     ...
Code Table Services                Client                                                       Server        model       ...
XRX Dynamic Forms Generation                                                             Application Server               ...
Model Driven                                  • XForms enables the                                    developer to reuse b...
View and Model are Trees            Model           • The view is a tree of a presentation                              da...
Models and View Are Linked with "Bind"                                                    HTML                            ...
Just “Do The Right Thing”                                                      HTML                                     he...
Example of Automatic UI Generation    • All true/false data types      (xs:boolean) automatically      become a checkbox  ...
Structure of a XForms File                                • XForms tags are just XML        Namespaces                    ...
REST    • REpresentation State Transfer    • Create applications based on well      designed URLs    • Take advantage of w...
Five RESTFull Friends    1. In-resident memory cache in your       browser    2. You local hard drive cache    3. Your loc...
Shallow REST vs. Deep REST    • You can start taking advantage of ReST      buy just doing      b j t d i well th ll thoug...
Benefits of REST    •   Provides improved response time    •   Reduced server load    •   Improves server scalability     ...
Sample XFormsM    D                                              48        Copyright 2011 Dan McCreary & Associates
Requirements Editor      Code      Table    Selection      ListsRepeatingR    tiElementsM    D                            ...
Drag-and-Drop XForms Builders                             • Several options for                               drag-and-dro...
Application ModularityM    D                                                 51           Copyright 2010 Dan McCreary & As...
Financial InstitutionM    D                                                52          Copyright 2011 Dan McCreary & Assoc...
Federal IntegratorM                                                     53    D   Copyright 2011 Kelly-McCreary & Associates
Minnesota Historical SocietyM    D                                                    54              Copyright 2010 Dan M...
XRX Apps Demo Site                                       •   Demo site for                                           danmc...
Structured Retrieval is Better                            Introduction to                              Information Retriev...
Table 10.1 - Revised                                                    unstructured          structured                  ...
Retain Document Structure in Search                 "Bag of Words"                         "Retained Structure"           ...
Empower the Non Programmer!          Before XRX                                      After XRX                            ...
Using the Right Architecture        Start                                                      Finish                Find ...
Six "S"s of XRX Agility    1. Semantics – build around shard metadata       registry services          i t       i    2. S...
XRX Resources    •   XRX Wikipedia Page        – http://en.wikipedia.org/wiki/XRX(web_application_architecture)           ...
References                                XForms                                XQuery                                XQ  ...
Thank You!    Dan McCreary, President    Kelly-McCreary & Associates    dan@danmccreary.com    (952) 931-9198             ...
Upcoming SlideShare
Loading in …5
×

Agile NoSQL With XRX

3,449 views
3,248 views

Published on

What if all members of your software development team from Project Managers, Business Analysts, Testing and documentation members could create and modify web applications and web services? With traditional SQL solutions this was difficult because of the need to convert web pages to objects, objects to tables as well as the reverse functions. But now with native XML databases and drag-and-drop forms builders, data can flow from the XML model of a web form to the database and back again without translation. This radically simpler process combined with standardized query languages makes it easier for non-programmers to build and maintain their own applications and web services.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,449
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Agile NoSQL With XRX

  1. 1. Agile NoSQL with XRX Dan McCreary President M Dan McCreary & Associates dan@danmccreary.com D (952) 931 9198 931-9198Metadata Solutions
  2. 2. Session Description What if all members of your software development team from Project Managers, Business Analysts, Testing and Documentation Managers Analysts members could create and modify web applications and web services? With traditional SQL solutions this was difficult because of th need t convert web pages t objects, objects t t bl as f the d to t b to bj t bj t to tables well as the reverse functions. But now with native XML databases and drag-and-drop forms builders, data can flow from the XML model of a web form to the database and back again without translation. This radically simpler process combined with standardized query languages makes it easier for non- non programmers to build and maintain their own applications and web services.M D 2 of Copyright 2011 Dan McCreary & Associates N
  3. 3. During this session viewers will: • Understand the challenges associated with traditional four- translation web/object/RDBMS systems • Understand the elegant simplicity of zero-translation systems • Understand the role of web standards in the XRX environment • See how each component of XRX plays key roles in the application • See how organizations have leveraged XRX for agile web development • See variations of XRX architectures • Create and objective p j process to evaluate the benefits of XRX • Get resources for building XRX pilot projectsM D 3 Copyright 2011 Dan McCreary & Associates
  4. 4. Executive Summary • Schema-free, zero translation NoSQL systems have the ability to have a big impact on overall system agility p y g y • XRX systems dramatically increase overall agility and can also empower non-progammers to build and maintain NoSQL systemsM D 4 Copyright 2011 Dan McCreary & Associates
  5. 5. Background for Dan McCreary • Circuit Designer at Bell Labs • UNIX/Supercomputers • NeXT Computer (Steve Jobs) • Owner of 75-person software consulting firm with a focus on Object-Oriented systems and Object Oriented Object-relational mapping frameworks • US Federal data integration (National Information Exchange Model NIEM.gov) • NativeXML/XQuery for metadata management since 2006 • Advocate of web standards, OpenSource, NoSQL d N SQL and XRX systems t • W3C invited expert (forms working group) • Likes functional programmingM D 5 Copyright Kelly-McCreary & Associates, LLC
  6. 6. Origins: The Humble Data DictionaryM 6 D Copyright 2011 Kelly-McCreary & Associates
  7. 7. Electronic Certificate of Real Estate Summer 2006 1 Document = 44 SQL insertsM D 7 Copyright 2011 Kelly-McCreary & Associates
  8. 8. 250 Data Elements XForms MockupM D 8 Copyright 2011 Kelly-McCreary & Associates
  9. 9. Four Translations T1 T2 T4 T3 Relational Web Browser Object Middle Database Tier • T1 – HTML into Java Objects • T2 – Java Objects into SQL Tables • T3 – Tables into Objects • T4 – Objects into HTMLM 9 D Copyright 2011 Kelly-McCreary & Associates
  10. 10. Kurts Suggestion Use a A Native XML Database! Web Form Save Web Browser Kurt Cagle store($collection, $file-name, $data)M eXist D 10 Copyright 2011 Kelly-McCreary & Associates
  11. 11. Zero Translation XForms Web Browser XML database • XML lives in the web browser (XForms) • REST interfaces • XML in the database (Native XML, XQuery) ( , y) • XRX Web Application Architecture • No translation!M D Copyright 2010 Dan McCreary & Associates 11
  12. 12. Key Question: Impact on Agility • What impact do zero translation NoSQL N SQL systems h t have on system agility? • Agility: the ability to quickly react to changing business requirements at any stage of the software development lifecycle • Question: Big impact or little impact?M • Answer: Big impact D 12 Copyright 2011 Kelly-McCreary & Associates
  13. 13. No-Shredding! My Form Data • Relational databases take a single hierarchical document and shred it into many pieces so it will fit in tabular structures • Many document-oriented NoSQL databases prevent this shreddinggM D 13 Copyright 2008 Dan McCreary & Associates
  14. 14. Is Shredding Really Necessary? • Every time you take hierarchical data and put it into a traditional database you have to put repeating groups in separate tables and use SQL “joins” to joins reassemble the dataM D 14 Copyright 2008 Dan McCreary & Associates
  15. 15. Many Processes Today Are Driven By… The constraints of yesterday… Challenge: Ask ourselves the question… Do D our current method of solving problems with t b l d t t th d f l i bl ith tabular data… Reflect the storage of the 1950s… Or our actual business requirements? What structures best solve the actual business problem? pM D 15 Copyright 2008 Dan McCreary & Associates
  16. 16. "Schema Free" • Systems that automatically determine how to index data as the data is loaded into the database • No a priori knowledge of data structure • N need f up-front logical d t modeling No d for f t l i l data d li – …but some modeling is still critical • Adding new data elements or changing g g g data elements is not disruptive • Searching millions of records still has sub- second response timeM D 16 Copyright 2010 Dan McCreary & Associates
  17. 17. Monoculture and Mono-architectureM Image Source: Wikipedia Monoculture 17 D Copyright 2011 Dan McCreary & Associates
  18. 18. Storage Architectural Patterns gTables Trees Triples Ti l StarsM D 18 Copyright 2010 Dan McCreary & Associates
  19. 19. Finding the Right Match Schema-Free Standards Compliant Mature Query LanguageM Use CMUs Architectural Tradeoff and Modeling (ATAM) Process D 19 Copyright 2010 Dan McCreary & Associates
  20. 20. Architectural Summary Four Translation Zero Translation T T T T web browser XML databaseweb browser database • HTML web pages eb • XForms Client • Object middle tier • Native XML Database • RDBMS database Which system more agile and by how much?M How can this help us manage enterprise metadata? D 20 Copyright 2010 Dan McCreary & Associates
  21. 21. Origins: The XML Data DictionaryM D 21 Copyright 2010 Dan McCreary & Associates
  22. 22. Electronic Certificate of Real Estate Summer 2006 1 Document = 44 SQL insertsM D 22 Copyright 2010 Dan McCreary & Associates
  23. 23. 250 Data Elements XForms MockupM D 23 Copyright 2010 Dan McCreary & Associates
  24. 24. The NO-SQL Universe Key-Value Stores Document Stores XML Graph Stores Object StoresM 24 D Copyright 2010 Dan McCreary & Associates
  25. 25. Three Core Processes 1. Add new data to the database 1. Use XML example but JSON and other formats could also be used 2. Query the d 2 Q h data 3. Create an XML web service • Analyze the effort for each step • Compare SQL and NoSQL XRX systems NoSQL-XRX • Analyze impact on participation of non-programmers • Rate the relative agility of each systemM D 25 Copyright 2010 Dan McCreary & Associates
  26. 26. It is Easy to Import Data SQL XQuery 1. Analyze data for all parent child 1. Drag XML files into folder relationships and repeating g p p p g groups 2. Design logical and physical ER diagrams 3. For each table create a Data Definition File using a data definition language (DDL) 4. Create indexes using DDL 5. Create one table for each set of repeating set of data 6. Run DDL on database creating tables using the appropriate data types 7. Create indexes 8. Create Insert statements 9. Create separate insert statements for each repeating group 10. Run Insert statements on primary structures in database 11. Use primary keys of the first data inserts as foreign keys of dependantM data structures D 26
  27. 27. The XML File system • XML File system – a way of storing information in XML that can be quickly searched • You can drag-and-drop almost any fil onto thi fil system files t this file t • You access it by using a WebDAV connector • Microsoft Windows “My Network Places” functionM D 27
  28. 28. Functional Programming y = f(x) • Computer programs are like mathematical functions • Developers do not manipulate states and variables (things that change value), but focus entirely on constants and functions g ) y (things that never change) and transformations of data • Makes it very easy to build modular programs • Software written in functional programming languages tend to be very concise and easy to port to highly parallel systemsM http://en.wikibooks.org/wiki/Computer_programming/Functional_programming D 28 Copyright 2011 Kelly-McCreary & Associates
  29. 29. Its Easy to Query XML DataSELECT COL1, Col2 for $r in doc(‘t.xml’)//rowFROM TABLE where col1=1 col1 1WHERE COL1=1 return $r/col1, $r/col2 <root> <row> Col1 Col2 <col1>1</col1><col2>A</col2> </row> 1 A <row> <col1>1</col1><col2>B</col2> 1 B </row> <row> 1 C <col1>1</col1><col2>C</col2> </row> 1 D <row> <col1>1</col1> <col2>D</col2>M </row> </root> D 29
  30. 30. It is Easy to Create A Web Service The Java/JDBC/SQL Way The XRX Way 1. All XQuerys are web services 1. 1 Learn Java or find a Java Developer 2. Install TomCat Web Server 3. Install Java AXIS Web Server 4. Write a JDBC program that sends SQL queries to a database 5. Get the results back in Java Result Object structures 6. Go through the Java Results Structues and use print statements to wrap XML tags aro nd the strings rap around in the result objects 7. Rename your class files to .jws files 8. Add the .jws files to the TomCat deploy folders 9. The WSDL files will automatically be generatedM D 30
  31. 31. Insert/Select/Publish Comparison SQL SQL Java logical Tomcat data AXIS modeling JDBC SQL XQuery XQuery XQuery Insert Query Web Service NoSQLM D Total Effort 31
  32. 32. Steps for Adding a New Element • Add the element to the "new instance instance" a form uses and set Takes about one minute the default value there • Add the UI control • Optionally – run an XQuery update to update existing data Takes about 10 minutes or add an insert to the "enricher" when you get an y g instanceM D 32 Copyright 2011 Kell-McCreary & Associates
  33. 33. Six Translation – Web Service Web Service T5 T6 T1 T2 T3 T4 Relational Web Browser Object Middle Database Tier • T1 – HTML into Java Objects • T2 – Java Objects into SQL Tables • T3 – Tables into Objects • T4 – Objects into HTML • T5 – Objects to XMLM • T6 – XML to Objects D Copyright 2011 Dan McCreary & Associates 33
  34. 34. XML Stored in XForms Model Browser Database model save update viewM D Copyright 2011 Kelly-McCreary & Associates 34
  35. 35. XRX Core Process Browser Database model save/edit update viewM 35 D Copyright 2011 Kelly-McCreary & Associates
  36. 36. Code Table Services Client Server model Form Data Code Table Service all-codes.xq view Code C d TablesM Code tables are separated from form instance data D 36 Copyright 2011 Kelly McCreary & Associates
  37. 37. XRX Dynamic Forms Generation Application Server Client Application XForms Model Session Form Data Form Data User Team Document Collection Status Code Tables Role Group Views DataElement Registry Binding Rules Code Table Services Required Context filters Read-only Suggest Services Data Types Business Rules Editor Calculations Calculations Inference Submissions Constraints XForms View XF Vi XML Schema Registry Static Controls Subschema Service Dynamic Controls Constraint Schemas Semantic SchemasM Design Time D Run Time
  38. 38. Model Driven • XForms enables the developer to reuse business XForms rules encapsulated in XML Application Schemas (xsd) and XML Transforms (XQuery Typeswitch) • XForms reduces duplication Meta XML Data and ensures that a change in Schema Registry the underlying business logic does not require rewriting inM another language D Copyright 2011 Kelly-McCreary & Associates 38
  39. 39. View and Model are Trees Model • The view is a tree of a presentation data element • Models are comprised of one or more trees • XForms supplies the control layer Control (Bind) that moves data elements to and from the model View (Presentation) • Users don’t have to worry about moving things to and from the g g screenM D 39 Copyright 2011 Kelly McCreary & Associates
  40. 40. Models and View Are Linked with "Bind" HTML head h d xf:model body Person P form f Name fieldset label label first last input <bind> input • Both the model and the views are trees ofM data l d t elements t D 40 Copyright 2011 Dan McCreary & Associates
  41. 41. Just “Do The Right Thing” HTML head body xf:model Person form PersonCurrentOnTaxes type="xs:boolean" fieldset label PersonBirthDate type="xs:date" label input input <bind> • Data types from the model just do the right thing • Boolean variables become checkboxesM • Dates have d t selectors D t h date l t 41 D Copyright 2008 Dan McCreary & Associates
  42. 42. Example of Automatic UI Generation • All true/false data types (xs:boolean) automatically become a checkbox • All dates (xs:date) have a date selector to the right of the date field • All codes can be selectedM from lists 42 D Copyright 2008 Dan McCreary & Associates
  43. 43. Structure of a XForms File • XForms tags are just XML Namespaces tags imbedded in a CSS Imports (View) standard XHTML file with Model a different namespace Constraints (Bindings) ( g ) • Most HTML form tags are g UI (View) exactly the same but some attributes have been Submit Controls promoted to be full MyForm.xhtml y elementsM 43 D Copyright 2008 Dan McCreary & Associates
  44. 44. REST • REpresentation State Transfer • Create applications based on well designed URLs • Take advantage of web caching • Migrate toward Resource Oriented Resource-Oriented Computing (ROC) • REST evangelists: RESTifariansM D Copyright 2011 Kelly-McCreary & Associates 44
  45. 45. Five RESTFull Friends 1. In-resident memory cache in your browser 2. You local hard drive cache 3. Your local enterprise cache 4. 4 The cache on the web server farm 5. The cache on the database Please make sure to check with your RESTfull friends BEFORE you bother the database.M D 45 Copyright 2011 Kelly-McCreary & Associates
  46. 46. Shallow REST vs. Deep REST • You can start taking advantage of ReST buy just doing b j t d i well th ll thought-out URL ht t design • To take advantage of deep ReST you must consider the subtleties of the HTTP protocol – GET vs POST vs PUT – DELETEM D 46 Copyright 2008 Dan McCreary & Associates
  47. 47. Benefits of REST • Provides improved response time • Reduced server load • Improves server scalability p y • Requires less client-side software • Depends less on vendor dependencies • Promotes discovery • Provides better long-term compatibility • y Better and evolvabilityM D 47 Copyright 2008 Dan McCreary & Associates
  48. 48. Sample XFormsM D 48 Copyright 2011 Dan McCreary & Associates
  49. 49. Requirements Editor Code Table Selection ListsRepeatingR tiElementsM D 49
  50. 50. Drag-and-Drop XForms Builders • Several options for drag-and-drop f d dd forms builder • Open Source – Orbeon – BetterFORM • Commercial – IBM FormsM D 50 Copyright 2011 Kelly-McCreary & Associates
  51. 51. Application ModularityM D 51 Copyright 2010 Dan McCreary & Associates
  52. 52. Financial InstitutionM D 52 Copyright 2011 Dan McCreary & Associates
  53. 53. Federal IntegratorM 53 D Copyright 2011 Kelly-McCreary & Associates
  54. 54. Minnesota Historical SocietyM D 54 Copyright 2010 Dan McCreary & Associates
  55. 55. XRX Apps Demo Site • Demo site for danmccreary.com • Some programs written under an hour with student data • Several utility programs that start with template and add transformations to other formats • Focus on metadata managementM D 55 Copyright 2011 Kelly-McCreary & Associates
  56. 56. Structured Retrieval is Better Introduction to Information Retrieval by Christopher D. Manning, y p g Prabhakar Raghavan and Hinrich Schütze Cambridge University C b id U i i Press, 2008M http://nlp.stanford.edu/IR-book/information-retrieval-book.html http://nlp stanford edu/IR-book/information-retrieval-book html D 56
  57. 57. Table 10.1 - Revised unstructured structured RDB search retrieval retrieval unstructured trees with text at objects records documents leaves vector space & model relational model XML hierarchy others trees with node- main data table inverted index ids for structure t t document ids queries SQL free text queries XQuery fulltext XML - Table 10.1 and structured information retrieval. SQLRDB (relational database) search, unstructured information retrievalM D 57
  58. 58. Retain Document Structure in Search "Bag of Words" "Retained Structure" keywords doc-id keywords love keywords keywords hate keywords new new keywords fear • All keywords in a single container • Keywords associated with each • Only count frequencies are stored sub-document component with each word • Assign higher weight for titles andM names D • Set by non-programmer 58
  59. 59. Empower the Non Programmer! Before XRX After XRX SUPER PM, BA! , Sorry, we have no idea Let me just search our XRX system… what code 47 means. Ill have your answer in 150 milliseconds.M D 59 Copyright 2011 Kelly-McCreary & Associates
  60. 60. Using the Right Architecture Start Finish Find ways to remove barriers to empowering the non programmers on your team team.M D 60 Copyright 2011 Kelly-McCreary & Associates
  61. 61. Six "S"s of XRX Agility 1. Semantics – build around shard metadata registry services i t i 2. Search – structured search 3. Standards – CSS, XML, XPath, XQuery, XForms, XML Schemas 4. Services – all XQ i are REST services 4 S i ll XQueries i 5. Solutions - that are quickly customized 6. Super – Empower the non-programmersM 61 D Copyright 2010 Kelly-McCreary & Associates
  62. 62. XRX Resources • XRX Wikipedia Page – http://en.wikipedia.org/wiki/XRX(web_application_architecture) p p g ( _ pp _ ) • XRX Resources – http://www.danmccreary.com/xrx/ • LinkedIn XRX Group – http://www.linkedin.com/groups/XRX-Web-Application-Architecture-1056777 • Beginners Guide to XRX – http://www danmccreary com/xrx/beginners-guide http://www.danmccreary.com/xrx/beginners-guide • XRX Wikibook – http://en.wikibooks.org/wiki/XRX • Google Code – http://code.google.com/p/xrx/M D 62 Copyright 2011 Kelly-McCreary & Associates
  63. 63. References XForms XQuery XQ XRX A Beginners Guide to XRX Send e-mail to dan@danmccreary com for extended e mail dan@danmccreary.comM list of "getting started" resources. D 63 Copyright 2011 Kelly-McCreary & Associates
  64. 64. Thank You! Dan McCreary, President Kelly-McCreary & Associates dan@danmccreary.com (952) 931-9198 eXist Meeting Prague March 12th, 2010M D 64 Copyright 2007 Dan McCreary & Associates

×