Agile NoSQL With XRX

Agile NoSQL with XRX

Dan McCreary
President
M Dan McCreary & Associates
dan@danmccreary.com
D
(952) 931 9198
931-9198
Metadata Solutions

Session Description
What if all members of your software development team from
Project Managers, Business Analysts, Testing and Documentation
Managers Analysts
members could create and modify web applications and web
services? With traditional SQL solutions this was difficult because
of th need t convert web pages t objects, objects t t bl as
f the d to t b to bj t bj t to tables
well as the reverse functions. But now with native XML databases
and drag-and-drop forms builders, data can flow from the XML
model of a web form to the database and back again without
translation. This radically simpler process combined with
standardized query languages makes it easier for non-
non
programmers to build and maintain their own applications and web
services.
M

D 2 of
Copyright 2011 Dan McCreary & Associates
N

During this session viewers will:
• Understand the challenges associated with traditional four-
translation web/object/RDBMS systems
• Understand the elegant simplicity of zero-translation systems
• Understand the role of web standards in the XRX environment
• See how each component of XRX plays key roles in the
application
• See how organizations have leveraged XRX for agile web
development
• See variations of XRX architectures
• Create and objective p
j process to evaluate the benefits of XRX
• Get resources for building XRX pilot projects

M

D 3

Executive Summary
• Schema-free, zero translation NoSQL
systems have the ability to have a big
impact on overall system agility
p y g y
• XRX systems dramatically increase
overall agility and can also empower
non-progammers to build and maintain
NoSQL systems

M

D 4

Background for Dan McCreary
• Circuit Designer at Bell Labs
• UNIX/Supercomputers
• NeXT Computer (Steve Jobs)
• Owner of 75-person software consulting firm
with a focus on Object-Oriented systems and
Object Oriented
Object-relational mapping frameworks
• US Federal data integration (National
Information Exchange Model NIEM.gov)
• NativeXML/XQuery for metadata management
since 2006
• Advocate of web standards, OpenSource,
NoSQL d
N SQL and XRX systems t
• W3C invited expert (forms working group)
• Likes functional programming
M

D 5
Copyright Kelly-McCreary & Associates, LLC

Origins: The Humble Data Dictionary

M
6
D Copyright 2011 Kelly-McCreary & Associates

Electronic Certificate of Real Estate
Summer 2006

1 Document
= 44 SQL inserts

M

D 7
Copyright 2011 Kelly-McCreary & Associates

250 Data Elements
XForms
Mockup

M

D 8

Four Translations

T1 T2

T4 T3
Relational
Web Browser Object Middle
Database
Tier

• T1 – HTML into Java Objects
• T2 – Java Objects into SQL Tables
• T3 – Tables into Objects
• T4 – Objects into HTML
M
9

Kurt's Suggestion
Use a
A Native XML
Database!
Web Form

Save

Web Browser
Kurt Cagle

store($collection, $file-name, $data)
M
eXist
D 10

Zero Translation

XForms

Web Browser XML database

• XML lives in the web browser (XForms)
• REST interfaces
• XML in the database (Native XML, XQuery)
( , y)
• XRX Web Application Architecture
• No translation!
M

D Copyright 2010 Dan McCreary & Associates 11

Key Question: Impact on Agility
• What impact do zero translation
NoSQL
N SQL systems h
t have on
system agility?
• Agility: the ability to quickly
react to changing business
requirements at any stage of
the software development
lifecycle
• Question: Big impact or little
impact?
M
• Answer: Big impact
D 12

No-Shredding!

My Form
Data

• Relational databases take a single hierarchical
document and shred it into many pieces so it will fit in
tabular structures
• Many document-oriented NoSQL databases prevent
this shreddingg
M

D 13

Is Shredding Really Necessary?

• Every time you take
hierarchical data and
put it into a traditional
database you have to
put repeating groups in
separate tables and
use SQL “joins” to
joins
reassemble the data
M

D 14

Many Processes Today Are Driven By…

The constraints of yesterday…

Challenge:
Ask ourselves the question…
Do
D our current method of solving problems with t b l d t
t th d f l i bl ith tabular data…
Reflect the storage of the 1950s…
Or our actual business requirements?
What structures best solve the actual business problem?
p
M

D 15

"Schema Free"
• Systems that automatically determine
how to index data as the data is loaded
into the database
• No a priori knowledge of data structure
• N need f up-front logical d t modeling
No d for f t l i l data d li
– …but some modeling is still critical
• Adding new data elements or changing
g g g
data elements is not disruptive
• Searching millions of records still has sub-
second response time

M

D 16

Monoculture and Mono-architecture

M
Image Source: Wikipedia Monoculture
17
D

Storage Architectural Patterns
g
Tables Trees

Triples
Ti l Stars

M

D 18

Finding the Right Match
Schema-Free

Standards Compliant

Mature Query Language

M Use CMU's Architectural Tradeoff and Modeling (ATAM) Process
D 19

Architectural Summary
Four Translation Zero Translation
T T

T T web browser XML database
web browser
database

• HTML web pages
eb • XForms Client
• Object middle tier • Native XML Database
• RDBMS database

Which system more agile and by how much?
M How can this help us manage enterprise metadata?
D 20

Origins: The XML Data Dictionary

M

D 21

Electronic Certificate of Real Estate
Summer 2006

1 Document
= 44 SQL inserts

M

D 22

250 Data Elements
XForms
Mockup

M

D 23

The NO-SQL Universe
Key-Value Stores Document Stores

XML

Graph Stores

Object Stores

M
24
D Copyright 2010 Dan McCreary & Associates

Three Core Processes
1. Add new data to the database
1. Use XML example but JSON and other
formats could also be used
2. Query the d
2 Q h data
3. Create an XML web service

• Analyze the effort for each step
• Compare SQL and NoSQL XRX systems
NoSQL-XRX
• Analyze impact on participation of non-programmers
• Rate the relative agility of each system
M

D 25

It is Easy to Import Data
SQL XQuery
1. Analyze data for all parent child 1. Drag XML files into folder
relationships and repeating g p
p p g groups
2. Design logical and physical ER
diagrams
3. For each table create a Data Definition
File using a data definition language
(DDL)
4. Create indexes using DDL
5. Create one table for each set of
repeating set of data
6. Run DDL on database creating tables
using the appropriate data types
7. Create indexes
8. Create Insert statements
9. Create separate insert statements for
each repeating group
10. Run Insert statements on primary
structures in database
11. Use primary keys of the first data
inserts as foreign keys of dependant
M data structures

D 26

The XML File system
• XML File system – a way of
storing information in XML that
can be quickly searched
• You can drag-and-drop almost
any fil onto thi fil system
files t this file t
• You access it by using a
WebDAV connector
• Microsoft Windows “My Network
Places” function

M

D 27

Functional Programming

y = f(x)
• Computer programs are like mathematical functions
• Developers do not manipulate states and variables (things that
change value), but focus entirely on constants and functions
g ) y
(things that never change) and transformations of data
• Makes it very easy to build modular programs
• Software written in functional programming languages tend to
be very concise and easy to port to highly parallel systems

M http://en.wikibooks.org/wiki/Computer_programming/Functional_programming

D 28

It's Easy to Query XML Data
SELECT COL1, Col2
for $r in doc(‘t.xml’)//row
FROM TABLE where col1=1
col1 1
WHERE COL1=1 return $r/col1, $r/col2

<root>
<row>
Col1 Col2 <col1>1</col1><col2>A</col2>
</row>
1 A <row>
<col1>1</col1><col2>B</col2>
1 B </row>
<row>
1 C <col1>1</col1><col2>C</col2>
</row>
1 D <row>
<col1>1</col1> <col2>D</col2>
M </row>
</root>
D 29

It is Easy to Create A Web Service
The Java/JDBC/SQL Way The XRX Way
1. All XQuerys are web services
1.
1 Learn Java or find a Java Developer
2. Install TomCat Web Server
3. Install Java AXIS Web Server
4. Write a JDBC program that sends
SQL queries to a database
5. Get the results back in Java Result
Object structures
6. Go through the Java Results
Structues and use print statements
to wrap XML tags aro nd the strings
rap around
in the result objects
7. Rename your class files to .jws files
8. Add the .jws files to the TomCat
deploy folders
9. The WSDL files will automatically be
generated

M

D 30

Insert/Select/Publish Comparison
SQL

SQL
Java
logical Tomcat
data AXIS
modeling JDBC

SQL XQuery
XQuery XQuery

Insert Query Web Service NoSQL

M

D Total Effort 31

Steps for Adding a New Element
• Add the element to the "new
instance
instance" a form uses and set
Takes about
one minute the default value there
• Add the UI control
• Optionally – run an XQuery
update to update existing data
Takes about 10
minutes
or add an insert to the
"enricher" when you get an
y g
instance

M

D 32
Copyright 2011 Kell-McCreary & Associates

Six Translation – Web Service
Web Service

T5 T6
T1 T2

T3 T4
Relational
Web Browser Object Middle
Database
Tier
• T1 – HTML into Java Objects
• T2 – Java Objects into SQL Tables
• T3 – Tables into Objects
• T4 – Objects into HTML
• T5 – Objects to XML
M
• T6 – XML to Objects
D Copyright 2011 Dan McCreary & Associates 33

XML Stored in XForms Model
Browser Database

model save
update

view

M

D Copyright 2011 Kelly-McCreary & Associates 34

XRX Core Process
Browser Database

model
save/edit

update
view

M
35

Code Table Services
Client Server

model Form Data

Code Table Service

all-codes.xq
view
Code
C d
Tables

M
Code tables are separated from form instance data
D 36
Copyright 2011 Kelly McCreary & Associates

XRX Dynamic Forms Generation
Application Server
Client Application

XForms Model
Session
Form Data Form Data
User Team Document Collection
Status
Code Tables
Role Group
Views DataElement
Registry
Binding Rules
Code Table Services
Required Context
filters
Read-only Suggest Services

Data Types
Business Rules Editor
Calculations
Calculations Inference
Submissions
Constraints

XForms View
XF Vi
XML Schema Registry
Static Controls
Subschema Service
Dynamic Controls
Constraint Schemas Semantic Schemas
M
Design Time
D Run Time

Model Driven
• XForms enables the
developer to reuse business
XForms rules encapsulated in XML
Application
Schemas (xsd) and XML
Transforms (XQuery
Typeswitch)
• XForms reduces duplication
Meta
XML
Data and ensures that a change in
Schema
Registry the underlying business logic
does not require rewriting in
M another language

View and Model are Trees
Model • The view is a tree of a presentation
data element
• Models are comprised of one or
more trees
• XForms supplies the control layer
Control (Bind) that moves data elements to and
from the model
View (Presentation) • Users don’t have to worry about
moving things to and from the
g g
screen

M

D 39
Copyright 2011 Kelly McCreary & Associates

Models and View Are Linked with "Bind"

HTML
head
h d
xf:model body

Person
P form
f

Name fieldset

label

label
first last input
<bind> input

• Both the model and the views are trees of
M data l
d t elements
t
D 40

Just “Do The Right Thing”
HTML
head
body
xf:model

Person form

PersonCurrentOnTaxes type="xs:boolean" fieldset

label

PersonBirthDate type="xs:date" label
input
input
<bind>

• Data types from the model just do the right thing
• Boolean variables become checkboxes
M • Dates have d t selectors
D t h date l t
41

Example of Automatic UI Generation

• All true/false data types
(xs:boolean) automatically
become a checkbox
• All dates (xs:date) have a
date selector to the right of
the date field
• All codes can be selected
M from lists
42

Structure of a XForms File
• XForms tags are just XML
Namespaces
tags imbedded in a
CSS Imports (View) standard XHTML file with
Model a different namespace
Constraints (Bindings)
( g ) • Most HTML form tags are
g
UI (View)
exactly the same but some
attributes have been
Submit Controls
promoted to be full
MyForm.xhtml
y elements
M
43

REST
• REpresentation State Transfer
• Create applications based on well
designed URLs
• Take advantage of web caching
• Migrate toward Resource Oriented
Resource-Oriented
Computing (ROC)
• REST evangelists: RESTifarians

M


Five RESTFull Friends
1. In-resident memory cache in your
browser
2. You local hard drive cache
3. Your local enterprise cache
4.
4 The cache on the web server farm
5. The cache on the database
Please make sure to check with your RESTfull
friends BEFORE you bother the database.
M

D 45

Shallow REST vs. Deep REST
• You can start taking advantage of ReST
buy just doing
b j t d i well th ll thought-out URL
ht t
design
• To take advantage of deep ReST you
must consider the subtleties of the
HTTP protocol
– GET vs POST vs PUT
– DELETE

M

D 46

Benefits of REST
• Provides improved response time
• Reduced server load
• Improves server scalability
p y
• Requires less client-side software
• Depends less on vendor dependencies
• Promotes discovery
• Provides better long-term compatibility
• y
Better and evolvability
M

D 47

Sample XForms

M

D 48

Requirements Editor

Code
Table
Selection
Lists

Repeating
R ti
Elements

M

D 49

Drag-and-Drop XForms Builders
• Several options for
drag-and-drop f
d dd forms
builder
• Open Source
– Orbeon
– BetterFORM
• Commercial
– IBM Forms
M

D 50

Application Modularity

M

D 51

Financial Institution

M

D 52

Federal Integrator

M
53

Minnesota Historical Society

M

D 54

XRX Apps Demo Site
• Demo site for
danmccreary.com
• Some programs written
under an hour with
student data
• Several utility programs
that start with template
and add
transformations to
other formats
• Focus on metadata
management

M

D 55

Structured Retrieval is Better
Introduction to
Information Retrieval
by Christopher D. Manning,
y p g
Prabhakar Raghavan and
Hinrich Schütze
Cambridge University
C b id U i i
Press, 2008

M
http://nlp.stanford.edu/IR-book/information-retrieval-book.html
http://nlp stanford edu/IR-book/information-retrieval-book html
D 56

Table 10.1 - Revised
unstructured structured
RDB search
retrieval retrieval
unstructured trees with text at
objects records
documents leaves
vector space &
model relational model XML hierarchy
others
trees with node-
main data
table inverted index ids for
structure
t t
document ids
queries SQL free text queries XQuery fulltext

XML - Table 10.1 and structured information retrieval. SQLRDB (relational database) search,
unstructured information retrieval
M

D 57

Retain Document Structure in Search
"Bag of Words" "Retained Structure"
keywords
doc-id
keywords

'love' keywords
keywords

'hate' keywords
'new'
new
keywords
'fear'

• All keywords in a single container • Keywords associated with each
• Only count frequencies are stored sub-document component
with each word • Assign higher weight for titles and
M
names
D • Set by non-programmer 58

Empower the Non Programmer!
Before XRX After XRX
SUPER PM, BA!
,

Sorry, we have no idea Let me just search our XRX system…
what code 47 means. I'll have your answer in 150 milliseconds.

M

D 59

Using the Right Architecture

Start Finish

Find ways to remove barriers to empowering
the non programmers on your team
team.

M

D 60

Six "S"s of XRX Agility
1. Semantics – build around shard metadata
registry services
i t i
2. Search – structured search
3. Standards – CSS, XML, XPath, XQuery,
XForms, XML Schemas
4. Services – all XQ i are REST services
4 S i ll XQueries i
5. Solutions - that are quickly customized
6. Super – Empower the non-programmers

M
61

XRX Resources
• XRX Wikipedia Page
– http://en.wikipedia.org/wiki/XRX(web_application_architecture)
p p g ( _ pp _ )
• XRX Resources
– http://www.danmccreary.com/xrx/
• LinkedIn XRX Group
– http://www.linkedin.com/groups/XRX-Web-Application-Architecture-1056777
• Beginner's Guide to XRX
– http://www danmccreary com/xrx/beginners-guide
http://www.danmccreary.com/xrx/beginners-guide
• XRX Wikibook
– http://en.wikibooks.org/wiki/XRX
• Google Code
– http://code.google.com/p/xrx/

M

D 62

References

XForms
XQuery
XQ
XRX

A Beginner's Guide to XRX
Send e-mail to dan@danmccreary com for extended
e mail dan@danmccreary.com
M
list of "getting started" resources.
D 63

Thank You!
Dan McCreary, President
Kelly-McCreary & Associates
dan@danmccreary.com
(952) 931-9198

eXist Meeting Prague March 12th, 2010
M

D 64

Agile NoSQL With XRX

More Related Content

What's hot

Viewers also liked

Similar to Agile NoSQL With XRX

More from DATAVERSITY

Recently uploaded

Agile NoSQL With XRX