Dynamic archive management
Dynamic Archive Management
a framework for dynamic database schemas
Michiel Overeem
Center for Software Technology, Universiteit Utrecht
August 31, 2007
Dr. B. J. Heeren Drs. Ing. P. van Diermen
Drs. L. Herlaar
Center for Software Technology Michiel Overeem

Dynamic archive management
Context
1
Objectives
2
Using dynamic database schemas
3
Meta-data
GUI
Managing database schemas
4
Synchronization
Version control system
Conclusion
5
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
Database schema coupling
...
50% or more of programmer effort arises as a result of system
modifications after implementation
...
Database schemas are knowledge, compiled into applications
communicating with the database. As a result schema modifications
lead to application code modifications.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
Web applications
Web applications are shifting from dynamic websites to the WebOS.
Web applications consist of (X)HTML, CSS, and JavaScript. Design is
done with CSS, however, structure is defined in (X)HTML.
Different approaches exist to decouple the interface from the logic in
web applications.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
Web-based archive management
Archive management systems:
Management of documents (such as photos, certificates,
books, and newspapers). Not only bookkeeping, but also
opening up to a wide, anonymous audience.
Data intensive: tight coupling with database schema.
On the document level: basic CRUD functionality.
On the collection level: search, report, and more.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
Web-based archive management - cont’d
Intranet interface: branding not required
Intranet is used for management:
more advanced features
design does not have to be customizable
Internet interface: branding often required
Internet is used to open up documents to anonymous visitors:
design is more important, customers want to present their
collection in their style
focus is not on management, but on presentation
sometimes anonymous visitors can help the archive in a Wiki-like
style
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
How is the current framework used?
1 Database schema is designed in custom management application.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
How is the current framework used?
1 Database schema is designed in custom management application.
2 Management application generates code files containing all
meta-data.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
How is the current framework used?
1 Database schema is designed in custom management application.
2 Management application generates code files containing all
meta-data.
3 The code files are put in the codebase.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
How is the current framework used?
1 Database schema is designed in custom management application.
2 Management application generates code files containing all
meta-data.
3 The code files are put in the codebase.
4 Webpages are designed that show (part of) the database schema.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
How is the current framework used?
1 Database schema is designed in custom management application.
2 Management application generates code files containing all
meta-data.
3 The code files are put in the codebase.
4 Webpages are designed that show (part of) the database schema.
5 Application is compiled and deployed.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
DEVENTit & Existing framework
DEVENTit has developed a framework for generic database access:
Framework deals with retrieval and synchronization of changes.
Meta-data (the dictionary), containing database schema, is fed
into framework.
How is the current framework used?
1 Database schema is designed in custom management application.
2 Management application generates code files containing all
meta-data.
3 The code files are put in the codebase.
4 Webpages are designed that show (part of) the database schema.
5 Application is compiled and deployed.
6 If a change is required: start at 1.
Center for Software Technology Michiel Overeem

Dynamic archive management > Context
Framework diagram
Webpages Dictionary
'
E E
Framework
'
c
c
Database
Center for Software Technology Michiel Overeem

Dynamic archive management > Objectives
Why this thesis?
Why?
No longer deployment cycles for schema changes: no compile-time
linking to meta-data, or webpage.
No longer developers to make small modifications.
No longer manual synchronization of schema design environment
with live database.
⇓⇓
Flexible, document independent archive management application
Center for Software Technology Michiel Overeem

Dynamic archive management > Objectives
Project definition
1 How can a framework for archive management systems be
constructed that removes the tight, compile-time coupling
between
a database schema and application logic, and
b database schema and the user interface?
2 How can non-technical people modify a database schema used in
an application built with the framework as described by 1?
Center for Software Technology Michiel Overeem

Dynamic archive management > Objectives
Project definition
1 How can a framework for archive management systems be
constructed that removes the tight, compile-time coupling
between
a database schema and application logic, and
b database schema and the user interface?
2 How can non-technical people modify a database schema used in
an application built with the framework as described by 1?
We focus on the documents, not the entire database schema.
We focus on CRUD pages for those documents.
The document definitions become run-time mutable for
non-technical users.
Center for Software Technology Michiel Overeem

Dynamic archive management > Using dynamic database schemas > Meta-data
Extension of the framework, meta-data
We want to loose the compile-time coupling
No longer code generation: instead the framework accesses the
meta-data (stored in a database) directly.
⇓
The meta-data is no longer static, thus reloading is required.
Center for Software Technology Michiel Overeem

Dynamic archive management > Using dynamic database schemas > Meta-data
Extension of the framework, meta-data - cont’d
The framework has to use correct, not out-dated information!
The data model used in-memory has to contain the right information:
the data model is a proxy between the framework and the
1
database
the data model uses caching: pulling and pushing keeps the data
2
up to date
Center for Software Technology Michiel Overeem

Dynamic archive management > Using dynamic database schemas > GUI
Extension of the framework, interface
No longer webpages defined in a programming language
XForms template engine
Standardized Custom built
Too technical Only what and where
Bad support HTML, CSS output
Center for Software Technology Michiel Overeem

Dynamic archive management > Using dynamic database schemas > GUI
Extension of the framework, interface
No longer webpages defined in a programming language
XForms template engine
Standardized Custom built
Too technical Only what and where
Bad support HTML, CSS output
...
<div>
<field id=\"ADD CERTIFICATES DATE\" type=\"header\"/>:
<field id=\"ADD CERTIFICATES DATE\" />
</div>
...
Center for Software Technology Michiel Overeem

Dynamic archive management > Using dynamic database schemas > GUI
Framework diagram
Webpages Dictionary
'
E E
Framework
'
c
c
Database
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas
The meta-data
The dictionary represents the relational model: tables, columns, and
foreign keys. Lots of details that can be hard for non-educated users.
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas
The meta-data
Persons
The dictionary represents the relational model: tables, columns, and
foreign keys. Lots of details that can be hard for non-educated users.
ID
Certificates ' E Name
PersonAndRoles
ID
ID
Date E
CertificatesID Roles
Description
NameID
ID
RoleID
Role
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas
The meta-data
The dictionary represents the relational model: tables, columns, and
foreign keys. Lots of details that can be hard for non-educated users.
Abstraction from this model to simplify things: entities, fields, and
relations. Relations do not have a direction, but a multiplicity: 1 to
many, many to 1, or many to many.
Users modify the abstraction.
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas
The meta-data
The dictionary represents the relational model: tables, columns, and
Persons
foreign keys. Lots of details that can be hard for non-educated users.
1-to-m m-to-1 Name
Certificates
PersonAndRoles
Abstraction from this model to simplify things: entities, fields, and
Date
relations. Relations do not have a direction, but a multiplicity: 1 to
m-to-1
Description
many, many to 1, or many to many. Roles
Role
Users modify the abstraction.
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas
The application
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Synchronization
Synchronize dictionary and schema
The applications based on the framework use two things: dictionary
(meta-data) and the database schema, in which the data is stored.
Both have to be synchronized at all times: the meta-data is used to
build queries to pull data from the database. These queries fail if the
dictionary is different from the database.
Dictionary
Database ' '
E
Framework
c
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Synchronization
Synchronize dictionary and schema - cont’d
The user opens a dictionary and starts an edit session. The edit session
results in a collection of actions: addition, modification, and removal
of entities, fields, and relations.
These actions have to be applied on the database schema, but in order!
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Synchronization
Synchronize dictionary and schema - cont’d
 
MC AR
 


    
MT E RR E RC E RT E AT
    
d d
d d
d  d 
‚
d ‚
d
MR AC
 
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Synchronization
Synchronize dictionary and schema - cont’d
 
MC AR
 


    
MT E RR E RC E RT E AT
    
d d
d d
d  d 
‚
d ‚
d
MR AC
 
∀c, t : MT (t) ∈ A ∧ MC (c, t) ∈ A : MT (t) MC (c, t)
The table can be renamed, but the modification of columns
have to use the new name.
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Synchronization
Synchronize dictionary and schema - cont’d
∀c, t : RC (c, t) ∈ A ∧ RT (t) ∈ A : RC (c, t) RT (t)
First remove the columns, before the tables can be removed.
 
MC AR
 


    
MT E RR E RC E RT E AT
    
d d
d d
d  d 
‚
d ‚
d
MR AC
 
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Synchronization
Synchronize dictionary and schema - cont’d
∀c, t : AT (t) ∈ A ∧ AC (c, t) ∈ A : AT (t) AC (c, t)
First add the table, before we can add a column to it.
 
MC AR
 


    
MT E RR E RC E RT E AT
    
d d
d d
d  d 
‚
d ‚
d
MR AC
 
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
The users are just like developers: they make mistakes. Therefore a
version control system is built into the management application.
The control system is delta-based: only the diffs are stored.
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles
The users are just like developers: they make mistakes. Therefore a
Date
m-to-1
Description
version control system is built into the management application.
Roles
The control system is delta-based: only the diffs are stored.
Role

V0

Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles Address
The users are just like developers: they make mistakes. Therefore a
Date
m-to-1
Description
version control system is built into the management application.
Roles
The control system is delta-based: only the diffs are stored.
Role
 
V0 E δ1
 
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles Address
Date
The users are just like developers: they make mistakes. Therefore a
Description m-to-1
version control system is built into the management application.
Roles
Location
The control system is delta-based: only the diffs are stored.
Role
  
V0 E δ1 E δ2
  
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles Address
Date
The users are just like developers: they make mistakes. Therefore a
Description m-to-1
OrderNr
version control system is built into the management application.
Roles
Location
The control system is delta-based: only the diffs are stored.
Role
   
V0 E δ1 E δ2 E δ3
   
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles Address
Date
The users are just like developers: they make mistakes. Therefore a
Description m-to-1
OrderNr
version control system is built into the management application.
Roles
Location
The control system is delta-based: only the diffs are stored.
Role
   
V0 E δ1 E δ2 E δ3
   
Deltas can be switched off and on!
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles Address
Date
The users are just like developers: they make mistakes. Therefore a
Description m-to-1
OrderNr
version control system is built into the management application.
Roles
Location
The control system is delta-based: only the diffs are stored.
Role
   
V0 δ1 δ2 E δ3
   
Deltas can be switched off and on!
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Version control system
Persons
1-to-m m-to-1 Name
Certificates
PersonAndRoles Address
Date
The users are just like developers: they make mistakes. Therefore a
Description m-to-1
OrderNr
version control system is built into the management application.
Roles
Location
The control system is delta-based: only the diffs are stored.
Role
But deltas have dependencies!
If delta a is switched off,
    too!
all dependent deltas are switched off
V0 δ1 δ2 E δ3
   
Deltas can be switched off and on!
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Branching and merging
Branches are used to experiment with alternative solutions, or to
design in a modular way (not parallel!).

 
δ4
E δ5 E δ6
  

   
V0 E δ1 E δ2 E δ3
   
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Branching and merging
Branches are used to experiment with alternative solutions, or to
design in a modular way (not parallel!).

 
δ4
E δ5 E δ6
  

   
V0 E δ1 E δ2 E δ3
   
Merge the changes from the upper branch in the lower branch:
Merge V123 and V456 using V0 , the result is Vm .
Calculate the delta between V123 and Vm .
Store that delta as δ7 , connected to δ3 .
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Branching and merging
Branches are used to experiment with alternative solutions, or to
design in a modular way (not parallel!).

 
δ4
E δ5 E δ6
  

    
V0 E δ1 E δ2 E δ3 E δ7
    
Merge the changes from the upper branch in the lower branch:
Merge V123 and V456 using V0 , the result is Vm .
Calculate the delta between V123 and Vm .
Store that delta as δ7 , connected to δ3 .
Center for Software Technology Michiel Overeem

Dynamic archive management > Managing database schemas > Version control system
Branching and merging - cont’d
Merging is done in two situations:
Branches
Application updates
But conflict-free merging is impossible!
But conflicts are not user friendly!
The first case is rather easy: let the user decide how to solve the
conflict, he knows best . . .
The second case is solved with annotations: let the developers
decide . . .
Center for Software Technology Michiel Overeem

Dynamic archive management > Conclusion
Conclusion
1 How can a framework for archive management systems be
constructed that removes the tight, compile-time coupling
between
a database schema and application logic, and
b database schema and the user interface?
Center for Software Technology Michiel Overeem

Dynamic archive management > Conclusion
Conclusion
1 How can a framework for archive management systems be
constructed that removes the tight, compile-time coupling
between
a database schema and application logic, and
b database schema and the user interface?
This is done by extending the existing framework. The framework can
now load the meta-data at runtime, and keep it up-to-date. Templates
are used that are mutable after compilation.
Center for Software Technology Michiel Overeem

Dynamic archive management > Conclusion
Conclusion
1 How can a framework for archive management systems be
constructed that removes the tight, compile-time coupling
between
a database schema and application logic, and
b database schema and the user interface?
This is done by extending the existing framework. The framework can
now load the meta-data at runtime, and keep it up-to-date. Templates
are used that are mutable after compilation.
2 How can non-technical people modify a database schema used in
an application built with the framework as described by 1?
Center for Software Technology Michiel Overeem

Dynamic archive management > Conclusion
Conclusion
1 How can a framework for archive management systems be
constructed that removes the tight, compile-time coupling
between
a database schema and application logic, and
b database schema and the user interface?
This is done by extending the existing framework. The framework can
now load the meta-data at runtime, and keep it up-to-date. Templates
are used that are mutable after compilation.
2 How can non-technical people modify a database schema used in
an application built with the framework as described by 1?
The management application allows non-technical users to modify the
schema, keeping the database synchronized, in a save environment.
History reviewing and undo/redo makes it possible to experiment.
Center for Software Technology Michiel Overeem

Dynamic archive management > Conclusion
Implementation
Large part of is implemented:
The runtime looking of
meta-data.
The template engine.
The management application:
The synchronization algorithm.
Version control with separate
copies.
Undo/redo system.
Center for Software Technology Michiel Overeem

Dynamic archive management > Conclusion
Future work
Some things need more work:
The templates are not integrated into the version control system,
they are disconnected from the meta-data, they are by no means
user-friendly.
The management application could be used as off-line database
schema manager. It could offer automatic generation of update
scripts and so on, everything to help developers with their
schemas.
Center for Software Technology Michiel Overeem