How to manage a system in which the schema of data cannot be defined “a priori”? How to quickly search for entities whose data is on multiple lines? In this session we are going to address all these issues, historically among the most complex for those who find themselves having to manage yet very common and very delicate with regard to performance. From EAV to Sparse Columns, we'll see all the possible techniques to do it in the best way possible, from a usability, performance and maintenance points of view.

1. Schema-less table &
Dynamic Schema
Davide Mauri
dmauri@solidq.com
@mauridb

2. Davide Mauri
• Microsoft SQL Server MVP
• Works with SQL Server from 6.5, on BI from 2003
• Specialized in Data Solution Architecture, Database Design, Performance
Tuning, High-Performance Data Warehousing, BI, Big Data
• President of UGISS (Italian SQL Server UG)
• Regular Speaker @ SQL Server events
• Consulting & Training, Mentor @ SolidQ
• E-mail: dmauri@solidq.com
• Twitter: @mauridb
• Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

3. Agenda
• Schema, Schemaless & Implicit Schemas
• Possible solutions
• Conclusion

4. Schema
• “A priori” definition of data structures
• Allows data to be inserted if and only if it is compatible with the
schema
• Es: RDBMS Table, XML Schema, Class, Struct

5. Schemaless (?)
• No definition at all on the data you expect to have.
• Unstructured data.
• For example: text files, binary files
• with no metadata and no position-based format
• In one word: chaos

6. Implicit Schema
• In reality a schema always exists, albeit implicit
• Otherwise it would be impossible to handle data

7. Implicit Schema
Any data that doesn't fit this implicit schema will not be
manipulated properly, leading to errors.
(Schemaless data structures, Martin Fowler)

8. Pros
• Flexibility
• Easy to manage
• actually, almost no management at all 
• Easy to be extended
• Just add a new element and you’re done
• Easy to be used
• No mismatch between OOP and other models

9. Cons
• Schema information are hidden somewhere
• Scattered all across the codebase
• It’s really difficult to keep under control the chaos that can
emerge
• For example two different element that contains the same information
• CustomerName and Customer_Name
• You still need to have a sort of «First Normal Form»  in order to avoid
inconsistency and code inefficiencies

10. Cons
• It’s really difficult to define and maintain integrity constraints
• Data Integrity is a value that must be preserved!
• Otherwise we’ll have data, not information 
• XML Schema were born for that specific reason
• Without Data Integrity, the process of extracting information from data
becomes
• Difficult
• Expensive
• Untrustable

11. Words of Wisdom
«Schemaless => implicit schema = bad.
Prefer an explicit schema»
(Schemaless data structures, Martin Fowler)

12. But if we need it anyway?
• What if my use case is one that perfectly fits the need for a
implicit schema?
• The only possible solution are the so-called «No-SQL»
databases
• Document Database or Key-Value store?
• How can I integrate it into already existing database?
• Integration does not come for free!

13. Schemaless & RDBMS
• (Usually) Are the exact opposite extremes
• Still is a very common request
• CRM, eCommerce, ERPs….
• Schemaless is used not only for pure data persistence

14. Solution within an RDBMS
• «Custom» columns
• Custom1, Custom2
• In-Table Data Structures
• BLOB, XML, JSON, «Complex» columns
• Entity-Attribute-Value Models

15. «Custom» Columns
• A problem until SQL Server 2008
• Space is still used for fixed length column even if they contain a NULL
value
• With SQL Server 2008 the «Sparse Column» feature comes to
help
• Helps to make the schema easily modifiable, even in presence of
existing data
• Changes to the schema must still be done with «ALTER TABLE»

16. «Custom» Columns
• Sparse Columns
• Are Columns at 100%
• Optionally you can have *all* the Sparse Columns returned as a single
XML column
• «Column Set»
• Make development easier
• Do not take space if not used 
• But use more space when used 

17. Demo
Dynamic Schema & Sparse Columns

18. In-Table Data Structures
• Complete support for XML
• XPath/XQuery
• XML Index
• Performance «Good Enough»
• But not optimal (compared with the equivalent relational approach)
• Use a lot of space

19. In-Table Data Structures
• XML Sometimes needs some help to boost performance
• Would be nice to be able to «promote» elements to turn them
into real columns
• Must be done manually using a choice of
• Triggers
• Stored Procedure
• Data Access Layer
• Service Broker

20. In-Table Data Structures
• JSON support is still missing in SQL Server
• But others database like PostgreSQL already have it…
• …so we can see it coming to MS Platform too 
• Right now one solution is to use SQLCLR
• Solutions available surfing the web:
• http://www.sqlservercentral.com/articles/SQLCLR/74160/
• http://www.json4sql.com/examples.html
• There is also a pure T-SQL solution
• https://www.simple-talk.com/sql/t-sql-programming/consuming-json-strings-
in-sql-server/

21. In-Table Data Structures
• Blob is an option if you just need to do persistence
• Blob can be stored in different way
• «Classic» blob inside SQL Server pages & extents
• Blob in a filestream
• Blob in a filetable

22. Demo
Dynamic Schema & In-Table Data Structures

23. Entity-Attribute-Values
• Old and very common technique to store attribute-value pairs
• Some well-known samples: Wordpress
• Works on any RDBMS
• No «special» features required
• There’s a huge debate around it 
• http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_mo
del
• But until SQL 2005 no true alternative

24. Entity-Attribute-Values

25. Entity-Attribute-Values
• Offers maximum flexibility
• No real control over data types.
• Options to deal with data types
• All strings
• SQL Variant
• One-Column-Per-Type
• Complex query pattern for «AND» predicates between attributes
• «Return all the entities that have «CPU=i7» and «Display=15.4’»

26. Entity-Attribute-Values
• Queries requires the implementation of a relational operator not
implemented in common RDMBS
• «Relational Division»
• Document and well explained in theory
• It is quite easy to implement it. Follow theory + add some pepper to boost
performances 

27. Relational Division
• Let’s get back to theory a little bit, in order to see the problem
from a more open perspective:
Dividend
Divisor Result
Remainder
𝛼
𝛽

28. Relational Division
• How do we implement the division?
• Thanks to Codd and the relational theory we already have the
solution 
28

29. Relational Division
• Thanks to relational algebra we know that the division is
expressed as
• Generate all possibile pairings
• Remove existing pairing
• (Now we’ve found all pairings that are NOT answers)
• Remove the non-answers from the dividend
29

30. Demo
Dynamic Schema & EAV

31. Conclusions
• It works! 
• Performance more than good
• Choose the solution that better fits your use-case
• Search for attributes only?
• Persistence only?
• Search for attributes & values?
• Performance read, write, read/write?

32. Conclusions
• Use it if and only if when really needed
• Always remeber the «Words of Wisdom» 
• If you can define and use a schema.
• It may seem «not cool» and convoluted but in the long term is the best
solution.
• *data* *must* *be* *turned* *into* *information*
• Sooner or later 
• Without metadata (a schema) it’s really really really hard!

33. Questions?

34. Thanks!
• If you want to rate this session on my SpeakerScore page:
• www.speakerscore.com
• Feedback Key: TZQL

35. Demo Material
• Can be found here
• http://1drv.ms/1Av5mb5
• Everything is release under the Creative Common Attribution-
NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA
4.0)
• http://creativecommons.org/licenses/by-nc-sa/4.0/