Content Applications: The Key To Next-Generation Publishing Dave Kellogg Chief Executive Officer 29 November 2007 Check out my blog at http://marklogic.blogspot.com

Topics A brief history of databases Search engines: filling the gap Web 2.0 Content applications

A brief history of databases

Search engines: filling the gap

Web 2.0

Content applications

A Brief History of Database Management Systems (DBMSs) Hierarchical Data model: hierarchy Example: IMS Network Data model: directed graph Examples: IDMS, DBMS32 Relational Data model: table (i.e., relation) Examples: Oracle, DB2, MySQL Mainstream market evolution through three generations

Hierarchical

Data model: hierarchy

Example: IMS

Network

Data model: directed graph

Examples: IDMS, DBMS32

Relational

Data model: table (i.e., relation)

Examples: Oracle, DB2, MySQL

Aside: The Great Irony of RDBMS Evolution The value proposition for the RDBMS was decision support oriented Prior-generation systems were great at OLTP Answer any question – without knowing in advance you’ll ask That focus was completely lost in the late 1980s / early 1990s TP1, DebitCredit, TPC-x, etc. benchmark wars Use the new car to do what the old one did well Only a decade later did the decision support focus return Data warehousing Business intelligence Kellogg’s rule: IT repeatedly buys insight and get transactions Examples: RDBMS, ERP Only BI delivered on the insight value proposition

The value proposition for the RDBMS was decision support oriented

Prior-generation systems were great at OLTP

Answer any question – without knowing in advance you’ll ask

That focus was completely lost in the late 1980s / early 1990s

TP1, DebitCredit, TPC-x, etc. benchmark wars

Use the new car to do what the old one did well

Only a decade later did the decision support focus return

Data warehousing

Business intelligence

Kellogg’s rule: IT repeatedly buys insight and get transactions

Examples: RDBMS, ERP

Only BI delivered on the insight value proposition

Specialized DBMS Markets Object databases Network underlying structure with fantastic integration to object programming languages Died: when the programmers fought the DBAs, the DBAs won The poster-child for special-purpose DBMS failure Multi-dimensional databases Also known as OLAP (online analytic processing) servers Dimensionalize, aggregate, and pre-calculate answers to intersections (e.g., product, geography, time) Approx $1B DBMS market Products: Essbase, SQL Server Analysis Services, Qlikview The overlooked example of special-purpose DBMS success

Object databases

Network underlying structure with fantastic integration to object programming languages

Died: when the programmers fought the DBAs, the DBAs won

The poster-child for special-purpose DBMS failure

Multi-dimensional databases

Also known as OLAP (online analytic processing) servers

Dimensionalize, aggregate, and pre-calculate answers to intersections (e.g., product, geography, time)

Approx $1B DBMS market

Products: Essbase, SQL Server Analysis Services, Qlikview

The overlooked example of special-purpose DBMS success

Generational Differences and Commonalities Differences Degree of BI / query processing Ability to process unpredicted queries Multi-dimensional processing Elegance of language interfaces Commonalities Business data processing Data-orientation Assume consistent schema Documents ignored

Differences

Degree of BI / query processing

Ability to process unpredicted queries

Multi-dimensional processing

Elegance of language interfaces

Commonalities

Business data processing

Data-orientation

Assume consistent schema

Documents ignored

Schema Regularity vs. Query Flexibility Schema Regularity Query Flexibility Hi Low Low Hi NDBMS RDBMS

Topics A brief history of databases Search engines: filling the gap Web 2.0 Content applications

A brief history of databases

Search engines: filling the gap

Web 2.0

Content applications

Enter Search Engines Assume nothing Index the words, and “filter” the rest Don’t even know where the content is Spiders to find it Indexes without data Leave content in place and point to it -- not the copy of record But build a “cache” so it’s copied anyway 404 type errors: data integrity Read-only Designed to help you find documents that might be useful Imprecise results There is no one precise “right” answer No two ever return the same  meta-search engines Trade-offs in precision vs. recall The relevancy quest

Assume nothing

Index the words, and “filter” the rest

Don’t even know where the content is

Spiders to find it

Indexes without data

Leave content in place and point to it -- not the copy of record

But build a “cache” so it’s copied anyway

404 type errors: data integrity

Read-only

Designed to help you find documents that might be useful

Imprecise results

There is no one precise “right” answer

No two ever return the same  meta-search engines

Trade-offs in precision vs. recall

The relevancy quest

The Relevancy Quest TF/IDF Term frequency inverse document frequency Webmaster effectively controls Easily gamed by spammers Link authority Inbound link analysis Other webmasters effectively control Increasingly gamed by spammers Interactive query refinement Extraction, taxonomy, clustering, faceted navigation Search history Semantic analysis Hakia’s SemanticRank Humans Cha Cha guides You try to guess what someone wants when they blurt two words! Is it Quixotic? Consider BI: let users access a database via a tool that generates a query language. No guessing required.

TF/IDF

Term frequency inverse document frequency

Webmaster effectively controls

Easily gamed by spammers

Link authority

Inbound link analysis

Other webmasters effectively control

Increasingly gamed by spammers

Interactive query refinement

Extraction, taxonomy, clustering, faceted navigation

Search history

Semantic analysis

Hakia’s SemanticRank

Humans

Cha Cha guides

Aside: Fun with People-Powered Search Search: my wife … Status: Looking for a guide ... connected to Pamela C Pamela C: Welcome to ChaCha! You: I'm looking for my wife, can you help me? Pamela C: Nope You: I thought you guys were experts in searching? She left an hour ago but her cell phone is off. Pamela C: searchable searches online You: you mean to say with all the world's information at our fingertips we can't put our heads together and find my wife? Pamela C: nope You: chacha sucks! Pamela C: I am sorry You: will you marry me Pamela? Pamela C: You are married You: yeah, but she's gone. I live in Utah. Pamela C: Okay You: awesome. Pamela C: Awesome Pamela C: What are you doin You: crying Pamela C: Why You: I miss my other wife. Pamela C: What is her name You: Pamela B. You: you are the third pamela, but I promise to love you as much as the others.

Search: my wife …

Status: Looking for a guide ... connected to Pamela C

Pamela C: Welcome to ChaCha!

You: I'm looking for my wife, can you help me?

Pamela C: Nope

You: I thought you guys were experts in searching? She left an hour ago but her cell phone is off.

Pamela C: searchable searches online

You: you mean to say with all the world's information at our fingertips we can't put our heads together and find my wife?

Pamela C: nope

You: chacha sucks!

Pamela C: I am sorry

You: will you marry me Pamela?

Pamela C: You are married

You: yeah, but she's gone. I live in Utah.

Pamela C: Okay

You: awesome.

Pamela C: Awesome

Pamela C: What are you doin

You: crying

Pamela C: Why

You: I miss my other wife.

Pamela C: What is her name

You: Pamela B.

You: you are the third pamela, but I promise to love you as much as the others.

Search Viewed From a DBMS Perspective You can run one query (very well) Return links to documents where document contains [word | phrase] You doesn’t serve as the master repository for the content But seems to “cache” everything so it takes as much space as if it did Indexes can point to things that no longer exist Indexes are not updated in real-time There’s no locking or transactions; no read-consistent snapshots I like How you assume nothing about the content Your indexing and query processing for text Your scalability architecture and clustering My, what a funny-looking DBMS you have there!

You can run one query (very well)

Return links to documents where document contains [word | phrase]

You doesn’t serve as the master repository for the content

But seems to “cache” everything so it takes as much space as if it did

Indexes can point to things that no longer exist

Indexes are not updated in real-time

There’s no locking or transactions; no read-consistent snapshots

I like

How you assume nothing about the content

Your indexing and query processing for text

Your scalability architecture and clustering

Schema Regularity vs. Query Flexibility Schema Regularity Query Flexibility Hi Low Low Hi NDBMS RDBMS Enterprise Search Engine ?

Topics A brief history of databases Search engines: filling the gap Web 2.0 Content applications

A brief history of databases

Search engines: filling the gap

Web 2.0

Content applications

Enter Web 2.0 People don’t just want to find documents anymore They want to Search content At granular level (just give me what I want) Navigate content See dynamic content View content in context (of a task or problem) Understand the social graph associated with content Transcend the data / content boundary Integrate and mash-up content Interact with content In short, they want full-blown applications on content Which we, shockingly, call “content applications”

People don’t just want to find documents anymore

They want to

Search content

At granular level (just give me what I want)

Navigate content

See dynamic content

View content in context (of a task or problem)

Understand the social graph associated with content

Transcend the data / content boundary

Integrate and mash-up content

Interact with content

In short, they want full-blown applications on content

Which we, shockingly, call “content applications”

Some Thoughts on Web 2.0 Lots of funny little companies with one or two syllable names? Hype designed by venture capitals to fuel Bubble 2.0? What Tim 2 (O’Reilly) did while Tim 1 (Berners-Lee) waited for the semantic web? Transformation of the web from publishing to programming platform? In my opinion, it’s about The specialization of search: Retrevo, Kayak, PickPackGo The wisdom of crowds: Wikis, PageRank, feedback (e.g., Diggs, eBay) The re-emergence of community: Facebook, LinkedIn, MySpace The transformation of the web From an e-commerce and publishing platform To an interactive information platform The transformation to content applications

Lots of funny little companies with one or two syllable names?

Hype designed by venture capitals to fuel Bubble 2.0?

What Tim 2 (O’Reilly) did while Tim 1 (Berners-Lee) waited for the semantic web?

Transformation of the web from publishing to programming platform?

In my opinion, it’s about

The specialization of search: Retrevo, Kayak, PickPackGo

The wisdom of crowds: Wikis, PageRank, feedback (e.g., Diggs, eBay)

The re-emergence of community: Facebook, LinkedIn, MySpace

The transformation of the web

From an e-commerce and publishing platform

To an interactive information platform

The transformation to content applications

Topics A brief history of databases Search engines: filling the gap Web 2.0 Content applications

A brief history of databases

Search engines: filling the gap

Web 2.0

Content applications

Content Applications Task aware Put content in context of what you’re trying to do Role aware And in the context of who you are and what you can see Are read/write Have classical (e.g., workflow) and web 2.0 features Seamlessly blend software and content to support that task Help a pathologist diagnose Help a pilot land with one engine Help a salesperson negotiate a contract Help a solider defuse an explosive device Help a traveler assemble an integrated itinerary Help a worker perform maintenance on a PET scanner Help a researcher explore prior research

Task aware

Put content in context of what you’re trying to do

Role aware

And in the context of who you are and what you can see

Are read/write

Have classical (e.g., workflow) and web 2.0 features

Seamlessly blend software and content to support that task

Help a pathologist diagnose

Help a pilot land with one engine

Help a salesperson negotiate a contract

Help a solider defuse an explosive device

Help a traveler assemble an integrated itinerary

Help a worker perform maintenance on a PET scanner

Help a researcher explore prior research

Content Apps: Top-to-Bottom XML My browser speaks XML, and my content’s in XML So why I am doing all this mapping between ... XML = hierarchies Java = objects and classes RDBMS = tables Answer: you don’t have to XQuery = misnamed and underpositioned Not just a query language, a full application development language Transcends data and content

My browser speaks XML, and my content’s in XML

So why I am doing all this mapping between ...

XML = hierarchies

Java = objects and classes

RDBMS = tables

Answer: you don’t have to

XQuery = misnamed and underpositioned

Not just a query language, a full application development language

Transcends data and content

XML: Not Just Web Content, Office Content Real XML formats are coming to the enterprise / office document world Open Office XML (OOXML) Open Document Format (ODF) Basic entity extraction through SmartTags Quark and Adobe are increasingly XML oriented In 3-5 years, you will be able not only to Get web content in an XML format But also most enterprise content as well

Real XML formats are coming to the enterprise / office document world

Open Office XML (OOXML)

Open Document Format (ODF)

Basic entity extraction through SmartTags

Quark and Adobe are increasingly XML oriented

In 3-5 years, you will be able not only to

Get web content in an XML format

But also most enterprise content as well

The Future Content Application Platform? Bolted-together search engines and RDBMS The usual choice today Negatives: integration work; performance / can’t push processing close enough to the data; thick middle tiers Extended RDBMS Columns of type XML Franken-queries in SQL/XQuery hybrids Negatives: Will never be optimized for XML and content; impossibility of general purpose data processing and content optimization; limits on extendibility Even COBOL wasn’t forever XML servers Native XML, native XQuery Optimized for content } Methinks these

Bolted-together search engines and RDBMS

The usual choice today

Negatives: integration work; performance / can’t push processing close enough to the data; thick middle tiers

Extended RDBMS

Columns of type XML

Franken-queries in SQL/XQuery hybrids

Negatives: Will never be optimized for XML and content; impossibility of general purpose data processing and content optimization; limits on extendibility

Even COBOL wasn’t forever

XML servers

Native XML, native XQuery

Optimized for content

The End of One Size Fits All One Size Fits All: An Idea Whose Time Has Come and Gone “ The last 25 years of commercial DBMS development can be summed up in a single phrase: “One size fits all”. This phrase refers to the fact that the traditional DBMS architecture (originally designed and optimized for business data processing) has been used to support many data-centric applications with widely varying characteristics and requirements. In this paper, we argue that this concept is no longer applicable to the database market, and that the commercial world will fracture into a collection of independent database engines, some of which may be unified by a common front-end parser.” Stonebraker and Cetintemel http://www.cs.brown.edu/~ugur/fits_all.pdf

One Size Fits All: An Idea Whose Time Has Come and Gone

“ The last 25 years of commercial DBMS development can be summed up in a single phrase: “One size fits all”. This phrase refers to the fact that the traditional DBMS architecture (originally designed and optimized for business data processing) has been used to support many data-centric applications with widely varying characteristics and requirements.

In this paper, we argue that this concept is no longer applicable to the database market, and that the commercial world will fracture into a collection of independent database engines, some of which may be unified by a common front-end parser.”

Stonebraker and Cetintemel

http://www.cs.brown.edu/~ugur/fits_all.pdf

The Rise of Special-Purpose DBMSs Streams Streambase, Skyler, Coral8 Huge memory stores TimesTen Hypercubes Essbase, Qlikview XML contentbases MarkLogic Column-orientation Vertica Data warehouses Greenplum, Hyperroll, Teradata XML data / messages Ipedo, Tamino RDF triples Things Codd wasn’t thinking about when he invented the relational model

Streams

Streambase, Skyler, Coral8

Huge memory stores

TimesTen

Hypercubes

Essbase, Qlikview

XML contentbases

MarkLogic

Column-orientation

Vertica

Data warehouses

Greenplum, Hyperroll, Teradata

XML data / messages

Ipedo, Tamino

RDF triples

Wall Street Journal 11/14/07, Excerpts Start-Ups Mine Database Field Most databases are based on technology that originated 30 years ago. But change is in the air. A mob of start-ups have been developing variants of the software, which provides the equivalent of filing cabinets for corporate information. Customers say the offerings are generating faster answers to questions that require sifting through huge volumes of business information. Some predict specialized products will find a niche. " One kind of database is not going to suit all of the different applications we are coming up with," said Donald Feinberg, [head database] analyst at market researcher Gartner Inc . http://online.wsj.com/article/SB119491593552290689.html

Start-Ups Mine Database Field

Most databases are based on technology that originated 30 years ago. But change is in the air.

A mob of start-ups have been developing variants of the software, which provides the equivalent of filing cabinets for corporate information. Customers say the offerings are generating faster answers to questions that require sifting through huge volumes of business information.

Some predict specialized products will find a niche. " One kind of database is not going to suit all of the different applications we are coming up with," said Donald Feinberg, [head database] analyst at market researcher Gartner Inc .

Schema Regularity vs. Query Flexibility Schema Regularity Query Flexibility Hi Low Low Hi NDBMS RDBMS Enterprise Search Engine XML Server

Conclusions I’m a 25 year career database guy Content has always been viewed as “funny data” Relational databases have had a great run But only diamonds are forever The same attributes that drove RDBMS success will drive XML server success Database platform needed for a new generation of applications Ability to answer any question Those applications must Handle varying (XML) schema-ed content Support users in tasks Deliver insight (content analytics), not just content Enable web 2.0 features Check out my blog at http://marklogic.blogspot.com

I’m a 25 year career database guy

Content has always been viewed as “funny data”

Relational databases have had a great run

But only diamonds are forever

The same attributes that drove RDBMS success will drive XML server success

Database platform needed for a new generation of applications

Ability to answer any question

Those applications must

Handle varying (XML) schema-ed content

Support users in tasks

Deliver insight (content analytics), not just content

Enable web 2.0 features