MarkLogic User Group - Best of MLW and Search + Semantics

<presentation/>
<presenter>Matt Turner</presenter>
<title>Chief Technologist, Media Solutions</title>

Copyright © 2013 MarkLogic® Corporation. All rights reserved.
<MLUGL>
<intro/>
<talk>
<bit>Mission Impossible</bit>
<story>Wiley</story>
<story>Springer</story>
<story>Mitchell1</story>
<bit>Search and Semantics<bit>
<demo>Old Skool</demo>
</talk>
</MLUGL>

Mission(s) Impossible

<story>http://www.marklogic.com/resources/slides-gearing-up-for-the-
content-factory-to-quickly-create-innovate-and-monetize/</story>

Why is it Mission Impossible?
Start Revenue Earning January 2013
• Publish new content from 1 Jan 2013
• Accepted Articles : 20/day; 100/week; 400/month
• Early View Articles: 20/day; 100/week; 400/month
• Issues : 19/month; 77/quarter; 230/year
Give AGU customers access to all licensed content by 1 January 2013
• 21 journals (160,000 articles)
• 33 personal choice products (aka virtual journals) based on AGU index terms
• 743 special sections
• Migrate customers, users, products, licenses, alerts data
Vendors, systems & business processes in Editorial & Production ready to
publish 2013 Content
• Integration with new editorial system
• Changes to work flow
And… it needs to work like how it works on AGU site with over 60 enhancements

KeyChallenges
•Content with no issue number and no pagination
•Journal with 7 parts, of which 3 of those parts have sub-parts!
•Many moving parts within Wiley - 17 systems to check
•Content completeness and quality (and external vendor)
•Unknown unknowns - coping with changing and emerging requirements
throughout development phase
Challenges to overcome
• 4 months left!

Examples:
“Coastal Ocean
Observatories”
“The 11 March 2011
Tohoku-Oki Earthquake
and Tsunami”
Content-Driven Functionality – Special Section Search

How MarkLogic Helped - S/W Development
Search Service
•As a search engine, doesn't need manual/additional re-indexing after loading new
content. Everything is done on fly – saves time and effort
•Enabled reuse and only had to add some enhancements to search service for AGU
Save Searches
•Search service processing request in XML is easy to save whole search and reuse it
for either alerts or loading the saved search
Index Terms
•Reuse vocabulary service to help with hierarchy of index terms. This was more
valuable for faceting for index terms. Can easily fetch any sub-structure of index
terms
Faceting
•MarkLogic supports faceting, so no need to do anything special, just add proper
configuration according AGU specification

What Variations/Non Standard
Practices were introduced
• New licensing model (e.g. multi choice product for personal subscribers)
• Create Special Sections as another slice of content view
• New workflow for handling daily society data updates via feeds
• Changing content workflow for legacy vs current content
• Improvements to content (not just conversion)
• Start development before requirements were clear
• Complete testing before we had all the content
• Cannot complete certain types of testing
• Break some rules
Recipe for Disaster?

Conclusion
•Mission Impossible? Choose not to accept
•Mission Impossible? Deal with it – that’s life but may not succeed
•Mission Impossible? New organizational capability
•Embrace challenge, but put your best people with experience on it
•Be brave to break the rules when required
•People over Process
•Enabling technologies like MarkLogic
Develop as new capability to handle the unexpected and unknowns

<story>http://www.marklogic.com/resources/betting-the-company-how-
springer-successfully-insourced-its-flagship-content-platform/</story>

Betting the Company | 4/6/2013 | 18
Growth in electronic sales
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
2007 2008 2009 2010 2011 2012
Bud
Total Online
Total Print66
33

So...
Springer decided to
build its own platform

36 man-years of effort to reproduce
36 man-yearsHow much time independent software auditor
estimated it would take to reproduce
the existing code base

A risky move?
MetaPress
code base

Oh, and have it ready
in 11 months

Where we were in April 2011
• People
• 1 Executive Champion
• 1 Product Owner
• 1 Dir. of Dev
• 1 Tech Lead
• 2 Developers
• 1 BA
• 0 QA
• 0 DevOps
• 0 UX/design/front-end
• 0 architect
• Hardware/Software/Data
• 0 databases
• 0 servers
• 0 documents
7 staff*
*3 managers – who don’t count
Jan-Erik de Boer
Brian Bishop Georg Nold
EVP of IT
Product Owner Dir. of Development

Where we are today
• 1 Executive champion
• 1 Product Owner
• 1 Dir. of Dev
• 2 Tech Leads
• 16 Developers
• 2 Dev Ops
• 4 BAs
• 6 QAs
• 2 UX
• 2 Design/Front-end
• 1 Architect
• 16 servers
• 2 live environments
• 1 database
• 12 pairing stations
• 2 Build Agents
• 2 dashboard machines
• 5.7 million documents
• 60 million PNGs
• 11TB of data
31staff

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
New platform release schedule
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release
Release

MarkLogic
cluster
RESTful APIs realtime.springer.com
citations.springer.com
iPhone apps

Goals are
prioritized
(top to bottom) and
stories
are prioritized
(left to right)
Velocity is measured
every week, allowing
us to accurately
forecast when a
certain level of work
can be completed

MarkLogic IS agile

MarkLogic agility
• Schema-less means we can use our complex XML content as-is
• E.g. Different attributes for books, journals, chapters, articles, protocols, etc.
• You can decide later if you need to add indexes at very little cost
• You don’t have to know everything up front
• Ingestion is relatively pain-free
• You are free to come up with features without worrying about back-end
• Modifying content via Record Loader makes it easy to manipulate data
• Handles various types of native content
• You don’t even have to use Xquery!

What if you could subscribe to
a search query?

Content Entitlements
2TB
Storing entitlements as queries means any new content loaded
automatically becomes available to authorized users
Customers
<material_ID=“001”>
Subject : Engineering
<content>
Journal_ID:0001
ContentType: Article
DatePublished: 4/4/2012
Subject:Mathematics
Author: John Smith
Language: English
Keywords: “k theory” <material_ID=“002”>
Journal_ID: 0001-0099
<material_ID=“003”>
Subject: Engineering
SearchTerm: “carbon nanotube”
DatePublished: 2000-2012
<customer=“001”>
material_ID : 001
These are stored as
serialized queries

How did it go?

0
2
4
6
8
10
12
Old New
Average Page Load Time (sec)

Weekly visits to SpringerLink (millions, Aug 4, 2012 – Mar 2, 2013)
Source: Google Analytics
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
4,000,000
4,500,000
5,000,000
link.springer.com
SpringerLink.com
Total

<story>http://www.marklogic.com/resources/the-journey-from-
print-to-online/</story>

2011

 65 OEM Auto and Part
Manufacturers
 Data on every modern car sold in
US
 Repair
 Diagnostics
 Maintenance
 Technical Service Bulletins (TSBs)
 Wiring
 Estimator
Mitchell1: Data

What’ s in the data store today?
• Articles – 408,892
– 209,987 Narratives
– 103,416 Technical Service Bulletins and Recalls
– 15,179 Maintenance Schedules
• Images – 6,193,647
– 5,924,959 Narrative
– 268,688 Technical Service Bulletins and Recalls
• When it’ s all broken down, it becomes roughly
16,000,000 MarkLogic Documents

And how do we describe it?
• Preferred Terms
– Tends to be the ASE term
– Used to describe Components (12,261), Diagnostic Trouble
Codes (65,525), and Information Types (98)
• Non-Preferred Terms
– Tends to be OEM specific terminology
– Alternate terms for Components (22,733) and Information Types
(757)
– Codes do not have Non-Preferred Terms
• Spatial References
– Because “ Replace the window motor” just isn’ t precise enough

Mitchell1: Data Then, Data Now

Mitchell1: Market Reaction
https://www.youtube.com/watch?v=IfM8v-8NY_4&list=UUIOYnh6LBFooV_YxlPVPLvA&index=36

Search . . . and Semantics

One Question . . .

Who’s Smarter?
VS

Do domestic dogs interpret pointing as a command?
Animal Cognition (2012): 1-12 , November 09, 2012
By Scheider, Linda; Kaminski, Juliane; Call, Josep; Tomasello, Michael

What if . . .

HOW?

The Basic Idea
Get some triples . . . if you haven’t already
• Grabbed DBPedia
• Dumped in Linked Data Consortium
• Loaded Lehigh
• and NYT’s open data
You are behind!
But what if you could add in documents?

Rich MarkLogic Applications .. Made Richer

Rich MarkLogic Applications .. Made Richer
Name: John Smith
Affiliation: IBM
Timezone: PST
Committer: Hadoop

Semantics Architecture
TRIPLE
XQY XSLT SQL SPARQL
GRAPH
SPARQL

Triple Index
• 3 triple orders
• Cached for performance
• Works seamlessly with other indexes
• Security
• 350 bytes per triple on disk
• 1 billion+ triples per host
TRIPLE

SPARQL
• Executed using the triple index
• SPARQL 1.0
• Cost-based optimization
• Join ordering and algorithms
• More in the lightning talks
select * where {
?person :birth-place ?place;
:first-name “John”
}
SPARQL

Demo

Old Skool
- Quickie Framework
- Circa 2006ish
- HTML tables -> 1997 style
- ‘action’ controller
- <query/> state -> from the query string
- No sessions
- No CSS
- No Javascript
- No Adaptive Design
- No Facets?

Search

Facets!

Semantics

Just Semantics?

Thank You!

MarkLogic User Group - Best of MLW and Search + Semantics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MarkLogic User Group - Best of MLW and Search + Semantics

Similar to MarkLogic User Group - Best of MLW and Search + Semantics (20)

More from Matt Turner

More from Matt Turner (20)

Recently uploaded

Recently uploaded (20)

MarkLogic User Group - Best of MLW and Search + Semantics

Editor's Notes