Agile Integration Using an Enterprise Data Hub by Michael Malgeri, Principal Technologist at MarkLogic, Corp. Today, business processes and decisions are driven from information contained in a variety of internal systems of record (SoRs) and other external sources (social media, partners, suppliers, customers). Enterprises are discovering that out of the box (OOTB) solutions often don't exist for their operational and observational requirements. In addition, traditional integration solutions often take too long to implement and are brittle. This presentation will discuss how an enterprise operational data hub (ODH) can be an agile, flexible solution for integrating key information sources in a timely manner for driving mission critical applications and reporting.

1. Agile Integration Using an Enterprise
Data Hub
Michael Malgeri
Principal Technologist
MarkLogic

2. SLIDE: 2
The World is Changing

3. SLIDE: 3
People Certainly Change

4. SLIDE: 4 3 August 2018© MARKLOGIC CORPORATION
CHANGE IS CONSTANT
Expect That Over Time,
Everything Can Change, at Any Time
DATA WHERE DATA
COMES FROM
HOW DATA IS
ACCESSED
WHO ACCESSES
THE DATA

5. SLIDE: 5
What’s this 360 Stuff?
Consumer 360

6. SLIDE: 6
What’s this 360 Stuff?
Product 360

7. SLIDE: 7
What’s this 360 Stuff?
Patient 360

8. SLIDE: 8
What’s this 360 Stuff?
Supplier 360

9. SLIDE: 9
What’s this 360 Stuff?
Employee 360

10. SLIDE: 10
A Unified, Actionable
360 View of Data
WHAT ORGANIZATIONS WANT
WHAT ARCHITECTS, DBAS, DEVELOPERS WANT
Technical Innovation,
Agility, Reduced Cost,
& Reduced Complexity

11. SLIDE: 11
How do enterprises deal with constant change?

12. SLIDE: 12
By Being Agile

13. SLIDE: 13
By Being Flexible

14. SLIDE: 14
THE IT CHALLENGE
Relational Databases
With ETL Sacrifice Agility,
Timeliness, and Cost
§ All future data needs must be predictable
§ Database changes require ETL re-writes
§ New SQL queries require re-indexing
§ Data security risk increases with silos
Agility and Flexibility Killers

15. SLIDE: 15
The Problem With the Relational Approach
The Organization Changes,
The Requirements Change,
The Source Data Changes
TAKE A CURRENT
STATE SNAPSHOT
DESIGN THE NEW
DATA MODEL
PERFORM ETL
CREATE THE
INDEXES
BUILD THE
APPLICATION
RESTART
PROCESS

16. SLIDE: 16
An Operational & Transactional
Enterprise NoSQL Database
Easy Data Load
Flexible Data Model
____________________________________
Data ingested as is (no ETL)
______________________________________________________
Structured data, unstructured data,
and metadata together
______________________________________________________
Adapts to changing data
and changing data structures
Fast Data Access
Ask Anything Universal Index
_____________________________________
Index once, query endlessly
______________________________________________________
Real-time, lightning fast
______________________________________________________
Open standards and APIs for
multi-model data – XML, JSON, RDF,
Geospatial, and text
100% Trusted
Enterprise Ready
_____________________________________
Most secure NoSQL database
______________________________________________________
True, full ACID transactions
______________________________________________________
Cloud neutral, run it anywhere
_____________________________________________________
Integrates with your environment
A New Approach

17. SLIDE: 17 3 August 2018© MARKLOGIC CORPORATION
An Operational and Transactional Enterprise NoSQL Database
That Integrates Data Better, Faster, With Less Cost
An Enterprise NoSQL Database
RELATIONAL
DATABASES
MAINFRAMES
FILE SYSTEMS
ANY OTHER
SOURCE
TRANSACTIONAL
APPLICATIONS
ANALYTICAL
APPLICATIONS
DOWNSTREAM
SYSTEMS
FLEXIBLE DATA MODEL
ASK ANYTHING UNIVERSAL INDEX
ENTERPRISE RELIABILITY
Enterprise NoSQL DB

18. SLIDE: 18
4. DELIVER
2. DISCOVER & VALIDATE
3. MODEL & HARMONIZE
ENTERPRISE NOSQL APPROACH TO
DATA INTEGRATION
1. LOAD AS IS
DATA GOVERNANCE
& DATA SECURITY

19. SLIDE: 19 3 August 2018© MARKLOGIC CORPORATION
Trusted to Run Your Organization
Exceeding the enterprise requirements to run mission-critical, operational systems
Must Deploy in Any Environment
Must Scale Up & Down
Must Be Secure
Must Never Lose Data
Must Be Cost Effective
Must Be Transactionally Consistent
Must Address Compliance Needs Must Handle Data Changes

20. SLIDE: 20
SCALABILITY & ELASTICITY
Horizontal scaling on commodity
hardware to reach massive scale
HA/DR
Automated failover and database
replication
ACID TRANSACTIONS
100% ACID transactions, not
eventually or strongly consistent
FLEXIBLE DEPLOYMENT
Write the application once, run it
anywhere
Advanced Enterprise Features
TIERED STORAGE
Minimize the cost of storage as
database expands
BITEMPORAL
Time-based audit trail to meet
compliance guidelines
REAL-TIME ALERTING
Unlimited alerts on new data at time
of ingestion
DATA SECURITY
Certified, granular security using
role-based access control

21. SLIDE: 21 3 August 2018© MARKLOGIC CORPORATION
Load As Is
Quickly ingest data from multiple silos
to form the basis for a 360 view of your organization
1. LOAD AS IS 2. DISCOVER & VALIDATE 3. MODEL & HARMONIZE 4. DELIVER

22. SLIDE: 22 3 August 2018© MARKLOGIC CORPORATION
Load Data As Is
ENTERPRISE NOSQL APPROACH
§ Load structured and unstructured data
§ Keep data and metadata together
§ Harmonize data in the database
§ Ability to skip:
– Upfront modeling
– ETL toolsOTHER METHODS
REST APIs, Java Client API, Node.js Client API, Java / .NET XCC
JSON
XML
SEMANTIC
DATA
GEOSPATIAL
DATA
MLCP
BINARY
Enterprise NoSQL DB

23. SLIDE: 23 3 August 2018© MARKLOGIC CORPORATION
The Document Model
ENTERPRISE NOSQL APPROACH
§ More natural and human-readable
§ Heterogeneous data is okay (schema-agnostic)
§ Query across data harmoniously (e.g., search
for zip code, “94111”, returns both records)
§ Group documents into collections (e.g., create
a collection for each source system)
§ Insert/update/delete documents in a single
transaction – even if it changes the schema
{
"Customer_ID”: 1001,
"Fname": "Paul",
"Lname": "Jackson",
"Phone": "415-555-1212",
"SSN": "123-45-6789",
"Addr": "123 Avenue ",
"City": "Someville",
"State": "CA",
"Zip": 94111
}
{
"Cust_ID" : 2001 ,
"Given_Name" : "Karen" ,
"Family_Name" : "Bender" ,
"Shipping_Address" : {
"Street" : "324 Some Road" ,
"City" : "San Francisco" ,
"State" : "CA" ,
"Postal" : "94111" ,
"Country" : "USA" } ,
"Billing_Address" : {
"Street" : "847 Another Ave" ,
"City" : "San Carlos" ,
"State" : "CA" ,
"Postal" : "94070" ,
"Country" : "USA" }
}
JSON
DOCUMENTS

24. SLIDE: 24 3 August 2018© MARKLOGIC CORPORATION
Documents Are
Everywhere
WHY DOCUMENTS?
§ Most data does not start or end as relational…
- Microservices, SOA, SPA messages
- ORM, JPA, JAXB, etc.
- Search indexing
- Custom SQL views or CSV
- Application code forensics

25. SLIDE: 25 3 August 2018© MARKLOGIC CORPORATION
Discover & Validate the Data
Provide ad-hoc, lightning fast query and search across integrated data
1. LOAD AS IS 2. DISCOVER & VALIDATE 3. MODEL & HARMONIZE 4. DELIVER

26. SLIDE: 26 3 August 2018© MARKLOGIC CORPORATION
Sophisticated Indexing
§ Ask Anything Universal Index – Indexes data,
structure on ingest
§ Advanced search – Relevance weighting,
proximity boosting, type-ahead suggestions,
co-occurrence, snippeting, and much more
§ More indexes for scalar (e.g., datetime,
numeric), geospatial, and semantic triple data
TERMS TERM LIST
123, 125, 129…
129, 130, 152 …
129, 130, 152 …
129, 130, 152 …
…
…
…
…

27. Indexing

28. SLIDE: 28 3 August 2018© MARKLOGIC CORPORATION
Hierarchical Data Model
§ Enterprise NoSQL document-centric database
- Supports any-structured data via hierarchical data model
- Stores compressed binary trees
Document
Title
Author Section
Section
Section
Section
Section
First Last
Metadata
Trade Cashflows
Party
Identifier Net Payment
Payment
Date
Party
Reference Payer
Party
Trade
ID
Payment
AmountReceiver
Party

29. SLIDE: 29
Schema-Agnostic
<SAR>
<title>Suspicious vehicle…Suspicious vehicle near airport
<date>
<type>
<threat>
2012-11-12Z
observation/surveillance
<type>suspicious activity
<category>suspicious vehicle
<location>
<lat>37.497075
<long>-122.363319
<subject>IRIID
<subject>IRIID
<predicate>
<predicate>
isa
value
<triple>
<triple>
<object>license-plate
<object>ABC 123
<description>A blue van…A blue van with license plate ABC 123 was observed parked behind the airport sign…
</title>
</date>
</type>
</type>
</category>
</threat>
</lat>
</long>
</location>
</subject>
</subject>
</predicate>
</predicate>
</object>
</object>
</description>
</SAR>
</triple>
</triple>
JSON and XML are self-describing:

30. SLIDE: 30
Schema-Agnostic
<SAR>
<title>
Suspicious vehicle…
<date>
2012-11-12Z
<type>
<threat>
suspicious activity
<category>
suspicious vehicle
<location>
<lat>
37.497075
<long>
-122.363319
<description>
A blue van…
<subject>
<subject>
<predicate>
<object>
IRIID
IRIID
isa
value
license-plate
ABC 123<predicate>
<object>
observation/surveillance
<type>
<triple>
<triple>

31. SLIDE: 31 3 August 2018© MARKLOGIC CORPORATION
Data Modeling & Harmonization
Leverage multi-model capabilities to create a unified view of the data, enrich it, and
further enhance it by adding semantic facts and relationships
1. LOAD AS IS 2. DISCOVER & VALIDATE 3. MODEL & HARMONIZE 4. DELIVER

32. SLIDE: 32 3 August 2018© MARKLOGIC CORPORATION
Data Harmonization
AGILE APPROACH
§ Harmonize data with different schemas in
order to create a common view:
E.g., Standardizing “male” and “female” versus, “m”
and “f” or “M” and “F”)
§ Only harmonize what you need to
§ Preserve and query data and metadata as is
§ Update the model later without re-ingesting
§ New data and schemas are okay
SCHEMA 2
SCHEMA 3
SCHEMA 1
ENVELOPE
PATTERN

33. SLIDE: 33 3 August 2018© MARKLOGIC CORPORATION
"Address": "123 Avenue ",
"address": "123 Avenue",
"location": "37.52 -122.25",
GEO UPDATE
Data Enrichment
AGIILE APPROACH
§ Provide additional context and meaning for
your data, without data modeling or ETL
§ Update/Annotate documents in place
§ Examples:
– Turn addresses into geospatial coordinates
– Tag entities in unstructured text (e.g.,
people, places, dates, etc.)
– Add active ingredient information to brand
name drug information

34. SLIDE: 34
Data Modeling in Parallel
AGILITY WITH ENTERPRISE NOSQL
TEAM 1
TEAM 2
TEAM 3
TEAM 10
SCHEMA v1.1
SCHEMA v2.0
Q1
SCHEMA v1.2
SCHEMA v1.3
SCHEMA v1.1
SCHEMA v3.0
Q2
SCHEMA v4.0
Q3
v2.1-1 v2.1-2
v2.2-1 v2.2-2 v2.2-3
v2.3-1
v3.1-1 v3.1-2
v3.2-1 v3.2-2
v3.3-1 v3.3-2
v2.x v3.x

35. SLIDE: 35
Data Modeling in Parallel
AGILITY WITH ENTERPRISE NOSQL
TEAM 1
TEAM 2
TEAM 3
TEAM 10
SCHEMA v1.1
SCHEMA v2.0
Q1
SCHEMA v1.2
SCHEMA v1.3
SCHEMA v1.1
SCHEMA v3.0
Q2
SCHEMA v4.0
Q3
v2.1-1 v2.1-2
v2.2-1 v2.2-2 v2.2-3
v2.3-1
v3.1-1 v3.1-2
v3.2-1 v3.2-2
v3.3-1 v3.3-2
v2.x v3.x

36. SLIDE: 36 3 August 2018© MARKLOGIC CORPORATION
Semantics
FEATURE
§ Identify and leverage data relationships
§ Add context and meaning to your 360 view
§ Deliver more intelligent search and business
insights
<suspect>A</suspect>
<suspect>B</suspect>
LOCATIONS
FIELD REPORT
FINANCES
INVESTIGATION
CRIMINAL
OPERATIONS

37. SLIDE: 37 3 August 2018© MARKLOGIC CORPORATION
"Order_ID": 8001,
"Customer_ID": 1001,
Semantics
FEATURE
§ Integrated Triple Store for managing facts and
relationships as semantic triples
§ Support for RDF and SPARQL standards
§ Scales to hundreds of billions of triples
§ Sub-second response via the Triple Index
§ Enrich and classify your data with partners:
OrderplacedCustomer
contains
Product
bought
TRIPLES
MULTI-MODEL: DOCUMENTS & TRIPLES TOGETHER
JSON, XML, & RDF

38. SLIDE: 38
Acme

39. SLIDE: 39
NBC SNL: Recommendation Engine with
Semantics
Talent
Kristen Wiig
Acted in
Episode 4
Anne Hathaway and Killers
Part of
Played
Character
Maharelle Sister
Season 34
Segment
The Lawrence Welk Show
Aired on
Date
10/4/08
Era
Acted in
Includes
Part of

40. SLIDE: 40 3 August 2018© MARKLOGIC CORPORATION
Deliver the Data
Run your organization on a 360 view
1. LOAD AS IS 2. DISCOVER & VALIDATE 3. MODEL & HARMONIZE 4. DELIVER

41. SLIDE: 41
§ Variety of Interfaces – Modern programming
languages familiar to developers
§ Application Services – Simplifies architecture
and speeds up development
§ Developer Productivity – Tools and
frameworks to speed delivery (e.g., Data Hub
Framework)
§ Adoption – Everyone just needs to know the
API
Fast, Powerful App
Development
ENTERPRISE NOSQL APPROACH
JAVASCRIPT
XQUERY
REST API
SPARQL
SQL
JAVA API
TRANSACTIONAL
APPS
ANALYTICAL
APPS
NODE.JS API
OPTIC API DOWNSTREAM
SYSTEMS
Enterprise NoSQL
DB

42. SLIDE: 42
TRANSACTIONAL ANALYTICAL
Enterprise NoSQL
DB

43. SLIDE: 43
An Operational Data Hub
ENTERPRISE ARCHITECTURE PATTERN