Download to read offline
Uploaded byMatt Turner
Operationalize Your Linked Data
This document summarizes a presentation about operationalizing linked data to transform industries using a multi-model approach. It discusses the importance of data and challenges with traditional data approaches. It promotes using linked data and semantic techniques to create flexible, contextual data layers that can be used across business units. Examples are provided of companies using these approaches for regulatory compliance, integrated digital delivery for auto repair, and open data sharing without data silos.
More Related Content
Similar to Operationalize Your Linked Data
More from Matt Turner
Operationalize Your Linked Data
- 1. 7 November 2018©MARKLOGIC CORPORATION Operationalize Your Linked Data To Transform Your Industry via a Multi-Model Approach Matt Turner CTO Media & Manufacturing November 7th, 2018 @matt_turner_nyc #MarkLogic
- 2.
- 3. Dr. Sven Fund CEOFullStopp 2013 Industrialize your data! Michel de Ru MarkLogic, 2018
- 4.
- 6. Alan Morrison Senior ResearchFellow PWC, 2018
- 7. Alan Morrison Senior ResearchFellow PWC
- 8.
- 9.
- 10. Do domestic dogsinterpret pointing as a command? Animal Cognition (2012): 1-12 , November 09, 2012 By Scheider, Linda; Kaminski, Juliane; Call, Josep; Tomasello, Michael DOGS GET Context!
- 11. SLIDE: 11 7November 2018© MARKLOGIC CORPORATION Machines Don’t Get Context . . . Manu Sporny Founder/CEO - Digital Bazaar, Inc. http://www.cambridgesemantics.com/semantic-university/what-is-linked-data
- 12. SLIDE: 12 7November 2018© MARKLOGIC CORPORATION Traditional Approach Strips Context Define everything in rows and columns - Narrows to specific purpose Fix categories into hierarchies - Strips general context Result: Data can’t be used across all parts of the business “Running on opinion not data” Title ProductionDate Category AssetType Length Film1 3/1/14 Feature HD Master 2:40 Show1 6/4/13 Series HD720 0:40 Film2 6/4/05 Feature Archive 1:55 Category Feature Series Action Drama Comedy Documentary … Cable Broadcast Drama Comedy … Action Drama Family Documentary …
- 13. SLIDE: 13 7November 2018© MARKLOGIC CORPORATION Schema Flexibility with NoSQL What if you didn’t need to define everything up front? To extend and customize you can just add elements A start towards solving the problem … Asset Title Asset Type Date Production Date Editing Date Release Date International Date Categories
- 14. SLIDE: 14 7November 2018© MARKLOGIC CORPORATION Enter Linked Data! Manu Sporny Founder/CEO - Digital Bazaar, Inc. http://www.cambridgesemantics.com/semantic-university/what-is-linked-data
- 15. SLIDE: 15 7November 2018© MARKLOGIC CORPORATION NoSQL and Semantics: Using CONTEXT to Create Data Layer Asset Title HD Master Dates Production Date Editing Date Release Date International Date <Character> London <place> <time period> Film Series <collection> Actor1 <performer> Film1 <work> is lives in lived in appears in is ais part of Crime-Thriller <category> played
- 16. SLIDE: 16 7November 2018© MARKLOGIC CORPORATION MarkLogic Operational Data Hub ENTERPRISE ARCHITECTURE PATTERN
- 17. Semantic Data Layerin Action
- 18. SLIDE: 18 7November 2018© MARKLOGIC CORPORATION ________________________________________________ Operational Data Hub(s) for Regulatory Compliance Success after multi-year traditional approach failure Real time analysis of trades, met the critical MiFID II Tackled GDPR consent with PROV-O ___________________________________________________________________________________________________________________ “We need the ability to respond quickly to changing regulatory requirements. We chose MarkLogic because trade data is notoriously difficult to handle in relational databases.” Jaap Boersma Principal Architect ABN AMRO
- 19. SLIDE: 19 7November 2018© MARKLOGIC CORPORATION Trace How Customer Data Is Used 10:14 Started At 10:15 Ended At Message 3527f751s Derived From PROV-O Ontology W3C Standard Implement via Graph/RDF Track ALL required Provenance Full lineage ENTITY Ingest Module Generated by ACTIVITY Branch User 34156 Attributed to AGENT Harrys Profile
- 20. SLIDE: 20 7November 2018© MARKLOGIC CORPORATION Record and Manage Actual Usage Breach Impacts Data Usage Data Access PII Locations Harrys Profile Consent Portal Generated by Decline Consent Purpose OfHcu1235 04-12-2017 18:00 Attributed to Started at Harry Has Profile Marketing System 04-12-2017 18:00 18-12-2017 18:02 Used By Ended at Started at Campaign Purpose Of
- 21. SLIDE: 21 7November 2018© MARKLOGIC CORPORATION Operational Data Hub(s) Regulatory Compliance Challenges ____________________________________ Traditional data integration too rigid and slow to meet MiFID II deadline Tackled GDPR permissions Needed flexibility for future regulatory changes Ability to leverage data for multiple purposes Security, audit, and high availability requirements PRICING ENGINE POSITION DATA MARKET DATA REFERENCE DATA DOWNSTREAM APPS - REGULATORS - PUBLISHING APP - OTHER SYSTEMS MONITORING APPS - COMPLIANCE - OPERATIONS HADOOP STORAGE MULTI-MODEL ACID TRANSACTIONS ADVANCED SECURITY ELASTICITY SEARCH & INDEXING ORDER DATA MUREX TRADE DATA
- 22. SLIDE: 22 7November 2018© MARKLOGIC CORPORATION Integrated Digital Delivery for Streamlined Auto Repair Challenges ____________________________________ Structured, unstructured data Needed advanced search Needed scalability for growing data Multi-device delivery (desktop, tablet, Snap-On ProDemand device) MITCHELL1 PRODEMAND EXPERT ADVICE REPAIR ORDERSREPAIR INFO
- 23.
- 25. SLIDE: 25 7November 2018© MARKLOGIC CORPORATION
- 26. SLIDE: 26 7November 2018© MARKLOGIC CORPORATION Mitchell1 Data Hub Shop Manuals Repair Schedules Diagnostic Data Parts Ordering Expert Feedback Additional Data DIAGNOSTIC SYSTEMS FUTURE DATA FLOWS PRODEMAND PRODUCTS
- 27. Our Mission makingavailable best-of-breedtechnologies to conserve ecosystemsand protect wildlife
- 28. Strengthening analytical capabilitiesSTEP 2 content impression
- 29. Elephants killed byarrow vs firearms STEP 2
- 30. Warthog bycatch ofcharcoalers? STEP 2
- 31. Want to help? FUNDOUR WORK03 ● Donation ● Investment HIREUS02 WORK WITH US01 ● Products ● Services ● Experts ● Products ● Services ● Experts
- 32. SLIDE: 32 7November 2018© MARKLOGIC CORPORATION Sharing Without Silos
- 34. 7 November 2018©MARKLOGIC CORPORATION Thank You! Matt Turner, MarkLogic CTO Media & Manufacturing @matt_turner_nyc #MarkLogic
- 35. SLIDE: 35 7November 2018© MARKLOGIC CORPORATION Resources • Importance of Data • Integrated Publishing, Sven Fund: https://onlinelibrary.wiley.com/doi/abs/10.1087/20130111 • Rich Data, Poor Data, Shelly Palmer: https://www.shellypalmer.com/2016/05/rich-data-poor-data- data-rich-data-poor-data-middle-class-not/ • Industrialze your Data:, Michel de Ru: https://www.slideshare.net/MicheldeRu/industrializing-data • Semantic Data Layer • Alan Morrison Keynote - https://www.slideshare.net/AlanMorrison/collapsing-the-it- stack-clearing-a-path-for-ai-adoption?from_action=save • Plus recording of the talk (+18min) -> https://www.facebook.com/fhstp/videos/308669336596727 / • Why Ontology – Kurt Kagel - https://www.forbes.com/sites/cognitiveworld/2018/07/20/why- ontology-will-be-a-big-word-in-your-companys-future/ • AI + Graph: https://www.zdnet.com/article/google-ponders-the- shortcomings-of-machine-learning/ • Pre-print article: https://arxiv.org/abs/1806.01261 • Semantics Primer: http://www.cambridgesemantics.com/semantic- university/what-is-linked-data • Sensing Clues: https://sensingclues.com/ • ABM Amro + Sensing Clues at MarkLogic 360: https://www.youtube.com/watch?v=RzJymikvrKs • Mitchell1 Webinar: https://www.marklogic.com/resources/mitchell1s- celebrates-century-success-connecting-customers-data/ • Dogs and Chimps • Pointing Study: https://doglab.shh.mpg.de/pdf/Scheider_et_al_2013_interpret_ pointing_as_a_command.pdf • Who’s Smarter, Matt Turner: https://www.marklogic.com/blog/dog-chimp/
- 36. SLIDE: 36 7November 2018© MARKLOGIC CORPORATION Last Word: Open Data Taylor Swift
Editor's Notes
- #2 Hi everyone, I’m Matt Turner, CTO for Media & Manufacturing. That means I take care of all our customers that make the great things you all know and love And I’ve very glad to be here today to talk about the impact of Linked Data and in particular how investing in this data can help transform your industry.
- #3 I’m going to start with something that seems obvious … the importance of data But, before I get into this, I do want to remind everyone that going back even 5 years, this was not an obvious topic. We were talking about apps and especially the mobile experience and the new wave of BI … but not about the data itself as a topic
- #4 But there were people having the conversation in their industry We do a lot of work with publishers and one of the primary voices for change has been Dr. Sven Fund – then the CEO of DeGruyter a publisher over in Germany. He wrote what I think of as a battle plan for the modern publisher called integrating publishing. It’s a data driven approach to rethinking every part of the business around using data across every part of the business. From planning what content to invest in, to creating it and tailoring it to knowing how it impacts your customers, data can and should play a role … and Sven laid out the plan to get publishers to that point. This was quite a change for an industry still just thinking about content. Shelly Palmer is another voice that was early with a message about data. He worked mostly within Media but his message was to every organization highlighting how the game has changed. He says “Data Rich or Data Poor” that is the ONLY game. Every company is now competing on the battleground of data. Its not your revenue, your number of customers or their engagement. It’s the data you gather that actually matters. What’s more, you aren’t competing against what you think of as your competitors. Its Google, Apple, Facebook … and way above all of them Amazon. Shelly says this to bring people’s attention to the importance of data. And he’s not alone – he is joined by my colleague Michel de Ru. Michel works across a number of industries and at the MarkLogic 360 event last year he issued a call to arms: Industrialize your data! You invest in your processes, your machinery, your people and take care of your capital. And you need to do the same thing your data. Think about how you manage it and, just like your machinery and other assets, industrialize how you deal with it
- #5 And they aren’t alone. Who has heard this phrase Data is the new Oil? Its everywhere … there is even someone saying it’s the not the new oil it the new nuclear. I guess because it keeps delivering value forever? In fact there is so much about this, if you search for Data is the new oil infographic you get 13 million hits! This is my favorite – see the data in the ground – just pump it out and – presto – you get your value! Right? Its that easy, right?
- #6 And they aren’t alone. Who has heard this phrase Data is the new Oil? Its everywhere … there is even someone saying it’s the not the new oil it the new nuclear. I guess because it keeps delivering value forever? In fact there is so much about this, if you search for Data is the new oil infographic you get 13 million hits! This is my favorite – see the data in the ground – just pump it out and – presto – you get your value! Right? Its that easy, right?
- #7 And on this topic, we are just starting to hear from the experts. I hope Alan Morrison as one of these visionaries. He gave a keynote at the Semantic conference in August that was a real call to arms for everyone in THIS room to evangelize that you do need more than just data. He was specifically talking about the vast gap between the vision of a unified IT stack and being able to leverage AI and the reality of the many silos of applications. He specifically is looking at who is out there paying attention to this problem and he put up this slide – the top 10 companies in the world And of them, fully 9 are doing more than just collecting data. They are investing in Linked Data – creating knowledge graphs and connecting their data to realize its value He isn’t alone – Kurt Cagle makes a bold statement about the rise of Ontology will be a critical business advantage. And then there is this paper about the state of AI. Alan also goes into this in his talk – AI without the meaning and connections in the data is just going to fall short. Specifically they make this statement – that nearly every problem comes down to graphs of relationships among entities!
- #8 And on this topic, we are just starting to hear from the experts. I hope Alan Morrison as one of these visionaries. He gave a keynote at the Semantic conference in August that was a real call to arms for everyone in THIS room to evangelize that you do need more than just data. He was specifically talking about the vast gap between the vision of a unified IT stack and being able to leverage AI and the reality of the many silos of applications. He specifically is looking at who is out there paying attention to this problem and he put up this slide – the top 10 companies in the world And of them, fully 9 are doing more than just collecting data. They are investing in Linked Data – creating knowledge graphs and connecting their data to realize its value He isn’t alone – Kurt Cagle makes a bold statement about the rise of Ontology will be a critical business advantage. And then there is this paper about the state of AI. Alan also goes into this in his talk – AI without the meaning and connections in the data is just going to fall short. Specifically they make this statement – that nearly every problem comes down to graphs of relationships among entities!
- #9 So what’s going on here? Well to get to the heart of this, I want to ask you ONE question. Get ready because I’m going to ask you to raise your hand and take this very seriously
- #10 Who’s smarter, a dog or a chimpanzee? --- OK – everybody has to vote -- And it was a trick. Of course chimpanzees are much smarter. They can drive cars, talk in sign language … they are way way smarter .. But … there is something a dog can do that a chimp just can’t do
- #11 And that is understand context. If you hide a treat and then point to it, the chimp will totally ignore you and randomly guess. No matter what you do, the chimp just doesn’t care what you think. But if you do the same thing with the dog, the dog will look at you to understand what is going on. In fact they will look at the right side of your face which is how we understand emotions and act on what they see. This means you can point to the treat and they will go right to it. Puppies will do it. Dogs are so good at this, you can just move your eyes. So what does this all mean for data?
- #12 Well, machines don’t get context! They are the chimps of the technology world they can tell you it’s a link or a picture on that page … but they can’t tell you that they fit together. Or what its all about!
- #13 When you take this to the world of data, and in particular the data layer that can run your business this is what you get – traditional data structures that just fall short You have to define everything up front – all your data and everything your organization does … And then categorize it. In no way will this work – you will end up stripping off context sometimes in layer. You can’t share this data across your organization and so you get what Alan was talking about in terms of the multiple layers of appliations and data One of our customers talks about the result of all this changing of data as operating on opinion, not data!
- #14 We at MarkLogic started to address this with a document data model. This foundation of NoSQL means that you can much more flexibly store the data. You didn’t have to define it all up front and now you could adapt as things changed. But it was still cumbersome to keep track of meaning and context … more elements could be very ticky
- #15 Enter Linked Data – this concept of describing the linkages in data – and specifically of using triples or RDF as an additional data mode, is a way to bring, with data, what the machine is lacking You can now actually describe the concepts around the data. And what is more important – you can also describe the source, the provenance and even the usage of the data.
- #16 Combined, these two data models are key to creating that data layer and enabling you to actually make data the foundation of your business This is critical – because there is a balance here. In the world of semantics and linked data, there are huge gains to be made in creating ontologies that match the real world and then linking data to those ontologies But there is also a role for just a document – things that belong just to the document like dates, titles and of course the actual –these are perfectly OK in the world of XML. And also in this model the connecting triple is part of the document – also making it a graph that enables you to have data integrity.
- #17 At MarkLogic we’ve been using this combined model in an architectural pattern called an Operational Data Hub. This pattern details (and I’m not going to go throught it all) taking in data, and then with many different approaches, curating that data and creating the context around it. You end up with documents and triples that can provide that single view of the data that, as Alan and others talk about, can be the foundation for a data driven organization
- #18 Lets take a look at how these patterns are letting organizations leverage their data
- #19 Lets start in the complex regulatory compliance sector of financial services ABN Amro wanted to take a new approach to compliance. Instead of building data sets for each issue and process they wanted to create an platform for regulatory compliance so they could respond to future needs This required them to think about the problem differently – create data that then be used in what they called multiple compliance schemas. They also realized this data would be a powerful asset across the compay. They did this first to meet the tradestore regularion MiFID II and then created another hub for GDPR This projects all used semantics to describe the entities and their relationships in the bank But really be able to use the data, you need to understand the provenance of the data
- #20 To do this they did map all the entities But the also added Prov-O to the picture to record that data lineage. Now they know not just about customers and their interactions inside the bank and on their own They also know where that data came from
- #21 This lets them see a much richer set of data – tieing together internal and external systems With this data they can now delvier the much more complete view of the customer required by GDPR, describe the entities correctly AND know the source and all the details about that data Keep in mind this also follows the document and semantics route – so that profile is also in the system enabling them to see the details of the customer
- #22 This platform delivers that vision of universal enterprise data – or the semantic data layer as Alan talks about it. Focused on compliance, it is an engine for handling compliance data rather than just a single solution But it is now also a view of data across the organization that is a considerable asset to the bank
- #23 In addition to improving your own organization, you can also use this approach to improve the customer experience. Mitchell1 is in the car repair business. They provide the tools to garages to help the shops and the mechanics make the repair experience better.
- #24 They stared with the acutal information on how to make the repair. This information went from paper to digital and is exploding in complexity as cars get more complex. But they also provide the systems that run the shop – scheduling, diagnostics, parts ordering, the actual repairs and billing. One cool thing is that they also get tips from the mechanics about how to fix the car To put this data into action, they created a data hub that linked all this data And the link is this – every part on every car that is sold going back 30 years or more! They invested in ontology development to create this foundational data – how parts, compnents and systems fit together And then spend 2+ years creating the data set and linking the data.
- #25 Using this, they are trying to make the mechanics job better – for instance knowing what is wrong almost as soon as they see the car. This is a composite screen that shows the parts that you will probably have to order from a set of error codes. This saves everyone time getting to answers And as cars get connected, this is starting to happen outside of the shop - cars sending codes and getting ready for repair and maybe even making appointments You can only do this if all the data is connected
- #26 And they are also giving information about the long term – for instance what is likely to break on my car. This used to be in the heads of experts … but now it done with data – Don’t forget this is your car, with your exact specification
- #27 This data hub lets Mitchell1 make car repair a much better experience
- #28 And finally lets talk about using data to make a difference Sensing clues is an organization here in the Netherlands dedicated to conversation and specifically protecting endangered species They are focused on the interactions between people and these animals This is both when people go into their environment -> and this is often very bad for instance poachers But also protecting the animals when they go into human environments
- #29 To help with this mission they have collected a lot of data. Incident reports, field notes, signal data and even pictures and videos And they have integrated all this data so they can look at an area. Part of the project is to get information to the right people so this lets them send updates and warnings to rangers and other workers when they are actually out in the field – or actually going out since there isn’t a lot of connectivity This also allows them to undersnad what is happening. And to do this they created a semantic layer that created concepts and linkd the data
- #30 This helps them gain insights into what is happening – this is a zoom into the data for a specific area And it shows the different types of interactions – the professional incidents, likely poachers, and then where arrows are used which is a very different case They can then go back to the map and be prepared for what the situation is and also use this information for different programs to prevent both types of incidents
- #31 And this also helps them understand other incidents. For instance why were warthogs showing up? Does this mean there are poachers? Well actually because of the context of the data, they can tell that these are not usually poachers – they are people coming in to the area and camping out for 6-7 days and making charcoal. Not as high a priority and, some good news in the data
- #32 We volunterred our efforts and as they continue to develop ther data layer to make an impact on animals I’m sure they could use more help
- #33 One more factor in creating this type of semantic data layer is that in addition to using the data for your internal uses, you can share it. This is what Springer does with their rich database of scientific information … they have the actual products SprinterLink and then they just take the UI off and offer APIs for text mining and data access.
- #34 So lets take a look back – you can impact your organization, help your customers and even help the world if you have your data together So lets go back to that infographic. Maybe it is actually just so easy – if you add a few things. First consider the context of the data where you find it. And capture all of that Then lets think about the usage and the different contexts everyone will need when the access it! Using this as a guide, lets think about putting a data hub right here in the middle. This won’t be the only source of data … and even for a single ‘well’ you need a place to collect the data and make it universally accessible. I think if you do this – if you apply the principles of linked data to your operational data … well maybe you can bring some of those doggie smarts to your organization. Thank you!