The document describes an architecture for semantically integrating enterprise data lakes. It proposes a knowledge graph that links metadata, data models and key performance indicators to provide a common meaning for data. Raw data is stored in a data lake and ingested from various sources. A metadata layer captures dataset metadata, ontologies and integration rules to link disparate data. An interface allows users to access consolidated views generated by executing queries on Hadoop. The process involves cataloging, discovering, lifting, linking and validating datasets to integrate them based on rules into the knowledge graph.
Semantic technologies offer a wide range of benefits in an increasing number of application fields such as data management, business intelligence, machine learning etc.
from Christian Opitz | Head of innovation at Netresearch GmbH & Co. KG
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Coherent and consistent tracking of provenance data and in particular update history information is a crucial building block for any serious information system architecture.
Marvin Frommhold | AKSW, Universität Leipzig
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
A large part of the free knowledge existing on the Web is available as heterogeneous, semi-structured data, which is only weakly interlinked and in general does not include any semantic classification.
Michael Krug | Technische Universität Chemnitz
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Large corporations have to master vast amounts of heterogeneous data in order to stay competitive. While existing approaches have attempted to consolidate and manage the data by forcing it into a single shared data model, data lakes recently emerged that instead provide a central storage point for holding all data sets in their original form. In this talk, we present eccenca Corporate Memory, which extends the data lake paradigm with a semantic integration layer for managing diverse, but semantically enriched data. In addition to that, we depict our vision for public / private data co-evolution and how we research this topic in the joint project Linked Enterprise Data Services (LEDS) together with the University of Leipzig and other partners.
from René Pietzsch | Head of Product Management, Eccenca
and Dr. Michael Martin | AKSW, Universität Leipzig, LEDS Project
Presentation at Sachsentag der Angewandten informatik 2016 in Leipzig in the context with the results of the LEDS project
Semantic technologies offer a wide range of benefits in an increasing number of application fields such as data management, business intelligence, machine learning etc.
from Christian Opitz | Head of innovation at Netresearch GmbH & Co. KG
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Coherent and consistent tracking of provenance data and in particular update history information is a crucial building block for any serious information system architecture.
Marvin Frommhold | AKSW, Universität Leipzig
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
A large part of the free knowledge existing on the Web is available as heterogeneous, semi-structured data, which is only weakly interlinked and in general does not include any semantic classification.
Michael Krug | Technische Universität Chemnitz
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Large corporations have to master vast amounts of heterogeneous data in order to stay competitive. While existing approaches have attempted to consolidate and manage the data by forcing it into a single shared data model, data lakes recently emerged that instead provide a central storage point for holding all data sets in their original form. In this talk, we present eccenca Corporate Memory, which extends the data lake paradigm with a semantic integration layer for managing diverse, but semantically enriched data. In addition to that, we depict our vision for public / private data co-evolution and how we research this topic in the joint project Linked Enterprise Data Services (LEDS) together with the University of Leipzig and other partners.
from René Pietzsch | Head of Product Management, Eccenca
and Dr. Michael Martin | AKSW, Universität Leipzig, LEDS Project
Presentation at Sachsentag der Angewandten informatik 2016 in Leipzig in the context with the results of the LEDS project
This presentation, hold during Semantcs conference, introduce Ontos' current achievement towards a Streaming-based Text Mining solution by using Deep Learning and Semantic Web technologies.
Knowledge graphs - it’s what all businesses now are on the lookout for. But what exactly is a knowledge graph and, more importantly, how do you get one? Do you get it as an out-of-the-box solution or do you have to build it (or have someone else build it for you)? With the help of our knowledge graph technology experts, we have created a step-by-step list of how to build a knowledge graph. It will properly expose and enforce the semantics of the semantic data model via inference, consistency checking and validation and thus offer organizations many more opportunities to transform and interlink data into coherent knowledge.
How to Reveal Hidden Relationships in Data and Risk AnalyticsOntotext
Imagine risk analysis manager or compliance officer who can discover easily relationships like this: Big Bucks Café out of Seattle controls My Local Café in NYC through an offshore company. Such discovery can be a game changer if My Local Café pretends to be an independent small enterprise, while recently Big Bucks experiences financial difficulties.
II-SDV 2016 Patrick Beaucamp - Data Science with R and Vanilla AirDr. Haxel Consult
Companies are facing challenges to analyze Big data databases & data Lakes in uncertain technologies environment, in order to provide accurate analysis and build forecast model.
In a context of budgdet constraints, the R project is a reliable alternative to legacy commmercial software to develop and deploy business analytics data model. R has a worldwide recognation and fast adoption from companies everywhere in the world. Together with Vanilla Air, everybody can start now a Data Science project and share his analysis in an instant.
This session comes along with a presentation of the Vanilla Air, a new "cloud / on premise" Data Science platform to develop & deploy R data model at enterprise level
Enabling Low-cost Open Data Publishing and ReuseMarin Dimitrov
In the space of just a few years we’ve seen the transformational power of open data; both for transparency and accountability in public data, and efficiency and innovation with businesses in private data. In its first year, institutions and individuals throughout Europe have supported public sector bodies in releasing data and numerous start-ups, developers and SMEs in reusing this data for economic benefit.
However, we are still at the beginning of the open data movement, and there is still more that can be done to make open data simpler to use and to make it available to a wider audience.
The core goal of the DaPaaS project is to provide a Data- and Platform-as-a-Service environment, where 3rd parties (such as governmental organisations, SMEs, developers and larger companies) can publish and host both data sets and data-intensive applications, which can then be accessed by end-user applications in a cross-platform manner. You can find out more about DaPaaS on the detailed about page.
Essentially, DaPaaS aims to make publishing, consumption, and reuse of open data, as well as deploying open data applications, easier and cheaper for SMEs and small public bodies which otherwise may not have sufficient technical expertise, infrastructure and resources required to do so.
see also http://www.slideshare.net/eswcsummerschool/wed-roman-tutopendatapub-38742186
Open Data and News Analytics Demo from the 4th Sofia Open Data & Linked Data meetup
http://www.meetup.com/Sofia-Open-Data-Linked-Data-Meetup/events/228747999/
Mar'2016, Sofia | BG
The Power of Semantic Technologies to Explore Linked Open DataOntotext
Atanas Kiryakov's, Ontotext’s CEO, presentation at the first edition of Graphorum (http://graphorum2017.dataversity.net/) – a new forum that taps into the growing interest in Graph Databases and Technologies. Graphorum is co-located with the Smart Data Conference, organized by the digital publishing platform Dataversity.
The presentation demonstrates the capabilities of Ontotext’s own approach to contributing to the discipline of more intelligent information gathering and analysis by:
- graphically explorinh the connectivity patterns in big datasets;
- building new links between identical entities residing in different data silos;
- getting insights of what type of queries can be run against various linked data sets;
- reliably filtering information based on relationships, e.g., between people and organizations, in the news;
- demonstrating the conversion of tabular data into RDF.
Learn more at http://ontotext.com/.
On-Demand RDF Graph Databases in the CloudMarin Dimitrov
slides from the S4 webinar "On-Demand RDF Graph Databases in the Cloud"
RDF database-as-a-service running on the Self-Service Semantic Suite (S4) platform: http://s4.ontotext.com
video recording of the talk is available at http://info.ontotext.com/on-demand-rdf-graph-database
Semantic Graph Databases: The Evolution of Relational DatabasesCambridge Semantics
In this webinar, Barry Zane, our Vice President of Engineering, discusses the evolution of databases from Relational to Semantic Graph and the Anzo Graph Query Engine, the key element of scale in the Anzo Smart Data Lake. Based on elastic clustered, in-memory computing, the Anzo Graph Query Engine offers interactive ad hoc query and analytics on datasets with billions of triples. With this powerful layer over their data, end users can effect powerful analytic workflows in a self-service manner.
Using the Semantic Web Stack to Make Big Data SmarterMatheus Mota
This presentation will discuss how just a few parts of the Semantic Web Cake can already boost your analytics by making your (big) data smarter and even more connected.
Linked Data Experiences at Springer NatureMichele Pasin
An overview of how we're using semantic technologies at Springer Nature, and an introduction to our latest product: www.scigraph.com
(Keynote given at http://2016.semantics.cc/, Leipzig, Sept 2016)
slides from our talk "Low-Cost Open Data as-a-service" from the Semantic Web Developers workshop of ESWC'2015 (full paper: http://ceur-ws.org/Vol-1361/paper7.pdf)
Smarter content with a Dynamic Semantic Publishing PlatformOntotext
Personalized content recommendation systems enable users to overcome the information overload associated with rapidly changing deep and wide content streams such as news. This webinar discusses Ontotext’s latest improvements to its Dynamic Semantic Publishing (DSP) platform NOW (News on the Web). The Platform includes social data mining, web usage mining, behavioral and contextual semantic fingerprinting, content typing and rich relationship search.
With the increasing adoption of NoSQL data base systems like MongoDB or CouchDB more and more applications store structured data according to a non-relational, document oriented model. Exposing this structured data as Linked Data is currently inhibited by a lack of standards as well as tools and requires the implementation of custom solutions. While recent efforts aim at expressing transformations of such data models into RDF in a standardized manner, there is a lack of approaches which facilitate SPARQL execution over mapped non-relational data sources. With SparqlMap-M we show how dynamic SPARQL access to non-relational data can be achieved. SparqlMap-M is an extension to our SPARQL-to-SQL rewriter SparqlMap that performs a (partial) transformation of SPARQL queries by using a relational abstraction over a document store. Further, duplicate data in the document store is used to reduce the number of joins and custom optimiza-tions are introduced. Our showcase scenario employs the Berlin SPARQL Benchmark (BSBM) with different adap-tions to a document data model. We use this scenario to demonstrate the viability of our approach and compare it to different MongoDB setups and native SQL.
Jörg Unbehauen | AKSW, Universität Leipzig
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Collaboration is one of the most important topics regarding the evolution of the World Wide Web and thus also for the Web of Data. In scenarios of distributed collaboration on datasets it is necessary to provide support for multiple different versions of datasets to exist simultaneously, while also providing support for merging diverged datasets. In this paper we present an approach that uses SPARQL 1.1 in combination with the version control system Git, that creates commits for all changes applied to an RDF dataset containing multiple named graphs. Further the operations provided by Git are used to distribute the commits among collabora-tors and merge diverged versions of the dataset. We show the advantages of (public) Git repositories for RDF datasets and how this represents a way to collaborate on RDF data and consume it. With SPARQL 1.1 and Git in combination, users are given several opportunities to participate in the evolution of RDF data.
Natanael Arndt | AKSW, Universität Leipzig
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
This presentation, hold during Semantcs conference, introduce Ontos' current achievement towards a Streaming-based Text Mining solution by using Deep Learning and Semantic Web technologies.
Knowledge graphs - it’s what all businesses now are on the lookout for. But what exactly is a knowledge graph and, more importantly, how do you get one? Do you get it as an out-of-the-box solution or do you have to build it (or have someone else build it for you)? With the help of our knowledge graph technology experts, we have created a step-by-step list of how to build a knowledge graph. It will properly expose and enforce the semantics of the semantic data model via inference, consistency checking and validation and thus offer organizations many more opportunities to transform and interlink data into coherent knowledge.
How to Reveal Hidden Relationships in Data and Risk AnalyticsOntotext
Imagine risk analysis manager or compliance officer who can discover easily relationships like this: Big Bucks Café out of Seattle controls My Local Café in NYC through an offshore company. Such discovery can be a game changer if My Local Café pretends to be an independent small enterprise, while recently Big Bucks experiences financial difficulties.
II-SDV 2016 Patrick Beaucamp - Data Science with R and Vanilla AirDr. Haxel Consult
Companies are facing challenges to analyze Big data databases & data Lakes in uncertain technologies environment, in order to provide accurate analysis and build forecast model.
In a context of budgdet constraints, the R project is a reliable alternative to legacy commmercial software to develop and deploy business analytics data model. R has a worldwide recognation and fast adoption from companies everywhere in the world. Together with Vanilla Air, everybody can start now a Data Science project and share his analysis in an instant.
This session comes along with a presentation of the Vanilla Air, a new "cloud / on premise" Data Science platform to develop & deploy R data model at enterprise level
Enabling Low-cost Open Data Publishing and ReuseMarin Dimitrov
In the space of just a few years we’ve seen the transformational power of open data; both for transparency and accountability in public data, and efficiency and innovation with businesses in private data. In its first year, institutions and individuals throughout Europe have supported public sector bodies in releasing data and numerous start-ups, developers and SMEs in reusing this data for economic benefit.
However, we are still at the beginning of the open data movement, and there is still more that can be done to make open data simpler to use and to make it available to a wider audience.
The core goal of the DaPaaS project is to provide a Data- and Platform-as-a-Service environment, where 3rd parties (such as governmental organisations, SMEs, developers and larger companies) can publish and host both data sets and data-intensive applications, which can then be accessed by end-user applications in a cross-platform manner. You can find out more about DaPaaS on the detailed about page.
Essentially, DaPaaS aims to make publishing, consumption, and reuse of open data, as well as deploying open data applications, easier and cheaper for SMEs and small public bodies which otherwise may not have sufficient technical expertise, infrastructure and resources required to do so.
see also http://www.slideshare.net/eswcsummerschool/wed-roman-tutopendatapub-38742186
Open Data and News Analytics Demo from the 4th Sofia Open Data & Linked Data meetup
http://www.meetup.com/Sofia-Open-Data-Linked-Data-Meetup/events/228747999/
Mar'2016, Sofia | BG
The Power of Semantic Technologies to Explore Linked Open DataOntotext
Atanas Kiryakov's, Ontotext’s CEO, presentation at the first edition of Graphorum (http://graphorum2017.dataversity.net/) – a new forum that taps into the growing interest in Graph Databases and Technologies. Graphorum is co-located with the Smart Data Conference, organized by the digital publishing platform Dataversity.
The presentation demonstrates the capabilities of Ontotext’s own approach to contributing to the discipline of more intelligent information gathering and analysis by:
- graphically explorinh the connectivity patterns in big datasets;
- building new links between identical entities residing in different data silos;
- getting insights of what type of queries can be run against various linked data sets;
- reliably filtering information based on relationships, e.g., between people and organizations, in the news;
- demonstrating the conversion of tabular data into RDF.
Learn more at http://ontotext.com/.
On-Demand RDF Graph Databases in the CloudMarin Dimitrov
slides from the S4 webinar "On-Demand RDF Graph Databases in the Cloud"
RDF database-as-a-service running on the Self-Service Semantic Suite (S4) platform: http://s4.ontotext.com
video recording of the talk is available at http://info.ontotext.com/on-demand-rdf-graph-database
Semantic Graph Databases: The Evolution of Relational DatabasesCambridge Semantics
In this webinar, Barry Zane, our Vice President of Engineering, discusses the evolution of databases from Relational to Semantic Graph and the Anzo Graph Query Engine, the key element of scale in the Anzo Smart Data Lake. Based on elastic clustered, in-memory computing, the Anzo Graph Query Engine offers interactive ad hoc query and analytics on datasets with billions of triples. With this powerful layer over their data, end users can effect powerful analytic workflows in a self-service manner.
Using the Semantic Web Stack to Make Big Data SmarterMatheus Mota
This presentation will discuss how just a few parts of the Semantic Web Cake can already boost your analytics by making your (big) data smarter and even more connected.
Linked Data Experiences at Springer NatureMichele Pasin
An overview of how we're using semantic technologies at Springer Nature, and an introduction to our latest product: www.scigraph.com
(Keynote given at http://2016.semantics.cc/, Leipzig, Sept 2016)
slides from our talk "Low-Cost Open Data as-a-service" from the Semantic Web Developers workshop of ESWC'2015 (full paper: http://ceur-ws.org/Vol-1361/paper7.pdf)
Smarter content with a Dynamic Semantic Publishing PlatformOntotext
Personalized content recommendation systems enable users to overcome the information overload associated with rapidly changing deep and wide content streams such as news. This webinar discusses Ontotext’s latest improvements to its Dynamic Semantic Publishing (DSP) platform NOW (News on the Web). The Platform includes social data mining, web usage mining, behavioral and contextual semantic fingerprinting, content typing and rich relationship search.
With the increasing adoption of NoSQL data base systems like MongoDB or CouchDB more and more applications store structured data according to a non-relational, document oriented model. Exposing this structured data as Linked Data is currently inhibited by a lack of standards as well as tools and requires the implementation of custom solutions. While recent efforts aim at expressing transformations of such data models into RDF in a standardized manner, there is a lack of approaches which facilitate SPARQL execution over mapped non-relational data sources. With SparqlMap-M we show how dynamic SPARQL access to non-relational data can be achieved. SparqlMap-M is an extension to our SPARQL-to-SQL rewriter SparqlMap that performs a (partial) transformation of SPARQL queries by using a relational abstraction over a document store. Further, duplicate data in the document store is used to reduce the number of joins and custom optimiza-tions are introduced. Our showcase scenario employs the Berlin SPARQL Benchmark (BSBM) with different adap-tions to a document data model. We use this scenario to demonstrate the viability of our approach and compare it to different MongoDB setups and native SQL.
Jörg Unbehauen | AKSW, Universität Leipzig
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Collaboration is one of the most important topics regarding the evolution of the World Wide Web and thus also for the Web of Data. In scenarios of distributed collaboration on datasets it is necessary to provide support for multiple different versions of datasets to exist simultaneously, while also providing support for merging diverged datasets. In this paper we present an approach that uses SPARQL 1.1 in combination with the version control system Git, that creates commits for all changes applied to an RDF dataset containing multiple named graphs. Further the operations provided by Git are used to distribute the commits among collabora-tors and merge diverged versions of the dataset. We show the advantages of (public) Git repositories for RDF datasets and how this represents a way to collaborate on RDF data and consume it. With SPARQL 1.1 and Git in combination, users are given several opportunities to participate in the evolution of RDF data.
Natanael Arndt | AKSW, Universität Leipzig
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
Since the most of the world’s data is unstructured, the mining of required information from text was, is and will be essential.
Martin Voigt | CEO Ontos GmbH
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
We discuss and demonstrate how selected governmental data from the city of Leipzig,
published as Linked Data, can play a decisive factor for realizing Digital Agenda goals of the European Commission.
As an example, Lecos present the current state and planned future developments of:
(a) data sources,
(b) data conversions,
(c) publishing technologies and
(d) incubating visualizations about data concerning the accessibility of governmental edifices depending on the degree of disabilities of visiting citizen.
Holger Wollschläger | IT-Consultant at Lecos GmbH
Presentation at Semantics 2016 in Leipzig in the context with the results of the LEDS project
mu.semte.ch - A journey from TenForce's perspective - SEMANTICS2016Aad Versteden
mu.semte.ch, a framework for building microservices-powered applications on top of Linked Data, presented from TenForce's perspective. This presentation was given at Semantics2016.
Whereas in the past, research and industry mainly focused on technological aspects related to the exposure of data and its processing from remote sources, the concern in Semantic Web Data Management started to shift over to other non-technical challenges when dealing with so called Linked Enterprise Data. Information Quality is such an aspect that plays an important role in the process of selecting the best available data source in the Web, consolidating it with already existing information and thereby improving the business value of the own data stock. Throughout the last decades, research has already comprehensively dealt with the question of what quality is and how it can be interpreted through different approaches. Surprisingly, it is all the more astounding that there are only vague and no concise and formal definitions of the quality concept so far, especially in the Semantic Web context. A formalization would help to make quality calculations more comparable and implementable. The paper addresses this challenge and raises the question on how to compute the quality of a particular data source by combining different aspects from existing approaches resulting in a more concise model. As a finding, data quality is expressed as the percentage to which degree a particular data source fulfills a set of specified requirements in a certain context. The formalized definition of data quality helps to discuss specific assessment aspects, and is exemplarily applied to a scenario from the CRM application domain.
from Prof. Dr. Martin Gaedke | Vice Dean of the Department of Computer Science at the Chemnitz University of Technology (TUC)
and Dipl.-Inf. André Langer | LEDS project at University of Technology (TUC)
Presentation at WWW/Internet 2016 (ICWI) in the context with the results of the LEDS project.
Joe Caserta, President at Caserta Concepts, presented "Setting Up the Data Lake" at a DAMA Philadelphia Chapter Meeting.
For more information on the services offered by Caserta Concepts, visit our website at http://casertaconcepts.com/.
Incorporating the Data Lake into Your Analytic ArchitectureCaserta
Joe Caserta, President at Caserta Concepts presented at the 3rd Annual Enterprise DATAVERSITY conference. The emphasis of this year's agenda is on the key strategies and architecture necessary to create a successful, modern data analytics organization.
Joe Caserta presented Incorporating the Data Lake into Your Analytics Architecture.
For more information on the services offered by Caserta Concepts, visit out website at http://casertaconcepts.com/.
BDVe Webinar Series - Designing Big Data pipelines with Toreador (Ernesto Dam...Big Data Value Association
In the Internet of Everything, huge volumes of multimedia data are generated at very high rates by heterogeneous sources in various formats, such as sensors readings, process logs, structured data from RDBMS, etc. The need of the hour is setting up efficient data pipelines that can compute advanced analytics models on data and use results to customize services, predict future needs or detect anomalies. This Webinar explores the TOREADOR conversational, service-based approach to the easy design of efficient and reusable analytics pipelines to be automatically deployed on a variety of cloud-based execution platforms.
Why Your Data Science Architecture Should Include a Data Virtualization Tool ...Denodo
Watch full webinar here: https://bit.ly/35FUn32
Presented at CDAO New Zealand
Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python, and Scala put advanced techniques at the fingertips of the data scientists.
However, most architecture laid out to enable data scientists miss two key challenges:
- Data scientists spend most of their time looking for the right data and massaging it into a usable format
- Results and algorithms created by data scientists often stay out of the reach of regular data analysts and business users
Watch this session on-demand to understand how data virtualization offers an alternative to address these issues and can accelerate data acquisition and massaging. And a customer story on the use of Machine Learning with data virtualization.
Unlock Your Data for ML & AI using Data VirtualizationDenodo
How Denodo Complement’s Logical Data Lake in Cloud
● Denodo does not substitute data warehouses, data lakes,
ETLs...
● Denodo enables the use of all together plus other data
sources
○ In a logical data warehouse
○ In a logical data lake
○ They are very similar, the only difference is in the main
objective
● There are also use cases where Denodo can be used as data
source in a ETL flow
Augmentation, Collaboration, Governance: Defining the Future of Self-Service BIDenodo
Watch full webinar here: https://bit.ly/3zVJRRf
According to Dresner Advisory’s 2020 Self-Service Business Intelligence Market Study, 62% of the responding organizations say self-service BI is critical for their business. If we look deeper into the need for today’s self-service BI, it’s beyond some Executives and Business Users being enabled by IT for self-service dashboarding or report generation. Predictive analytics, self-service data preparation, collaborative data exploration are all different facets of new generation self-service BI. While democratization of data for self-service BI holds many benefits, strict data governance becomes increasingly important alongside.
In this session we will discuss:
- The latest trends and scopes of self-service BI
- The role of logical data fabric in self-service BI
- How Denodo enables self-service BI for a wide range of users - Customer case study on self-service BI
Advanced Analytics and Machine Learning with Data VirtualizationDenodo
Watch full webinar here: https://bit.ly/32c6TnG
Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way.
Attend this webinar and learn:
- How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice
- How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo
- How you can use the Denodo Platform with large data volumes in an efficient way
- About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization
Using a Semantic and Graph-based Data Catalog in a Modern Data FabricCambridge Semantics
Watch this webinar to learn about the benefits of using semantic and graph database technology to create a Data Catalog of all of an enterprise's data, regardless of source or format, as part of a modern IT or data management stack and an important step toward building an Enterprise Data Fabric.
Virtualisation de données : Enjeux, Usages & BénéficesDenodo
Watch full webinar here: https://bit.ly/3oah4ng
Gartner a récemment qualifié la Data Virtualisation comme étant une pièce maitresse des architectures d’intégration de données.
Découvrez :
- Les bénéfices d’une plateforme de virtualisation de données
- La multiplication des usages : Lakehouse, Data Science, Big Data, Data Service & IoT
- La création d’une vue unifiée de votre patrimoine de données sans transiger sur la performance
- La construction d’une architecture d’intégration Agile des données : on-premise, dans le cloud ou hybride
ADV Slides: Building and Growing Organizational Analytics with Data LakesDATAVERSITY
Data lakes are providing immense value to organizations embracing data science.
In this webinar, William will discuss the value of having broad, detailed, and seemingly obscure data available in cloud storage for purposes of expanding Data Science in the organization.
When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY
Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020.
Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms.
Data lakes will be built in cloud object storage. We’ll discuss the options there as well.
Get this data point for your data lake journey.
Guest Speaker in the 2nd National level webinar titled "Big Data Driven Solutions to Combat Covid 19" on 4th July 2020, Ethiraj College for Women(Auto), Chennai.
Analytical Innovation: How to Build the Next Generation Data PlatformVMware Tanzu
There was a time when the Enterprise Data Warehouse (EDW) was the only way to provide a 360-degree analytical view of the business. In recent years many organizations have deployed disparate analytics alternatives to the EDW, including: cloud data warehouses, machine learning frameworks, graph databases, geospatial tools, and other technologies. Often these new deployments have resulted in the creation of analytical silos that are too complex to integrate, seriously limiting global insights and innovation.
Join guest speaker, 451 Research’s Jim Curtis and Pivotal’s Jacque Istok for an interactive discussion about some of the overarching trends affecting the data warehousing market, as well as how to build a next generation data platform to accelerate business innovation. During this webinar you will learn:
- The significance of a multi-cloud, infrastructure-agnostic analytics
- What is working and what isn’t, when it comes to analytics integration
- The importance of seamlessly integrating all your analytics in one platform
- How to innovate faster, taking advantage of open source and agile software
Speakers: James Curtis, Senior Analyst, Data Platforms & Analytics, 451 Research & Jacque Istok, Head of Data, Pivotal
In this slidedeck, Infochimps Director of Product, Tim Gasper, discusses how Infochimps tackles business problems for customers by deploying a comprehensive Big Data infrastructure in days; sometimes in just hours. Tim unlocks how Infochimps is now taking that same aggressive approach to deliver faster time to value by helping customers develop analytic applications with impeccable speed.
Transforming Data Management and Time to Insight with Anzo Smart Data Lake®Cambridge Semantics
This webinar is targeted to Federal Government CIOs and
staff that are researching enterprise data management and
mining tools to help them understand how Smart Data Lakes
enable a viable mechanism for addressing their top priorities.
What is OLAP -Data Warehouse Concepts - IT Online Training @ NewyorksysNEWYORKSYS-IT SOLUTIONS
NEWYORKSYSTRAINING are destined to offer quality IT online training and comprehensive IT consulting services with complete business service delivery orientation.
ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY
Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020.
Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms.
Data lakes will be built in cloud object storage. We’ll discuss the options there as well.
Get this data point for your data lake journey.
Experimental transformation of ABS data into Data Cube Vocabulary (DCV) form...Alistair Hamilton
Presentation by Al Hamilton and Cody Johnson to Canberra Semantic Web Meetup Group on why producers of official statistics are interested in semantic web community (including Linked Open Data) and outlining experimental work by Cody Johnson on transforming selected Population Census data released by the ABS in SDMX-ML to RDF Data Cube Vocabulary format.
Similar to eccenca CorporateMemory - Semantically integrated Enterprise Data Lakes (20)
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
2. MOTIVATION
Enterprise Data Management Objective:
“Ensure all data is aligned to a common meaning
in order to achieve automation in performing
complex analytics and generating trusted
reports.”
Source:
2015 Data Management Industry Benchmark -
EDM Council
September 26,
2016
2
In 2015 only 7% of
respondents claim to
already be using shared
and unambiguous
definitions of data across
the firm and have it
accessible as operational
metadata.
7%
3. ARCHITECTURE
September 26,
2016
3
Management
Accounting
Risk Management
Regulatory Reporting
Treasury MarketingAccounting
Corporate
Memory
Inbound
Data Sources
Outbound and
Consumption
Inbound Raw Data Store
Knowledge Graph for Meta Data, KPI Definition and Data Models
Frontend to Access Relationship and KPI Definition /
Documentation
Frontend to Access (ad hoc) Reports Outbound Data Delivery to Target Systems
Big Data DWH-
Infrastructure
4. ARCHITECTURE
Management
Accounting
Risk Management
Regulatory Reporting
Treasury MarketingAccounting
Inbound Raw Data Store
Knowledge Graph for Meta Data, KPI Definition and Data Models
Frontend to Access Relationship and KPI Definition /
Documentation
Frontend to Access (ad hoc) Reports
Outbound Data Delivery to
Target Systems
Big Data
DWH-
Infrastructure
Data Ingestion
• Files in the data lake (CSV, XML, Excel)
• (relational) Databases
5. ARCHITECTURE
Management
Accounting
Risk Management
Regulatory Reporting
Treasury MarketingAccounting
Inbound Raw Data Store
Knowledge Graph for Meta Data, KPI Definition and Data Models
Frontend to Access Relationship and KPI Definition /
Documentation
Frontend to Access (ad hoc) Reports
Outbound Data Delivery to
Target Systems
Big Data
DWH-
Infrastructure
Data Lake
• Emerging approach to handle large amounts
of data
• Cost-effective storage
• Data is held in their native formats
Good
Does not force an up-front integration of the
ingested data sets
Bad
Retaining an overview of disparate data silos in
the lake without having a coherent shared view
is a challenging issue
6. ARCHITECTURE
Management
Accounting
Risk Management
Regulatory Reporting
Treasury MarketingAccounting
Inbound Raw Data Store
Knowledge Graph for Meta Data, KPI Definition and Data Models
Frontend to Access Relationship and KPI Definition /
Documentation
Frontend to Access (ad hoc) Reports
Outbound Data Delivery to
Target Systems
Big Data
DWH-
Infrastructure
Data Warehouses
• Existing infrastucture
• Typically relational databases
7. ARCHITECTURE
Management
Accounting
Risk Management
Regulatory Reporting
Treasury MarketingAccounting
Inbound Raw Data Store
Knowledge Graph for Meta Data, KPI Definition and Data Models
Frontend to Access Relationship and KPI Definition /
Documentation
Frontend to Access (ad hoc) Reports
Outbound Data Delivery to
Target Systems
Big Data
DWH-
Infrastructure
Metadata Layer
• Dataset Metadata
• Ontologies
• Integration Rules
8. ARCHITECTURE
Management
Accounting
Risk Management
Regulatory Reporting
Treasury MarketingAccounting
Inbound Raw Data Store
Knowledge Graph for Meta Data, KPI Definition and Data Models
Frontend to Access Relationship and KPI Definition /
Documentation
Frontend to Access (ad hoc) Reports
Outbound Data Delivery to
Target Systems
Big Data
DWH-
Infrastructure
Graphical User Interface
Customer Applications
9. INTEGRATION PROCESS
Dataset
Management
•Catalog Datasets
•Catalog Ontologies
•Manage Metadata
Dataset Discovery
•Data Profiling
•Dataset Exploration
Dataset Integration
•Dataset Lifting
•Dataset Linking
•Data Quality Validation
Data Access
•Domain Specific
Consolidated Views
•Execution on Hadoop
September 26,
2016
9
10. DATASET MANAGEMENT
Dataset
Management
•Catalog Datasets
•Catalog Ontologies
•Manage Metadata
Dataset Discovery
•Data Profiling
•Dataset Exploration
Dataset Integration
•Dataset Lifting
•Dataset Linking
•Data Quality Validation
Data Access
•Domain Specific
Consolidated Views
•Execution on Hadoop
September 26,
2016
10
11. DATASET CATALOG
• Enables the user to explore and manage datasets in the data lake
• Files in the data lake (CSV, XML, Excel)
• Databases (Apache Hive or external databases)
September 26,
2016
11
12. MANAGING METADATA
• Exploring and editing dataset metadata
• Semantic content information, like textual
descriptions, tags and related Persons
• Technical information and parameters, like
formats, data model and encoding
• Access information, like access path or URL,
source system or API call
• Organizational provenance, like
organizational units owning or maintaining
the dataset
September 26,
2016
12
13. DATASET DISCOVERY
Dataset
Management
•Catalog Datasets
•Catalog Ontologies
•Manage Metadata
Dataset Discovery
•Data Profiling
•Dataset Exploration
Dataset Integration
•Dataset Lifting
•Dataset Linking
•Data Quality Validation
Data Access
•Domain Specific
Consolidated Views
•Execution on Hadoop
September 26,
2016
13
14. DATASET DISCOVERY
• Goal: Augment a dataset with data from related datasets
• Automatic discovery of dataset with overlapping information
• Explorative interface
• Discovery is based on two data parts
• Business meta data
• Profiling summary
September 26,
2016
14
15. DISCOVERY VIEW
• Datasets are matched based on their metadata (profiling + business data)
September 26,
2016
15
16. DATASET PROFILING
• Datasets often contain implicit and explicit schema information
• Column names, data formats, enumerated values etc.
• Example: column contains formatted dates
• Idea: Extract a dataset summary
• For each column / property the summary contains:
1. Data type (e.g., number, date, industry classification)
2. Data format (e.g., date format)
3. Data statistics (e.g., range, distribution, most frequent values)
• Materialized as RDF with UI view
September 26,
2016
16
17. DETECTING DATA TYPES
• Detecting common datatypes as well as user-defined types
• Common datatypes
• Numbers
• Dates / Times
• Geographic locations (geo-coordinates, states, countries)
• User-defined data types can be integrated by adding an ontology /
taxonomy
• Usually a SKOS taxonomy
• Managed as another dataset in the dataset management
• Example: Industry taxonomy
• Standard taxonomy (NACE, SIC, NAICS) or company specific
September 26,
2016
17
18. FORMATS AND STATISTICS
• For some types, the data format is detected
• Example: Dates are formatted in DD-MM-YYYY
• Two functions are generated:
1. Parser that is able to read the detected representation
2. Normalizer that converts the parsed values into a configurable, organization-wide
target representation
• Statistics summarize the values:
• Value range and distribution
• Most frequent values
• Data selectivity
September 26,
2016
18
19. DISCOVERY VIEW
• Datasets are matched based on their metadata (profiling + business data)
Septemb
er 26, 2016
19
20. INTEGRATION PROCESS
Dataset
Management
•Catalog Datasets
•Catalog Ontologies
•Manage Metadata
Dataset Discovery
•Data Profiling
•Dataset Exploration
Dataset Integration
•Dataset Lifting
•Dataset Linking
•Data Quality Validation
Data Access
•Domain Specific
Consolidated Views
•Execution on Hadoop
September 26,
2016
20
21. DATA INTEGRATION
• The integration process is driven by a set of rules
• Lifting Rules map the source datasets to a ontology
• Linking Rules connect different datasets to a knowledge graph
• Rules are operator trees, consisting of four types of operators
• Data Access Operators
• Transformation Operators
• Similarity Operators
• Aggregation Operators
• Rules can be learned using genetic programming algorithms
• Rules are human understandable and can be edited
September 26,
2016
21
22. DATASET LIFTING
• Objective: Map the datasets in the data lake to a consistent vocabulary.
• A lifting rule consists of a number of mappings
• Each mapping assigns a term in the original data set (such as a column for tabular data)
to a term in the target ontology (such as a property provided by an ontology).
• Multiple mappings for each dataset can be managed to allow different
views on the same data.
• Initial mappings are generated automatically based on the profiling results
from where the user can continue to build on.
September 26,
2016
22
23. LIFTING EXAMPLE
September 26,
2016
23
Bond ISIN Country Industry
NEDWBK CAD 5,2%25 CA639832AA25 Canada Banking
SIEMENSF1.50%03/20 DE000A1G85B4 Germany Electrical
Equipment
Electricite de France
(EDF), 6,5% 26jan2019
USF2893TAB29 France Utilities
NEDWBK CAD 5,2%25
fibo:hasSecurityIdentifier
Utilities
Industry Ontology
Banking
France
Country Ontology
Germany
EMEA
“CA639832AA25”
fibo:legallyRecordedIn
fibo:industrySector
24. LINKING
• Goal: Connect individual datasets to a knowledge graph
• Identify related entities in different datasets and link them
• Either entities describing the same real world object or another relation
September 26,
2016
24
NEDWBK CAD 5,2%25
ratingScore
Industry OntologyCountry Ontology
EMEA
“AAA”
fibo:legallyRecordedIn
fibo:industrySector
Rating CAD 5,2%25
hasRating
fibo:industrySector
fibo:legallyRecordedIn
25. LINKAGE RULES
• Linking is based on domain-specific rules
• Specify the conditions that must hold true for two entities to be linked
September 26,
2016
25
26. LEARNING LINKAGE RULES
Problem: Manually writing rules is time-consuming and requires expertise
Approach: Interactive machine learning algorithm for generating rules
• Generates a rule based on a number of user-confirmed link candidates.
• Link candidates are actively selected by the learning algorithm to include link candidates
that yield a high information gain.
• The user does not need any knowledge of the characteristics
of the dataset or any particular similarity computation techniques.
September 26,
2016
26
27. INTEGRATION PROCESS
Dataset
Management
•Catalog Datasets
•Catalog Ontologies
•Manage Metadata
Dataset Discovery
•Data Profiling
•Dataset Exploration
Dataset Integration
•Dataset Lifting
•Dataset Linking
•Data Quality Validation
Data Access
•Domain Specific
Consolidated Views
•Execution on Hadoop
28. VIEW GENERATION
• The user selects a set of lifted and linked datasets
September 26,
2016
28
29. Hadoop
Data Lake
DATA ACCESS
• Generate data flows based on
Apache Spark
• The data flows utilize Resilient
Distributed Datasets (RDDs)
• RDDs derive new data sets from
existing data sets by applying a
chain of transformations
• A derived data set can either
• be recomputed on-the-fly
• persisted on stable storage
• Data flows can be executed
efficiently on Hadoop clusters.
September 26,
2016
29
Corporate
Bonds
Data Lifting 1
(Apache Spark
RDD)
Data Linking
(Apache Spark RDD)
Internal
Ratings
Data Lifting 2
(Apache Spark
RDD)
External
Ratings
Data Lifting 3
(Apache Spark
RDD)
eccenca
Corporate
Memory
Data
Consumer
SQL CSV
Excel
Spark
API