Wouldn't it be nice if... an introduction to Enterprise Data Mashups

Wouldn’t it be nice if…? An Introduction to Enterprise Data Mashups at the School of Information, UC Berkeley Feb 11th, 2008 Justo N. Hidalgo VP Product Management and Consulting Denodo

… I could use information for all these sources without going crazy? CRM (MySQL) Billing System (Web Service - Rest) Incidences System (Web Application) Inventory System (MS SQL Server) Product Catalog (Web Service -SOAP) Knowledge Base (Internet) Product Data (CSV) Log files (.txt/.log files)

Extracción de Datos Web Estructuración y Navegación Automática. Estado del Arte y Propuesta Investigadora de Denodo

Justo N. Hidalgo, VP Product Management and Consulting

21 de Enero de 2009

… I could do it in real time, where required? CRM (MySQL) Billing System (Web Service - Rest) Incidences System (Web Application) Inventory System (MS SQL Server) Product Catalog (Web Service -SOAP) Knowledge Base (Internet) Product Data (CSV) Log files (.txt/.log files)

… I could do it in batch mode, where required?

… I could react rapidly to changes?

… I could do this without having to code it all?

… , finally, I could do this for actual ENTERPRISE PROBLEMS AND NEEDS???

Mashups are Crossing the Chasm

Agenda

Background

Enterprise Mashups

Web mashups and Enterprise Data Mashups

Some real-world examples

Research Areas

Denodo

Company

Founded 1999, Europe 2005, NA HQ in 2006, APAC SIs 2007, East, Midwest 2009

Profitable in Europe; Venture-backed global expansion

Operations

Americas & EMEA; VARs in Asia

90+ Customers – Telco, Fin Svcs, Energy, Health, Govt / Defense

Expanding Partners - Focused ISVs, SaaS, SI/VARs

Product (Denodo Platform 4.5 - Enterprise Hardened)

Enterprise Data Mashups & Web Automation

Data Integration of Structured, Web and Unstructured Information

Fast, Agile Deployments in under 30 days

Technology Focus: Data Integration, Web 2.0 & Mashup Enabler

Targeted Solutions: Intelligence, Single View, Web Automation

Tenths of projects in the last 18 months to learn from

Q4 2009: Denodo Platform 5.0

Justo

VP Product Management and Consulting, Denodo Technologies

Responsible for Product Management and Sales Engineering

Ph.D. Computer Science, Query Optimization in Mediators of Web Sources

Working in Enterprise Data Integration and Web Process Automation since 2000

jhidalgo at domain denodo dot com

If you read spanish: http://justohidalgo.blogspot.com

This presentation is available at:

http://www.slideshare.net/justohidalgo/

Enterprise Mashups

Enterprise Mashup Architecture, according to the Analysts

A little bit more detail

What, Enterprise Mashups are not for cool things? 1. Aggregate Data 3. Exchange Data Enterprise Suppliers Customers 2. Enrich Data Apps Data Services

Web and Data Mashups

Web Mashups vs. Data Mashups

Web Mashups

Visual centric

Data relationships are simple

End-user driven enabled by APIs / tools

No QoS guarantees – as is

Simple Web security

Data Mashups

Data & Information centric

Deep transformations & semantic relationships

Interact with Enterprise ecosystem

Business analyst driven w/ user control options

Enterprise QoS- Query Optimization, Reliable, Failover, etc.

Policy-based security

ENTERPRISE ECOSYSTEM Enterprise Infrastructure / Web Platform Enterprise & Web Applications RIA, Social Software, GUI, Portals

Mutual Benefits Web Mashups Data Mashups Enterprise Data Integration BPM ESB EAI ETL Data Warehouses Enterprise Infrastructure / Web Platform Enterprise & Web Applications RIA, Social Software, GUI, Portals ENTERPRISE ECOSYSTEM WEB MASHUPS DATA MASHUPS User experience / global access Enterprise-class features ?

Enterprise Areas

Mashups for Business Users / For Power Users and IT

User experience

Pipes&Filters, Data Federation

Heterogeneous granularity for different levels

Enterprise-class features…

Security

LDAP access, user and role permission levels, …

But also: Secure VPNs, secure communications, encryption, SSO

HA/Scalability

Load balancing

Federated and distributed architectures

Performance

Asynchronous access

Parallelism

Execution Plan Optimization techniques

… Enterprise-class features

Data Transformations

Textual similarity, contextual summaries, patterns and regular expressions

Data Management… don´t get me started 

What are the boundaries between mashups and MDM, Data Quality tools?

Standards and Integration

JSR-168, JMX, JMX, JDBC, ODBC, WS-*, microformats, … don´t you love standardization bodies?

Automatic Browsing and Structure

Maintenance

Sources change… specially Web Sites!!!

Automatic Extraction

Schema matching

Automatic wrapper induction

Automatic Browsing

Taxonomy-based path recognition

Form updates

Research

Architecture

A new Mashup Reference Architecture

Web Data Extraction

Automatic Web Data Extraction

Automatic Web Maintenance

Extraction Templates

Enterprise-class

Query Optimization Techniques

A few references

Information Structuring

Manuel Álvarez, Alberto Pan, Juan Raposo, Fernando Bellas, Fidel Cacheda. Finding and Extracting Data Records from Web Pages. Journal of Signal Processing Systems (Special issue on EUC 2007).

Manuel Álvarez, Alberto Pan, Juan Raposo, Fernando Bellas, Fidel Cacheda: Extracting lists of data records from semi-structured web pages. Data Knowl. Eng. 64(2): 491-509 (2008)

Manuel Álvarez, Alberto Pan, Juan Raposo, Fernando Bellas, Fidel Cacheda: Using Clustering and Edit Distance Techniques for Automatic Web Data Extraction. WISE 2007: 212-224

Juan Raposo, Alberto Pan, Manuel Álvarez, Justo Hidalgo: Automatically maintaining wrappers for semi-structured web sources. Data Knowl. Eng. 61(2): 331-358 (2007)

Focused Crawling

Manuel Álvarez, Juan Raposo, Alberto Pan, Fidel Cacheda, Fernando Bellas, Victor Carneiro: DeepBot: a focused crawler for accessing hidden web content. DEECS 2007: 18-25

Manuel Álvarez, Juan Raposo, Alberto Pan, Fidel Cacheda, Fernando Bellas, Victor Carneiro: Crawling the Content Hidden Behind Web Forms. ICCSA (2) 2007: 322-333

Cost Optimization

Justo Hidalgo, Alberto Pan, José Losada, Manuel Álvarez. Preloading Browsers for Optimizing Automatic Access to Hidden Web: a Ranking-based Repository Solution. ADBIS 2006. pp.171-183.

Justo Hidalgo, Alberto Pan, Manuel Álvarez, Jaime Guerrero. Efficiently Updating Cost Repository Values for Query Optimization on Web Data Sources in a Mediator/Wrapper Environment. NGITS 2006. Vol. 4032, pp. 1-12.

Q&A Justo Hidalgo VP Product Management and Consulting [email_address]

All images shown that are not property of Denodo come from the search engine search.creativecommons.org, that searches Flickr images with CreativeCommons license

Credits