A Guide to Modern BPM Data Integration
Kimberly Palko, Product Manager
Eric Schabell, Technical Evangelist
Red Hat JBoss
Agenda
• Data integration challenges with business
processes
• Providing context with JBoss Middleware
– JBoss Data Virtualization
– JBoss BRMS
– JBoss BPM Suite
• Business process data virtualization
• Demo combining JBoss Data Virtualization with
JBoss BPMS
Data in disconnected enterprise data
silos
NoSQL
Mainframes
Enterprise Applications
Datawarehouse
and Databases
Cloud Apps
Hadoop
Files
Data Control Challenges Getting Bigger
with Big Data, Cloud, and Mobile
• Security capabilities are tightly coupled to data sources
• Extracting and moving data adds risk
• Every project solves data access and integration in a different way
• Inconsistent and decentralized control of data
BI Reports
Business
Process
Enterprise
Applications
SOA
Applications
Mobile
Applications
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
Different security capabilities for each
data source
Constant
Change
Siloed &
Complex
How to align?
Data Quality issues impact BPM
projects
RedPoint – MDM for the Modern Data Architecture, Sept 2014
Process model complexity slows
change
Point to point connections to multiple data sources inhibit agility
Business and IT both want control of
the data
Business wants autonomy IT wants security and control
Consider...
How would your organization change…
●If data were readily reusable in place rather than
requiring significant effort to build new intermediary
data tiers?
●If data could be repurposed quickly into new
applications and business processes?
●If all applications and business processes could get all
of the information needed in the form needed, where
needed and when needed?
Inconsistent,
Incomplete
Information
Uninformed, Delayed
Decisions
Costly Business Risk
and Exposure
Combine process with
data in a flexible and secure way for
agile business processes
What does Data Virtualization software do?
Turn Fragmented Data into Smart Data
Data Virtualization software virtually
unifies data spread across various disparate
sources; and makes it available to
applications as a single consolidated data
source.
The data virtualization software
implements 3 steps process to bridge data
sources and data consumers:
• Connect: Fast access to data from
diverse data sources
• Compose: Easily create unified virtual
data models and views by combining
and transforming data from multiple
sources.
• Consume: Expose consistent
information to data consumers in the
right form thru standard data access
methods.
Virtual Consolidated Data Source
BI Reports
Data Virtualization
Software
• Consume
• Compose
• Connect
SAP Salesforce.comOracle DW XML, CSV
& Excel files
Siloed &
Complex
Virtualize
Abstract
Federate
Easy,
Real-time
Information
Access
SOA Applications
DATA CONSUMERS
DATA SOURCES
JBoss Data Virtualization
DESIRED STATE
Data as a Service
●Standard based interface
●Single view of disparate source
data
●Single point of access / integration
●Reuse of Data
But you cannot achieve this by
writing more application code…
Data Sources
Siloed & Complex
Data as a Service
BI Reports
Business
Process
Enterprise
Applications
SOA
Applications
Mobile
Applications
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
Turn Siloed Data into Actionable Information
Connect
Compose
Consume
BI Reports & Analytics
Mobile Applications
SOA Applications & PortalsESB, ETL
Native Data Connectivity
Standard based Data Provisioning
JDBC, ODBC, SOAP, REST, OData
JBoss
DataVirtualizationData
Consumers
Data
Sources
Design Tools
Dashboard
Optimization
Caching
Security
Metadata
Data Warehouse
& Databases
XML, CSV
& Excel FilesHadoop NoSQL Cloud Apps Mainframe
Enterprise Apps
Siloed &
Complex
Virtualize
Transform
Federate
Easy,
Real-time
Information
Access
Unified Virtual Database / Common Data Model
Data Transformations
Integration of Big Data with traditional data
sources
• Integrating traditional
data with big data is easy
• Integration specifications
can be shared or be
developed for individual
reports
• Caching capabilities for
improved performance
MapReduce
HDFS
Hive ApplicationDatabase
Server
JBoss Data Virtualization:
Supported Data Sources
Enterprise RDBMS:
• Oracle
• IBM DB2
• Microsoft SQL Server
• Sybase ASE
• MySQL
• PostgreSQL
• Ingres
Enterprise EDW:
• Teradata
• Netezza
• Greenplum
Hadoop:
• Apache
• HortonWorks
• Cloudera
• More coming…
Office Productivity:
• Microsoft Excel
• Microsoft Access
• Google Spreadsheets
Specialty Data Sources:
• ModeShape Repository
• Mondrian
• MetaMatrix
• LDAP
NoSQL:
• JBoss Data Grid
• MongoDB
• Cassandra
• More coming…
Enterprise & Cloud
Applications:
• Salesforce.com
• SAP
Technology Connectors:
• Flat Files, XML Files,
XML over HTTP
• SOAP Web Services
• REST Web Services
• OData Services
SECURITY CAPABILITIES
• Authentication
– Kerberos, LDAP, SAML, SSL, WS-
UsernameToken, HTTP Basic
• Authorization
– Virtual data marts, Role based access
control
• Audit
• Centralized audit logging and dashboard
• Administration
– Centralized management of users and
groups
• Protection
– Row and column masking
Optional ways to control data access
using Virtual Data Marts or various
views of the data
either can be combined with security features like user
role access
Data Warehouse
& Databases
XML, CSV
& Excel FilesHadoop NoSQL Cloud Apps Mainframe
Enterprise Apps
Base Virtual Database (VDB)
Dept1
VDB
Dept2
VDB
View
2
View
1
Data Virtualization Designer
Model Driven Development
Eclipse-based graphical
tool for
• modeling,
• analyzing,
• Integrating,
• resolving semantic
differences and
• testing
multiple data sources
to produce
• Relational,
• XML and
• Web Service Views
that expose your
business data without
any programming.
• Shows structural
transformations and
dependencies
• Defines
transformations
Business Dashboard
Quickly Visualize your Data
What does Data Virtualization software do?
Turn Fragmented Data into Smart Data
Data Virtualization software virtually
unifies data spread across various disparate
sources; and makes it available to
applications as a single consolidated data
source.
The data virtualization software
implements 3 steps process to bridge data
sources and data consumers:
• Connect: Fast access to data from
diverse data sources
• Compose: Easily create unified virtual
data models and views by combining
and transforming data from multiple
sources.
• Consume: Expose consistent
information to data consumers in the
right form thru standard data access
methods.
Virtual Consolidated Data Source
BI Reports
Data Virtualization
Software
• Consume
• Compose
• Connect
SAP Salesforce.comOracle DW XML, CSV
& Excel files
Siloed &
Complex
Virtualize
Abstract
Federate
Easy,
Real-time
Information
Access
SOA Applications
DATA CONSUMERS
DATA SOURCES
JBoss BRMS and BPM
Business challenge
travel booking
review
approval
pricing
accounting
payment
final approval
Business challenge
travel booking
review
approval
pricing
accounting
payment
final approval
Visibility – difficult to
know what work is
where and at what
stage
Agility - it takes too
long to adapt the
process as business
needs change
Consistency –
different people
route work
differently
Problems:
Agile Processes with BPM
BPMS
Business experts create a model of
the business process
Agile Processes with BPM
A BPMS automates the process – driving claims
through the organization according to the model, and
managing interaction with people and applications.
BPMS
Data collection Data registration
The BPMS Solution
A BPMS tracks the status of work, providing dashboards and reports that
ensure visibility into every aspect of business operations
JBoss BRMS 6
JBoss BPM Suite 6
JBoss BRMS 6EventsRules
Process
EventsRules
Business Rules Management Business Process Management
JBoss BRMS and JBoss BPM Suite
What does Data Virtualization software do?
Turn Fragmented Data into Smart Data
Data Virtualization software virtually
unifies data spread across various disparate
sources; and makes it available to
applications as a single consolidated data
source.
The data virtualization software
implements 3 steps process to bridge data
sources and data consumers:
• Connect: Fast access to data from
diverse data sources
• Compose: Easily create unified virtual
data models and views by combining
and transforming data from multiple
sources.
• Consume: Expose consistent
information to data consumers in the
right form thru standard data access
methods.
Virtual Consolidated Data Source
BI Reports
Data Virtualization
Software
• Consume
• Compose
• Connect
SAP Salesforce.comOracle DW XML, CSV
& Excel files
Siloed &
Complex
Virtualize
Abstract
Federate
Easy,
Real-time
Information
Access
SOA Applications
DATA CONSUMERS
DATA SOURCES
Business Process Data Virtualization
With JBoss Data Virtualization
and JBoss BPM Suite
Process Data Virtualization
“Process data virtualization provides virtual views of
systems-of-record data in forms that process models,
process applications, and users can consume while
hiding the true complexity of data.”
TechRadar: Customer-centric BPM Technology, Q4 2014
Clay Richardson
Complex business process using data
from multiple sources
BPMS
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
Virtualized data layer simplifies BPM
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
Base Virtual Database (VDB)
BPMS
Process data virtualization
Business value-add: HIGH
“The primary goal of process data virtualization is to move
integration complexity out of process models and thereby support
faster change of business processes. This is a crucial requirement for
transitioning BPM to focus on customer engagement.
Process data virtualization also allows process changes to be pushed
closer to the business, since it reduces the level of of involvement
required by tech management for connecting processes and data.”
TechRadar: Customer-centric BPM Technology, Q4 2014
Clay Richardson
What does Data Virtualization software do?
Turn Fragmented Data into Smart Data
Data Virtualization software virtually
unifies data spread across various disparate
sources; and makes it available to
applications as a single consolidated data
source.
The data virtualization software
implements 3 steps process to bridge data
sources and data consumers:
• Connect: Fast access to data from
diverse data sources
• Compose: Easily create unified virtual
data models and views by combining
and transforming data from multiple
sources.
• Consume: Expose consistent
information to data consumers in the
right form thru standard data access
methods.
Virtual Consolidated Data Source
BI Reports
Data Virtualization
Software
• Consume
• Compose
• Connect
SAP Salesforce.comOracle DW XML, CSV
& Excel files
Siloed &
Complex
Virtualize
Abstract
Federate
Easy,
Real-time
Information
Access
SOA Applications
DATA CONSUMERS
DATA SOURCES
Travel Agency Demo
Insert Screen Share Here
Demo – Intro JBoss BPM Travel Agency
• The project
• The workflow
• The rules
Demo – Acquisition integration
• Data virtualization
• External connection
• Dashboard
Demo – Bookings registration
Data virtualization mapped to dashboard
What does Data Virtualization software do?
Turn Fragmented Data into Smart Data
Data Virtualization software virtually
unifies data spread across various disparate
sources; and makes it available to
applications as a single consolidated data
source.
The data virtualization software
implements 3 steps process to bridge data
sources and data consumers:
• Connect: Fast access to data from
diverse data sources
• Compose: Easily create unified virtual
data models and views by combining
and transforming data from multiple
sources.
• Consume: Expose consistent
information to data consumers in the
right form thru standard data access
methods.
Virtual Consolidated Data Source
BI Reports
Data Virtualization
Software
• Consume
• Compose
• Connect
SAP Salesforce.comOracle DW XML, CSV
& Excel files
Siloed &
Complex
Virtualize
Abstract
Federate
Easy,
Real-time
Information
Access
SOA Applications
DATA CONSUMERS
DATA SOURCES
Conclusion
Benefits of Business Process Data
Virtualization
• Easy access to data in real-time from multiple
different sources
• Improved data quality with a single version of the
truth
• More agility with simplified process models
• Re-use of data integrations from multiple sources
• Increased autonomy for lines of business with
departmental virtual data bases
• Centralized IT control and auditing of data sources
Questions

A Guide to Modern BPM Data Integration

  • 1.
    A Guide toModern BPM Data Integration Kimberly Palko, Product Manager Eric Schabell, Technical Evangelist Red Hat JBoss
  • 2.
    Agenda • Data integrationchallenges with business processes • Providing context with JBoss Middleware – JBoss Data Virtualization – JBoss BRMS – JBoss BPM Suite • Business process data virtualization • Demo combining JBoss Data Virtualization with JBoss BPMS
  • 3.
    Data in disconnectedenterprise data silos NoSQL Mainframes Enterprise Applications Datawarehouse and Databases Cloud Apps Hadoop Files
  • 4.
    Data Control ChallengesGetting Bigger with Big Data, Cloud, and Mobile • Security capabilities are tightly coupled to data sources • Extracting and moving data adds risk • Every project solves data access and integration in a different way • Inconsistent and decentralized control of data BI Reports Business Process Enterprise Applications SOA Applications Mobile Applications Hadoop NoSQL Cloud Apps Data Warehouse & Databases Mainframe XML, CSV & Excel Files Enterprise Apps Different security capabilities for each data source Constant Change Siloed & Complex How to align?
  • 5.
    Data Quality issuesimpact BPM projects RedPoint – MDM for the Modern Data Architecture, Sept 2014
  • 6.
    Process model complexityslows change Point to point connections to multiple data sources inhibit agility
  • 7.
    Business and ITboth want control of the data Business wants autonomy IT wants security and control
  • 8.
    Consider... How would yourorganization change… ●If data were readily reusable in place rather than requiring significant effort to build new intermediary data tiers? ●If data could be repurposed quickly into new applications and business processes? ●If all applications and business processes could get all of the information needed in the form needed, where needed and when needed? Inconsistent, Incomplete Information Uninformed, Delayed Decisions Costly Business Risk and Exposure
  • 9.
    Combine process with datain a flexible and secure way for agile business processes
  • 10.
    What does DataVirtualization software do? Turn Fragmented Data into Smart Data Data Virtualization software virtually unifies data spread across various disparate sources; and makes it available to applications as a single consolidated data source. The data virtualization software implements 3 steps process to bridge data sources and data consumers: • Connect: Fast access to data from diverse data sources • Compose: Easily create unified virtual data models and views by combining and transforming data from multiple sources. • Consume: Expose consistent information to data consumers in the right form thru standard data access methods. Virtual Consolidated Data Source BI Reports Data Virtualization Software • Consume • Compose • Connect SAP Salesforce.comOracle DW XML, CSV & Excel files Siloed & Complex Virtualize Abstract Federate Easy, Real-time Information Access SOA Applications DATA CONSUMERS DATA SOURCES JBoss Data Virtualization
  • 11.
    DESIRED STATE Data asa Service ●Standard based interface ●Single view of disparate source data ●Single point of access / integration ●Reuse of Data But you cannot achieve this by writing more application code… Data Sources Siloed & Complex Data as a Service BI Reports Business Process Enterprise Applications SOA Applications Mobile Applications Hadoop NoSQL Cloud Apps Data Warehouse & Databases Mainframe XML, CSV & Excel Files Enterprise Apps
  • 12.
    Turn Siloed Datainto Actionable Information Connect Compose Consume BI Reports & Analytics Mobile Applications SOA Applications & PortalsESB, ETL Native Data Connectivity Standard based Data Provisioning JDBC, ODBC, SOAP, REST, OData JBoss DataVirtualizationData Consumers Data Sources Design Tools Dashboard Optimization Caching Security Metadata Data Warehouse & Databases XML, CSV & Excel FilesHadoop NoSQL Cloud Apps Mainframe Enterprise Apps Siloed & Complex Virtualize Transform Federate Easy, Real-time Information Access Unified Virtual Database / Common Data Model Data Transformations
  • 13.
    Integration of BigData with traditional data sources • Integrating traditional data with big data is easy • Integration specifications can be shared or be developed for individual reports • Caching capabilities for improved performance MapReduce HDFS Hive ApplicationDatabase Server
  • 14.
    JBoss Data Virtualization: SupportedData Sources Enterprise RDBMS: • Oracle • IBM DB2 • Microsoft SQL Server • Sybase ASE • MySQL • PostgreSQL • Ingres Enterprise EDW: • Teradata • Netezza • Greenplum Hadoop: • Apache • HortonWorks • Cloudera • More coming… Office Productivity: • Microsoft Excel • Microsoft Access • Google Spreadsheets Specialty Data Sources: • ModeShape Repository • Mondrian • MetaMatrix • LDAP NoSQL: • JBoss Data Grid • MongoDB • Cassandra • More coming… Enterprise & Cloud Applications: • Salesforce.com • SAP Technology Connectors: • Flat Files, XML Files, XML over HTTP • SOAP Web Services • REST Web Services • OData Services
  • 15.
    SECURITY CAPABILITIES • Authentication –Kerberos, LDAP, SAML, SSL, WS- UsernameToken, HTTP Basic • Authorization – Virtual data marts, Role based access control • Audit • Centralized audit logging and dashboard • Administration – Centralized management of users and groups • Protection – Row and column masking
  • 16.
    Optional ways tocontrol data access using Virtual Data Marts or various views of the data either can be combined with security features like user role access Data Warehouse & Databases XML, CSV & Excel FilesHadoop NoSQL Cloud Apps Mainframe Enterprise Apps Base Virtual Database (VDB) Dept1 VDB Dept2 VDB View 2 View 1
  • 17.
    Data Virtualization Designer ModelDriven Development Eclipse-based graphical tool for • modeling, • analyzing, • Integrating, • resolving semantic differences and • testing multiple data sources to produce • Relational, • XML and • Web Service Views that expose your business data without any programming. • Shows structural transformations and dependencies • Defines transformations
  • 18.
  • 19.
    What does DataVirtualization software do? Turn Fragmented Data into Smart Data Data Virtualization software virtually unifies data spread across various disparate sources; and makes it available to applications as a single consolidated data source. The data virtualization software implements 3 steps process to bridge data sources and data consumers: • Connect: Fast access to data from diverse data sources • Compose: Easily create unified virtual data models and views by combining and transforming data from multiple sources. • Consume: Expose consistent information to data consumers in the right form thru standard data access methods. Virtual Consolidated Data Source BI Reports Data Virtualization Software • Consume • Compose • Connect SAP Salesforce.comOracle DW XML, CSV & Excel files Siloed & Complex Virtualize Abstract Federate Easy, Real-time Information Access SOA Applications DATA CONSUMERS DATA SOURCES JBoss BRMS and BPM
  • 20.
  • 21.
    Business challenge travel booking review approval pricing accounting payment finalapproval Visibility – difficult to know what work is where and at what stage Agility - it takes too long to adapt the process as business needs change Consistency – different people route work differently Problems:
  • 22.
    Agile Processes withBPM BPMS Business experts create a model of the business process
  • 23.
    Agile Processes withBPM A BPMS automates the process – driving claims through the organization according to the model, and managing interaction with people and applications. BPMS Data collection Data registration
  • 24.
    The BPMS Solution ABPMS tracks the status of work, providing dashboards and reports that ensure visibility into every aspect of business operations
  • 27.
    JBoss BRMS 6 JBossBPM Suite 6 JBoss BRMS 6EventsRules Process EventsRules Business Rules Management Business Process Management JBoss BRMS and JBoss BPM Suite
  • 28.
    What does DataVirtualization software do? Turn Fragmented Data into Smart Data Data Virtualization software virtually unifies data spread across various disparate sources; and makes it available to applications as a single consolidated data source. The data virtualization software implements 3 steps process to bridge data sources and data consumers: • Connect: Fast access to data from diverse data sources • Compose: Easily create unified virtual data models and views by combining and transforming data from multiple sources. • Consume: Expose consistent information to data consumers in the right form thru standard data access methods. Virtual Consolidated Data Source BI Reports Data Virtualization Software • Consume • Compose • Connect SAP Salesforce.comOracle DW XML, CSV & Excel files Siloed & Complex Virtualize Abstract Federate Easy, Real-time Information Access SOA Applications DATA CONSUMERS DATA SOURCES Business Process Data Virtualization With JBoss Data Virtualization and JBoss BPM Suite
  • 29.
    Process Data Virtualization “Processdata virtualization provides virtual views of systems-of-record data in forms that process models, process applications, and users can consume while hiding the true complexity of data.” TechRadar: Customer-centric BPM Technology, Q4 2014 Clay Richardson
  • 30.
    Complex business processusing data from multiple sources BPMS Hadoop NoSQL Cloud Apps Data Warehouse & Databases Mainframe XML, CSV & Excel Files Enterprise Apps
  • 31.
    Virtualized data layersimplifies BPM Hadoop NoSQL Cloud Apps Data Warehouse & Databases Mainframe XML, CSV & Excel Files Enterprise Apps Base Virtual Database (VDB) BPMS
  • 32.
    Process data virtualization Businessvalue-add: HIGH “The primary goal of process data virtualization is to move integration complexity out of process models and thereby support faster change of business processes. This is a crucial requirement for transitioning BPM to focus on customer engagement. Process data virtualization also allows process changes to be pushed closer to the business, since it reduces the level of of involvement required by tech management for connecting processes and data.” TechRadar: Customer-centric BPM Technology, Q4 2014 Clay Richardson
  • 33.
    What does DataVirtualization software do? Turn Fragmented Data into Smart Data Data Virtualization software virtually unifies data spread across various disparate sources; and makes it available to applications as a single consolidated data source. The data virtualization software implements 3 steps process to bridge data sources and data consumers: • Connect: Fast access to data from diverse data sources • Compose: Easily create unified virtual data models and views by combining and transforming data from multiple sources. • Consume: Expose consistent information to data consumers in the right form thru standard data access methods. Virtual Consolidated Data Source BI Reports Data Virtualization Software • Consume • Compose • Connect SAP Salesforce.comOracle DW XML, CSV & Excel files Siloed & Complex Virtualize Abstract Federate Easy, Real-time Information Access SOA Applications DATA CONSUMERS DATA SOURCES Travel Agency Demo
  • 34.
  • 35.
    Demo – IntroJBoss BPM Travel Agency • The project • The workflow • The rules
  • 36.
    Demo – Acquisitionintegration • Data virtualization • External connection • Dashboard
  • 37.
    Demo – Bookingsregistration Data virtualization mapped to dashboard
  • 38.
    What does DataVirtualization software do? Turn Fragmented Data into Smart Data Data Virtualization software virtually unifies data spread across various disparate sources; and makes it available to applications as a single consolidated data source. The data virtualization software implements 3 steps process to bridge data sources and data consumers: • Connect: Fast access to data from diverse data sources • Compose: Easily create unified virtual data models and views by combining and transforming data from multiple sources. • Consume: Expose consistent information to data consumers in the right form thru standard data access methods. Virtual Consolidated Data Source BI Reports Data Virtualization Software • Consume • Compose • Connect SAP Salesforce.comOracle DW XML, CSV & Excel files Siloed & Complex Virtualize Abstract Federate Easy, Real-time Information Access SOA Applications DATA CONSUMERS DATA SOURCES Conclusion
  • 39.
    Benefits of BusinessProcess Data Virtualization • Easy access to data in real-time from multiple different sources • Improved data quality with a single version of the truth • More agility with simplified process models • Re-use of data integrations from multiple sources • Increased autonomy for lines of business with departmental virtual data bases • Centralized IT control and auditing of data sources
  • 43.