Collecting and analyzing data for valuable decision
making in a service oriented business scenario
Stefano SCAMUZZO
ENGINEERING Group (www.eng.it)
Technical, Innovation & Research Division
Architectures & Consulting

SPAGOWORLD, Executive Board (www.spagoworld.org)
SPAGOBI, Ecosystem Manager (www.spagobi.com)
Engineering Group

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Everyone needs services

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
But … what is a service ?
The Core concept in aaService Oriented model is the concept of Service which is subject to
The Core concept in Service Oriented model is the concept of Service which is subject to
interpretations that make its meaning quite ambiguous
interpretations that make its meaning quite ambiguous
Different points of view are possible:
Different points of view are possible:
The Business Analyst point of view: aaservice is aafunctionality, part of aaBusiness
The Business Analyst point of view: service is functionality, part of Business
Process
Process
The Designer point of view: aaservice is aasoftware module with aaprecise functional
The Designer point of view: service is software module with precise functional
identity that can be invoked through aawell defined and documented public interface
identity that can be invoked through well defined and documented public interface

We prefer to talk in terms of processes and components
We prefer to talk in terms of processes and components
The Business Analyst analyzes the processes so as to detect the functional
The Business Analyst analyzes the processes so as to detect the functional
components he needs to build the process
components he needs to build the process
The Designer translates the identified components into “services”
The Designer translates the identified components into “services”

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
What methodology to use ?
Bottom-up

Top-down

Leverage the existing applications and
start building services as an assembly
of existing components

Start from business models to identify a
complete plan of the services to be built,
then iterate on their design and
implementation

benefits: time reduction
risks: the developed services have a
shorter lifecycle and demand frequent
maintenance and refactoring

benefits: tidy and complete architecture
risks: huge initial investment for analysis
an identification of the services plan

Meet-in-the-Middle (Agile Delivery):
Gradual definition of the services plan and parallel implementation of highest priority
services. When the service plan has sufficiently progressed, services can be revised to
make them compliant with the plan.
The goal: to balance the need for a tidy design with the demands of a rapid development
and time to market.

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Synthesis of the methodology
Enterprise
Enterprise
Business Models
Business Models
Definition
Definition

Technological
Technological
Architecture
Architecture
Definition
Definition

Service-Oriented
Service-Oriented
Analysis
Analysis

Service Plan
Service Plan
Definition
Definition

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
The Universal approach
Universal Middleware
Integration

Processes

Integration
ESB

IT & Business
goals

Cooperation
Connectors

Orchestration
Collaboration
BPM

eForm Lifecycle Business Rules SOA Application
PDF
management (Web – Mobile)

Governance
Innovation / reuse
(FI – mobile)

Management
& Intelligence

Adaptable Middleware Services

Design
& develop.
Tools IDE

Process
& services
management

Business
Activity
monitor

Core components (service bundles) for the integration and management of services; for the
Core components (service bundles) for the integration and management of services; for the
definition of business rules and the realization of application modules in SOA architectures.
definition of business rules and the realization of application modules in SOA architectures.
By parameterizing each component, the solution gains aaconsiderable modularity, allowing
By parameterizing each component, the solution gains considerable modularity, allowing
to meet even the most complex requirements.
to meet even the most complex requirements.

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case history: a University
Defining the architectural framework to support new projects following aacollaborative
Defining the architectural framework to support new projects following collaborative
development model based on modular software components that interact using consistent,
development model based on modular software components that interact using consistent,
shared and integrated information.
shared and integrated information.
The development of aaservice is split into phases: initial integration of the existing systems
The development of service is split into phases: initial integration of the existing systems
followed by aare-engineering process
followed by re-engineering process
Requirements ?

Where is the information ?

New components ?

Reuse ?
Deploying components
on infrastructure nodes

Standards
Connectors
ARIS -> BPMN
Application services
Infrastructural services

Monitoring &
Management

New Platform

Unified Information
Methodology

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case history: a University
Explicitly expressed
Explicitly expressed
No application silos
Workflow
No application silos
Workflow
Interoperability
Document Management
Interoperability
Document Management
Scalability
Unified data model
Scalability
Unified data model
Flexibility
SSO & Security/Profiling
Flexibility
SSO & Security/Profiling
Standardization
Governance
Standardization
Governance
Proposed
Iterative development - initial integration of existing applications
ARIS processes modeled by the Customer
Low organizational impact
Performance
Scalability (SLA)
Availability
Knowledge gain

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case history: a University
Functional areas

Processes
& services

Unified data
model

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case history: a University
Reusing Jar Library

Jar Library

R

Develop new
component

Transformation

New
Requirement

OSGi Bundles

Implements
Spagic Interface

Spagic Bundles

Config
Publishing rules

Deploy

Supported by Eclipse Visual Tools
Spagic Service
Manager...

Spagic MetaDB
Registration

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case history: a University
G
o
v
e
r
n
a
APPLICATION
n
GOVERNANCE
c
e
PROJECT
MANAGEMENT

Services and Contract
Manager

GOVERNANCE SERVICES
KNOWLEDGE
MANAGEMENT

SERVICES
GOVERNANCE

SECURITY
GOVERNANCE

INFRASTRUCTURE
GOVERNANCE

Analysis and
Development Team
Users
Services/Applications

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.

Operations
Governance requires BI / BAM
Real time monitor for processes, services and relevant information
Error management and process restart
Events generated by systems monitor
Support to decision making

MetaDB
(real time)
Data
Warehouse

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Business Activity Monitoring
BAM systems detect events and make them
meaningful to business users
Data Source(s)

Console &
Dashboards

Data

EVENT VISUALIZATION

Alarms &
Notifications

Complex Event
Processing (CEP)

Events

EVENT DETECTION

EVENT NOTIFICATION

RT & Historical
Analysis

DWH STORAGE

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.

BUSINESS USERS
Case History: BAM
Web-based authenticated access

Data Sources:
SAP, CRM, backoffice, technical
Real Time Console
& Dashboards

BPM System
(TIBCO)

Other SpagoBI
Engines

REAL TIME
HISTORICAL
ANALYSIS

Event Manager
DB (events)

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case History: BAM

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Case History: BAM
Alarms for critical situations (bottleneck, SLA)
Overview and detail

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Data as a service
Structured Data

Transactional
Time phased data

Unstructured Data

Text, Social
User feedbacks

VOLUME
VELOCITY
VARIABILITY

Sensor Data

RFID, GPS
QR codes

New data types

Video, voice
Images

VVVV…

Big Data
Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Data as a service

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
(Big) Data as a service
Infrastructure
On Site
IaaS

Data Management
Capture
Clean
Load
Store

ETL

View and Analyze
Exploration, navigation, presentation
Text analysis
Text mining

BI

Application
Cloud
SaaS

SERVICES
Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Big Data Application fields
Brand Reputation
Sentiment analysis
Social listening
Social CRM
Complaint Analysis
Competitive intelligence
Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Big Data Scenarios

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Big Data approach

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Open Data

Out-of-date
Inhomogeneous
Multi-format
Different detail levels
Dispersed

Growing
Public and private sectors
Structured
Measurable quality
It’s the momentum …

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Open data: case history

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Open data: case history

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Open data: case history

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Open data: case history

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Open data: case history
USAGE SCENARIOS

in Ser vices. . .
...t o help companies
per f or m mar ket
r esear ches

in Trade. . .
...t o analyse demands
and emer ging t r ends

in Tur ism. . .
...t o compar e exist ing
r ange wit h available
of f icial st at ist ics

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
How open source helps

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
New skills requires

Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.

Collecting and analyzing data for valuable decision making in a service oriented business scenario

  • 1.
    Collecting and analyzingdata for valuable decision making in a service oriented business scenario Stefano SCAMUZZO ENGINEERING Group (www.eng.it) Technical, Innovation & Research Division Architectures & Consulting SPAGOWORLD, Executive Board (www.spagoworld.org) SPAGOBI, Ecosystem Manager (www.spagobi.com)
  • 2.
    Engineering Group Copyright ©2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 3.
    Everyone needs services Copyright© 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 4.
    But … whatis a service ? The Core concept in aaService Oriented model is the concept of Service which is subject to The Core concept in Service Oriented model is the concept of Service which is subject to interpretations that make its meaning quite ambiguous interpretations that make its meaning quite ambiguous Different points of view are possible: Different points of view are possible: The Business Analyst point of view: aaservice is aafunctionality, part of aaBusiness The Business Analyst point of view: service is functionality, part of Business Process Process The Designer point of view: aaservice is aasoftware module with aaprecise functional The Designer point of view: service is software module with precise functional identity that can be invoked through aawell defined and documented public interface identity that can be invoked through well defined and documented public interface We prefer to talk in terms of processes and components We prefer to talk in terms of processes and components The Business Analyst analyzes the processes so as to detect the functional The Business Analyst analyzes the processes so as to detect the functional components he needs to build the process components he needs to build the process The Designer translates the identified components into “services” The Designer translates the identified components into “services” Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 5.
    What methodology touse ? Bottom-up Top-down Leverage the existing applications and start building services as an assembly of existing components Start from business models to identify a complete plan of the services to be built, then iterate on their design and implementation benefits: time reduction risks: the developed services have a shorter lifecycle and demand frequent maintenance and refactoring benefits: tidy and complete architecture risks: huge initial investment for analysis an identification of the services plan Meet-in-the-Middle (Agile Delivery): Gradual definition of the services plan and parallel implementation of highest priority services. When the service plan has sufficiently progressed, services can be revised to make them compliant with the plan. The goal: to balance the need for a tidy design with the demands of a rapid development and time to market. Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 6.
    Synthesis of themethodology Enterprise Enterprise Business Models Business Models Definition Definition Technological Technological Architecture Architecture Definition Definition Service-Oriented Service-Oriented Analysis Analysis Service Plan Service Plan Definition Definition Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 7.
    The Universal approach UniversalMiddleware Integration Processes Integration ESB IT & Business goals Cooperation Connectors Orchestration Collaboration BPM eForm Lifecycle Business Rules SOA Application PDF management (Web – Mobile) Governance Innovation / reuse (FI – mobile) Management & Intelligence Adaptable Middleware Services Design & develop. Tools IDE Process & services management Business Activity monitor Core components (service bundles) for the integration and management of services; for the Core components (service bundles) for the integration and management of services; for the definition of business rules and the realization of application modules in SOA architectures. definition of business rules and the realization of application modules in SOA architectures. By parameterizing each component, the solution gains aaconsiderable modularity, allowing By parameterizing each component, the solution gains considerable modularity, allowing to meet even the most complex requirements. to meet even the most complex requirements. Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 8.
    Case history: aUniversity Defining the architectural framework to support new projects following aacollaborative Defining the architectural framework to support new projects following collaborative development model based on modular software components that interact using consistent, development model based on modular software components that interact using consistent, shared and integrated information. shared and integrated information. The development of aaservice is split into phases: initial integration of the existing systems The development of service is split into phases: initial integration of the existing systems followed by aare-engineering process followed by re-engineering process Requirements ? Where is the information ? New components ? Reuse ? Deploying components on infrastructure nodes Standards Connectors ARIS -> BPMN Application services Infrastructural services Monitoring & Management New Platform Unified Information Methodology Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 9.
    Case history: aUniversity Explicitly expressed Explicitly expressed No application silos Workflow No application silos Workflow Interoperability Document Management Interoperability Document Management Scalability Unified data model Scalability Unified data model Flexibility SSO & Security/Profiling Flexibility SSO & Security/Profiling Standardization Governance Standardization Governance Proposed Iterative development - initial integration of existing applications ARIS processes modeled by the Customer Low organizational impact Performance Scalability (SLA) Availability Knowledge gain Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 10.
    Case history: aUniversity Functional areas Processes & services Unified data model Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 11.
    Case history: aUniversity Reusing Jar Library Jar Library R Develop new component Transformation New Requirement OSGi Bundles Implements Spagic Interface Spagic Bundles Config Publishing rules Deploy Supported by Eclipse Visual Tools Spagic Service Manager... Spagic MetaDB Registration Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 12.
    Case history: aUniversity G o v e r n a APPLICATION n GOVERNANCE c e PROJECT MANAGEMENT Services and Contract Manager GOVERNANCE SERVICES KNOWLEDGE MANAGEMENT SERVICES GOVERNANCE SECURITY GOVERNANCE INFRASTRUCTURE GOVERNANCE Analysis and Development Team Users Services/Applications Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved. Operations
  • 13.
    Governance requires BI/ BAM Real time monitor for processes, services and relevant information Error management and process restart Events generated by systems monitor Support to decision making MetaDB (real time) Data Warehouse Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 14.
    Business Activity Monitoring BAMsystems detect events and make them meaningful to business users Data Source(s) Console & Dashboards Data EVENT VISUALIZATION Alarms & Notifications Complex Event Processing (CEP) Events EVENT DETECTION EVENT NOTIFICATION RT & Historical Analysis DWH STORAGE Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved. BUSINESS USERS
  • 15.
    Case History: BAM Web-basedauthenticated access Data Sources: SAP, CRM, backoffice, technical Real Time Console & Dashboards BPM System (TIBCO) Other SpagoBI Engines REAL TIME HISTORICAL ANALYSIS Event Manager DB (events) Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 16.
    Case History: BAM Copyright© 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 17.
    Case History: BAM Alarmsfor critical situations (bottleneck, SLA) Overview and detail Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 18.
    Data as aservice Structured Data Transactional Time phased data Unstructured Data Text, Social User feedbacks VOLUME VELOCITY VARIABILITY Sensor Data RFID, GPS QR codes New data types Video, voice Images VVVV… Big Data Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 19.
    Data as aservice Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 20.
    (Big) Data asa service Infrastructure On Site IaaS Data Management Capture Clean Load Store ETL View and Analyze Exploration, navigation, presentation Text analysis Text mining BI Application Cloud SaaS SERVICES Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 21.
    Big Data Applicationfields Brand Reputation Sentiment analysis Social listening Social CRM Complaint Analysis Competitive intelligence Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 22.
    Big Data Scenarios Copyright© 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 23.
    Big Data approach Copyright© 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 24.
    Open Data Out-of-date Inhomogeneous Multi-format Different detaillevels Dispersed Growing Public and private sectors Structured Measurable quality It’s the momentum … Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 25.
    Open data: casehistory Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 26.
    Open data: casehistory Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 27.
    Open data: casehistory Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 28.
    Open data: casehistory Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 29.
    Open data: casehistory USAGE SCENARIOS in Ser vices. . . ...t o help companies per f or m mar ket r esear ches in Trade. . . ...t o analyse demands and emer ging t r ends in Tur ism. . . ...t o compar e exist ing r ange wit h available of f icial st at ist ics Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 30.
    How open sourcehelps Copyright © 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 31.
    New skills requires Copyright© 2013 Engineering Group, SpagoBI Competency Center. All rights reserved.
  • 32.
    Copyright © 2013Engineering Group, SpagoBI Competency Center. All rights reserved.