The Story of How an Oracle Classic Stronghold successfully embraced SOA (ODTUG 2009, actual presentation)

The Story of How an Oracle Classic Stronghold Successfully Embraced SOA
ODTUG Kaleidoscope 2009
Lucas Jellema
SOA

Overview
What is an Oracle stronghold?
Triggers to start moving towards Services
Levels of embracing Services and SOA
Objectives, benefits, costs & challenges
Demonstration
Pitfalls, Lessons Learned & Best Practices
Summary

What is an Oracle stronghold?
The typical Oracle stronghold
Using Oracle RDBMS & Oracle Development tools
Lot of SQL and PL/SQL
Probably Oracle Forms and maybe APEX as well
Possibly Oracle Designer, tools for BI & Reporting
Several databases with many years of essential corporate data
IT staff has Oracle veterans – 5-15 years or more
Internet development may have taken place largely separate from the Oracle technology stack

My central case…
Oracle Forms application tightly integrated with an Oracle RDBMS
The database is known by the name of the app
Containing jobs, workers, timesheets, payments,..
Used in hundreds of branches as well as the central (back)office
Main business driver that required attention
Business partners requested a programmatic interface to load multiple timesheets – to save on time and hassle (time again)

Similar cases …
Oracle Forms application for assigning homes offered SaaS-style
SaaS Customers want the details on houses and their availability published on a website
.NET applications need access to data in the Oracle Database
Customers want their local applications to interface (programmatically) with the SaaS application

Then wat happened… (2)
Car Lease company has various custom applications and databases per department
Business requires IT to support processes that go across those applications and databases
through a single, unified User Interface
involving a legacy database and a 3rd party ERP system
with eventually some self service web modules
Insurance company sells policies through agents using a Forms application
New direct channel: On-line policy selling

Then wat happened… (3)
Agricultural company supports ‘cow insemination’ process with Forms application
Farmers and inspectors need to be able to record data anytime and anywhere through PDAs (that run a .Net application)
Mid-sized chemical pharmaceutical company uses BoB stand-alone systems and databases
To allow for faster (near real-time) responses to customer demands and logistical challenges, tighter integration between the systems is needed

Common Characteristics
Cross Boundaries
Cross Technology - .Net, Uniface, Java, Tibco
Cross Channel – Back Office, Web, PDA, API
Cross User Group – Internal, Agents, Self Service
Cross Domain – Multiple departments & systems
Cross Enterprise – Interact with external partners
Data synchronization or consolidation
Management whim Vision
From Database angle: providing services

Objectives
Business Agility
(Faster) responses to changing demands
Or at least an urgent, currently pending demand
Creating new business from existing resources
IT Flexibility
Optimize locally without impact ‘globally’
Prepare for future developments
Lower costs
Through reuse, better integration, decoupling

Objectives
(longer term) Higher Quality and Faster Process execution
Automated data exchange cross boundaries
Workflow and task orientation
integrated, cross department, in a controlled way based on sound understanding of the business processes
Business Event driven interaction
Manage risks and fear

SOA = BAD

Business
Agility through
Decoupling
SOA =

Decoupling≈Managing Dependencies
minimize impact of change while maximizing reusability

Types of decoupling
Functional
Interface - Encapsulation of implementation
Design by Contract, Implement by Design
Technical
No proprietary technology, protocol, message format
Standards based (XML, HTTP, RSS, WSDL…)
Temporal
Asynchronous communication (separate response)
Development
Separate teams working in parallel based on mutually agreed interface definitions

Decoupling Applications & Data
Application(User Interface)
Data
Data

Data

Application
(User Interface)
Data

Application
(User Interface)
WorkflowEngine
CMS
Email
IM
Fax

Data Ownership
Data no longer exclusively owned by a single application
Data (query and manipulation) available via APIs, (web)services and open standards
For example based on XML, XSD, WSDL, SOAP, HTTP
Data Hubs are formalized, structured approach where data is completely separated from applications
All access is through services
No data duplication at all in the enterprise
Data ownership is separate process

Decoupling from Table to ESB+
http
WEBDAV
FTP
WSRP
http
WS/SOAP
WS*
WS*
WS*

Tables in Database
SQL for retrieval and manipulation
Data Model in plain “view”
Decoupling between DML and Retrieval
Retrieve data:
select e.empno, e.ename, d.dnamefrom emp join dept using (deptno)
Manipulate data:
insert into dept;insert into emp;

View in Database
Hide (encapsulate) Data Model
Manage access privileges
SQL for retrieval and manipulation
Instead Of trigger decouples DML operations
select id, name, department from emp_vw
Insert into emp_vw
Use case: new UI on top of ‘legacy’ data model

Package in Database
Hide (encapsulate) SQL
Procedure calls for retrieval and manipulation
Potentially complex data structures using Object Types and (nested) Cursors
HRM_MGR.get_emp( id) return emp_t
HRM_MGR.create_emp( emp_t);
Use case: tailor made business services to support clients that understand types
Or core service implementation around which a non-type wrapper is applied

Database Object Types

Package in Database (2)
Hide (encapsulate) SQL and Oracleand user defined Types
Procedure calls for retrieval and manipulation
Input and output parameters standard types only (string and number)
Complex datastructures: XML passed as string
HRM_MGR.get_emp( id) return string
HRM_MGR.create_emp( string);
Use case: packaged business services to support any client (that can access the DB)

Handling XML
Structured, multi-level data in a string: XML
Oracle Database has XMLType
Can be created from a String, will parse XML
Can validate against a schema definition (the XML data design)
Support XPath operations to retrieve specific bits and pieces from the XML document
Can do XSLT transformations of the incoming or outgoing XML

XML based PL/SQL interface

AQ for Asynchronous
Decouple consumer and providerin time – asynchronuous processing
Consumer is registered on the AQ
Usually a package that processes User Defined Type that is sent as payload in the AQ message
Use case: asynchronously processed one-way (fire-n-forget) requests
Potentially lenghty requests
High volume of requests

Hiding the database
http
WEBDAV
FTP
http
WS/SOAP
WS*
WS*
WS*

Publish package as http-based API using dbms_epg
http
Hide database protocol
Not its physical location nor the schema, and user authentication
HTTP communication is truly cross technology
Browser, Java, .Net, JavaScript & RIA clients, …
Approximation of RESTful services (very du jour)
Can publish in various formats
Text, HTML, CSV, JSON, XML, RSS
Use case:cross-technology, internal no WS*/ESB

HTTP API – directly on top of the RDBMS

Publish static resources on various protocols with XMLDB
Hide database protocol
support FTP, WEBDAV, HTTP(S)
Run scheduled batch jobs (PL/SQL) to periodically create & expose resources
Can also consume and process resources
Use case: cross technology need for retrieving slowly changing resources (CSV, XML)
Possibly uploading resources for batch processing
Use case:
http
WEBDAV
FTP

Publishing RDBMS resources in a decoupled fashion

SOAP based WebServices
http
WEBDAV
FTP
WSRP
http
WS/SOAP
WS*
WS*
WS*

SOAP WebServices
All messages (input and output) are XML
The message consists of two parts inside an envelope (a SOAP XML wrapper)
The header with meta-data
The body with the contents to be handled by or returned by the service
The WebService Definition Language (WSDL) document describes the service
An XML Schema Document (XSD) describes the structure of the XML messages
XSD is like an ERD or Table Design

Oracle RDBMS 11g - Native Database WebServices
WS/SOAP
Schema can be published throughnative database web services
Each package corresponds with a WSDL
Every program unit with an operation
WSDL and XSD are dynamically generated

http and https is supported
Limited control over WSDL & XSD
Use case: internal, cross technology, WS enabled client, no ESB or Application Server available

WebService in App Server based on PL/SQL package
Hide database
Protocol, location, authentication:everything handled by the application server
Use JPublisher (embedded in JDeveloper) to publish a PL/SQL package as WebService
JPublisher creates JAX-WS-annotated class, utility classes and possibly helper types in the database
Alternatively: create ADF BC Application Module & publish it as a WebService (Bulldog)
With support for SDO (Service Data Objects)
Use case: WS (SDO!) enabled client, no ESB
WS*

Enterprise Service Bus
App 2
External Partner
Service
Service
Service
Service
Service
App 1
Data
Data

The Enterprise Service Bus
Should first of all be considered ‘a pattern’
Virtualizes services – hides the real service from consumers
Deals with the physical location of the services
Adapts synchronous to a-synchronous and vv.
Can use multiple real services to offer one virtual (composite) service
Allows callers to use a generic, canonical message structure that it will transform to the service contract
It may even allow callers to use their own “lingo”

Canonical Data Model
Common Business Language
Esperanto for service invokers
Service
Service
Data
Data
Clients
Customers

The Enterprise Service Bus (2)
Handles various QoS & SLA aspects
sometimes in concert with tools like OWSM
Encryption, signing, authentication
Retry and fallback
“Throttle” (prevent peak loads)
Does monitoring, tracking & auditing, reporting, notification and escalation
Works with Adapters to access technologies
like RDBMS (SQL, PL/SQL), AQ and JMS, File System, FTP, Java, E-Business Suite

Publish PL/SQL Package through ESB
WS*
Use Database Adapter to createService, combine with ESB Routing Service
Use case:
external access to services
virtualize location of service –
route to service based on content of the request
virtualize part of contract of service
Package-derived XSD not suitable for consumers
handle peak loads
monitor service levels and trace service access

Enterprise Service Busin action
One team responsible for exposing services to external consumers
Working with those customers for establishing the contract and testing across the firewall
This team built from internally provided services
Team two worked inside database – providing package based API for granular services
Team three created ESB level database adapter and routing/mapping services
WS*

Evaluation – end of phase 1
Done well
Working “application” in production!
Worked together (across teams and departments) on the Service definition and the ‘XSD’
To determine the scope and granularity of services and operations for optimal reuse
To define
Management sponsorship (though IT mainly)
To be improved
More involvement from the business for defining both services and especially the canonical model

Done well
Achieved good way of working with database team developing packages underneath services
Developed a feel for using services rather than immediate database access
Built up XML and ESB skills
To be improved
Involvement of administrators
Setting up Dev-Test-Acceptance-Production
Managing Service End Points

Done well
Ambitions, drive - However: technology driven
To be improved
Define the Enterprise Architecture/BluePrint and work within its context
“Think globally, act locally”
Set up CEA – mandated by business and IT
Reduce number of service calls
Externally exposed services do not need to be 1:1 with database packages!
Granularity and Composite Services

To be improved
Share success with business and other IT teams
Establish a mechanism for sharing, exposing, finding, adopting SOA artefacts
Services, Canonical Data Model, Policies/SLAs, …
And also for ‘governing’ the life cycle: how to change services already being used?
(automated) Testing
Monitoring service traffic (some focus on SLA)
Peak load, response time/overhead, availability

End of Phase 1
Working service implementation and infrastructure – across departments
End-to-end from backoffice database to customer
“Foundation for innovation”
Lost the fears, ready for the next step
Basis for reuse and widening the scope
Shimmering light at the end of the tunnel for the Forms application

Improve & Broaden the scope of the Canonical Data Model
Involve business representatives for all of business
Add business terminology and (some) business rules that help describe the model
Build up translations of domain values and business object identies across ‘systems’
More accessible and consistent structure
using namespaces (domains), naming conventions and guidelines (element vs. attribute, data types)
Annotate the model and provide examples
Eridicate database model legacy or technology specific elements

Pending “Research”
How to make use of cache infrastructure to prevent services being called unnecessarily
Yet guaranteeing non-stale data
How do we process binary attachments through the layers of our architecture
Do we have some low-hanging-fruit to pick in order to gain some additional benefits
Design patterns, best practices, short-cuts, …
Security and Policies – what levels, mechanism

Phase 2
A new user interface (internet website) on top of the back office database
Partially covered by existing services and an existing Web Application and CMS
Business is also looking for Web 2.0, Social/Community-style interaction in Portal
Running process flows that potentially go from external workforce through branch to (multiple entities within the) back office
How to move from peak load batch processing

Introducing BPEL for Service
WS*
BPEL adds to service
Long running (stateful) ‘service instances’
Composite services that include
Multiple service calls (including asynchronous)
Exception handling including retry and compensation
Human Task for manual steps & integration Rule Engine
Process flow logic
Use case:
data request must be fulfilled by various services;
DML impacts several systems and/or requires human approval

Publish Business Events
Extremely Decoupled Architecture
Any system – including database – reports events that may be interesting to other parties
The Event backbone (could be the ESB)
Defines Event Types (name, structure of payload)
Registers Event Listeners (“please call me when the event occurs and send the details”)
Receives events – instances of the predefined event type with payload and timestamp
Propagate events to all registered listeners
Without blocking the event producer

Event Driven Architecture (EDA)
External Partner
App 2
Service
Service
Service
Service
Service
App 1
Data
Data

Database publishing events
WS*
Table Trigger intercepts DML
Checks for Business Events such as new employee
Sends them to package EVENT_PRODUCER
Package EVENT_PRODUCER sends events
Via UTL_HTTP to a WebService
Via AQ to a listener (ESB AQ Adapter)
EventProducer
EMP

Publishing Service with UI
Instead of only publishing a programmatic service interface
A service can be published with a User Interface; the service-with-UI is called: Portlet
The standard approach:
Portlet Container in Application Server exposes WSRP services for the portlets
The Portlet produces (X)HTML and handles HTTP requests
A Portal consumes the WSRP Portlets in a web page
WSRP

Decoupling from Table to ESB+
http
WEBDAV
FTP
WSRP
http
WS/SOAP
WS*
WS*
WS*

Increasingly decoupled
More hiding of the implementation
More Formal Interface Contract
Less (proprietary) technology & more standards for interacting
Less exposure of (legacy) data model
More support for asynchronous interaction
More reuse potential
Pervasive throughout enterprise
More suitable for external consumption

Comes at a cost…
More run time overhead
Additional tiers
XML serialization and deserialization
More infrastructure
Burden of Administration
License Costs
Hardware
Broader skills palette – more stuff to master
Harder to get started

Pitfalls
Do SOA from the IT/technology side only
Inconsistent, illegible, unstructured namespaces and (XML) canonical model
Having the database (table and column names) shine through in the canonical data model
And forgetting the database design wisdom
Reusable SOA artefacts are not found, understood nor trusted
Lack of Balance between reusability and usefulness (Fine grained vs. coarse grained)

Pitfalls
The greedy clutches of enterprise architects
Think globally, talk, (high level) design, draw & write, present, think, talk, …. (no real action)
… versus the technology driven, ’think and act locally’ approach from the ‘developer squad
Introducing new unmanaged dependencies
Hard coded endpoints (service URLs)
Calling external services without proper SLA or fallback option
Using complex technology without proper skills

Pitfalls
No DTAP process & environment
And/or completely manual
Involving DBAs/Administrators way too late
Inappropriate use of the SOA infrastructure
Web applications retrieving each individual record (or even field) through a separate service call
Running Forms on top of the ESB!
Sending debug and trace messages via the ESB
“Package calls other package via ESB”

Useful
Use Mock Service implementations during development and test
Developers that depend on services not yet available easily get stuck
Automated Functional and Performance Test of individual Services
Automatic Service ‘ping’ utility
Early detection of service unavailability

Summary
Objective: agility through decoupling
Managing dependencies
Crossing boundaries – functional, technology, time
Just do it! (well, “think big, do (small)”)
Get started – at the right level for your situation
Do not go off and buy BPEL just like that
Even though it won’t be perfect the first time round – you will learn (only) through experience
Do it explicitly, visibly and with all involved
SOA

Q &A
SOA