Dutch Railways and SOA Jack van Hoof JvH-Y2K+7 v1.0

This presentation Who am I What is Dutch Railways Vision of Dutch Railways on the application landscape Application level: Role of Web services technology at Dutch Railways Infrastructure level: Role of the ESB as a Web services platform at Dutch Railways Our view on the innovation roadmap - High level, no technical details -

Who am I

What is Dutch Railways

Vision of Dutch Railways on the application landscape

Application level: Role of Web services technology at Dutch Railways

Infrastructure level: Role of the ESB as a Web services platform at Dutch Railways

Our view on the innovation roadmap

Who am I? Jack van Hoof… The 21st century: Dutch Railways (Utrecht) IT-architect – Application Integration, SOA, ESB The 90-s: Dutch Defence (The Hague) Business consultancy System development Leading developmentteams The 80-s: IBM (Amsterdam/Uithoorn) Leading analyst of development teams Programming/design/analyses

The 21st century: Dutch Railways (Utrecht)

IT-architect – Application Integration, SOA, ESB

The 90-s: Dutch Defence (The Hague)

Business consultancy

System development

Leading developmentteams

The 80-s: IBM (Amsterdam/Uithoorn)

Leading analyst of development teams

Programming/design/analyses

http://soa-eda.blogspot.com - Thoughts on SOA and EDA - My weblog

http://soa-eda.blogspot.com

- Thoughts on SOA and EDA -

What is Dutch Railways? Passengers transportation by train (no freight transport) Owner of railway stations (maintenance and operation) Owner of buildings and terrains (biggest real estate owner in The Netherlands) Time tables Maintenance and repair of trains (not only Dutch Railways trains) Commerce (price of the tickets) Training of drivers and conductors

Passengers transportation by train (no freight transport)

Owner of railway stations (maintenance and operation)

Owner of buildings and terrains (biggest real estate owner in The Netherlands)

Time tables

Maintenance and repair of trains (not only Dutch Railways trains)

Commerce (price of the tickets)

Training of drivers and conductors

How big is Dutch Railways? Number of coaches: 3000 Number of stations: 400 Number of travellers per day : 1.000.000 Number of traveller-kilometers per day : 45.000.000 (1.000 x around the world) Number of “front-liners” (drivers, conductors, station-personel): 10.000 Total number of employees: 20.000

Number of coaches: 3000

Number of stations: 400

Number of travellers per day : 1.000.000

Number of traveller-kilometers per day : 45.000.000 (1.000 x around the world)

Number of “front-liners” (drivers, conductors, station-personel): 10.000

Total number of employees: 20.000

Our Vision

We strive for “new” technology Application infrastructure Enterprise Service Bus Portals Open standards XML Web services: SOAP / WSDL / WS-* WSRP JSR168 Architectural approaches SBI (Service Based Integration) SOA (Service Oriented Architecture) EDA (Event Driven Architecture) Driver: Not cost reduction, but necessary to survive

Application infrastructure

Enterprise Service Bus

Portals

Open standards

XML

Web services: SOAP / WSDL / WS-*

WSRP

JSR168

Architectural approaches

SBI (Service Based Integration)

SOA (Service Oriented Architecture)

EDA (Event Driven Architecture)

Why should we strive for “ new ” technology? The whole IT-world is converging into this direction Market analysts (Gartner): SOA / EDA Software vendors: XML / WSDL / SOAP / WS-* Organizations: Flexible business-to-business value chains with use of uniform/standard technology Ignoring these trends means getting away from your partners Modern software relies on modern technology (expensive to adapt new software to legacy) Environment demands uniform connectivity with regard to B2B value chains: take-it or leave-it Because everone does… so if we don’t want to get isolated we have to join Driver: Not cost reduction, but necessary to survive Innovation is no option, but a must in the current era

The whole IT-world is converging into this direction

Market analysts (Gartner): SOA / EDA

Software vendors: XML / WSDL / SOAP / WS-*

Organizations: Flexible business-to-business value chains with use of uniform/standard technology

Ignoring these trends means getting away from your partners

Modern software relies on modern technology (expensive to adapt new software to legacy)

Environment demands uniform connectivity with regard to B2B value chains: take-it or leave-it

Does Dutch Railways have an SOA? No… But we will, as everyone! All software suppliers are service enabling their application interfaces All software tools will enforce Web services technologies You can’t (in future) implement ERP (SAP) without implementing SOA (SOA out-of-the-box) Business partners will force us to communicate via standard technology (web services) SaaS providers will (in future) use SOA to allow for customized business process definitions accross multiple providers We anticipate by service enabling our legacy, and putting an adquate infrastructure in place

All software suppliers are service enabling their application interfaces

All software tools will enforce Web services technologies

You can’t (in future) implement ERP (SAP) without implementing SOA (SOA out-of-the-box)

Business partners will force us to communicate via standard technology (web services)

SaaS providers will (in future) use SOA to allow for customized business process definitions

accross multiple providers

Current situation: 500 “known” applications Every application has own solution/technology for : Access control User interface mechanism Data storage Application server Interface-mechanisms shared databases file shares (accross domains) Mail FTP remote calls No structure! No overview! Monolithic applications User interface and access control Business logic Data storage Interface

Every application has own solution/technology for :

Access control

User interface mechanism

Data storage

Application server

Interface-mechanisms

shared databases

file shares (accross domains)

Mail

FTP

remote calls

Getting structure: separation of concerns Generic solutions for : Application access (Portal) Data storage (DBMS, datawarehouse, backup/recovery) Standardized interfaces (ESB, secured, reliable) Application platform (application servers) Interaction based on open standards : XML, SOAP, WSDL, JSR168, WSRP, WS-* Effects : Better manageable Consistency (data, user interfaces, business logic) Cheaper deployments (one-time deployment) More flexible to change Business logic services Access services Exchange area Data storage services User interface and access control Business logic Data storage Interface

Generic solutions for :

Application access (Portal)

Data storage (DBMS, datawarehouse, backup/recovery)

Standardized interfaces (ESB, secured, reliable)

Application platform (application servers)

Interaction based on open standards :

XML, SOAP, WSDL, JSR168, WSRP, WS-*

Effects :

Better manageable

Consistency (data, user interfaces, business logic)

Cheaper deployments (one-time deployment)

More flexible to change

One virtual Enterprise application All business logic implementations must fit within framework : ERP - Enterprise oriented applications like SAP COTS - Function oriented packages Home made software Legacy - Old fashioned monolithic applications Flexibility and uniformity, but also basis for : Business Process Management Flexible definition of processes outside applications Business Activity Monoriting Real-time insight in operational business state Event processing React proactive to potential bottle-necks Business logic services ERP Access services Exchange area Data storage services COTS Home made Legacy Business Process Management Business Activity Monitoring (Complex) Event processing

All business logic implementations must fit within framework :

ERP - Enterprise oriented applications like SAP

COTS - Function oriented packages

Home made software

Legacy - Old fashioned monolithic applications

Flexibility and uniformity, but also basis for :

Business Process Management

Flexible definition of processes outside applications

Business Activity Monoriting

Real-time insight in operational business state

Event processing

React proactive to potential bottle-necks

Implementation technologies Business logic services ERP Access services Exchange Area Data storage services COTS Home made Legacy Business Process Management Business Activity Monitoring (Complex) Event Processing Web 2.0 clients (web browsers) AJAX mashups Portal with portlets Databases SOA and EDA ESB

Two important challenges H eterogenous technologies within Dutch Railways Programming languages Databases Operating systems Hardware Devices Gates at stations Camera’s Railpockets (PDA’s for front-liners) Information displays Heterogenous environments of Dutch Railways Geographical locations Stations (Moving) trains Data Centers External partners ProRail (rail infrastructure) Other transportation companies (time tables) Service providers (cleaning, maintenance) Retailers 1 2 Putting together:

H eterogenous technologies within Dutch Railways

Programming languages

Databases

Operating systems

Hardware

Devices

Gates at stations

Camera’s

Railpockets (PDA’s for front-liners)

Information displays

Heterogenous environments of Dutch Railways

Geographical locations

Stations

(Moving) trains

Data Centers

External partners

ProRail (rail infrastructure)

Other transportation companies (time tables)

Service providers (cleaning, maintenance)

Retailers

Current market trends are helping us Our needs Standards to harmonize different technologies Infrastructure to virtualize different locations Market trends Suppliers strive for uniformity and standardization - Web services Generic distributed integration technologies are coming to market - ESB

Our needs

Standards to harmonize different technologies

Infrastructure to virtualize different locations

Market trends

Suppliers strive for uniformity and standardization - Web services

Generic distributed integration technologies are coming to market - ESB

We go for Web services and ESB By using standard Web services technology everything can be connected with everything Enterprise Service Bus (ESB) is our deployment platform for Web services technology Application level Infrastructure level

The application level

Application innovation: 3 different focus points Service Based Integration (SBI): focus on monolithic applications – wrapping interfaces Service Oriented Architecture (SOA): focus on modular application construction - “call” services Event-driven Architecture (EDA): focus on business events – publish and consume messages

Service Based Integration (SBI): focus on monolithic applications – wrapping interfaces

Service Oriented Architecture (SOA): focus on modular application construction - “call” services

Event-driven Architecture (EDA): focus on business events – publish and consume messages

SBI: focus on monolithic application New Connect monolithic applications (legacy, COTS, ERP) using Web services technology to harmonize different technologies Keywords: Legacy, stove-pipes, packaged software (COTS, ERP) Old Looking for common technologies to communicate e.g: B with C via fileshare on server of A Application B Application C Application A Web Services Application B Application C Application A

SOA: focus on modular application construction Reusable components deliver services to each other Process definitions outside the components (Process Manager) Ultimate: one enterprise wide box of building blocks Keywords: Strong cohesion, command-and-control, reuse, functional decomposition Calling services using Web services technology B1 B2 C3 C1 C2 A3 SOA A1 A4 B3 A2

Reusable components deliver services to each other

Process definitions outside the components (Process Manager)

Ultimate: one enterprise wide box of building blocks

Calling services using Web services technology

EDA: focus on event messaging Events Web Services (SOAP) Applications (legacy, SOA’s, workflows, transactions, processes, UI’s, portals, databases, gateways, devices) Keywords: Loose coupling, linking autonomous processes, workflow Publish and consume messages using Web services technology

Simple example of EDA (illustrative) ESB Passenger x Train Routing table Gates at stations a, b, c, d Back-end systems SOAP SOAP SOAP SOAP SOAP … buys ticket from A to B via Internet (business event) … Determines stations on route A-D (enrichment) … allow access to passenger x on date y … register transaction Passenger: x Date: y Stations: a, b, c and d Passenger: x Date: y Route: A-D Data- warehouse SOAP … logs data ibo analyses and rapporting 1 2 3 4 5 Concurrent with: Concurrent with:

Holistic approach at Dutch Railways Access to old and packaged systems: Service based integration Development of new systems (processes): Service oriented architecture Connecting systems (processes) into chains: Event-driven architecture Not subsequently, but all three concurrently! And all with the same common technology: Web services

Web Services: lubricating oil between old and new Common technology allows for integration of legacy with SOA and EDA Legacy Z Legacy Y S1 S4 S3 S5 S2 SOA / EDA SBI Web services technology “ Old” “ New”

Common technology allows for integration of legacy with SOA and EDA

The infrastructure level - distributed web services platform -

Web services alone are not enough How do you deal with semantics ? Is “customer name” only last name, or first name AND last name …? How do you deal with security ? Who is allowed to see or change a message? (confidentiallity, integrity) Who is allowed to send or receive a message? (authorization, authentication) How do you prove a message has been sent/received? (non-repudiation) How do you deal with transactions ? How do you define a transaction and how will it be performed and rolled back? How do you deal with workflow ? How is routing of messages defined and monitored? (intelligent routing when paths are blocked) How do you deal with format transformation between sender and receiver ? How and where is data transformed during transport? How do you deal with reliability ? How is guaranteed that a message reaches its destination? How is guaranteed that a message is processed only once? How is guaranteed that a message is processed in time? Where does a published message persist before it is consumed? ? ? ? ? ? ?

How do you deal with semantics ?

Is “customer name” only last name, or first name AND last name …?

How do you deal with security ?

Who is allowed to see or change a message? (confidentiallity, integrity)

Who is allowed to send or receive a message? (authorization, authentication)

How do you prove a message has been sent/received? (non-repudiation)

How do you deal with transactions ?

How do you define a transaction and how will it be performed and rolled back?

How do you deal with workflow ?

How is routing of messages defined and monitored? (intelligent routing when paths are blocked)

How do you deal with format transformation between sender and receiver ?

How and where is data transformed during transport?

How do you deal with reliability ?

How is guaranteed that a message reaches its destination?

How is guaranteed that a message is processed only once?

How is guaranteed that a message is processed in time?

Where does a published message persist before it is consumed?

Supporting platform: The Enterprise Service Bus SOA (Process X) Gateway External systems Legacy (cusotm, package) SOA (Process Y) Package (COTS, ERP) Enables support for : Secured routing of messages Guaranteed delivery of messages Instant availability of messages within the enterprise Semantic mapping between systems (CDM) Format harmonisation between systems (CDM) Explicit definition of busniness processes (BPM) Insight in actual state of business processes (BAM) Detect correlations between messages (CEP) Access to services (SOA) Publish and consume messages Global Dataspace - ESB -

Enables support for :

Secured routing of messages

Guaranteed delivery of messages

Instant availability of messages within the enterprise

Semantic mapping between systems (CDM)

Format harmonisation between systems (CDM)

Explicit definition of busniness processes (BPM)

Insight in actual state of business processes (BAM)

Detect correlations between messages (CEP)

Access to services (SOA)

Reach of our ESB (future) ESB Stations Trains PDA’s Data Centers Railpockets Gates … Partners Gateway All places where applications are running

Appearance of our ESB ESB SAP-FIN Netweaver XI SAP-CRM Netweaver XI SAP-HR Netweaver XI IBM WebSphere (Corporate) BEA WebLogic ESB Partner Gateway Etcetera (future) BizTalk Ddatagateway (MQ-Series) Remaining applications External environment (business partners)

ESB: Intelligent layer on the network Messages flowing through ESB safely cross firewalls over HTTP port 80: No connectivity issues anymore! Connectivity Enterprise Service Bus Network Security Reliability Transactions … User-defined Host-config: DHCP Name: DNS Time: SNTP Netwerkmngt: SNMP … Technical infrastructure oriented services Business oriented services

ESB virtualizes location and technology ESB Model of application landscape Location and technology virtualization Connectivity Location and technology of applications are made invisible (unimportant) by the ESB Visibility and overview of interaction between applications Application landscape becomes manageable by visual tools ESB: - Tools to get overview - Virtualizes locations and technology - Distributed presence at locations Network: - Technical connection between locations Locations: - Places where applications run

Location and technology of applications are made invisible (unimportant) by the ESB

Visibility and overview of interaction between applications

Application landscape becomes manageable by visual tools

ESB:

- Tools to get overview

- Virtualizes locations and technology

- Distributed presence at locations

Network:

- Technical connection between locations

Locations:

- Places where applications run

Infrastructural characteristics of an “ideal” ESB Based on open standards Also internal mechanismes based on web services technology Makes interoperability with other ESB’s easy Makes distributed architectures possible Light weight Adding functionality on demand over time when needed Start on small scale and grow over time when needed Distributed architecture Intelligent components deployed near the endpoints (like agents) Communicating components, no central hub processing (very scalable) Only central rules repository; changes are published to local components

Based on open standards

Also internal mechanismes based on web services technology

Makes interoperability with other ESB’s easy

Makes distributed architectures possible

Light weight

Adding functionality on demand over time when needed

Start on small scale and grow over time when needed

Distributed architecture

Intelligent components deployed near the endpoints (like agents)

Communicating components, no central hub processing (very scalable)

Only central rules repository; changes are published to local components

Current technology trends In a few years the network will be the ESB The network (CISCO) Network devices (routers) are getting XML-aware Network decives are getting SOAP-aware Network devices will understand Web services standards (WS-*) The ESB (IBM) XML-processors are getting build into appliances (hardware) SOAP-message security is getting managed by appliances SOAP-messages are filtered, blocked and routed by appliances Web services standards (WS-*) are getting implemented by appliances

The network (CISCO)

Network devices (routers) are getting XML-aware

Network decives are getting SOAP-aware

Network devices will understand Web services standards (WS-*)

The ESB (IBM)

XML-processors are getting build into appliances (hardware)

SOAP-message security is getting managed by appliances

SOAP-messages are filtered, blocked and routed by appliances

Web services standards (WS-*) are getting implemented by appliances

Innovation roadmap

Two innovation approaches Top down Start from business functions perspective Requires highly structured and mature IT-organization Bottom up Start from application integration perspective Dutch Railways takes this approach

Top down

Start from business functions perspective

Requires highly structured and mature IT-organization

Bottom up

Start from application integration perspective

Dutch Railways takes this approach

Our bottom-up approach to innovation: ESB as the platform FTP file transports over the ESB Record (message) oriented data transport over the ESB – slicing into SOAP messages and rebuild Shift focus from applications and batch-files to real-time point-to-point messages Scale up - start limited and let it grow Define intermediate (canonical) layer of message types to decouple senders from receivers Decompose applications into documented – reusable - components (services) Start using tools for Business Process Management and Business Activity Monitoring By connecting applications to the ESB, every application can have its own pace to innovation without affecting the other applications Old world can talk with new world via the ESB Increasing maturity over time A C D B = Transformation Canonical F F E A C D B E Receiving applications Sending applications Step 3 Step 5

FTP file transports over the ESB

Record (message) oriented data transport over the ESB – slicing into SOAP messages and rebuild

Shift focus from applications and batch-files to real-time point-to-point messages

Scale up - start limited and let it grow

Define intermediate (canonical) layer of message types to decouple senders from receivers

Decompose applications into documented – reusable - components (services)

Start using tools for Business Process Management and Business Activity Monitoring

SOA SOA SOA SOA Event- msg Event- msg Event- msg ERP COTS Legacy Event- msg External Gateway External Systems Event- msg Event- msg SOA, COTS, ERP, legacy and external systems = heterogenous and flexible application landscape with EDA = Heterogenous systems are loosely (asynchronously) coupled via triggering event messages

Competence Center of Integration

Competence Center of Integration to help Integration process Define message flows (data analyses) Build message flows through ESB Buy/build/deploy adapters/wrappers Buy/build/adapt/decompose business applications Operate/manage message flows through ESB Test and user acceptance CCI CCI advises development teams and delivers specialists (designers/developers) CCI does the work Focus is on message flows

Deliverables of the CCI Toward development teams : Deliver (templates, solution patterns, guidelines) Give advise (guidance during development projects) Coordinate communciations between departments and with external parties Enterprise wide metadatamanagement (XML-schema’s, semantics) Deliver specialists (developers) to help with: Adapt legacy to connect to the ESB Connect new applicaties/services to the ESB Choose/build/configure adapters and wrappers Configurate message flows (e.g. routing, security, enrichment, transformations) Facilitate test channels in the ESB Facilitate separation of functional domains in the ESB Functional adminstration of cross domain communication (interface-configuration) If desired: functional administration of communication within domains Functional life-cycle management of the ESB-infrastructure plus appurtenant tools Deploy infrastructural ESB-facilities at distributed locations - final slide -

Toward development teams :

Deliver (templates, solution patterns, guidelines)

Give advise (guidance during development projects)

Coordinate communciations between departments and with external parties

Enterprise wide metadatamanagement (XML-schema’s, semantics)

Deliver specialists (developers) to help with:

Adapt legacy to connect to the ESB

Connect new applicaties/services to the ESB

Choose/build/configure adapters and wrappers

Configurate message flows (e.g. routing, security, enrichment, transformations)

Facilitate test channels in the ESB

Facilitate separation of functional domains in the ESB

Functional adminstration of cross domain communication (interface-configuration)

If desired: functional administration of communication within domains

Functional life-cycle management of the ESB-infrastructure plus appurtenant tools

Deploy infrastructural ESB-facilities at distributed locations

FIN

Questions?

Hidden sheets

Basic principles on IT Organizational flexibility IT-support of business processes must be agnostic for future reorganizations IT-support of business processes must be able to easy adapt to process changes We think new technology will help Technological flexibility IT-support of business processes must be able to easy adapt to technological (r)evolutions This leads to : Organizational Strive for loose coupling between separate business functions (redundant data/services, asynchronous data exchange) Allow for strong cohesion within coherent business functions (shared data/services, synchronous data exchange) Technological IT will be based on modular architectures and open standards (= open architectures) Software components will map one-to-one with autonomous business functions Autonomy Efficiency

Organizational flexibility

IT-support of business processes must be agnostic for future reorganizations

IT-support of business processes must be able to easy adapt to process changes

Technological flexibility

IT-support of business processes must be able to easy adapt to technological (r)evolutions

This leads to :

Organizational

Strive for loose coupling between separate business functions (redundant data/services, asynchronous data exchange)

Allow for strong cohesion within coherent business functions (shared data/services, synchronous data exchange)

Technological

IT will be based on modular architectures and open standards (= open architectures)

Software components will map one-to-one with autonomous business functions

Loose coupling required Autonomous business function 1 Sub-function 1a Sub-function 1b Sub-function 1e Sub-function 1c Sub-function 1d Autonomous business function 2 Basic principles illustrated Supporting software component 1 (service) Supporting software component 2 (service) Reusable components One-to-one mapping Autonomy Strong cohesion allowed Sub-function 2a Sub-function 2b Sub-function 2e Sub-function 2c Sub-function 2d

Event- msg Event- msg Event- msg Business process chain: EDA pub pub pub sub sub sub Command and Control: SOA Data and services reuse domain Data and services reuse domain Data and services reuse domain Data and services reuse domain Final pattern Decoupling borders Loose coupling Asynchronous communication (publish-subscribe) Strong cohesion Synchronous communication (request-reply) Specific per situation “ craftsmanship” of the architect