Session CON8736

Handling Large Files with
Oracle SOA Suite and Managed
File Transfer Pass-by-Reference
Dave Berry, Oracle
Senior Manager, Integration Products
Robert van Molken, AMIS
Senior Oracle Integration Specialist
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle Fusion Middleware
BUSINESS INNOVATION PLATFORM

Web

Social

Mobile

User Engagement
Business
Process
Management

Content
Management

Service Integration

 Complete
 Integrated

Business
Intelligence

Data Integration

 Best-in-class
 Open

Identity Management
Development
Tools

Cloud Application
Foundation

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Enterprise
Management

 On-premise and Cloud
Program Agenda

 What is MFT and why you need it?
 Functional Use Cases
 Real World Use Case and Demo
 Discussion

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The growing problem with FTP in the enterprise
Lack of control, visibility, security & reliability
Manufacturing
Design partners

 Uncontrolled proliferation

of FTP servers & clients
Product
specifications

 No global visibility into

exchange of crucial data files
– including customer data

Distributors
Legal
firm

Catalogs

Enterprise

Media
(photos,
videos)
PR company

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Contracts

Employee
Records
HR consulting
firm

 Not integrated with

enterprise security
standards: FTP servers are
rarely integrated with
directories
 Single points of failure: FTP

servers rarely offer HA
capabilities
What Is Managed File Transfer?
Market Maturing, Consolidating and approaching $1B annually
 Simple and Secure End-to-End Managed File Gateway
– Large files, encryption, auditing, monitoring, pass-by-reference

 Standards Based Middleware Integrations
– (s)FTP, SOA, B2B, Service Bus, Web Services …

 Lightweight Web based Design Time Interface
– Easily build, edit and deploy end-to-end transfers

 Additional Characteristics
– Scheduling, Embedded sFTP server, FTP proxy, no transformations
– Advanced Management: Pause, Resume, Resubmit

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle MFT Feature Set
 Core
– “Embedded” FTP(S)/SSH, FTP / FTP-SSL / SSH

server

 File Delivery
– Very Large Files ~ 500GB+

– Pass by Reference FTP or File, Inline

– File transfer auditing

– Scheduling and Auto Retry

– Extensible framework – pre/post processing

– Pause and Resume Target endpoints

– HA clustering

– HTTP, JCA, FTP, In-memory

– SOA/B2B integration and delivery headers

– Notifications

– Scalable architecture

– Actions

 Monitoring and Management
– Web UI: Managed Server
 Designer, Monitor, Administration
– Enterprise Manager: Admin Server

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

– Zip Compress/Decompress
– PGP Encrypt/Decrypt
– Callouts
– Archive, Delete, Move, Custom
MFT Use Cases
Use Case Categories

 Internal Business Integrations
– Application to Application File sharing: SOA/SB integrations

 External Partner Integrations
– (s)FTP File gateway outside firewall. B2B integration, EDI ...

 COTS/Cloud
– OOTB End-to-end delivery channel, A2A …

 Other
– File management beyond business integration.
– Media, software delivery, smart phones …
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Functional Use Case Patterns
MFT

Chained

Fan Out

Direct

MFT

MFT

MFT

MFT

SOA
MFT

Direct

Indirect

SOA

SOA

MFT

MFT

B2B

B2B

SOA

MFT

MFT

MFT

B2B
Direct

MFT

B2B

Enterprise
Hybrid

MFT

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

B2B

SOA

MFT

MFT
Web-based UI Functionality (MFT UI + EM)
 Design Configuration
– Set up Transfers, Sources, Targets

 Monitoring and Operations Console
– Application-level functionality, diagnosis, resubmit …
– Dashboard: Metrics, Find File, Activity Deliveries, Recent Errors, Flow instance Reports

 System Administration/EM
– System-level functionality, Server Parameters, EXP/IMP …

 JDeveloper Bindings
– Composite Bindings/BPEL Services

 Enterprise Manager
– MFT App Configuration, Mbeans, global defaults

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT JDeveloper Binding
Integrated MFT JDeveloper Component

• SOA Browse and MFT
• Creates an SOA SCA Reference to MFT

• MFT Browse and Consume SOA
• Creates an SOA SCA Service from MFT
MFT

SOA

SOA

MFT

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Composite
Bindings

BPEL Services
Transfer Flow Design

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Runtime Transfer Flow Report

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Monitoring Dashboard

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Transfer Report

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Partner Profile
 Robert van Mölken
 Specialist in SOA Suite & OSB since 2007
 Oracle SOA Suite 12c and MFT Beta tester
 Linkedin: www.linkedin.com/in/rvmolken
 Blog: technology.amis.nl
 Twitter: @robertvanmolken

 AMIS was founded in 1991
 90+ employees & 1000+ years Oracle expertise

 2 ACE directors

3 ACEs

 Oracle specialized partner in Database, SOA en ADF
 Oracle Middleware partner 2011 & 2013
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Real World Use Case Overview
Title
Requesting building permit by submitting web form and (required) attachments

Scenario
A person or company can request a building permit online. When submitting the web form a minimum
of one attachment is included. The web form data is send to a web service (SOA Suite) which creates
a new case in the backend. The uploaded files are saved to a file share where the final MFT picks
them up and transfers them separately to a second SOA Suite web service based on MFT service
definition. The received file references are correlated to the case and are passed to Webcenter
Content. Webcenter Content, which is the final destination, downloads the referenced files.

Business Value
Keep large files from interrupting transactions and gobbling system resources

Features
1.
2.
3.

Transfer files by using Pass-By-Reference
File content is not visible in instance tracing and secured with user account
Transferred files can be monitored and resubmitted if a transaction has failed

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Real World Use Case Details
Customer Requirements
 Submit large files
– Submit web form with max 10 attachments with each 20Mb max file size

 Performance
– Processing form data and attachment should not impact performance of

transactions and system resources
 Security
– Files should only be accessed by authorized personal / system integrators

– Files should only be stored in Webcenter Content

 Track and Trace
– Transferred files should be monitored and able to resubmit when failed
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Real World Use Case Details
Oracle Managed File Transfer Objectives
 Consolidate ad-hoc FTP transfers into a managed, fully audited and HA platform

 Working with Design Partners and Beta customers on a new Oracle 12c product to address

high demand for a well integrated MFT solution
 Handle large files with flexible payload semantics including pass-by-reference for just-in-time

file processing
 Easy to use Web Based Design and Runtime environment
 Deliver extensible framework allowing custom callout file manipulation
Target

Source
FTP/File
System

MFT
Domain

SOA
Domain

MFT

By Ref

______
______
______
______

SOAP
MFT
File
store

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

______
______
______
______

SOA
MFT Real World Use Case Details
Architecture
 Front-end
– ADF Web / Mobile Application with Web Forms

 Network File Share
– File store for files uploaded when sending Web Form

 SOA Suite
– Web services for Case and Document management

 Database
– PL/SQL packages for storing/selecting Case data

 Managed File Transfer (MFT)
– Transfer files, uploaded through Front-end, to DocumentService (Pass-By-Reference)

 Webcenter Content
– Collect and store files from MFT using received reference
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Real World Use Case Details
Architecture

SCA Composite

BPEL
CaseProcess

BPEL
Document
Process

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Demo: Frontend > FTP > SOA > Repository
Scenario
eGoverment, a citizen request a building permit and adds some required documents as attachments. The requested
permit is then received and processed using a SOA Suite Composite service. This SCA composites uses a BPEL
process to create a case and it spawns, per attached document, a second BPEL process. This BPEL process uses an
intermediate web service binding to receive the uploaded files (pass-by-reference) from MFT. Using correlation the
files, send by MFT, can be matched to the correct permit. Using a third web service the file references are downloaded
en saved to a file-based repository.

Features
1.

MFT Composer: create and deploy MFT Transfer artifacts. 1 Source + 1 Target.

2.

MFT Server: transfer files from FTP Server to file system and creates instance data

3.

MFT Dashboard: shows transfer results and “Find File” portlet

4.

HTML5 Frontend: fill in web form and send data to SOA Suite Composite web service

5.

BPEL Process w/ intermediate MFT Binding: process waits till it receives service call from MFT

6.

File Repository: the received file(s) are downloaded, by SOA Suite, to a second environment

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Demo: Managed File Transfer

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
MFT Demo: HTML5 Frontend
JDeveloper 12c HTML5 Editor

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

HTML5 Frontend
MFT Demo: JDEV with MFT Binding

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Q&A
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MFT_SOASuite_Pass_by_reference

  • 1.
    Session CON8736 Handling LargeFiles with Oracle SOA Suite and Managed File Transfer Pass-by-Reference Dave Berry, Oracle Senior Manager, Integration Products Robert van Molken, AMIS Senior Oracle Integration Specialist Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 2.
    The following isintended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 3.
    Oracle Fusion Middleware BUSINESSINNOVATION PLATFORM Web Social Mobile User Engagement Business Process Management Content Management Service Integration  Complete  Integrated Business Intelligence Data Integration  Best-in-class  Open Identity Management Development Tools Cloud Application Foundation Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Enterprise Management  On-premise and Cloud
  • 4.
    Program Agenda  Whatis MFT and why you need it?  Functional Use Cases  Real World Use Case and Demo  Discussion Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 5.
    The growing problemwith FTP in the enterprise Lack of control, visibility, security & reliability Manufacturing Design partners  Uncontrolled proliferation of FTP servers & clients Product specifications  No global visibility into exchange of crucial data files – including customer data Distributors Legal firm Catalogs Enterprise Media (photos, videos) PR company Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Contracts Employee Records HR consulting firm  Not integrated with enterprise security standards: FTP servers are rarely integrated with directories  Single points of failure: FTP servers rarely offer HA capabilities
  • 6.
    What Is ManagedFile Transfer? Market Maturing, Consolidating and approaching $1B annually  Simple and Secure End-to-End Managed File Gateway – Large files, encryption, auditing, monitoring, pass-by-reference  Standards Based Middleware Integrations – (s)FTP, SOA, B2B, Service Bus, Web Services …  Lightweight Web based Design Time Interface – Easily build, edit and deploy end-to-end transfers  Additional Characteristics – Scheduling, Embedded sFTP server, FTP proxy, no transformations – Advanced Management: Pause, Resume, Resubmit Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 7.
    Oracle MFT FeatureSet  Core – “Embedded” FTP(S)/SSH, FTP / FTP-SSL / SSH server  File Delivery – Very Large Files ~ 500GB+ – Pass by Reference FTP or File, Inline – File transfer auditing – Scheduling and Auto Retry – Extensible framework – pre/post processing – Pause and Resume Target endpoints – HA clustering – HTTP, JCA, FTP, In-memory – SOA/B2B integration and delivery headers – Notifications – Scalable architecture – Actions  Monitoring and Management – Web UI: Managed Server  Designer, Monitor, Administration – Enterprise Manager: Admin Server Copyright © 2012, Oracle and/or its affiliates. All rights reserved. – Zip Compress/Decompress – PGP Encrypt/Decrypt – Callouts – Archive, Delete, Move, Custom
  • 8.
    MFT Use Cases UseCase Categories  Internal Business Integrations – Application to Application File sharing: SOA/SB integrations  External Partner Integrations – (s)FTP File gateway outside firewall. B2B integration, EDI ...  COTS/Cloud – OOTB End-to-end delivery channel, A2A …  Other – File management beyond business integration. – Media, software delivery, smart phones … Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 9.
    MFT Functional UseCase Patterns MFT Chained Fan Out Direct MFT MFT MFT MFT SOA MFT Direct Indirect SOA SOA MFT MFT B2B B2B SOA MFT MFT MFT B2B Direct MFT B2B Enterprise Hybrid MFT Copyright © 2012, Oracle and/or its affiliates. All rights reserved. B2B SOA MFT MFT
  • 10.
    Web-based UI Functionality(MFT UI + EM)  Design Configuration – Set up Transfers, Sources, Targets  Monitoring and Operations Console – Application-level functionality, diagnosis, resubmit … – Dashboard: Metrics, Find File, Activity Deliveries, Recent Errors, Flow instance Reports  System Administration/EM – System-level functionality, Server Parameters, EXP/IMP …  JDeveloper Bindings – Composite Bindings/BPEL Services  Enterprise Manager – MFT App Configuration, Mbeans, global defaults Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 11.
    MFT JDeveloper Binding IntegratedMFT JDeveloper Component • SOA Browse and MFT • Creates an SOA SCA Reference to MFT • MFT Browse and Consume SOA • Creates an SOA SCA Service from MFT MFT SOA SOA MFT Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Composite Bindings BPEL Services
  • 12.
    Transfer Flow Design Copyright© 2012, Oracle and/or its affiliates. All rights reserved.
  • 13.
    Runtime Transfer FlowReport Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 14.
    Monitoring Dashboard Copyright ©2012, Oracle and/or its affiliates. All rights reserved.
  • 15.
    Transfer Report Copyright ©2012, Oracle and/or its affiliates. All rights reserved.
  • 16.
    Partner Profile  Robertvan Mölken  Specialist in SOA Suite & OSB since 2007  Oracle SOA Suite 12c and MFT Beta tester  Linkedin: www.linkedin.com/in/rvmolken  Blog: technology.amis.nl  Twitter: @robertvanmolken  AMIS was founded in 1991  90+ employees & 1000+ years Oracle expertise  2 ACE directors 3 ACEs  Oracle specialized partner in Database, SOA en ADF  Oracle Middleware partner 2011 & 2013 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 17.
    MFT Real WorldUse Case Overview Title Requesting building permit by submitting web form and (required) attachments Scenario A person or company can request a building permit online. When submitting the web form a minimum of one attachment is included. The web form data is send to a web service (SOA Suite) which creates a new case in the backend. The uploaded files are saved to a file share where the final MFT picks them up and transfers them separately to a second SOA Suite web service based on MFT service definition. The received file references are correlated to the case and are passed to Webcenter Content. Webcenter Content, which is the final destination, downloads the referenced files. Business Value Keep large files from interrupting transactions and gobbling system resources Features 1. 2. 3. Transfer files by using Pass-By-Reference File content is not visible in instance tracing and secured with user account Transferred files can be monitored and resubmitted if a transaction has failed Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 18.
    MFT Real WorldUse Case Details Customer Requirements  Submit large files – Submit web form with max 10 attachments with each 20Mb max file size  Performance – Processing form data and attachment should not impact performance of transactions and system resources  Security – Files should only be accessed by authorized personal / system integrators – Files should only be stored in Webcenter Content  Track and Trace – Transferred files should be monitored and able to resubmit when failed Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 19.
    MFT Real WorldUse Case Details Oracle Managed File Transfer Objectives  Consolidate ad-hoc FTP transfers into a managed, fully audited and HA platform  Working with Design Partners and Beta customers on a new Oracle 12c product to address high demand for a well integrated MFT solution  Handle large files with flexible payload semantics including pass-by-reference for just-in-time file processing  Easy to use Web Based Design and Runtime environment  Deliver extensible framework allowing custom callout file manipulation Target Source FTP/File System MFT Domain SOA Domain MFT By Ref ______ ______ ______ ______ SOAP MFT File store Copyright © 2012, Oracle and/or its affiliates. All rights reserved. ______ ______ ______ ______ SOA
  • 20.
    MFT Real WorldUse Case Details Architecture  Front-end – ADF Web / Mobile Application with Web Forms  Network File Share – File store for files uploaded when sending Web Form  SOA Suite – Web services for Case and Document management  Database – PL/SQL packages for storing/selecting Case data  Managed File Transfer (MFT) – Transfer files, uploaded through Front-end, to DocumentService (Pass-By-Reference)  Webcenter Content – Collect and store files from MFT using received reference Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 21.
    MFT Real WorldUse Case Details Architecture SCA Composite BPEL CaseProcess BPEL Document Process Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 22.
    MFT Demo: Frontend> FTP > SOA > Repository Scenario eGoverment, a citizen request a building permit and adds some required documents as attachments. The requested permit is then received and processed using a SOA Suite Composite service. This SCA composites uses a BPEL process to create a case and it spawns, per attached document, a second BPEL process. This BPEL process uses an intermediate web service binding to receive the uploaded files (pass-by-reference) from MFT. Using correlation the files, send by MFT, can be matched to the correct permit. Using a third web service the file references are downloaded en saved to a file-based repository. Features 1. MFT Composer: create and deploy MFT Transfer artifacts. 1 Source + 1 Target. 2. MFT Server: transfer files from FTP Server to file system and creates instance data 3. MFT Dashboard: shows transfer results and “Find File” portlet 4. HTML5 Frontend: fill in web form and send data to SOA Suite Composite web service 5. BPEL Process w/ intermediate MFT Binding: process waits till it receives service call from MFT 6. File Repository: the received file(s) are downloaded, by SOA Suite, to a second environment Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 23.
    MFT Demo: ManagedFile Transfer Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 24.
    MFT Demo: HTML5Frontend JDeveloper 12c HTML5 Editor Copyright © 2012, Oracle and/or its affiliates. All rights reserved. HTML5 Frontend
  • 25.
    MFT Demo: JDEVwith MFT Binding Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
  • 26.
    Q&A Copyright © 2012,Oracle and/or its affiliates. All rights reserved.
  • 27.
    Copyright © 2012,Oracle and/or its affiliates. All rights reserved.