Workflow Automation: Applications, Technology and Research


Published on

Amit Sheth, "Workflow Automation: Applications, Technology and Research," Tutorial presented at SIGMOD, May 1995.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Workflow Automation: Applications, Technology and Research

  1. 1. LSDIS Workflow Automation: Applications, Technology and Research Prof. Amit Sheth Large Scale Distributed Information Systems Lab Dept. of Comp. Science, The Univ. of Georgia 415 Graduate Studies Research Center Athens GA 30602-7404 USA Tel. +1 706-542-2310, Fax: -2966 Email: Tutorial notes, SIGMOD Conference, May 1995, California. c 1995, Amit P. Sheth. Collaborations/Acknowledgments: Bellcore (N. Krishnakumar,.. ), U. of Houston (M. Rusinkiewicz, ..), GTE (D. Georgakopoulos) MCC (M. Singh,..), ETH-Zurich, LSDIS/UGA (J. Miller, K. Kochut), Clients: Ameritech, CHREF Future revisions of this presentation can be found at: Interoperation SIGMOD95 LSDIS Preliminaries This talk will emphasize Workflow Management for Mission Critical and Enterprise-wide Applications involving heterogeneous informa- tion systems.... We will only look at some of the issues under the “Workflow Umbrella” [Georgakopoulos, Hornick, Sheth 95]. Workflow Business Process Workflow specification Business Process Workflow implementation specification/map Business Process Workflow automation re-engineering Workflow management Business Processes automation Workflow management system Note: Trademarks are those of respective owners. ActionWorkflow: Action Tecnologies; Encina: Transarc Corp., FloMark: IBM; InCon- cert: X Soft; Lotes Notes: Lotus Corp.; ProcessIT: Digital Equipment Corp., SAP Business SQL Server: Microsoft; Workflow: SAP AG; WorkFlo: Filenet. Any missing reference to trademark is unintentional.C 1995, Amit P. Sheth 1
  2. 2. Multidatabase Interoperation SIGMOD95 LSDIS Overview • What’s workflow? Origin, Advantages, and Example Applications • Review of Commercial State-of-the-Art, Markets and Related Technologies • Basic concepts and specification of workflows • Components of a Workflow Management System • Research: past and in progress • TrendsC 1995, Amit P. Sheth 2Multidatabase Interoperation SIGMOD95 LSDIS Origins of Workflow Ten years ago, a team of engineers conceived the idea that computer software could be used to automate paper-driven business processes. They called it “workflow software.” [Smith 93] imaging document flow enhanced emails workgroup support multi-system apps.C 1995, Amit P. Sheth 3
  3. 3. Multidatabase Interoperation SIGMOD95 LSDIS Definition of Workflow (Management) (some samples) “Workflow refers to a new set of software and tools for automating and improving business processes.” [Dyson 1992] “Workflow is a process by which individual tasks come together to complete a “transaction” -- a clearly defined business process -- within an enterprise.” [Silver in [2]] “Workflow is the sequence of actions or steps used in business process. Automated workflow applies technology to process, though not necessary to every action.” [Marshak in [2]] “Workflows are computerized models of business processes...” [Hollingsworth 94] “Workflow management is an important tool for structuring and optimizing business processing... and for supporting the practical implementation of business process re-engineering.” [Fritz 95]C 1995, Amit P. Sheth 4Multidatabase Interoperation SIGMOD95 LSDIS The lure of workflow: it fits the trend Workflow fits nicely with other trends such as re-engineering, downsizing, network computing, groupware, and client-server computing.C 1995, Amit P. Sheth 5
  4. 4. Multidatabase Interoperation SIGMOD95 LSDIS The lure of workflow: a large potential market Market/Revenue Forecast for Workflow Software($-mil) Workflow- Production “Transactional Year All Workflow Workflow” 1992 226 (ID) 115 (DL) 186 (DL) 1993 628 (ID) 250 44 1994 1200 (ID) 540 106 1995 1800 (ID) 810 184 1996 2500 (IDC) 1120 (DL) 293 2500 (DL) Sources: ID = IDC & Advante; DL = Delphi Consulting; IT = International Data Corp. Forecast data used above is old (1993), but it shows an important reason for vendors’ interest in this market.C 1995, Amit P. Sheth 6Multidatabase Interoperation SIGMOD95 LSDIS Trade Press Characterization of Workflows Multi-system Applications & Transactional Workflows Complex Production Insurance Claims Task Complexity Loan Applications Ad Hoc Product Documentation Sales Proposals Administrative Press Releases Expenses Travel Requests, Purchase Requests Simple Messages Simple Complex Task Structure Source: IDC/Avante Technologies, Inc. Administrative workflows involve repetitive, predictable processes with simple task coordination rules, such as routing an expense report or travel request through an authorization process. Ad-hoc workflows involve human coordination, collaboration, co-decision, and often appear in office pro- cesses such as product documentation or sales proposal. Production workflows involve repetitive and predictable business processes, such as loan applications or insurance claims. Unlike administrative workflow, production workflow encompasses an information process involving access to one or more distributed/heterogeneous/autonomous information systems.C 1995, Amit P. Sheth [McCready 92, Georgakopoulos, Hornick and Sheth 94] 7
  5. 5. Multidatabase Interoperation SIGMOD95 LSDIS Trade Press Characterization of Workflows (contd..) Process Automation Organizational Productivity Benefits Document Flow Task Automation Ad-hoc Workgroup Tools Application Integration Complexity Source: BSG Corp. (see [Als94]) Yet another classification: Mail-centric, document-centric and process-centric (see [Fry94]).C 1995, Amit P. Sheth 8Multidatabase Interoperation SIGMOD95 LSDIS Workflow (WF) Automation Software: Example Products WorkFlo (Filenet) InConcert (XSoft) WorkMan (Reach) FloWare (Plexis) AWS (Action Tech) WorkManager (HP) LinkWorks, ProcessIT, ObjectFlow (DEC) SAP Business Workflow (SAP AG) FloMark (IBM Vienna) About 200 products claim to support workflow features and/or workflow management! See [2] and [Georgakopoulos, Hornick, Sheth 95] for a list of vendors and products.C 1995, Amit P. Sheth Trademarks are those of respective owners. 9
  6. 6. Multidatabase Interoperation SIGMOD95 LSDIS Overview of Current Commercial State-of-the-Art • Emphasis on office processes: – imaging – document flow – enhanced mail • Reasonable support for administrative and ad-hoc workflows • Many products are little more than fancy diagramming tools (Dataflow, Digraph, Flowchart, Network, Orgchart, Pertchart,...), with layout support, capture/import/export of data from/to databases, spreadsheets, simulation tools • Some are specialized electronic data management systems: e-mail, imaging, databases, electronic forms, text, engineering drawing,.. • Alliances between image/document management, GUI builder and tools companies (e.g., simulation) are commonC 1995, Amit P. Sheth 10Multidatabase Interoperation SIGMOD95 LSDIS Maturing Infrastructure: A Driving Force Communication Infrastructure used by Commercial Workflow Management Systems e-mail Early 90s - already mature Work-group (Notes) 1993 - almost mature Distributed Object Management 1995 - very active Current Generation Transaction 1996? Processing Monitors Agents 1997?C 1995, Amit P. Sheth 11
  7. 7. Multidatabase Interoperation SIGMOD95 LSDIS Workflow Management System for Office Automation vs Enterprise Automation (Typical Case) Current workflow/workgroup software supports office automation functions (involving user tasks) (most products are in PC and mainframe env.). There is little support for application automation (involving both user and application tasks with varied level of transactional properties). User Task User Task User Task Appl. Task User Task Appl. Task workflows supported by workflows supported by most current WMSs some new/emerging WMSsC 1995, Amit P. Sheth 12Multidatabase Interoperation SIGMOD95 LSDIS More Demanding Workflows some examples-- • Business loan processing in Banks requires coordinating user tasks such as loan application entry and risk exception, and application tasks such as risk evaluation, risk update, and loan decision recording. • Patient health care support in a health care group practice requires coordinating user tasks such as patient registration, doctor’s record review/update, lab work, and application tasks such as automated billing and statistics compilation. • Service provisioning in Telcos require the coordinated execution of heterogeneous tasks on heterogeneous systems: • Product life-cycle management. Support for most mission-critical multi-system enterprise applica- tions is lacking (Finance, Healthcare, Manufacturing,...)C 1995, Amit P. Sheth 13
  8. 8. Multidatabase Interoperation SIGMOD95 LSDIS An Example Application in Business Data Processing (Loan Processing) Client credit worthiness Enter loan Risk Record request exception decision Risk Risk Risk update evaluation update compensation Workflow may involve both user and application tasks, as well as different types of application tasks. Some tasks can be compensated.C 1995, Amit P. Sheth [Breitbart, et al. 93: ETH] 14Multidatabase Interoperation SIGMOD95 LSDIS An Application (segment) in a Healthcare Environment internet Insurance Reference Info. company Billing Patient Provider registration encounter Labwork Request Lab Specialist review Workflow may involve multiple organizations, data of different types/media, different notions of transactions (Domain specific [e.g., HL7 in Healthcare], EDI, DB), may take several days...C 1995, Amit P. Sheth 15
  9. 9. Multidatabase Interoperation SIGMOD95 LSDIS An Application in Telecommunication: Provisioning a Telephone Service Customer Loop Inter-Office/Trunk Inter-Office/Trunk Central Office Central Office L-D Carrier L-D CarrierC 1995, Amit P. Sheth 16Multidatabase Interoperation SIGMOD95 LSDIS Provisioning a Telephone Service (contd..) Outside Plant Engineer 2 Customer 1 Planning Design Customer Representative Work-Force Adm. Service Order Processor 3a 3b Billing Controller Operations 5a 5b Support 7 4a 6 4b Manual error resolutions 8 Loop Intra-Off. Trunk Assignment Assignment Assignment Use of legacy application/information systems is unavoidable. Migration to new systems and modern distributed computing environment continue.C 1995, Amit P. Sheth [Ansari et al 92,Aslo see: Georgakopoulos et al 93] 17
  10. 10. Multidatabase Interoperation SIGMOD95 LSDIS Business Challenges • Improve flexibility for re-engineering • Increase automation to reduce cost and improve response time • Support evolution and migration (accommodate both existing systems and new systems; increase use of the latter) Example (Case Study of a Service Automation): • 40 persons -> 12 persons (work center reorganization, re-engineering) -> 2-3 person (automation) • Six Weeks -> Next Day -> Few MinutesC 1995, Amit P. Sheth 18Multidatabase Interoperation SIGMOD95 LSDIS Some of the key new requirements • Automation involving heterogeneous software tasks/processes and user tasks. • Integration with legacy systems and heterogeneous information systems; support with-in enterprise and across enterprises (local, wide area, and wireless). Performance and scalability. • Use of transaction concepts and technology, reliability, failure handling/recovery. • Support for domain-specific tasks (e.g., HL7 in Healthcare), objects (e.g., EDI) and repositories. Task and info. sys. heterogeneity Object/ Data complexity Coordination/ CorrectnessC 1995, Amit P. Sheth 19
  11. 11. Multidatabase Interoperation SIGMOD95 LSDIS Key Promising Technologies (and Challenges) Distributed Object Management Legacy System Interoperability Customized Transaction ManagementC 1995, Amit P. Sheth 20Multidatabase Interoperation SIGMOD95 LSDIS Another Characterization (that is sensitive to new requirements) Human-oriented System-oriented Transactional workflows CSCW Commercial WFM Systems Commercial TP Systems [Georgakopoulos, Hornick and Sheth 94]C 1995, Amit P. Sheth 21
  12. 12. Multidatabase Interoperation SIGMOD95 LSDIS On Transactional Workflows Transactional workflows are activities that involve coordinated execution of multiple related tasks on distributed/heteroge- neous/autonomous information systems and support (provide) selective use of transaction properties at individual task and (intra- and inter-) workflow levels. In particular, they use transaction management concepts and technology for specifying and ensuring workflow correctness and reliability in distributed/heterogeneous/autonomous infor- mation system environments. More on this later... [Georgakopoulos, Hornick and Sheth 94; Early use: Sheth and Rusinkiewicz 93. Also see Mohan et al 95]C 1995, Amit P. Sheth 22Multidatabase Interoperation SIGMOD95 LSDIS (Distributed) Transaction Processing (DTP) vs. Advanced WFS (Typical Case) DTP/D-OLTP is focused on efficient execution of relatively simple tasks with no coordination across heterogeneous tasks (in different task groups). Advanced (transactional) workflows require coordinated execution of heterogeneous tasks, with varied levels of transactional properties, on a variety of systems. Task Groups Task2 Task1 Task4 Task3 Task Workflow Group In DTP In TWF • Earlier queued message systems and “chaining of transactions”. Problems: insufficient control over transaction properties, one type of task, interactions among concurrent activities difficult.C 1995, Amit P. Sheth [STDL, Encina,...] 23
  13. 13. Multidatabase Interoperation SIGMOD95 LSDIS Related Work: Database Literature • ACID transactions and their nested derivatives Problems: inflexible, difficult to implement in multi-systems. • Extended/Relaxed Transaction Models: Sagas and Nested Sagas [Garcia-Molina et al. 88, 90], ConTracts [Reuter 89]. Flexible Transactions [Elmagarmid et al 90, Rusinkiewicz et al 90], Multi-transaction Activities [Garcia-Molina et al. 90], Open Nested Transactions [Weikum & Schek 92] and Others (e.g., in [Elmagarmid 92]), ACTA framework [Chrysanthis & Ramamritham 91/92] • “Workflow” and hybrid models: Long-Running Activities [Dayal et al. 91], DOM model/TSME [Buchmann et al 92, Georgokopoulos et al. 93], Third Generation TP Monitors [Dayal et al. 93], ASSET [Biliris et al. 94] [Rusinkiewicz and Sheth 94, Mohan tutorial; ETM vs Workflow Model-- see Breitbart et al 94]C 1995, Amit P. Sheth 24Multidatabase Interoperation SIGMOD95 LSDIS Workflow Management WFM involves: • defining workflows, i.e., describing those aspects of processes that are relevant to controlling and coordinating the execution of its tasks (and possibly the skills of individuals or information systems required to perform each task), and • providing for automation and re-engineering (fast (re)design and (re)implementation) of the processes as business needs and information systems change. [Georgakopoulos, Hornick and Sheth 94]C 1995, Amit P. Sheth 25
  14. 14. Multidatabase Interoperation SIGMOD95 LSDIS Workflow Management Issues Workflow Workflow workflow Process Specification Implementation = application • Business Process Modeling/Reengineering (BPM/R) • Workflow model & specification language • Executable application code • Run time support [Georgakopoulos, Hornick and Sheth 94]C 1995, Amit P. Sheth 26Multidatabase Interoperation SIGMOD95 LSDIS WMS Conceptual Architecture (system components) BPM toolkit Workflow WMS run-time – process view Development toolkit system and tools – organization view – graphical design tool – scheduler – data view – developer’s workbench – task managers/interfaces – re-engineering analyzer – testing tool – processing entities – TQM advisor – simulation tool – monitoring tool – ... – tracking tool – ... – reporting tool – ...C 1995, Amit P. Sheth 27
  15. 15. Multidatabase Interoperation SIGMOD95 LSDIS Process Modeling Methodologies There are many methodologies ... Communication-based methodologies stem from Winograd/Flores “Conversation for Action Model” [WF87] 1.preparation - a customer requests an action to be performed or a per- former offers to do some action 2.negotiation - both customer and performer agree on the action to be per- formed and define the terms of satisfaction 3.performance - the action is performed according to the terms established 4.acceptance - the customer reports satisfaction (or dissatisfaction) with the action Figure 1. preparation negotiation Customer Workflow Loop Performer acceptance performance [2], [Georgakopoulos, Hornick and Sheth 94]C 1995, Amit P. Sheth 28Multidatabase Interoperation SIGMOD95 LSDIS Process Modeling Methodologies.. continued Example of Communication-based Model: Modeling Materials Procurement Process investigator procurement Procure Materials office Verify Status accounts office vendors Get Bids Place Order vendor [2], [Georgakopoulos, Hornick and Sheth 94]C 1995, Amit P. Sheth 29
  16. 16. Multidatabase Interoperation SIGMOD95 LSDIS Process Modeling Methodologies.. continued Activity-based methodologies focus on modeling the work instead of modelling the commitments among humans. Example of Activity based process modeling: Procure Materials task nesting Verify Status Get Bids Place Order Process modeling does not capture significant computational aspects [Georgakopoulos, Hornick and Sheth 94]C 1995, Amit P. Sheth 30Multidatabase Interoperation SIGMOD95 LSDIS Process modeling to Workflow Implementation-- an example (Action WorkFlow) Process Modeling Analyst Application Builder Implementation Implementation Manager on Lotus Notes on SQL ServerC 1995, Amit P. Sheth 31
  17. 17. Multidatabase Interoperation SIGMOD95 LSDIS Workflow Modeling: The example of METEOR model start task task task end task filter Aux. Sys. interface interface proc. interface entity proc. entity proc. entity METEOR: Managing End-To-End OpeRations [Krishnakumar and Sheth 94]C 1995, Amit P. Sheth 32Multidatabase Interoperation SIGMOD95 LSDIS About the environment: Types of Tasks • user tasks involving humans in processing task • application tasks: • scripts involving terminal emulations to remote systems • predefined interfaces to legacy application systems (e.g., Bellcore “contracts”) • stored procedure calls • client programs or servers invoking other servers • database transactions [Krishnakumar and Sheth 94]C 1995, Amit P. Sheth 33
  18. 18. Multidatabase Interoperation SIGMOD95 LSDIS About the environment Processing Entities • humans (may appear as a GUI; may use document/image processing systems and applications) • script interpreters and compilers (for processing scripts and application programs) • (legacy) application systems • servers in client-server and transaction processing systems • DBMSs [Rusinkiewicz and Sheth 94, Krishnakumar and Sheth 94]C 1995, Amit P. Sheth 34Multidatabase Interoperation SIGMOD95 LSDIS About the environment Types of Interfaces • RPC and t-RPC mechanisms using transaction processing systems • queue managers • proprietary workstation to mainframe interfaces for – “contracts” – terminal emulation • (distributed computing/communication infrastructure: CORBA, DCE, Notes-like) Workflow management can be seen as a new distributed computing paradigm... [Sheth and Krishnakumar 94]C 1995, Amit P. Sheth 35
  19. 19. Multidatabase Interoperation SIGMOD95 LSDIS Workflow Model and WMS wish list (Requirements and Features) • Modeling heterogeneous tasks – task behavior/structure: externally visible states of the task, initial state, termination states, significant events and their attributes – task inputs and outputs – task (operation) semantics, e.g., compatibility, relaxed isolation • Modeling Interfaces and Processing Entities: – type of interface/processing entity: communication infrastructur(s) and associated APIs – interface/processing entity (system) properties/semantics-- e.g., isolation granularity, order preservation, idempotency, monotonicity [Krishnakumar and Sheth 94/95, Sheth and Rusinkiewicz 93/94]]C 1995, Amit P. Sheth 36Multidatabase Interoperation SIGMOD95 LSDIS Workflow Model and WMS wish list (Requirements and Features) • Coordination and Inter-relationships: Routing, Rules, Policies, Practices, .. • Inter-task dependencies – state-based – value-based: I/O objects and external variables • Roles • Work-lists: Work-prioritizing, Dynamic Work Distribution • Data Management – different task formats: message, contract, form, transaction; EDI; use of auxiliary systems for complex data manipulation – different types of data: structured, text, image, voice, video,... [Krishnakumar and Sheth 94/95, Sheth and Rusinkiewicz 93/94, Dayal and Shan 93]C 1995, Amit P. Sheth 37
  20. 20. Multidatabase Interoperation SIGMOD95 LSDIS Workflow Model and WMS wish list (Requirements and Features) • Dynamic Aspects – processing entity not known at design time – new tasks can be added dynamically – multiple concurrent invocation of the same task types – ... • Intra- and Inter-workflow Execution requirements: – failure atomicity (A) – execution atomicity (I) – workflow recovery – inter-workflow concurrencyC 1995, Amit P. Sheth [Krishnakumar and Sheth 94/95, Sheth and Rusinkiewicz 93/94, Dayal and Shan 93] 38Multidatabase Interoperation SIGMOD95 LSDIS Workflow Model and WMS wish list (Requirements and Features) • support for long running workflows and tasks • Error Handling – Systems Errors – Logical Errors • Forward Recovery • Monitoring • Status tracking • ReportingC 1995, Amit P. Sheth 39
  21. 21. Multidatabase Interoperation SIGMOD95 LSDIS Some Technical Challenges • Different types of tasks => homogeneous modeling; modeling execution behavior • Correct and Executable (hence well defined) specifications and correct and safe execution (including recovery) • Heterogeneous processing environments and systems => semantics of applications and processing entities, their impact on correct execution and performance • Performance- many more messages/tasks in “application/ operation automation” as compared to “office/user task automation”C 1995, Amit P. Sheth 40Multidatabase Interoperation SIGMOD95 LSDIS Modeling Heterogeneous Tasks (Task Structures) Different state transition diagrams for different types of tasks representing what is observable and what is controllable by the WMS (i.e., can WMS enable that transition) Initial start Initial Executing Initial start start done Executing Executing abort Done fail done abort commit prepared Prepared Failed Done Aborted Committed commit abort A non-transactional task A transactional task Aborted Committed An open 2PC transactional task [Attie et al 93, Rusinkiewicz & Sheth 93, Sheth and Krishnakumar 94]]C 1995, Amit P. Sheth 41
  22. 22. Multidatabase Interoperation SIGMOD95 LSDIS Intertask Dependencies Preconditions for initiating each scheduler-controllable transition in a task. Klein’s primitives [KL91]: • Order Dependency: e1 < e2. If both e1 and e2 occur, then e1 precedes e2. • Existence Dependency: e1 -> e2. If event e1 occurs sometimes, then event e2 also occurs sometimes. – Conditional Existence Dependency [KL91]: e1 -> (e2 -> e3) Examples from multidatabase transaction models: • Commit Dependency [CR92]: cmB < cmA • Abort Dependency [CR92]: abB -> abA [Attie et al 93, also see ACTA [Chrysanthis & Ramamritham 92], Georgakopoulos et al 94, Bilris et al 94,...]C 1995, Amit P. Sheth 42Multidatabase Interoperation SIGMOD95 LSDIS Intertask Dependencies There are many ways to model/specify dependencies, routing, pol- icies, etc. Examples of the types of dependencies from database (extended transaction model) literature include, execution dependencies, data/value dependencies, temporal dependencies. For example, Flexible Transactions [Elmagarmid et al 90], ConTracts [Reuter 89], ACTA [Chrysanthis and Ramamritham 92], Multitransactions [Gar- cia-Molina et al 90], Multidatabase Transactions [Rusinkiewicz et al 92, Mauro thesis 1993], DOM [Georgakopoulos et al 94].... Rule-based (ECA) specification [Dayal et al., others] is popular, especially in research.C 1995, Amit P. Sheth 43
  23. 23. Multidatabase Interoperation SIGMOD95 LSDIS Simple Workflow Example Compound Task task B Initial task A task A task C (SD1,DD1) Initial Initial Initial Executing Executing Executing Executing (SD3,DD3) Failed Completed Failed Completed Failed Completed (SD2,DD2) Failed CompletedC 1995, Amit P. Sheth 44Multidatabase Interoperation SIGMOD95 LSDIS Another Example Task Graph DELETE BOOKING delete decrement update booking summary alarm (dB) (ds) (u?a) increment update summary alarm (is) (u?a) Intertask Dependencies -> -> < s(dB) -> s(dS) c(ds) -> s(u?a) c(iS) -> s(u?a) (a(dB) & c(dS)) -> s(iS) -> (a(dB) < d(dS)) -> a(dS) & -> [Woelk et al 93]C 1995, Amit P. Sheth 45
  24. 24. Multidatabase Interoperation SIGMOD95 LSDIS A somewhat complex example CKT-Type_1 SO start executing abort done AGG_task_1 ADD_COMPONENTS NEW_ADD ROUTE=YES “CHG_GR” ROUTE=NO NEW_SWITCH SEGMENT-1 SEGMENT-2 SEGMENT-3C 1995, Amit P. Sheth 46Multidatabase Interoperation SIGMOD95 LSDIS Desirable Specification Language Features • definition of individual tasks (basic operation definition) • support for application as well as use tasks (incl. transactions and nonelectronic operations) • state-based and value based intertask dependencies and data management (control and data flow definitions) • failure and exception handling • business rules and constraints • security and role resolution [Dayal and Shan 93 (terms in parentheses), Krishnakumar and Sheth 94,]C 1995, Amit P. Sheth 47
  25. 25. Multidatabase Interoperation SIGMOD95 LSDIS Several Approaches to Language Specification • Based on Extended Transaction Models (e.g., [ASSET [Biliris et al 94]), based on rule-based specification, based on logic-based specification • Extending Multidatabase Query Language (e.g., MSQL extensions [Rusinkiewicz et al]) • Extending a script or “agent-based” language (e.g., ConTract/APRICOT [Watcher & Reuter 92, Schwenkreis 93], work at MCC-Carnot/InfoSleuth) • Extending general programming languages (e.g., as in Tranactional-C [see Encina manuals], or IPL [Chen et al 93, Bukhres et al 95]) • Develop a Enterprise-wide Multi-system Application Development Environment (tasks in different languages, executed through/by different interfaces/processing entities) (e.g., Graphical Intefrace -> METEOR “intermediate-level” sub-languages) • Add graphical interface and visual programming paradigm to the aboveC 1995, Amit P. Sheth 48Multidatabase Interoperation SIGMOD95 LSDIS An Approach to Specification METEOR (Sub-)Languages start task task task end task filter WFSL TSL Aux. Sys. interface interface proc. interface entity proc. entity proc. entity WFSL: WorkFlow Specification Language TSL: Task Specification Language [Sheth and Krishnakumar 94]C 1995, Amit P. Sheth 49
  26. 26. Multidatabase Interoperation SIGMOD95 LSDIS Components of WFSL (partial) • Task types: task structures, data input/output • Task classes, Task instances • Inter-task dependencies (logical error handling) • Data exchange statements • Filter (interface def.) Components of TSL (partial) • processing entity specific statements • statements for revealing task structures • statements for identifying interfaces and dealing with systems errors Languages can be targeted for end user programming or developers. For example, WFSL and TSL are “intermediate” languages targeted to developers/administrators. [Krishnakumar and Sheth 94/95]C 1995, Amit P. Sheth 50Multidatabase Interoperation SIGMOD95 LSDIS Run-time Architecture (METEOR Approach) Application Developer and System Analyst USER Work flow Workflow initiation, Specification Workflow monitoring and Workflow Group Simulation Workflow Controller TP Sys. TP Sys. TP Sys. TP Sys. Log Open_2PC User Tran. Contract Script . Task Task Programs TP Sys. Log Log Log Log Task Logs DBMS QMS Term. Interfaces Resource Mgr. Emulator DBMS OSS OSS GUI Proc. Entities [Krishnakumar and Sheth 94/95]C 1995, Amit P. Sheth 51
  27. 27. Multidatabase Interoperation SIGMOD95 LSDIS Enforceable Dependencies • Dependencies may not be enforceable. For example, ab(A) -> cm(B) • Event attributes determine whether a dependency is enforceable. For example, – e1 -> e2 is run-time enforceable if rejectable(e1) [delay e1 until e2 is submitted, reject e1 if task 2 terminated without submitting e2], or forcible(e2) [force execution of e2 when e1 is accepted for execution]. – e1 < e2 is run-time enforceable if rejectable(e1) [let e2 be executed when it is submitted, thereafter reject e1 if submitted], or delayable(e2) [delay e2 until either e1 has been accepted for execution, or task 1 has terminated without issuing e1]. [Attie et al 93]C 1995, Amit P. Sheth 52Multidatabase Interoperation SIGMOD95 LSDIS Scheduling Approaches • Based on Petri-net Models [Elmagarmid et al 90] • Executor for Flex. Trans. in a logically parallel language L.0 [Ansari et 92] • Interpreter of MDB transaction specification Language (VLP) [Kuehn et al 91] • Interpreter of ECA rules [Dayal et al 92] • Games vs. Nature [Rusinkiewicz et al 92] • Fine-state Automata [Jin et al 93] • Scheduling and enforcing intertask dependencies using temporal propositional logic and finite automata [Attie et al 93] • Scheduling through Distributed Events [Singh and Tomlinson 94] [Rusinkeiwicz and Sheth 94]C 1995, Amit P. Sheth 53
  28. 28. Multidatabase Interoperation SIGMOD95 LSDIS Scheduler/Controller Implementations • Centralized (e.g., [Attie et al 93]) • Distributed (e.g., [Singh and Tomlinson 94]) • Scheduler per Workflow (e.g., [Jin et al 94]) • Scheduling by Workflow Objects (objects/tasks that carry state info.)-- “ambulatory” (i.e., INCA-style [Barbara et al 94])C 1995, Amit P. Sheth 54Multidatabase Interoperation SIGMOD95 LSDIS A Prototype Scheduler (a centralized approach) Submitted Requests Accepted Requests Task Agents & event notifications Scheduler Dispatcher Rejected Requests Delayed Requests Re-attempt Queries Replies Execution Dependency Pending Automata Set [Attie et al 93]C 1995, Amit P. Sheth 55
  29. 29. Multidatabase Interoperation SIGMOD95 LSDIS Correctness & Execution Requirements • Executable/ run-time enforceable specifications • Scheduling – correctness (no violation of intertask dependencies) – safety (progress towards acceptable states) – termination • Recovery – forward recovery • Concurrency Control – serializability ?? [Attie et al 93, Georgakopoulos et al 93, Krishnakumar and Sheth 94]C 1995, Amit P. Sheth 56Multidatabase Interoperation SIGMOD95 LSDIS Transactional Workflow-- What it is and is not Transactional workflows (try to) address application specific and user-defined correctness, reliability, and functionality requirements. Transactional workflows share the objectives of some extended/relaxed transaction models about selective relaxation of transactional properties based on application semantics. ... the term is likely to evolve, as it has significant appeal. For example, see DOM and Exotica projects ... [Early use: Sheth & Rusinkiewicz 93, Rusinkiewicz and Sheth 94, Georgakopoulos et al 94] [Additional discussions: Georgokopoulos, Hornick & Sheth 94/95, Krishnakumar & Sheth 94/95, Mohan et al 95]C 1995, Amit P. Sheth 57
  30. 30. Multidatabase Interoperation SIGMOD95 LSDIS Transactional Workflow-- What it is and is not Transactional workflows does not imply workflows are similar/ equivalent to DB transactions or support all ACID transaction properties. Usually WMSs do not support consistency of data across multiple databases, especially when there are failures. Typically WMSs do not support some of the important features supported by TP Monitors (e.g., concurrency control, backward recovery, consistency of data). WMSs today do not support concurrency control similar to those involved between “transaction groups” in TP monitors. WMS applicationsoften rely on local concurrency control.C 1995, Amit P. Sheth 58Multidatabase Interoperation SIGMOD95 LSDIS Transactional Workflow-- What it is and is not A WMS may provide transactional properties to support forward recovery, and/or use system and application semantics to support semantic-based “correct” multi-system application execution. Example levels of transactional support a WMS may provide are: – use of TM concepts/techniques (log input/output, before image, compensation...) to enable forward recovery and failure atomicity – part of a workflow has transactional properties (extended transactional model with component transactions) – support a “general” two phase commit (WMS schedule may provide commit coordination) or interface with an external commit coordinator.C 1995, Amit P. Sheth 59
  31. 31. Multidatabase Interoperation SIGMOD95 LSDIS Transactional Workflow-- What it is and is not A WMS may use transaction management technology, such as – transactional-RPC between two components of a WMS (e.g., scheduler and task manager), – an external commit coordinator – XA-like protocol between task manager and resource manager (Interface/Proc. Ent.). Scheduler Task Application Manager Interface/ Proc. Ent.C 1995, Amit P. Sheth 60Multidatabase Interoperation SIGMOD95 LSDIS Application/Task and System Semantics to simplify CC and Recovery Management Semantics (Application/Task, System) Impact (CC: Con. Control, R: Recovery) limited commutativity (apply) fewer exclusive locks (CC) relaxed isolation (appl) no global commitment (CC) order preservation + rigorousness (sys) early release of locks (CC) idempotency (sys) resubmit transactions (R) + monotonicity roll-forward recovery (R) [Jin et al 93, Jin et al 94]C 1995, Amit P. Sheth 61
  32. 32. Multidatabase Interoperation SIGMOD95 LSDIS Enabling technologies and standards DCE CORBA, OLE/COM Notes X/OPEN TxRPC, ... STDL EDI .... SIGs in some of the standardization bodies may address workflow issues. For example, Common Facilities in CORBA may address some workflow issues.C 1995, Amit P. Sheth 62Multidatabase Interoperation SIGMOD95 LSDIS Transaction Monitor as a building block Advantages second generation TP-monitors (e.g., Encina): – Ease of implementing fault tolerance: Transactional-RPCs, utilities such as Queue Managers or Structured File Systems – Limited support for persistence of server state Disadvantages: – Lack of interoperability between monitors – Products still lack good performance, stability, lack of ease of use/administration, OR Problems with support for state persistence and multi-threadingC 1995, Amit P. Sheth 63
  33. 33. Multidatabase Interoperation SIGMOD95 LSDIS CORBA as a Building Block Advantages • Distributed Computing Infrastructure with several features. • Support for distributed client-server system on different hardware and operating system platforms: ORBeline-- SunOS, Solaris, HP/UX, IBM/AIX, OSF, Unixware, MS-Windows, Windos-NT, etc.; Orbix-- Windows NT,Unix, etc.; ObjectBroker-- MS-windows, Unix, Mac, OpenVMS, OS/2. • Unified support for all data types defined in CORBA. • Support for low-level data format transformations between different systems. • Multiple concurrent invocation of tasks. Most ORBs support multi-threading. • Dynamic Interface Invocation.C 1995, Amit P. Sheth 64Multidatabase Interoperation SIGMOD95 LSDIS CORBA as a Building Block ... continued Advantages (continued) • Error Handling. Support for System Exceptions and User Exceptions (variable level of support by different products). • Security Service. Permission for every object; additional services (access permissions as user-programmed filters or at method invocation) • Reliability/Fault tolerance. Varied support, outside CORBA: ORBeline’s smart agent tracks all object & clients, notifies failures, reroutes lost dataC 1995, Amit P. Sheth 65
  34. 34. Multidatabase Interoperation SIGMOD95 LSDIS CORBA as a Building Block ... continued Disadvantages/Problems • Lack of interoperability between different commercial ORBs and/or other distributed infrastructure. Some exceptions: DOE--Orbix; Orbix--Tuxedo/MS OLE/ ISIS-RDO; ObjectBroker--OLE-COM/DDE; ORBeline--?? • Lack of access to Legacy Applications. Some exceptions: ObjectBroker-- script servers; DOE -- wrapper? • Limited Mapping of IDL to different languages. Orbix-- C++; ORBeline-- C++; ObjectBroker-- C++; HyperDesk-- C++, DOE-- C++ and C? • Lack of transaction management support. Not part of core specification. Common Facilities Architecture tries to build transaction support on ORB.C 1995, Amit P. Sheth 66Multidatabase Interoperation SIGMOD95 LSDIS Lotus Notes as Building Block Advantages • supports client-server application development • supports and incorporates across several hardware platforms: Windows, NT, OS/2, Solaris, AIX, SCO/ HP Unix, Macintosh • supports multiple network protocols (TCP/IP, Ethernet, IPX, NetBIOS, AppleTalk, Token Ring) • comprehensive email application included • support for enhanced email and document routing • provides a consistent interface for all applications written for Notes • ease of use for end users • large user baseC 1995, Amit P. Sheth 67
  35. 35. Multidatabase Interoperation SIGMOD95 LSDIS Lotus Notes as Building Block ... continued • supports multimedia data: formatted text, graphics, audio and video objects may be embedded in Notes documents • remote access of data – users may dial up or log in across a network • provides secure access to data – kerberos based data encryption; password protection • limited replication of databases allowed • can work with other applications (through OLE in Windows) • Notes API provides a good basis for creating workflow applications Notes provides a good infrastructure to build a WMS for Administrative and Ad-hoc workflows.C 1995, Amit P. Sheth 68Multidatabase Interoperation SIGMOD95 LSDIS Lotus Notes as Building Block ... continued Disadvantages/deficiencies • no support for transaction processing – no support for ACID properties – lack of locking mechanism – latency in replication • no real time collaboration support – no conferencing/ shared screen support • no native support for integrating legacy applications • supports only the Notes database format Currently Notes lacks good support for implementing Production workflows requiring integration with heterogenous information systems and transaction support.C 1995, Amit P. Sheth 69
  36. 36. Multidatabase Interoperation SIGMOD95 LSDIS Research Projects and Prototypes on Workflow and Related Issues (a partial list) APRICOT (Germany), METEOR (Bellcore, Georgia, UofH), Interbase (Purdue), ASSET (Bell Labs), TSME/DOM (GTE Lab), Pegasus (HP Lab), TriGSflow (Linz, Austria), Exotica (IBM Almaden), Aachen, INRIA, ETH ... Earlier projects: ETM (DEC), Carnot (MCC), INCA (MITL),C 1995, Amit P. Sheth 70Multidatabase Interoperation SIGMOD95 LSDIS Multi-paradigm Workflows Workflow applications in large/complex enterprises and those spanning multiple enterprises require support for multiple paradigms in terms of: • Types of workflows: production, ad-hoc,... • Communication infrastructures: – Async (e-mail, document flow/work-group based, message based) – Sync (PRC, t-RPC, ...) – local-area -- internet - wireless • Computing Structures/Semantics: e.g., transactions in Electronic Commerce, Transaction Processing Systems, and DBMSsC 1995, Amit P. Sheth 71
  37. 37. Multidatabase Interoperation SIGMOD95 LSDIS Going Forward (commercial technology: Development vs. Market Models A1 An A1 An A1 An W1 Wn WMS WMS Infrastructure Infrastructure I1 In Vertical Multi-paradigm Horizontal (infrastructure) WMS: different features, same infrastructure A: Market/Application Domain (Healthcare, Financial, Mfg.,...) W: Workflow Management System I: InfrastructureC 1995, Amit P. Sheth 72Multidatabase Interoperation SIGMOD95 LSDIS Trends/Standardization Efforts Horizontal “Interoperability” Focus – Workflow Coalition-- lack of “information system” perspective and “transaction” support so far Vertical “Market” Focus – not yet started, but likely to start soon-- Healthcare, Finance, Manufacturing – SIGs are formingC 1995, Amit P. Sheth 73
  38. 38. Infrastructure components for Multi-paradigm WMS Component infrastructures for future “multi-paradigm” workflow management systems: Trans. Workflow – e-mail CTM/ETM – internet DTP - Encina Workflow Software – Lotus Notes Distributed Proc. – CORBA Infrastructure Groupware e-mail - CORBA, DCE, Support - Lotus Notes internet – Transaction Monitors X.400, X.500Multidatabase Interoperation SIGMOD95 LSDIS A Small List of Research Challenges • Model: multiple views of what is modeled, when to use transaction properties/features, correctness issues • Language: ease of specification vs. features; logical error handling; use of visual programming • Development: next generation enterpirse application development scenario (multi-system applications running within and across business enterprises), testing, simulation • Run-time System: error handling, failure handling/recovery, correctness, heterogeneous objects, scalability (e.g., scheduling for environment with many concurrent workflows), performance • Standards: plenty of relevant efforts, what is useful now?C 1995, Amit P. Sheth 75
  39. 39. Conclusions • New paradigm for distributed computing? Perhaps it is the way to provide glue to handle legacy systems, and to to support migration/evolution. • Technology related to business processes and applications-- better relevance and visibility than heterogeneous DDBMS and extended transactions; still database and transaction management have important roles to play.Multidatabase Interoperation SIGMOD95 LSDIS Brief biography of the Tutorial Speaker: Dr. Amit Sheth directs the Large Scale Distributed Information Systems (LSDIS) Lab and is an Associ- ate Professor of Computer Science at the University of Georgia. Earlier he worked for nine years in the R&D labs at Bellcore, Unisys, and Honeywell. He has lead projects on heterogeneous DDBMS, factory information system, integration of AI-database systems (BrAID), transactional workflows (PROMT and METEOR), federated database tools (BERDI and TAILOR), multidatabase consistency, and data qual- ity (Q-Data). Dr. Sheth has published over 75 papers, given over 45 invited talks and 14 tutorials, and lead two international conferences and a workshop as a General/Program (Co-)Chair. He has also served twice as an ACM Lecturer, has been on over twenty five program and organization committees, and is on the editorial board of four journals. The LSDIS lab maintains very active collaboration with industry, and has won significant projects in the areas of interoperable information system and workflow management (under the Healthcare Informa- tion Infrastructure Program awarded by NIST) and global information system and management of het- erogeneous digital data (awarded in the Massive Digital Data Systems initiative). Industrial partners on these projects are Bellcore, MCC and CHREF. The lab acknowledges sponsorship/industrial affiliation of the HP Labs and the Persistence Software Inc. 1995, Amit P. Sheth 77