Intalio Project Dogfood And Doubleshot
Upcoming SlideShare
Loading in...5
×
 

Intalio Project Dogfood And Doubleshot

on

  • 3,897 views

 

Statistics

Views

Total Views
3,897
Views on SlideShare
3,897
Embed Views
0

Actions

Likes
2
Downloads
29
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Intalio Project Dogfood And Doubleshot Intalio Project Dogfood And Doubleshot Document Transcript

  • Project Dogfood Monday, January 19th 2009 | Ismael Ghalimi While the current recession is forcing many companies to go through painful rounds of layoffs, Intalio is actually growing its employee base, albeit at a slower rate than in the previous twelve months. There are two reasons for this: Number one, we want to remain on the conservative side, and continue outsourcing most non-core functions, which pretty much means everything outside of architecture design and customer training. Number two, we have started to aggressively use our own products in order to improve our operations, thereby allowing us to do more with less. We call this initiave Project Dogfood. WARNING: very long article, but full reading highly recommended. Project Introduction The idea for Project Dogfood emerged at a recent company meeting, and was refined during our yearly company offsite. The idea is pretty simple: every single employee (36 out of our 60+ full-time collaborators) will develop a new process, or improve an existing one, using our products. The project is managed as an internal competition, and the rules are relatively straightforward: 12 teams of 3 employees must be formed. Each team must include at least one engineer and one process expert (trainer). Each team must design and deploy at least one real-world business process. No two teams can implement the same processes (first come, first served). Processes must contribute to improve Intalio’s operations. Processes should not use confidential information (publishing encouraged). Teams must use off-the-shelf Intalio software. Any product edition can be used (combinations encouraged). All custom code must be deployed as a J2EE webapp or XPath function. Custom product features must be developed through the D3 process. D3 projects must be funded by external customers, or actual cost savings. D3 projects funded internally must be approved by the CFO. Projects should have a business sponsor, who does not have to be part of the team. Processes must be deployed on the latest versions of Intalio’s products. Processes must be used in production on an ongoing basis. Processes must be kept running when new product versions are deployed. New product versions cannot ship until all processes have been upgraded. Teams must capture time spent working on the project. Teams must create a business case before implementing the process. Teams must demonstrate Return on Investment after deploying the process. Teams should publicly communicate about their projects (website, blogs, etc.). Teams must write a detailed success story presenting their projects.
  • All teams must be formed by January 31, 2009. All projects must be introduced by February 28, 2009. All projects must be completed by June 30, 2009. Once all projects are completed, they will be presented to our Board of Directors, which will select the winning project, based on the following criteria: Best success story Highest Return on Investment Best utilization of Intalio’s products Highest respect of industry best practices Most contributions to product improvement The winning team will receive $10,000 in Amazon credits, to be shared equally among all 3 teams members. The CEO is eligible to participate, but shall donate his prize to a charity of the team’s choosing in the event that he would be part of the winning team. The prize will (hopefully) be funded by savings realized from the deployment of processes developed through the project. Sample Process Since I am eligible to participate, I have decided to automate the process that could have the most impact on Intalio’s success in 2009: our Marketing & Public Relations process. Historically, marketing has been Intalio’s weakest points, and our resources are extremely limited, with only one full-time employee in our marketing “team”. Our marketing budget is essentially spent on payroll and a very modest use of Google AdWords (about $4,000/month). We do not go to tradeshows, we do not advertise in trade publications, and we do not retain the services of a PR agency anymore. That’s what I call marketing on a shoestring budget. But lack of resources should not be an excuse for lack of execution. With today’s tools, there are many things that can be done with very limited resources, as long as you can put a good process in place, and this is what my team is about to implement. Talking about the team, I have the privilege of working with Antoine Toulme (Blog), lead developer for Intalio’s BPMN modeler (based on Eclipse BPMN Modeler, which we contributed three years ago), and Rick Geneva, process expert based in Atlanta, GA, and co-author of the book The Microguide to Process Modeling in BPMN. Process Discovery The process we are implementing is responsible for keeping track of all marketing and public relations activities that must be conducted in response to newsworthy events, such as the release of a new product, the acquisition of a new company, or the hire of a new executive. Using Google Docs (see spreadsheet in .xls format) for process discovery, we identified 25 such news item, and defined the set of actions that need to be conducted in response to them. We identified 20 such steps, including drafting marketing collaterals, conducting analyst briefings, or issuing a press release.
  • Since the downstream process is actually fairly linear, we used the spreadsheet editor as a way to capture all process steps, in a two dimensional way: news items are listed in rows in no particular order, and steps are listed in columns in sequential orders. Steps were then grouped in sequential phases (using cell merging, also known as colspan in HTML). Steps that are part of the same phase can be executed in parallel. We identified seven phases, plus the process instantiation phase (Phase 0). Once all news items, phases, and steps were identified, time came to fill up the matrix, indicating which steps were required for every news item. For doing this, we used the following conventions: ‘X’ sign: mandatory step ‘?’ sign: contextual step ‘O’ sign: optional step Contextual steps are not mandatory and are included in the list of steps to be completed for a given news item depending on the context of the news item. The decision to make a contextual step part of the process instance is taken by the Marketing Manager during the news item’s initial review (Pase 1/Review Item). Optional steps are always included in the process instance, but their completion is not required, and the process instance can continue its execution if the optional step has not been completed within a configurable time period. Completing the matrix of 500 cells gave us a full map of the process. If the process were to be mapped in BPMN directly, it would require 500 decision gateways, but using the matrix to feed a rules engine will reduce this number down to 20. Once this work was completed, we defined the Roles who would be involved in the process, and identified 5 of them: Vice President of Alliances Vice President of Marketing Marketing Manager IT|Redux Editor Contractor Process Implementation We then specified which notification mechanism would be used for each step, based on the preferences of each Role holder: the Vice President of Alliances is using Microsoft Exchange and a BlackBerry device, and wanted tasks to be created directly into Microsoft Exchange; the Vice President of Marketing (a role I am holding now) is doing pretty much everything withing Salesforce.com, and wants all his tasks to be created there; the Marketing Manager (Jonathan, who really holds the title of Director of Marketing, but it did not fit into the spreadsheet’s cell) offered to use Intalio’s Task Manager; the Editor of the IT|Redux blog is the same as the Vice President of Marketing (damn, I am wearing a lot of hats…); finally, the Contractor is also using Intalio’s Task Manager, thereby demonstrating that such a component can be used outside of the firewall.
  • Once all that was set, we defined how each step would be implemented. This allowed us to identify 6 types of step, that would be packaged as composable services: Data Input Task, with a form to be produced using Intalio|AJAX Approval Task, with three possible actions (Approve, Deny, and Comment) Manual Task, with two possible actions (Complete and Save) Multiple Tasks, which are nothing more than multiple Manual Tasks Email, sent to a small number of recipients (typically less than 100) Email Blast, sent to a large number of recipients (typically up to 100,000) At the exception of the Data Input Task and Multiple Tasks steps, most steps are pretty generic. Nevertheless, we felt the need to provide more details for some steps, such as the Email and Email Blast steps. For them, we indicated recipients to emails directly in the cell, sometime suffixed by the ‘?’ sign in order to indicate a contextual step. For the Data Input Task step, we listed the combined set of form elements used by forms to be displayed to the Marketing Manager during the initial review step (Phase 1/Review Item). For most form elements, we marked the required ones with the ‘X’ sign. In some cases, we provided more details, such as the expected content of the Description field (Biography, Executive Summary, etc.). For Links (URLs), we indicated the content of the linked page (Datasheet, Use Case, etc.), and the online service expected to be used for serving it (WordPress for example). We also did the same for attachments, indicating the software to be used for producing them (Apple Keynote or Microsoft Word for example). We also suffixed all form element labels with their datatypes, like [Text], [TextArea], or [URL]. This essentially gave us all the information we needed to dynamically produce contextual process instantiation forms. In order to complete our process design, we specified which kind of event would be responsible for creating a new process instance, and used comments on cells to indicate who in the company was expected to generate such events (in this particular case, comments are used for process documentation purposes). We identified four main kinds of process instantiation events: Web Form (WF) RSS Feed (RSS) Google Alert (GA) Salesforce.com Alert (SFDC) At this point, we had a complete model of our process. High-level modeling took our Vice President of Sales & Operations (business sponsor), Jonathan, and I about 1 hour (3 person-hour). Low-level modeling, including all deployment details presented in this article took me another 5 hours or so (I finished the spreadsheet and wrote this article on my way to Tokyo, for movies available on my flight were simply too bad to waste time watching them — I won’t give titles). Framework Design
  • What came next is the really interesting part of this article, and justifies the aforeindicated (neologism?) warning. When I say that we had a complete “model” of our process, I should say that we had a complete “design” instead, for everything needed to make this process executable was right there, in the spreadsheet. Assuming that conventions would be respected, all the information needed for making this process executable had been captured in the spreadsheet, using what is arguably one of the most primitive spreadsheet editors currently in use today — albeit a very convenient one, which I use for most of my spreadsheets, on a daily basis, thank you Google! The challenge then became: how do we turn this spreadsheet into an executable process by using off-the-shelf software and writing a minimal amount of code, using only publicly available APIs? Furthermore, how could we allow the process to be changed directly from the spreadsheet editor, and support dynamic process re-deployment? After a few rounds of discussions, we figured out a working strategy, which will require the following tasks: Task #1: Implement Composable Services As mentioned above, we identified six composable services, which can easily be implemented using Intalio’s connectors and workflow framework. These services will be packaged as WSDL Web Services, and deployed on Intalio|Server. Furthermore, they will be registered into Intalio|ESB, so that other projects could take advantage of them. Here is how each service will be implemented: Data Input Task: forms will be produced for Intalio|AJAX from XML Schemas Approval Task: standard workflow task (Comment implemented using Save) Manual Task: standard workflow task Multiple Tasks: multiple standard workflow tasks created within a FOR-EACH loop Email: standard Email connector Email Blast: custom connector for VerticalResponse By default, Approval Tasks and Manual Tasks should display all the data captured through Data Input Tasks. Furthermore, Approval Tasks should support data editing as well. Approval Tasks should also make it easy to change or edit attached documents. Google Apps should be used for the online editing of attachments which formats are supported by the application. Task #2: Create a Form Generator The form used for initial review (Step 1/Review Item) has 25 variations, one for each news item type. This form will be generated automatically from an XML Schema, which itself will be generated automatically from the master spreadsheet (later referenced as master spreadsheet). Unfortunately, while Intalio|AJAX supports the development of contextual forms for which some elements are optionally displayed based on some specific context, the XML Schema to Form converter does not support this feature yet (Feature Request). In order to work around this limitation, one form will be produced for each news item type.
  • In order to do so automatically, the spreadsheet will be published as a .csv feed, and a web application developed in PHP (a language I am familiar and comfortable with, unlike Java) will generate an XML schema for each news item type. This PHP application will run on top of Quercus, a Java-based implementation of the PHP language. The reason for this is the following: Intalio|Server does not support the execution of PHP code out of the box, and all custom applications must be deployed as a Java-based webapp (per the role of our competition). Fortunately, Quercus can be packaged as a webapp, and serve as container for the execution of custom PHP code. In the future, I would expect Intalio|Server to ship directly with the Quercus PHP interpreter, as well as Jython for Python and JRuby for Ruby (Feature Request). The PHP script will produce a set of XML Schemas to be parsed at deployment time by the XML Schema Import Wizard component of Intalio|Designer, which itself will be deployed as an OSGI runtime component alongside Intalio|Server. Task #3: Create a Decision Table Generator The spreadsheet contains all the information for the decision table (AJ7:BA31) to be fed to the business rules engine (Intalio|BRE). The decision table will be converted into a smaller .csv document using another PHP script, and deployed on the business rules engine using a new API to be provided by Intalio|Designer. This API will be developed through a Demand Driven Development (D3) project for which we are currently looking for sponsors. Task #4: Create a BPMN Generator This is where things get really interesting. The idea is to automatically generate the BPMN process map from the first five rows of the spreadsheet. Row 1: Phases, modeled as sequential sub-processes. Row 2: Steps, modeled as parallel sub-processes. Row 3: Roles, modeled as independent swimlanes. Row 4: Implementations, modeled as sub-processes invoking composable services. Row 5: Task Management, providing details on the composable service to be used. The process will be modeled through one swimlane responsible for orchestrating the end-to-end flow, plus one swimlane per Role. Each service invocation sub-process (Implementation) will be modeled as a an activity preceded by an invocation of the rules engine and a conditional gateway, which together will be responsible for defining whether the corresponding step should be part of the process instance or not. This pattern is equivalent to the concept of pre-condition for activities, which is not part of the standard BPMN specification. Nevertheless, proper use of sub-processes should hide this implementation detail, and depict each step as a single activity. The generation of the BPMN code will be done by another PHP script deployed on the Quercus runtime, on top of Intalio|Server. This script will translate all the information related to the process flow (B1:BA5) into a BPMN serialization format contributed by Intalio to the Eclipse BPMN Modeler project. Even though not a standard yet, this format is perfectly human readable (unlike XMI), and has the benefit of being available today. The PHP script will produce a serialized BPMN document to be parsed at deployment time by a server-side implementation of Intalio|Designer. Unfortunately, the BPMN parser currently does not support references and
  • binding to web services, therefore will have to be extended through a D3 project for which we are currently looking for sponsors. Task #5: Implement a Dynamic Deployment Process A Dynamic Deployment Process will be implemented using a custom BPMN process. This process will be instantiated automatically every 5 minutes, and will lookup the .csv feed provided by the master spreadsheet. If the .csv feed has been modified since the last time a new process version was deployed and the value of cell A1 is “Status: Deployment”, the deployment process will continue its execution. Otherwise, it will terminate. By convention, the developer of the process should change the value of cell A1 to “Status: Development” when making modifications to the process. If a new version of the .csv feed is available, the Dynamic Deployment Process will invoke the Form Generator and deploy a new set of forms on Intalio|AJAX, invoke the Decision Table Generator and deploy a new set of rules on Intalio|BRE, and invoke the BPMN Generator, import the BPMN process map, generate the corresponding BPEL code, and deploy a new set of BPEL processes on Intalio|Server. The Dynamic Deployment Process will then send an email to the process owner notifying her that a new version of the process has been generated and deployed. Expected Product Improvements If we manage to make it all work, we will have developed what might be the first spreadsheet-driven BPM tool, usable by any business analyst, and available as Software as a Service. As currently designed, it could support any process made of sequential phases comprised of multiple parallel steps. Nevertheless, more complex flows could be designed from the spreadsheet by using a construct equivalent to GOTO. In order to model such an arbitrary transition from one step to another, the user could reference a Phase to transition to by simply indicating the Phase’s column reference in the appropriate cell. Ten years ago, I suggested that BPM tools should do for process execution what spreadsheet editors did for computation. With this application, we’re about to make using Intalio’s BPM tools as easy to use as using a spreadsheet editor, literally. And the side benefits of the approach are quite interesting as well: Web 2.0 Collaboration By using an online spreadsheet editor such as Google Docs, we make it easy to share executable process models among collaborators, or publish them on the Internet for anyone to see. Access rights can be easily granted and revoked by simply adding and removing email addresses to and from a list of collaborators, across organization boundaries. Additionally, tools such as Google Docs provide full versioning capabilities, making it easy to keep track of all changes made to process definitions and revert to previous versions. Pre-processing Since the entire process model is designed within a spreadsheet, all parameters and rules driving the process execution can be the result of complex calculations executed at deployment time. Such calculations can be implemented as simple formulas when using any online spreadsheet editor, or complex ones making invocations to macros when using more sophisticated editors such as Zoho Sheet.
  • Integrated Dashboard Online spreadsheet editors such as Google Docs or Zoho Sheet allow the content of cells to be dynamically updated through Web Service interfaces. Using this feature, the process engine (Intalio|Server) could feed metrics and KPIs back to the master spreadsheet for real-time Business Activity Monitoring (BAM). As a result, process owners would have a fully integrated interface for defining the process, setting business rules, monitoring the process in real-time, and applying process optimizations on the fly, all from the intuitive spreadsheet metaphor that any business user is accustomed to. Service Composition While all the capabilities listed above are interesting, the most important by-product of this project should be a better understanding of how Intalio’s products can support the notion of Service Composition. Currently, Intalio|Designer allows processes to be packaged as reusable services, but requires that data mappings be specified at every step of the process. While the Data Mapper component of Intalio|Designer makes it easy to design both simple and complex mappings, the very requirement for them gets in the way of simple Service Composition that could be performed by non-technical users. The meta-data driven approach advocated by our project intrinsically relies on the availability of composable services designed against pre-defined data schemas and packaged with all required mappings. While such an approach makes it a little bit more difficult for a composable service to leverage a new data element defined for a Data Input Task in the master spreadsheet, it makes it a lot easier for new processes to be composed from existing services. But for such an approach to work , composable services must provide the right set of interfaces and built-in mappings, and much work remains to be done in order to discover what such interfaces should be to address a wide range of scenarios. Once such a discovery is made, many different metaphors could be used for supporting the notion of Service Composition, and the spreadsheet is only one of them, albeit possibly the most intuitive and flexible. Expected Business Benefits With respect to the actual Marketing & Public Relations process, the benefits we are expecting are many. First among them is the ability for our small marketing team to scale its output as our business is growing, without having to increase the size of the team, nor using the services of outside contractors. Properly handling our PR needs would in and by itself save us about $200,000 a year. Implementing the process as designed should also make it easier for some steps to be outsourced, such as lead generation for local events for example (Phase 3/Generate Leads). Second, we expect that by simply issuing a steady stream of press releases, blog posts, and white papers, we will increase Intalio’s visibility in the marketplace, thereby increasing the number of visits to our website. Third, by providing up to date content on the website, we expect conversion rates from visitors to users and users to paying customers to increase as well. Finally, by automating the end-to-end Marketing & Public Relations process, we hope to gain better visibility on the effectiveness of fine-grain marketing campaigns (such as the distribution of a Press Release or the publishing of a White Paper), which will allow us to better allocate our limited marketing resources. Ongoing Updates
  • Our intent is to fully document our progress online. We will record the time spent working on the project on a timesheet shared among collaborators, alongside new ideas and integration requirements captured through additional sheets on the master spreadsheet. We will publish regular updates on this blog and a section of the new Intalio website to be added soon. We will release our code under the Apache Public License, and share as many process artifacts as possible (master spreadsheet, .csv feeds, XML Schemas, generated forms, decision table, Serialized BPMN documents, generated BPMN maps, WSDL interfaces, etc.). We will also publish real-time metrics regarding the occurrences of news items and process cycle times once the process is deployed. Future Roadmap If we are successful, we will package the Form Generator, Decision Table Generator, BPMN Generator, Dynamic Deployment Process, and all generic composable services (Data Input Task, Approval Task, Manual Task, Multiple Tasks, and Email) as a process template, and make it part of Intalio|BPP Business Edition, Intalio|Community Edition, and Intalio|BPP Enterprise Edition. Wish us luck, and consider sponsoring some of our D3 projects! Entry filed under: BPM 2.0
  • A First Taste of Dogfood Sunday, January 25th 2009 | Ismael Ghalimi Our Project Dogfood was launched just two weeks ago, but it’s running full speed ahead. Among the most active projects, the Customer Support Process lead by Gene Grell — Intalio’s new Director of Support — and the Marketing & Public Relations Process, which I am working on with Rick Geneva and Antoine Toulme. Here is a quick update on both, plus a really cool side project that emerged from one of them. First, the Customer Support Process. It’s a pretty complex one originated from a custom user interface built in Ruby on Rails, integrated with our new trouble-ticketing system (itself integrated with Salesforce.com), our knowledge database (Confluence), our legacy bug tracking system (Jira), and our original Demand Driven Development (D3) portal (itself to be migrated to a collection of processes running on Intalio|BPP). This process is only a prototype right now, but should be fully implemented within a month or two. In the meantime, feel free to download the BPMN project for it, which you can import directly from Intalio|Designer (File/Import…). Customer Support Process Second, our Marketing Process, which was described in details in this article. After some internal discussions, we concluded that the process could be implemented through a generic BPMN pattern, rather than dynamically generating BPMN code. This is a much better solution, for three main reasons: One, it’s a lot easier to implement. Two, it will allow any changes to the Execution Table (the spreadsheet describing the
  • process) to be applied to running process instances in flight. Three, it will support the definition of any loops within the process, and the use of recursion within the process — a process instance can create a new one, like is the case when a Public Call or a Webinar have to be organized for example. What follows is a map of the generic process that will support the execution of any process designed with our Execution Table Template. Marketing & Public Relations Process The Execution Table Template has been cleaned up in order to support its design with Google Apps or Zoho (Google Apps version), the definition of arbitrary loops, recursion, and multiple forms, which are now captured through separate sheets (Data Input Form). Integration Points, Ideas, and Timesheet are also captured through separated sheets, and used for documentation purposes only. What follows is a screenshot of the Execution Table designed using Google Apps, also made publicly available (Decision Table | BPMN Model)
  • Execution Table What’s truly remarkable about this project is that Intalio|Designer and Intalio|Server could be used off-the-shelf for implementing a custom process design and execution framework built upon a proprietary process execution meta-model, in a matter of days, and without having to write custom code. So far, we spent 18.5 hours on the project, and we expect to have a first running version of the end-to-end process within two weeks, working on it in our spare time. Another interesting discovery made through this project is the fact that processes modeled with our Execution Table Template can be used by auditors working on SoX compliance projects. We presented our idea to half a dozen companies in Japan earlier this week, and every single one indicated that they could use our tool for their J-SOX projects (J-SOX is the Japanese equivalent of the Sarbanes-Oxley Act). As a result, we’re seriously considering packaging this template with our Business Rules Engine (Intalio|BRE) and upcoming Complex Event Processing (CEP) engine into a solution for next-generation Governance, Risk Management, and Compliance (GRC). Third, the side project I mentioned above. As part of our Marketing & Public Relations project, we had a need for an Enterprise Content Management system that could be used to develop and publish all our marketing collaterals. While we could have used Intalio|ECM for this purpose, integration with Google Apps and Zoho was lacking. Furthermore, we wanted a system that could not only store our marketing collaterals, but also all our process artifacts, fully integrated with Intalio|Designer. Last but not least, we wanted a system that could store XML documents natively, and provide integration with Apache Cocoon, in order to support the publishing of content on our existing website.
  • After some internal discussions, we decided to develop our own system. It is built on top of Apache Sling, which itself is based on Apache Jackrabbit. The funny story about Jackrabbit is that it is based on Apache Slide, which itself was contributed by Intalio to the Apache Software Foundation 8 years ago. Small world, very small world… Sling also includes Apache Felix, which is the OSGi framework that all Intalio’s runtime components will be deployed on top of moving forward. Jackrabbit now includes Apache Lucene, to be used as search engine. Finally, we’re looking at the Jena Semantic Web Framework in order to support the tagging and indexation of documents in a scalable manner. At present time, a prototype integration of Sling with Tempo has been developed by Nicolas (Cf. blog post and screencast), and we are considering creating a D3 project for it, while debating about the Open Source license this new project should be released under. If this sounds like fun, please take a look at our tentative roadmap and drop us a line. The codename for this project is Doubleshot.
  • Project Doubleshot Features On . Community Enterprise Demand Priority Component . Document create/read/update/delete/transform/move/copy/search/browse + + + 1 Sling . Automated document versioning + + + 1 Sling . Advanced version control (major/minor version) + + 2 Sling . Check-in/Check-out + + + 1 Sling . Document/Folder import/export + + + 1 Sling . Document/Folder history and rollback + + + 1 Sling . Document/Folder diff and merge + + + 2 ? . Document/Folder sharing (including temporary sharing) + + + 1 Sling . Document/Folder publishing + + + 1 Sling . Document/Folder syndication (RSS interface) + + 1 ? . Archiving engine