Intalio Project Dogfood And DoubleshotDocument Transcript
Monday, January 19th 2009 | Ismael Ghalimi
While the current recession is forcing many companies to go through painful rounds of layoffs, Intalio is
actually growing its employee base, albeit at a slower rate than in the previous twelve months. There are two
reasons for this: Number one, we want to remain on the conservative side, and continue outsourcing most
non-core functions, which pretty much means everything outside of architecture design and customer training.
Number two, we have started to aggressively use our own products in order to improve our operations, thereby
allowing us to do more with less. We call this initiave Project Dogfood.
WARNING: very long article, but full reading highly recommended.
The idea for Project Dogfood emerged at a recent company meeting, and was refined during our yearly
company offsite. The idea is pretty simple: every single employee (36 out of our 60+ full-time collaborators) will
develop a new process, or improve an existing one, using our products. The project is managed as an internal
competition, and the rules are relatively straightforward:
12 teams of 3 employees must be formed.
Each team must include at least one engineer and one process expert (trainer).
Each team must design and deploy at least one real-world business process.
No two teams can implement the same processes (first come, first served).
Processes must contribute to improve Intalio’s operations.
Processes should not use confidential information (publishing encouraged).
Teams must use off-the-shelf Intalio software.
Any product edition can be used (combinations encouraged).
All custom code must be deployed as a J2EE webapp or XPath function.
Custom product features must be developed through the D3 process.
D3 projects must be funded by external customers, or actual cost savings.
D3 projects funded internally must be approved by the CFO.
Projects should have a business sponsor, who does not have to be part of the team.
Processes must be deployed on the latest versions of Intalio’s products.
Processes must be used in production on an ongoing basis.
Processes must be kept running when new product versions are deployed.
New product versions cannot ship until all processes have been upgraded.
Teams must capture time spent working on the project.
Teams must create a business case before implementing the process.
Teams must demonstrate Return on Investment after deploying the process.
Teams should publicly communicate about their projects (website, blogs, etc.).
Teams must write a detailed success story presenting their projects.
All teams must be formed by January 31, 2009.
All projects must be introduced by February 28, 2009.
All projects must be completed by June 30, 2009.
Once all projects are completed, they will be presented to our Board of Directors, which will select the winning
project, based on the following criteria:
Best success story
Highest Return on Investment
Best utilization of Intalio’s products
Highest respect of industry best practices
Most contributions to product improvement
The winning team will receive $10,000 in Amazon credits, to be shared equally among all 3 teams members.
The CEO is eligible to participate, but shall donate his prize to a charity of the team’s choosing in the event that
he would be part of the winning team. The prize will (hopefully) be funded by savings realized from the
deployment of processes developed through the project.
Since I am eligible to participate, I have decided to automate the process that could have the most impact on
Intalio’s success in 2009: our Marketing & Public Relations process. Historically, marketing has been Intalio’s
weakest points, and our resources are extremely limited, with only one full-time employee in our marketing
“team”. Our marketing budget is essentially spent on payroll and a very modest use of Google AdWords (about
$4,000/month). We do not go to tradeshows, we do not advertise in trade publications, and we do not retain
the services of a PR agency anymore. That’s what I call marketing on a shoestring budget.
But lack of resources should not be an excuse for lack of execution. With today’s tools, there are many things
that can be done with very limited resources, as long as you can put a good process in place, and this is what
my team is about to implement. Talking about the team, I have the privilege of working with Antoine Toulme
(Blog), lead developer for Intalio’s BPMN modeler (based on Eclipse BPMN Modeler, which we contributed
three years ago), and Rick Geneva, process expert based in Atlanta, GA, and co-author of the book The
Microguide to Process Modeling in BPMN.
The process we are implementing is responsible for keeping track of all marketing and public relations
activities that must be conducted in response to newsworthy events, such as the release of a new product, the
acquisition of a new company, or the hire of a new executive. Using Google Docs (see spreadsheet in .xls
format) for process discovery, we identified 25 such news item, and defined the set of actions that need to be
conducted in response to them. We identified 20 such steps, including drafting marketing collaterals,
conducting analyst briefings, or issuing a press release.
Since the downstream process is actually fairly linear, we used the spreadsheet editor as a way to capture all
process steps, in a two dimensional way: news items are listed in rows in no particular order, and steps are
listed in columns in sequential orders. Steps were then grouped in sequential phases (using cell merging, also
known as colspan in HTML). Steps that are part of the same phase can be executed in parallel. We identified
seven phases, plus the process instantiation phase (Phase 0).
Once all news items, phases, and steps were identified, time came to fill up the matrix, indicating which steps
were required for every news item. For doing this, we used the following conventions:
‘X’ sign: mandatory step
‘?’ sign: contextual step
‘O’ sign: optional step
Contextual steps are not mandatory and are included in the list of steps to be completed for a given news item
depending on the context of the news item. The decision to make a contextual step part of the process
instance is taken by the Marketing Manager during the news item’s initial review (Pase 1/Review Item).
Optional steps are always included in the process instance, but their completion is not required, and the
process instance can continue its execution if the optional step has not been completed within a configurable
Completing the matrix of 500 cells gave us a full map of the process. If the process were to be mapped in
BPMN directly, it would require 500 decision gateways, but using the matrix to feed a rules engine will reduce
this number down to 20.
Once this work was completed, we defined the Roles who would be involved in the process, and identified 5
Vice President of Alliances
Vice President of Marketing
We then specified which notification mechanism would be used for each step, based on the preferences of
each Role holder: the Vice President of Alliances is using Microsoft Exchange and a BlackBerry device, and
wanted tasks to be created directly into Microsoft Exchange; the Vice President of Marketing (a role I am
holding now) is doing pretty much everything withing Salesforce.com, and wants all his tasks to be created
there; the Marketing Manager (Jonathan, who really holds the title of Director of Marketing, but it did not fit into
the spreadsheet’s cell) offered to use Intalio’s Task Manager; the Editor of the IT|Redux blog is the same as
the Vice President of Marketing (damn, I am wearing a lot of hats…); finally, the Contractor is also using
Intalio’s Task Manager, thereby demonstrating that such a component can be used outside of the firewall.
Once all that was set, we defined how each step would be implemented. This allowed us to identify 6 types of
step, that would be packaged as composable services:
Data Input Task, with a form to be produced using Intalio|AJAX
Approval Task, with three possible actions (Approve, Deny, and Comment)
Manual Task, with two possible actions (Complete and Save)
Multiple Tasks, which are nothing more than multiple Manual Tasks
Email, sent to a small number of recipients (typically less than 100)
Email Blast, sent to a large number of recipients (typically up to 100,000)
At the exception of the Data Input Task and Multiple Tasks steps, most steps are pretty generic. Nevertheless,
we felt the need to provide more details for some steps, such as the Email and Email Blast steps. For them, we
indicated recipients to emails directly in the cell, sometime suffixed by the ‘?’ sign in order to indicate a
For the Data Input Task step, we listed the combined set of form elements used by forms to be displayed to the
Marketing Manager during the initial review step (Phase 1/Review Item). For most form elements, we marked
the required ones with the ‘X’ sign. In some cases, we provided more details, such as the expected content of
the Description field (Biography, Executive Summary, etc.). For Links (URLs), we indicated the content of the
linked page (Datasheet, Use Case, etc.), and the online service expected to be used for serving it (WordPress
for example). We also did the same for attachments, indicating the software to be used for producing them
(Apple Keynote or Microsoft Word for example). We also suffixed all form element labels with their datatypes,
like [Text], [TextArea], or [URL]. This essentially gave us all the information we needed to dynamically produce
contextual process instantiation forms.
In order to complete our process design, we specified which kind of event would be responsible for creating a
new process instance, and used comments on cells to indicate who in the company was expected to generate
such events (in this particular case, comments are used for process documentation purposes). We identified
four main kinds of process instantiation events:
Web Form (WF)
RSS Feed (RSS)
Google Alert (GA)
Salesforce.com Alert (SFDC)
At this point, we had a complete model of our process. High-level modeling took our Vice President of Sales &
Operations (business sponsor), Jonathan, and I about 1 hour (3 person-hour). Low-level modeling, including
all deployment details presented in this article took me another 5 hours or so (I finished the spreadsheet and
wrote this article on my way to Tokyo, for movies available on my flight were simply too bad to waste time
watching them — I won’t give titles).
What came next is the really interesting part of this article, and justifies the aforeindicated (neologism?)
warning. When I say that we had a complete “model” of our process, I should say that we had a complete
“design” instead, for everything needed to make this process executable was right there, in the spreadsheet.
Assuming that conventions would be respected, all the information needed for making this process executable
had been captured in the spreadsheet, using what is arguably one of the most primitive spreadsheet editors
currently in use today — albeit a very convenient one, which I use for most of my spreadsheets, on a daily basis,
thank you Google!
The challenge then became: how do we turn this spreadsheet into an executable process by using off-the-shelf
software and writing a minimal amount of code, using only publicly available APIs? Furthermore, how could we
allow the process to be changed directly from the spreadsheet editor, and support dynamic process
re-deployment? After a few rounds of discussions, we figured out a working strategy, which will require the
Task #1: Implement Composable Services
As mentioned above, we identified six composable services, which can easily be implemented using Intalio’s
connectors and workflow framework. These services will be packaged as WSDL Web Services, and deployed
on Intalio|Server. Furthermore, they will be registered into Intalio|ESB, so that other projects could take
advantage of them. Here is how each service will be implemented:
Data Input Task: forms will be produced for Intalio|AJAX from XML Schemas
Approval Task: standard workflow task (Comment implemented using Save)
Manual Task: standard workflow task
Multiple Tasks: multiple standard workflow tasks created within a FOR-EACH loop
Email: standard Email connector
Email Blast: custom connector for VerticalResponse
By default, Approval Tasks and Manual Tasks should display all the data captured through Data Input Tasks.
Furthermore, Approval Tasks should support data editing as well. Approval Tasks should also make it easy to
change or edit attached documents. Google Apps should be used for the online editing of attachments which
formats are supported by the application.
Task #2: Create a Form Generator
The form used for initial review (Step 1/Review Item) has 25 variations, one for each news item type. This form
will be generated automatically from an XML Schema, which itself will be generated automatically from the
master spreadsheet (later referenced as master spreadsheet).
Unfortunately, while Intalio|AJAX supports the development of contextual forms for which some elements are
optionally displayed based on some specific context, the XML Schema to Form converter does not support this
feature yet (Feature Request). In order to work around this limitation, one form will be produced for each news
In order to do so automatically, the spreadsheet will be published as a .csv feed, and a web application
developed in PHP (a language I am familiar and comfortable with, unlike Java) will generate an XML schema
for each news item type. This PHP application will run on top of Quercus, a Java-based implementation of the
PHP language. The reason for this is the following: Intalio|Server does not support the execution of PHP code
out of the box, and all custom applications must be deployed as a Java-based webapp (per the role of our
competition). Fortunately, Quercus can be packaged as a webapp, and serve as container for the execution of
custom PHP code. In the future, I would expect Intalio|Server to ship directly with the Quercus PHP interpreter,
as well as Jython for Python and JRuby for Ruby (Feature Request). The PHP script will produce a set of XML
Schemas to be parsed at deployment time by the XML Schema Import Wizard component of Intalio|Designer,
which itself will be deployed as an OSGI runtime component alongside Intalio|Server.
Task #3: Create a Decision Table Generator
The spreadsheet contains all the information for the decision table (AJ7:BA31) to be fed to the business rules
engine (Intalio|BRE). The decision table will be converted into a smaller .csv document using another PHP
script, and deployed on the business rules engine using a new API to be provided by Intalio|Designer. This API
will be developed through a Demand Driven Development (D3) project for which we are currently looking
Task #4: Create a BPMN Generator
This is where things get really interesting. The idea is to automatically generate the BPMN process map from
the first five rows of the spreadsheet.
Row 1: Phases, modeled as sequential sub-processes.
Row 2: Steps, modeled as parallel sub-processes.
Row 3: Roles, modeled as independent swimlanes.
Row 4: Implementations, modeled as sub-processes invoking composable services.
Row 5: Task Management, providing details on the composable service to be used.
The process will be modeled through one swimlane responsible for orchestrating the end-to-end flow, plus one
swimlane per Role. Each service invocation sub-process (Implementation) will be modeled as a an activity
preceded by an invocation of the rules engine and a conditional gateway, which together will be responsible for
defining whether the corresponding step should be part of the process instance or not. This pattern is
equivalent to the concept of pre-condition for activities, which is not part of the standard BPMN specification.
Nevertheless, proper use of sub-processes should hide this implementation detail, and depict each step as a
The generation of the BPMN code will be done by another PHP script deployed on the Quercus runtime, on top
of Intalio|Server. This script will translate all the information related to the process flow (B1:BA5) into a BPMN
serialization format contributed by Intalio to the Eclipse BPMN Modeler project. Even though not a standard yet,
this format is perfectly human readable (unlike XMI), and has the benefit of being available today. The PHP
script will produce a serialized BPMN document to be parsed at deployment time by a server-side
implementation of Intalio|Designer. Unfortunately, the BPMN parser currently does not support references and
binding to web services, therefore will have to be extended through a D3 project for which we are currently
looking for sponsors.
Task #5: Implement a Dynamic Deployment Process
A Dynamic Deployment Process will be implemented using a custom BPMN process. This process will be
instantiated automatically every 5 minutes, and will lookup the .csv feed provided by the master spreadsheet. If
the .csv feed has been modified since the last time a new process version was deployed and the value of cell
A1 is “Status: Deployment”, the deployment process will continue its execution. Otherwise, it will terminate. By
convention, the developer of the process should change the value of cell A1 to “Status: Development” when
making modifications to the process.
If a new version of the .csv feed is available, the Dynamic Deployment Process will invoke the Form Generator
and deploy a new set of forms on Intalio|AJAX, invoke the Decision Table Generator and deploy a new set of
rules on Intalio|BRE, and invoke the BPMN Generator, import the BPMN process map, generate the
corresponding BPEL code, and deploy a new set of BPEL processes on Intalio|Server. The Dynamic
Deployment Process will then send an email to the process owner notifying her that a new version of the
process has been generated and deployed.
Expected Product Improvements
If we manage to make it all work, we will have developed what might be the first spreadsheet-driven BPM tool,
usable by any business analyst, and available as Software as a Service. As currently designed, it could
support any process made of sequential phases comprised of multiple parallel steps. Nevertheless, more
complex flows could be designed from the spreadsheet by using a construct equivalent to GOTO. In order to
model such an arbitrary transition from one step to another, the user could reference a Phase to transition to
by simply indicating the Phase’s column reference in the appropriate cell.
Ten years ago, I suggested that BPM tools should do for process execution what spreadsheet editors did for
computation. With this application, we’re about to make using Intalio’s BPM tools as easy to use as using a
spreadsheet editor, literally. And the side benefits of the approach are quite interesting as well:
Web 2.0 Collaboration
By using an online spreadsheet editor such as Google Docs, we make it easy to share executable process
models among collaborators, or publish them on the Internet for anyone to see. Access rights can be easily
granted and revoked by simply adding and removing email addresses to and from a list of collaborators,
across organization boundaries. Additionally, tools such as Google Docs provide full versioning capabilities,
making it easy to keep track of all changes made to process definitions and revert to previous versions.
Since the entire process model is designed within a spreadsheet, all parameters and rules driving the process
execution can be the result of complex calculations executed at deployment time. Such calculations can be
implemented as simple formulas when using any online spreadsheet editor, or complex ones making
invocations to macros when using more sophisticated editors such as Zoho Sheet.
Online spreadsheet editors such as Google Docs or Zoho Sheet allow the content of cells to be dynamically
updated through Web Service interfaces. Using this feature, the process engine (Intalio|Server) could feed
metrics and KPIs back to the master spreadsheet for real-time Business Activity Monitoring (BAM). As a result,
process owners would have a fully integrated interface for defining the process, setting business rules,
monitoring the process in real-time, and applying process optimizations on the fly, all from the intuitive
spreadsheet metaphor that any business user is accustomed to.
While all the capabilities listed above are interesting, the most important by-product of this project should be a
better understanding of how Intalio’s products can support the notion of Service Composition. Currently,
Intalio|Designer allows processes to be packaged as reusable services, but requires that data mappings be
specified at every step of the process. While the Data Mapper component of Intalio|Designer makes it easy to
design both simple and complex mappings, the very requirement for them gets in the way of simple Service
Composition that could be performed by non-technical users.
The meta-data driven approach advocated by our project intrinsically relies on the availability of composable
services designed against pre-defined data schemas and packaged with all required mappings. While such an
approach makes it a little bit more difficult for a composable service to leverage a new data element defined for
a Data Input Task in the master spreadsheet, it makes it a lot easier for new processes to be composed from
existing services. But for such an approach to work , composable services must provide the right set of
interfaces and built-in mappings, and much work remains to be done in order to discover what such interfaces
should be to address a wide range of scenarios. Once such a discovery is made, many different metaphors
could be used for supporting the notion of Service Composition, and the spreadsheet is only one of them,
albeit possibly the most intuitive and flexible.
Expected Business Benefits
With respect to the actual Marketing & Public Relations process, the benefits we are expecting are many. First
among them is the ability for our small marketing team to scale its output as our business is growing, without
having to increase the size of the team, nor using the services of outside contractors. Properly handling our PR
needs would in and by itself save us about $200,000 a year. Implementing the process as designed should
also make it easier for some steps to be outsourced, such as lead generation for local events for example
(Phase 3/Generate Leads). Second, we expect that by simply issuing a steady stream of press releases, blog
posts, and white papers, we will increase Intalio’s visibility in the marketplace, thereby increasing the number
of visits to our website. Third, by providing up to date content on the website, we expect conversion rates from
visitors to users and users to paying customers to increase as well. Finally, by automating the end-to-end
Marketing & Public Relations process, we hope to gain better visibility on the effectiveness of fine-grain
marketing campaigns (such as the distribution of a Press Release or the publishing of a White Paper), which
will allow us to better allocate our limited marketing resources.
Our intent is to fully document our progress online. We will record the time spent working on the project on a
timesheet shared among collaborators, alongside new ideas and integration requirements captured through
additional sheets on the master spreadsheet. We will publish regular updates on this blog and a section of the
new Intalio website to be added soon. We will release our code under the Apache Public License, and share
as many process artifacts as possible (master spreadsheet, .csv feeds, XML Schemas, generated forms,
decision table, Serialized BPMN documents, generated BPMN maps, WSDL interfaces, etc.). We will also
publish real-time metrics regarding the occurrences of news items and process cycle times once the process
If we are successful, we will package the Form Generator, Decision Table Generator, BPMN Generator,
Dynamic Deployment Process, and all generic composable services (Data Input Task, Approval Task, Manual
Task, Multiple Tasks, and Email) as a process template, and make it part of Intalio|BPP Business Edition,
Intalio|Community Edition, and Intalio|BPP Enterprise Edition.
Wish us luck, and consider sponsoring some of our D3 projects!
Entry filed under: BPM 2.0
A First Taste of Dogfood
Sunday, January 25th 2009 | Ismael Ghalimi
Our Project Dogfood was launched just two weeks ago, but it’s running full speed ahead. Among the most
active projects, the Customer Support Process lead by Gene Grell — Intalio’s new Director of Support — and
the Marketing & Public Relations Process, which I am working on with Rick Geneva and Antoine Toulme. Here
is a quick update on both, plus a really cool side project that emerged from one of them.
First, the Customer Support Process. It’s a pretty complex one originated from a custom user interface built in
Ruby on Rails, integrated with our new trouble-ticketing system (itself integrated with Salesforce.com), our
knowledge database (Confluence), our legacy bug tracking system (Jira), and our original Demand Driven
Development (D3) portal (itself to be migrated to a collection of processes running on Intalio|BPP). This
process is only a prototype right now, but should be fully implemented within a month or two. In the meantime,
feel free to download the BPMN project for it, which you can import directly from
Customer Support Process
Second, our Marketing Process, which was described in details in this article. After some internal discussions,
we concluded that the process could be implemented through a generic BPMN pattern, rather than
dynamically generating BPMN code. This is a much better solution, for three main reasons: One, it’s a lot
easier to implement. Two, it will allow any changes to the Execution Table (the spreadsheet describing the
process) to be applied to running process instances in flight. Three, it will support the definition of any loops
within the process, and the use of recursion within the process — a process instance can create a new one, like
is the case when a Public Call or a Webinar have to be organized for example. What follows is a map of the
generic process that will support the execution of any process designed with our Execution Table Template.
Marketing & Public Relations Process
The Execution Table Template has been cleaned up in order to support its design with Google Apps or Zoho
(Google Apps version), the definition of arbitrary loops, recursion, and multiple forms, which are now captured
through separate sheets (Data Input Form). Integration Points, Ideas, and Timesheet are also captured
through separated sheets, and used for documentation purposes only. What follows is a screenshot of the
Execution Table designed using Google Apps, also made publicly available (Decision Table | BPMN Model)
What’s truly remarkable about this project is that Intalio|Designer and Intalio|Server could be used off-the-shelf
for implementing a custom process design and execution framework built upon a proprietary process
execution meta-model, in a matter of days, and without having to write custom code. So far, we spent 18.5
hours on the project, and we expect to have a first running version of the end-to-end process within two weeks,
working on it in our spare time.
Another interesting discovery made through this project is the fact that processes modeled with our Execution
Table Template can be used by auditors working on SoX compliance projects. We presented our idea to half a
dozen companies in Japan earlier this week, and every single one indicated that they could use our tool for
their J-SOX projects (J-SOX is the Japanese equivalent of the Sarbanes-Oxley Act). As a result, we’re
seriously considering packaging this template with our Business Rules Engine (Intalio|BRE) and upcoming
Complex Event Processing (CEP) engine into a solution for next-generation Governance, Risk
Management, and Compliance (GRC).
Third, the side project I mentioned above. As part of our Marketing & Public Relations project, we had a need
for an Enterprise Content Management system that could be used to develop and publish all our marketing
collaterals. While we could have used Intalio|ECM for this purpose, integration with Google Apps and Zoho
was lacking. Furthermore, we wanted a system that could not only store our marketing collaterals, but also all
our process artifacts, fully integrated with Intalio|Designer. Last but not least, we wanted a system that could
store XML documents natively, and provide integration with Apache Cocoon, in order to support the publishing
of content on our existing website.
After some internal discussions, we decided to develop our own system. It is built on top of Apache Sling,
which itself is based on Apache Jackrabbit. The funny story about Jackrabbit is that it is based on Apache
Slide, which itself was contributed by Intalio to the Apache Software Foundation 8 years ago. Small world, very
small world… Sling also includes Apache Felix, which is the OSGi framework that all Intalio’s runtime
components will be deployed on top of moving forward. Jackrabbit now includes Apache Lucene, to be used
as search engine. Finally, we’re looking at the Jena Semantic Web Framework in order to support the tagging
and indexation of documents in a scalable manner.
At present time, a prototype integration of Sling with Tempo has been developed by Nicolas (Cf. blog post and
screencast), and we are considering creating a D3 project for it, while debating about the Open Source license
this new project should be released under. If this sounds like fun, please take a look at our tentative roadmap
and drop us a line. The codename for this project is Doubleshot.