1. Camunda Community Meeting
USING BUSINESS PROCESS
MANAGEMENT IN SPATIAL DATA
PROCESSING
www.ptvgroup.com
Siegfried Klausmann, Dominik Eisenberg
Karlsruhe, 22.01.2014
2. AGENDA
1.
2.
3.
4.
5.
6.
7.
8.
Motivation and goals
Main requirements
System Architecture
Demo of current solution
Process/Service concepts
Experiences with BPM introduction
Future plans
Discussion / Best practices
www.ptvag.com
3. MOTIVATION – PLANNING AND OPTIMISATION SOFTWARE
Planning and optimising
the flow of people
www.ptvgroup.com
Planning and optimising
the flow of goods
I Page 4
4. MOTIVATION – PRODUCTS NEED OPTIMIZED DATA
Provider raw data
Quality Assurance steps
Validation, Harmonization
Database
Conversion to binary formats
Vision Suite - PTV Visum, PTV Vissim, PTV Viswalk:
Traffic planning, traffic technology and simulation
PTV Map&Guide:
Transport route planning
PTV Smartour:
Transport planning and optimisation
PTV Map&Market:
Geomarketing, sales, marketing and sales force planning
PTV xServer:
Developer components with core algorithms
www.ptvgroup.com
I Page 5
5. MOTIVATION – CURRENT DATA PROCESSING
Globaltabellen
bearbeiten
PND
1
PremiumNetz
bearbeiten
eKarte
PND
Aufbereitung
Truck
POI
TMC
eKarten
Aufbereitung
Global
Tabellen
PremiumMap
TINFO
Diverse
Scripte
Premium
Netz MapInfo
Mautdaten
AND
Meerland
bearbeiten
ATF
AGF2MapInfo
Premium
Orte
ProDBIntegration
Map2ATF
MapBase
Premium
ATFAGF-ID
Zuordnung
Hamlet Filter
LSN-Korrektur
TauschNamen/RoutNr
ON Priorisierung
Map2MIF
MapInfo
Kontrollsch.
Map2AGF
MapBase
GDF2Map
StatistiX
Statistik
Files
AGF
Check
Log-Files
Meerland
Truck2agf
Projekt
Importer
MapInfo
Kontrollsch.
TA
Logistic
Projektdaten
bearbeiten
Projekt-/
Fremddaten
AND
MapInfo
Personenfähren
löschen
AND2AGF
ATFID2AGFID
PremiumOrte
bearbeiten
GDF
Projekt
Mid
MapAGF2AGF
Projektdaten
bearbeiten
Ableitungs
vorschriften
POI-Icon
Ableitungen.
Maut
Projektdaten
bearbeiten
Steuerungs
Globaldateien Daten
2
DEU LKW
Sperren
Projektdaten
bearbeiten
MapAGF
Tiler
AGF
1
Groß/KleinWandlung
Löschen env. Zonen
Typ
konvertierung
Regionennachberarbeitung
Meeressaum
bearbeiten
Teleporter
integration
AGF2GeoDB
finale
Teleporter
Meeressaum
feature_stat.pl
GeoDB
TBL neu
Creator
Höhenschicht
Australien
Vorverarbeitung
Statistik
Files
2
TBL
neu
Map2MIF
Mautdaten
Manueller Bearbeitungsschritt
Mit GIS-Systemen (MapInfo)
Manueller Bearbeitungsschritt
mit UltraEdit, Scripten, SQL,...
Automatischer Bearbeitungsschritt
Teleporter
nachbearbeitung
Greenzones
MapInfo
Kontrollsch.
Teleporter
abgleich
Maut
bearbeiten
Relevante Zwischenformate
Overviews
Teleporter
Analysesch.
Greenzones
bearbeiten
Fährenlayer
ASCII-Daten (intern)
MIF/MID-Daten (intern)
tw. mit zusätzlicher *.ini)
*.xls-Daten (intern)
„Externe“ Daten
TBL
alt
Datenbank
Fähr
bearbeitung
Wurmloch
xml-Daten (intern))
www.ptvgroup.com
I Page 6
6. MAIN GOALS OF THE NEW SOLUTION
Business perspective
Reduce overall data processing times by
increased productivity
Enable world wide coverage incl. new feature
layers with the same data processing team
But do not invent everything again / Limit
implementation costs
www.ptvgroup.com
I Page 7
7. MAIN REQUIREMENTS
How to achieve the goals?
Easy to use GUI (Graphical User Interface) with
transparent business workflows/processes
Homogeneous behavior of services /
processes (e.g. Logging, Error handling)
Robust processing (high availability, monitoring,
fault tolerance)
Increase automation
Parallel processing (support scaling)
Re-use of existing concepts and software
(preferred Open Source), already existing
business logic
www.ptvgroup.com
I Page 8
9. SYSTEM ARCHITECTURE
Web GUI
Loose coupling of
process control and
workers
(JavaServer Pages/jQuery/Spring MVC)
Process Control
(Tomcat container with BPM engine)
Load distribution by
using multiple
workers and
multiple data DBs
BPM engine DB
(PostgreSQL)
Messaging System
(Apache Active MQ JMS Message Broker)
Worker 1
Worker 2
Worker 2
(Apache Karaf OSGi container with Apache
Camel based Services)
(Apache Karaf OSGi container with Apache
Camel based Services)
(Apache Karaf OSGi container with Apache
Camel based Services)
Service
A
Service
B
Service
B
Service
A
Service
B
Service
C
Separating data DB
from process
control DB
No map data in
payload of BPM
engine
CDM
(PostGIS database)
www.ptvgroup.com
CDM
(PostGIS database)
CDM
(PostGIS database)
Main spatial data
processing in DB
I Page 10
10. DEMO – MAP DATA PROCESSING
http://172.23.71.11:8181/ImpactGui/
www.ptvgroup.com
I Page 11
11. PROCESS/SERVICE CONCEPTS
BPM – Workflow engine
We use an existing workflow engine which supports to
•
•
•
•
•
Persist process state
Store durations of process executions for later analysis and optimization
Profit of existing and new features (tasks, transitions, forks, joins)
Profit of tooling for process modeling, process monitoring, persistence
thus allowing an efficient and transparent data processing
2 kinds of BPM processes for spatial data processing at PTV
• Data Production Processes
• Technical Processes
www.ptvgroup.com
I Page 12
12. PROCESS/SERVICE CONCEPTS
BPM – Kinds of processes
Data Production Processes (DPPs)
•
•
•
•
•
Combining the Human Workflows with automatic data preparation
The only processes, directly started by users
Configured for the data preparation of multiple countries
Can trigger several technical processes as data preparation steps
Can be recovered in case of errors
www.ptvgroup.com
I Page 13
13. PROCESS/SERVICE CONCEPTS
BPM – Kinds of processes
Technical Processes (TPs)
•
•
•
•
•
•
Contain only service tasks for complete automatic data preparation
Not directly started by users
Started by DPPs
Cover a single step of the automatic data preparation
Configured to for data preparation of a single country
Can not be recovered in case of error
www.ptvgroup.com
I Page 14
14. PROCESS/SERVICE CONCEPTS
BPM – Extensions of Service Tasks
MultiProcessExecutionTask in Data Production Processes
www.ptvgroup.com
I Page 15
15. PROCESS/SERVICE CONCEPTS
BPM – Why extending the ServiceTask for this?
ServiceTask in Technical Processes
www.ptvgroup.com
I Page 16
16. EXPERIENCES
BPM Engine
Re-Use: BPM engine for process management is a powerful tool, has
many features which must not be implemented by us
Lightweight: Using the BPM Engine does not have too much impact on
architecture or application development
Transparency: Easy API for implementing visualization and monitoring
of process information, allowing a central overview of all processes
(historic, current, planned)
www.ptvgroup.com
I Page 17
17. EXPERIENCES
Pitfalls for developers using the BPM Engine
OptimisticLockingException
• Querying variables with RuntimeService increments revision rev_
• Exception occurs when quering the same variables in another thread
• E.g. for updating the process monitoring page in GUI
• Solution:
• Usage of HistoryService where ever possible
• Isolating concurrent variable updates by semaphor
HistoricVariableUpdateQuery
• Results of type HistoricVariableUpdate can contain the exact same Date returning by
getTime()
• Problem to determine the last variable update
• Solution:
• Workaround: Using the id or revision, which are strongly incremented
www.ptvgroup.com
I Page 18
18. EXPERIENCES
Co-Work with business users
If BPM is new for the business users, give introduction / benefits
First examples with small models (created by developers)
Hide generic / recurring workflows (especially technical ones)
Create an easy to use and attractive GUI
Train users and let them work with it
Ask for feedback / improvements
In common sessions with users, add more and more steps to
workflows where users see an added value (do not oversize things)
If possible: Integrate business users with dev skills into dev team
Do not give up, it takes time to experience the benefits
www.ptvgroup.com
I Page 19
19. FUTURE PLANS
Modeling
Add higher level processes representing the
overall workflow (all phases of map production
beginning from provider data checks and ending
with data delivery)
Transparency
Visualize process statistics to be able to find
improvements/speed up data production
Statistics depend on configuration
Technical
Supporting different process versions
„Real“ parallel processing
www.ptvgroup.com
I Page 20
