Presentation given at FME WT in Australia

1. Potent Potions
for Batch Data
Processing

2. 250,000 CAD
files & rasters
on mobile devices

4. Tip: Know
Your Potions
and Choose
Wisely

5. Today’s Potions
1. Wildcards
2. Batch Deploy
3. Parent/child Workspaces
4. Parent/child Server Workspaces

6. Potion 1:
One Wild<card>
Dataset

7. Multi-Dataset Picker

8. Multi-Dataset Picker

9. Multi-Dataset Picker

10. Multi-Dataset Picker
Shapefile
MapInfo
Most rasters
DWG
DGN
SQLite

11. Dataset
Wildcards
Extended glob syntax:
Symbol Matches
? Any single character
* Any sequence of zero or more
characters
[chars] Any single character in chars.
[a-d] Any character between a and
d inclusive
{a,b,...} Any of the sub-patterns a, b
/**/ 0 or more subdirectories

12. Time to brew
Potion 1

13. Potion 1:
Enticements

14. Wildcard Bulk
Data Processing
Enticements
� Simple to set up
� Can transform across
file boundaries
– Needs memory &
time

15. Potion 1:
Pitfalls

16. Wildcard Bulk
Data Processing
Pitfalls
☹Recovery from data
errors difficult
☹Feature Type vs File
vs Format Issues
☹No granular log
☹No ability to
parallelize

17. Potion 2:
Batch Deploy

18. Batch Deploy Script Writer

19. Batch Deploy Script Writer

20. Batch Deploy Script Writer

21. Batch Deploy Script Writer

22. Batch Deploy Script Writer

23. Batch Deploy
Script Writer

24. Time to brew
Potion 2

25. Potion 2:
Enticements

26. Batch Deploy
Enticements
�Simple to set up
�Runs quickly
�Can script via
command line
�Run on demand

27. Potion 2:
Pitfalls

28. Batch Deploy
Pitfalls
☹Recovery from data
errors difficult
☹No granular log
☹Destination dataset
naming can be tricky

29. Potion 3:
Parent/Child
Workspaces

30. Parent/Child
Workspace
Ingredients
●Parent Workspace:
– PathReader
– WorkspaceRunner
●Child Workspace:
– FeatureWriter

31. Parent
Ingredients

32. Parent
Ingredients

33. Child
Ingredients

34. Time to brew
Potion 3

35. Potion 3:
Enticements

36. Parent/Child
Workspace
Enticements
�Separate
transformation from
workflow
�Generate audit logs
�All authored within
Workbench

37. Potion 3:
Pitfalls

38. Parent/Child
Workspace
Pitfalls
☹Not all writers can be
used concurrently
☹Slow to run each child
workspace separately
☹Recovery from data
errors not easy if
concurrent runs used

39. Potion 4:
Parent/Child Server
Workspaces

40. Parent/Child
Server
Workspace
Ingredients
●Parent Workspace:
PathReader
FMEServerJobSubmitter
FMEServerJobWaiter
●Child Workspace:
FeatureWriter

41. Parent
Ingredients

42. Parent
Ingredients

43. Parent Ingredients

44. Child Ingredients

45. Time to brew
Potion 4

46. Potion 4:
Enticements

47. Parent/Child
Server
Workspace
Enticements
�Separate
transformation from
workflow
�Generate audit logs
�All authored within
Workbench
�Make full use of
parallelism = FAST

48. Potion 4:
Pitfalls

49. Parent/Child
Server
Workspace
Pitfalls
☹Not all writers can
be used concurrently
☹Data needs to be
accessible to Server
Engines
– Consider using
Server Data
Resources
☹Craft your
reload/audit plan

50. Summary
● Many ways to handle bulk data moves
● Choose your potion wisely - each has pluses and
minuses
● FME Server is the most robust automation choice

51. Questions?
Batch processing tutorial:
fme.ly/b59