This document discusses four potions or methods for batch data processing large datasets: 1) Wildcards, 2) Batch Deploy, 3) Parent/Child Workspaces, and 4) Parent/Child Server Workspaces. Wildcards provide simple setup but have limitations. Batch Deploy scripts are also simple but have log and error handling issues. Parent/Child Workspaces separate transformations from workflows but can be slow. Parent/Child Server Workspaces leverage parallelism for speed but require data accessibility to server engines. In summary, each potion has tradeoffs; FME Server is recommended for robust automation.

1. Potent Potions for
Batch Data
Processing

2. 250,000 CAD
files & rasters
on mobile devices

4. Tip: Know
Your Potions
and Choose
Wisely

5. Today’s Potions
1. Wildcards
2. Batch Deploy
3. Parent/child Workspaces
4. Parent/child Server Workspaces

6. Potion 1:
One Wild<card>
Dataset

7. Multi-Dataset Picker

8. Multi-Dataset Picker

9. Multi-Dataset Picker

10. Multi-Dataset Picker
Shapefile
MapInfo
Most rasters
DWG
DGN
SQLite

11. Dataset
Wildcards
Extended glob syntax:
Symbol Matches
? Any single character
* Any sequence of zero or more
characters
[chars] Any single character in chars.
[a-d] Any character between a and
d inclusive
{a,b,...} Any of the sub-patterns a, b
/**/ 0 or more subdirectories

12. Time to brew
Potion 1

13. Potion 1:
Enticements

14. Wildcard Bulk Data
Processing
Enticements
ü Simple to set up
ü Can transform across file
boundaries
- Needs memory & time

15. Potion 1:
Pitfalls

16. Wildcard Bulk Data
Processing Pitfalls
x Recovery from data
errors difficult
x Feature Type vs
File vs Format
Issues
x No granular log
x No ability to
parallelize

17. Potion 2:
Batch Deploy

18. Batch Deploy Script Writer

19. Batch Deploy Script Writer

20. Batch Deploy Script Writer

21. Batch Deploy Script Writer

22. Batch Deploy Script Writer

23. Batch Deploy
Script Writer

24. Time to brew
Potion 2

25. Potion 2:
Enticements

26. Batch Deploy
Enticements
ü Simple to set up
ü Runs quickly
ü Can script via
command line
ü Run on demand

27. Potion 2:
Pitfalls

28. Batch Deploy
Pitfalls
x Recovery from data
errors difficult
x No granular log
x Destination dataset
naming can be tricky

29. Potion 3:
Parent/Child
Workspaces

30. Parent/Child
Workspace
Ingredients
• Parent Workspace:
– PathReader
– WorkspaceRunner
• Child Workspace:
– FeatureWriter

31. Parent
Ingredients

32. Parent
Ingredients

33. Child
Ingredients

34. Time to brew
Potion 3

35. Potion 3:
Enticements

36. Parent/Child
Workspace
Enticements
ü Separate
transformation from
workflow
ü Generate audit logs
ü All authored within
Workbench

37. Potion 3:
Pitfalls

38. Parent/Child
Workspace
Pitfalls
x Not all writers can be
used concurrently
x Slow to run each child
workspace separately
x Recovery from data
errors not easy if
concurrent runs used

39. Potion 4:
Parent/Child Server
Workspaces

40. Parent/Child
Server
Workspace
Ingredients
• Parent Workspace:
– PathReader
– FMEServerJobSubmitter
– FMEServerJobWaiter
• Child Workspace:
– FeatureWriter

41. Parent
Ingredients

42. Parent
Ingredients

43. Parent Ingredients

44. Child Ingredients

45. Time to brew
Potion 4

46. Potion 4:
Enticements

47. Parent/Child
Server
Workspace
Enticements
ü Separate
transformation from
workflow
ü Generate audit logs
ü All authored within
Workbench
ü Make full use of
parallelism = FAST

48. Potion 4:
Pitfalls

49. Parent/Child
Server
Workspace
Pitfalls
x Not all writers can be
used concurrently
x Data needs to be
accessible to Server
Engines
- Consider using Server
Data Resources
x Craft your reload/
audit plan

50. Summary
● Many ways to handle bulk data moves
● Choose your potion wisely - each has pluses and
minuses
● FME Server is the most robust automation choice

51. Questions?
Batch processing tutorial:
fme.ly/b59