I like to give a technical presentation about how to synchronize and release ArcGIS Portal Items between different instances of ArcGIS Portal like migrating certain items between a Staging and a Production environment. Those Items can be service types, maps, apps, or groups. In doing so I am using FME as a central spot of interaction between those environments. The aim is to maintain and synchronize several ArcGIS Enterprise instances that belong to one bigger deployment including TEST and PROD environments without a lot of manual maintenance. The developed workspaces make us of ArcGIS functionality based on the Python and REST API for ArcGIS. In this project, I am using FME Desktop and FME Server (Automations, FME Server Apps, Dynamic Engines).

1. How to synchronize ArcGIS
Portal Items with FME

2. FME
User
Conference
20
22
Jan Middel
Software Engineer
- Joined con terra GmbH (Germany) in 2019
- FME Certified Professional (Desktop, Server)

3. 20
22
FME
User
Conference
The FME Plattform is a
strong addition to
ArcGIS Plattform.

4. 20
22
FME
User
Conference
Agenda
● Background Info & Main Objective
● Technical Solution Workflow
● Challenges & key takeaways

5. 20
22
FME
User
Conference
Background Information
● Project: provide publicly accessible Web
Application Maps via ArcGIS Portal
● DEV & PROD Environment
● Disaster Recovery Scenario

6. 20
22
FME
User
Conference
System Architecture
DEV
PROD
LIVE STAND BY / DISASTER RECOVERY
DEV LIVE
PROD LIVE
DEV
STANDBY
PROD
STANDBY
Source of graphic source:
https://www.esri.com/en-us/arcgis/products/index
∙ Live cluster and stand by
cluster
∙ Each cluster with DEV and
PROV environment
🡪 four ArcGIS Portals

7. 20
22
FME
User
Conference
Migration of ArcGIS Portal Items
Complex Architectures / Scenarios demand
migration mechanisms
● Release new item versions from DEV to
PROD within release cycles
● Manage to support Backup and Disaster
Recovery Scenarios by synchronizing items
between LIVE and STAND BY within
acceptable time ranges

8. 20
22
FME
User
Conference
Main Objective
∙ Migration of Web Mapping Applications and
all related items (Maps and Services)
∙ No manuell creation of ArcGIS Portal Items in
each environment
∙ Create the Web App once and migrate all to
other environments / Cluster
∙ Develop a mechanism that supports
automatic migration

9. 20
22
FME
User
Conference
Solution Workflow
∙ Create Services in ArcGIS Pro and export
them as Service Definition (*.sd) File
∙ Publishing of Services Definition File to ArcGIS
Portal
∙ In ArcGIS Portal:
∙ Create Web Map
∙ Create Web Mapping Application
∙ Migration of all components that are part of
the Web Mapping App from source to
(several) target ArcGIS Portal
∙ Adjust parameters depending on local environment
∙ Replace item Ids
🡪 FME Server is the central component
regarding publishing services and migration of
ArcGIS Portal Items!
.sd
Source of graphic source:
https://www.esri.com/en-us/arcgis/products/index
https://www.safe.com/

10. 20
22
FME
User
Conference
Technical Solution
∙ FME Server Apps and Automations to trigger Migration steps
∙ FME Server Apps: migration from DEV to PROD
∙ FME Server Autoamtion: synchronization between LIVE and STAND BY
∙ Interaction with ArcGIS Portal / ArcGIS Server via ArcGIS REST API and Python
API for ArcGIS
∙ Create token
∙ Get item meta data and configs
∙ Create and copy items
∙ Start Uploading and caching Jobs
∙ Share items
∙ Manage Item IDs in SQLite Database

11. 20
22
FME
User
Conference
Technical Solution
∙ Two big main Workspaces + smaller “helper” Workspaces
∙ Publish Services (SD-Files)
∙ Migrate Web Maps and Web Mapping Applications

12. 20
22
FME
User
Conference
Publishing of Service Definitions
Generate token for
target (and source)
portal
Get infos
from SD File
Publish Web Map
and Image Services
Publish Feature
Service and
Vector Tile Layer
Package
Update item ID
mapping table
Share published
Services with
Groups and
Organization
Create result
report in HTML
Cache service data

13. 20
22
FME
User
Conference
Publishing of Service Definitions
∙ Selection of used REST API Endpoints (<HOST>/<WEBADAPTOR>…)
∙ Generate Token
∙ POST /sharing/rest/generateToken
∙ Publish Services (Map / Image Service)
∙ POST admin/uploads/upload
∙ GET admin/uploads/upload/<ITEM_ID>/serviceconfiguration.json
∙ POST sharing/rest/services/System/PublishingTools/GPServer/Publish%20Service%20Definition/submitJob
∙ Publish Feature Service
∙ POST /sharing/rest/content/users/<USER>/addItem
∙ POST /sharing/rest/content/users/<USER>/publish
∙ Manage Services
∙ POST /admin/services/@Value(sd_serviceName).@Value(sd_type)/start
∙ POST /rest/services/System/CachingTools/GPServer/Manage%20Map%20Cache%20Tiles/submitJob
∙ GET /rest/services/System/CachingTools/GPServer/Manage%20Map%20Cache%20Tiles/jobs/<JOB_ID>
∙ GET /rest/services/System/PublishingTools/GPServer/Publish%20Service%20Definition/jobs/<JOB_ID>)
∙ Manage items and groups
∙ POST /sharing/rest/search
∙ POST /sharing/rest/content/users/<USER>/items/<ITEM_ID>/delete
∙ POST admin/services/<SERVICE>/delete
∙ POST /sharing/rest/content/users/<USER>/shareItems
∙ POST /sharing/rest/community/groups
∙ ….

14. 20
22
FME
User
Conference
Publishing of Service Definitions
∙ Selection of used Python API Endpoints
∙ portal = GIS(url, token=, referer, …) # connect to source portal via token….
∙ added_item = portal.content.add(item_properties, data) # add item to portal, e.g. SD / VTPK file
∙ published_added_item = added_item .publish(file_type, overwrite, item_id) # publish added item

15. 20
22
FME
User
Conference
Migration of Web Maps and Web Mapping Apps
Generate token for
target (and source)
portal
Share migrated
items with Groups
and Organization
Check for existence
of target item and
referenced Services
and Web Maps
Update item ID
mapping table
Invoke migration
of Web Map
Create new Item in
target Portal and
modify configuration

16. 20
22
FME
User
Conference
Migration of Web Maps and Web Mapping Apps
∙ Selection of used REST API Endpoints (<HOST>/<WEBADAPTOR>…)
∙ Manage items
∙ GET /sharing/rest/content/items/<ITEM_ID>
∙ GET /sharing/rest/content/items/<ITEM_ID>data
∙ POST /sharing/rest/content/users/<USER>/addItem
∙ GET /sharing/rest/content/users/<USER>/items/<ITEM_ID>
∙ POST /sharing/rest/content/users/<USER>)/items/<ITEM_ID>/update
∙ POST /sharing/rest/content/users/<USER>/shareItems
∙ POST /sharing/rest/community/groups/<GROUP_ID>
∙ ….

17. 20
22
FME
User
Conference
Key transformers
∙ HTTPCaller in combination with
JSONFlattener / JSONFragmenter
∙ PythonCaller
∙ SQLExecutor / SQLite Reader and Writer
∙ FeatureMerger

18. 20
22
FME
User
Conference
Synchronization Workflow between LIVE and
STAND BY cluster
∙ FME Server Automation triggers scheduled synchronization cycle
∙ Migration of Web Mapping Application and Web Maps that are shared with a
certain ArcGIS Portal group
∙ Get Services that are embedded within Web Maps
∙ Compare last updated timestamp off all selected Items in mapping table
∙ Publish and migrate all updated items (overwrite) / remove all removed items

19. 20
22
FME
User
Conference
Key takeaways
∙ FME complements ArcGIS Portal regarding migration and release of certain
Portal Items between separate Portals
∙ No full backup of ArcGIS Portal is needed to restore only certain items
∙ FME Server Apps and Automations are useful functionalities to trigger manual
or automated migration / release workflows

20. 20
22
FME
User
Conference
Challenges
∙ Design and creation of Workspaces independent of environment
∙ Interaction with ArcGIS REST API
∙ Figure out the right workflow order and API endpoints
∙ Documentation
∙ Manage item IDs in ArcGIS Portal (keep the same item id)
∙ User, Role and Ownership Management in ArcGIS Portal

21. 20
22
FME
User
Conference
Useful “by the way” developments
∙ Maintaining FME Server Synchronization by schedule and event based
∙ Backup and Restore
∙ Custom Transformer
∙ Error handling + Reporting

22. 20
22
FME
User
Conference
FME proves again to be a
valuable tool in
complementing the
ArcGIS Platform in order
to migrate and
synchronize Portal Items

23. 20
22
FME
User
Conference
Resources
ArcGIS Developer
∙ REST API:
https://developers.arcgis.com/rest/users-groups-and-items/working-with-users-groups-
and-items.htm
∙ Python: https://developers.arcgis.com/python/api-reference/arcgis.gis.toc.html

24. Thank You!
j.middel@conterra.de
fme@conterra.de