Auckland Council, like many organisations around the world relies on ArcGIS Online to host their public-facing mapping applications. By utilizing this platform, they’re able to off-load non-functional requirements such as scalability and resilience to Esri’s SaaS infrastructure. This approach avoids the hassle of maintaining internal-facing infrastructure enabling Auckland Council to focus on delivering exceptional user experiences. However, working with hosted feature services in ArcGIS Online can be a challenge. Data sets can be quite large, and frequent updates can cause service outages, resulting in maps being unavailable to the public for unacceptable periods of time. To minimize downtime, Auckland Council created a manual process that utilized Hosted Feature Services and Feature Layer Views via the Agol web UI. They then collaborated with Locus to build a proof of concept that automates the process using FME. The results of this project were nothing short of impressive. By automating the process, the Council was able to reduce server downtime from minutes to mere seconds and establish a new standard for local Government organizations looking to enhance mapping applications while streamlining their processing time.

1. Optimizing Bulk Data Updates in
ArcGIS Online

2. Presenter
The
Peak
of
Data
Integration
20
23
Jan
Roggisch
GIS & System
Integration Consultant
Locus Limited

3. The
Peak
of
Data
Integration
20
23
Agenda
1. Who is the client?
2. What is their challenge?
3. How did Locus help?
4. Conclusion

4. The
Peak
of
Data
Integration
20
23
The client:
Auckland Council

5. The
Peak
of
Data
Integration
20
23
Auckland Council (Org.)
● Established in 2010
● Service area ~5,000 km2
● Population >1.6 million
● ~12,000 staff
● Annual budget $3 billion
● Largest council in Oceania
https://en.wikipedia.org/wiki/Auckland_Region#/map/0

6. The
Peak
of
Data
Integration
20
23
Auckland Council (GIS)
● ArcGIS-based public maps & apps
● Datasets are
● diverse
● complex
● large
● dynamic
● Need for scalability and resilience (e.g. 2023 Floods)

7. The
Peak
of
Data
Integration
20
23
Auckland Anniversary Weekend Floods
Newsroom
Richard Easther/Twitter
NZ Herald

8. The
Peak
of
Data
Integration
20
23
The challenge:
Provide highly-available apps

9. The
Peak
of
Data
Integration
20
23
First steps
Host data and apps in ArcGIS Online
1. No infrastructure maintenance
2. Scalability, Resilience, Security built-in
3. Upgrades taken care of
Challenges around data updates
4. Duration 🡺 service outage
5. Timing 🡺 after-hours work
6. Dependent on on-premise data processes

10. The
Peak
of
Data
Integration
20
23
Hosted view
● Requires identical FGDBs
(schema)
● Manually update FDGBs and
services
● Manually switch view source

11. The
Peak
of
Data
Integration
20
23

12. The
Peak
of
Data
Integration
20
23
Hosted view
● Requires identical FGDBs
(schema)
● Manually update FDGBs and
services
● Manually switch view source

13. The
Peak
of
Data
Integration
20
23
The solution:
Automation

14. The
Peak
of
Data
Integration
20
23
What to do?
● ArcGIS Admin REST API
● Esri documentation
● Excel-pattern
● Trial & Error

15. The
Peak
of
Data
Integration
20
23
The result – FGDB replacement
● 61 transformers
● 7 x HttpCaller
● JsonX-Former
● Dedicated error stream

16. The
Peak
of
Data
Integration
20
23
The result – data source swap
● 67 transformers
● 8 x HttpCaller
● JsonX-Former
● Dedicated error stream

17. The
Peak
of
Data
Integration
20
23
The result – master workspace

18. The
Peak
of
Data
Integration
20
23
Conclusion

19. The
Peak
of
Data
Integration
20
23
1. Data updates fully automated
2. Downtime reduced to seconds
3. Improved public service delivery

20. ThankYou!
jan.roggisch@locus.co.nz