Automating Annexation Related Data and Map Production
•Download as PPTX, PDF•
1 like•457 views
When the City annexes an area a multitude of maps and data need to be updated accordingly. We'll cover how we've used FME to automate the bulk of our annexation related map and data production.
Recommended
More Related Content
Viewers also liked
Viewers also liked (10)
Similar to Automating Annexation Related Data and Map Production
Similar to Automating Annexation Related Data and Map Production (20)
More from Safe Software
More from Safe Software (20)
Recently uploaded
Recently uploaded (20)
Automating Annexation Related Data and Map Production
- 1. Annexation Related Map and Data Automation Aubrey Drescher Ben Vanderford John Cook
- 2. About Us Aubrey Drescher Ben Vanderford John Cook Enterprise Geospatial Services
- 3. Automated our annexation related map and data production with FME. Outline 1. Business Need 2. Old vs. New Workflows 3. Maps Produced 4. Workbench Components 5. Shutdown Script 6. Examples 7. Future Enhancements 8. Closing
- 5. Annexation Driven Information Jurisdiction Web Map GIS and Maps page Open Data Portal Government page 1 1 2 3 4 2 3 4
- 6. Collaboration Wanted more staff depth with FME and more staff knowledge of annexation process. Team: Aubrey Drescher Ben Vanderford Stacy Meeks Bruce Bacia Mark Hawkins Marna McLain Logan Pugh John Cook
- 7. What we did/project overview ArcGIS Workflow Manager
- 8. More Efficient & More Depth Old process • ½ FTE (~1000 hours annually) • One person who owned the entire process • Process was difficult to follow New process • 800 hours on project • Project to spend 40-80 hours annually maintaining • Multiple folks on the team who can do this work
- 9. Old vs. New Workflows
- 10. Previous Model
- 12. Simplified Workflow Old Workflow Current Workflow
- 13. Maps Produced
- 19. Shutdown Scripts
- 20. Automatically Update PDF Maps
- 21. FME Options > Runtime Python Shutdown Script
- 22. INI File Python Shutdown Script
- 23. Python Shutdown Script: Socrata Open Data Portal
- 24. Examples
- 26. Full Purpose Before Reduced Fees Before Legend Full Purpose Annexation Area Agricultural Development Agreement Annexation Area Full Purpose Jurisdiction 30% Reduced Fee Zone 20% Reduced Fee Zone Full Purpose After Reduced Fees After Reduced Fee Zones North East Austin Annexations
- 28. Hazardous Materials Storage Zone FM 812 Landfill Annexation Full Purpose Before HMS Before Legend Annexation Area Onion Creek Watershed Boundary Full Purpose Jurisdiction Hazardous Materials Storage Zone Full Purpose After HMS After
- 30. Future Enhancements Inspector Boundaries Run workspace on a schedule Automate Notifications 1 2 3
- 31. Closing We’re pretty happy with it. Any questions?
- 32. Thank you! Austin GIS Portal: http://austintexas.gov/department/gis- and-maps Aubrey – Aubrey.Drescher@austintexas.gov Ben – Ben.Vanderford@austintexas.gov John – John.Cook@austintexas.gov
Editor's Notes
- Thank FME for giving us this opportunity to show off our work and putting on this event. It’s good to see so many folks are interested in and doing this work.
- I’m John Cook Joining me today are Aubrey Drescher and Ben Vanderford. We work the IT department at the City of Austin. Aubrey -On our team for about 3 years -Non-defined role - troubleshooting specialist -Lived all over, both coasts, Huntsville now here Ben -Been with the City for a long time, joined our team ~5 years ago -Non-defined role - business processes -Grew up in Saudi Arabia where they had a money jar in the house; now he’s slumming it with us here John -I’ve been on theteam since 2007 -Non-defined role – Cheerleader and I can type fast. -I grew up in Houston and moved all the way to Austin, wow!
- What we did on this project. John -I’m going kick things off by covering the business needs that we addressed and wrap things up by going over some future enhancements we plan to make. Ben -Going to talk about how the workflow/workload has been updated, the basic structure of the workbench, and touch on all the maps it produces Aubrey -Going to cover our Shutdown Script and go over some examples from recent annexations to illustrate changes the workbench makes to our data
- Two business needs: 1) Update annexation related data and maps 2) Grow bench strength with FME and annexation processing -Only had two folks in the department proficient with FME at the start Those were the goals, get the work done and learn how to do it with FME.
- Slide speaks to the 1st business need. When the City annexes an area a multitude of maps and data are impacted and need to be updated accordingly. It is our responsibility to see that those updates are made and shown on the maps and in applications. Here are some screenshots from AustinTexas.gov that illustrate some of the ways this information is disseminated. Jurisdiction Web Map – Needs to show most current info GIS and Maps page of AustinTexas.gov – portal to City’s GIS data, lots of it impacted by jurisdiction Open Data Portal – Jurisdiction data is posted on this site for public consumption Government page – All this data impacts the delivery of government services; services like, voting, development fees, utilities, emergency services, library access, etc.. Had been doing this, but wanted to automate this work to save time.
- This slide represents the second business need. FME is powerful software, we could see lots of potential for it. As you can see, lots of folks worked on this project. We could have had a smaller team work on this, but we used this project as a mechanism to get folks on the team exposed to FME and these business processes.
- In this project we combined an FME workspace and an Esri WMX Job Type to track of and automate most of this work. The new process begins with some manual data editing of two GIS layers. Then we schedule an FME workspace to run which edits 5 of these data layers, creates the maps, and publishes the data/maps to our database, FTP site, and public data portal on the scheduled day.
- With the completion of the project we have more efficiency and depth with this work. We figure we’ll save over 900 hours of staff time annually. The process is still complex and difficult to troubleshoot, but we have more depth with it so we’re in better shape.
- Old Model was great for it’s time but had many short comings. It didn’t make maps, run on a schedule, post data to FTP or the cities Open data Portal. Worse of all almost every time we opened it was broken. You’ll see in a moment that this model is a lot prettier than the replacement workspace. What is deceptive is that this older method had a lot more manual work involved.
- The workbench to fix all the old models shortcomings looks like this. Not nearly as nice to look at as our old model, but the work gets done. The workspace begins with the jurisdiction feature class that has just experienced some change, A single jurisdiction change pushing a row of bookmarked dominoes down the transformation and mapping production line.
- Here are the old and current workflows for all the annexation work. The old workflow went on and on no one could understand this. I’m not sure anyone was ever able to finish this. The new workflow is a lot simpler with the workbench taking the place of most of these steps. In workflow manager each of the shapes reference a step in the process. Some of the steps are procedural, make edits, The schedule jurisdiction change workbench step runs the FME workspace mentioned previously. The step executes a batch file that then runs the workspace completing all the tasks required. For those that don’t know you can embed a workspace in a batch file. Handy tip) There is a canned up email that sends all the vested parties notification that the process is complete.
- Time sensitive – must be posted by the next day Dynamic - Jurisdiction changes a couple of times a year Produces 21 separate maps with each change Impacts people – Lots of eyes are watching
- Our creation is composed of a collection of bookmarks each with a task to performed. There is a collection of 6 readers; only the jurisdiction one is dynamic. The results are about a dozen different features being written. All of these updated features get consumed in the mapping process, published to our default SDE data base. And some will also go on to be posted on the open data portal. The number one transformer used in the workspace is by far the tester. The process involves a lot of selecting out attribute then performing some geometry and attribute transformation. Other transformers used extensively in this workspace are the clipper, dissolver, attribute creator, aggregator and the AreaOnAreaOverlay.
- Use your bookmarks! - Bookmarks hold all the work together for each step in the process. Here we are making the layer which will be used to create the Reduced Fee Zone Maps. There are inputs from other locations in the workbench, testers testing and clippers clipping then a dissolver and an aggregator. The bookmarks helped along the way in testing. If a map or feature didn’t bring the desired results; we can quickly go to the bookmarks and inspect.
- The transformers in these bookmarks create the geometry for the outer boundary for the city of Austin. This outer boundary is in about every standard map product we make. You can see by all the lines disseminating from the attribute keeper that many other transformers are waiting down the line.
- One of the main things that is different and better about our FME solution is that it automatically updates those 21 maps that Ben showed you flying in on a previous slide. We make those maps available to the public on our “GIS / Map Downloads” webpage. We call them our “Standard Map Products” and they’re a collection of PDF files. We create those PDFs from ESRI map document (mxd) templates. It used to be that when we finished running our old model, somebody had to go in and open the mxd for every map that had a data layer that changed, and re-export the PDF file manually, one by one. Then, that person had to go and upload all those new PDFs to the website.
- Now, we have a python shutdown script embedded in to the workspace which runs as the last thing the workspace does. It reads in an .ini configuration file where we list the names of all the mxds that we want it to process. It uses the ArcPy mapping module to loop through that list of mxds and export them to pdf. It took some trickery for us to get ArcPy working inside FME. We had to make it use a custom python interpreter which points to the ArcGIS desktop installation folder on the local machine. But then, it works just perfect.
- Not only do we export the maps automatically, we also upload them to the website automatically. This website is actually an FTP site, and we store a pointer to our FTP login credentials in that same .ini file. We use the FTPLib module to connect to the FTP directory and upload the PDFs as binary files.
- In addition to our GIS FTP site, where we make maps available to the public, we also make some of our data available to the public on the Socrata Open Data Portal. One of the layers we have up there is the watershed regulation areas, which gets clipped to the outer edge of the Austin extra territorial jurisdiction. Our python shutdown script uploads the new watershed regulation areas and the new jurisdiction boundary to Socrata. FME actually has a built in Socrata writer, which works for point layers, but these are polygon layers, so we had to take a different approach. Our writer writes to a shapefile, which is the format accepted by Socrata. The python script reads this shapefile and posts it to Socrata using the Socrata REST API.
- Now I’m going to give you examples of two of our maps which changed. I’m going to zoom in on the changed places and show you how they look before and after our workbench ran.
- Austin has a “Smart Growth” initiative where we want to encourage development in certain parts of the city, basically the ones that are off of the aquifer (so we have cleaner water) and closer to downtown (so we have less urban sprawl). In order to accomplish this, we have established three zones which have reduced fees for things like permits and other development-related costs. The green area closest to downtown has the biggest reduction in fees, the yellow area has the second biggest, and the blue area has the smallest reduction. Over the past couple years we’ve had several annexations in the area of North East Austin where I’ve drawn the orange boxes on the maps. These annexations are going to cause the yellow zone to change shape. I’m going to zoom in on this area on the next slide.
- The orange boxes on these maps show areas which are getting annexed in to the Full Purpose zone. The purple boxes show an area which got annexed into the Agricultural Development Agreement Zone – which is not Full Purpose – its still Extra Territorial Jurisdiction. Now, if you are building in the Full Purpose, you get a 30% reduction in fees, but if you are building in the ETJ you only get a 20% reduction in fees. So what you are going to see on the left is the Full Purpose growing in to the orange boxes, but not in to the purple box [play animation]. And then on the right you will see the yellow 30% zone grow into the orange boxes, but not in to the purple box [play animation]. I thought this was significant to show you, because its an example of a place where we used testers in our workspace to tell the difference between different types of annexations, and process them differently.
- This map shows where the City of Austin requires permits for Underground Storage Tanks. Permits are required in the Full Purpose, which is the yellow area on the map, and in the part of the ETJ that is on top of watersheds that recharge the aquifer, which is the blue area of the map. Back in December 2014, we annexed a landfill off of FM road 812, which is located where I’ve drawn the orange box on the maps. This annexation is going to cause the blue zone to change shape. I’m going to zoom in on this area on the next slide.
- The orange outline shows the boundary of the landfill, which is getting annexed in to the Full Purpose. The light blue area is the Hazardous Materials Storage Zone, or HMS zone – where permits are required in the ETJ. This zone only exists in the ETJ and it also only exists on top of certain watersheds. On the left you will see the Full Purpose growing in to the orange boundary [play animation]. And then on the right you will see the Hazardous Materials Storage zone moving out of the orange area – because the HMS zone doesn’t exist in the full purpose [play animation]. Now, if you were watching really closely, you might have noticed another change. The blue area disappeared from inside the orange box, but it grew bigger outside the orange box. This is because the Onion Creek watershed boundary, which is the dark blue outline, also changed slightly around this same time. And if you remember, the HMS zone is the part of the ETJ that is also inside the special watersheds. I’ll play it again, this time watch the area just to the left of the orange annexation area [play animation]. I thought this was significant to show you, because its an example of a place where we merged changes that happened in two different layers together in to the final product. None of this is hard for FME. And now I’ll hand it off to John to wrap things up.
- Inspector areas are not currently part of the workspace and are edited manually. We want to add them. Workspace is started manually. Eventually we’d like to set up a process where the workspace starts automatically based on the effective date field in our annexation layer. Couldn’t set up on a schedule because of data alignment issues that we’ll correct some day some how. Send notification to stakeholders. Currently in WMX workflow; would like to have in the workflow. Not many enhancements because it’s already great. Big pat on the back.
- Austin GIS Portal -mapping applications -data downloads -hardcopy maps -link to City’s Open Data Portal