Joey Avery, Laurens SC Matt Mole , URS11 February 2013SDE & Database Maintenance /Business System & Integration & QualityA...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County Overview2• 354 Full-Time Employees• P...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County Overview3• Hardware & Software• GIS W...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data4• Arc...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data5• Web...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data6
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data7
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data8...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data9
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data1...
Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data11
Database Maintenance / Business System Integration & Quality AssuranceDatabase Maintenance: Why is it Important?12 GIS is...
Database Maintenance / Business System Integration & Quality AssuranceWhat is the most important part of your vehicle? En...
Database Maintenance / Business System Integration & Quality AssuranceDatabase Maintenance: What causes the need for it?14...
Database Maintenance / Business System Integration & Quality AssuranceCommon Maintenance Tasks & Reoccurring Operations15•...
Database Maintenance / Business System Integration & Quality AssuranceCommon Maintenance Tasks & Reoccurring Operations16•...
Database Maintenance / Business System Integration & Quality AssuranceCommon Maintenance Tasks & Reoccurring Operations17•...
Database Maintenance / Business System Integration & Quality AssuranceGIS CAMA Updater: Business System Integration & Qual...
Database Maintenance / Business System Integration & Quality AssuranceGIS CAMA Updater: Business System Integration & Qual...
Database Maintenance / Business System Integration & Quality AssuranceBlood, Sweat, & Tears: Manual Maintenance Tasks & re...
Database Maintenance / Business System Integration & Quality AssuranceBlood, Sweat, & Tears: Manual Maintenance Tasks & re...
Database Maintenance / Business System Integration & Quality AssuranceNo Sweat: Automated Maintenance Tasks & reoccurring ...
Database Maintenance / Business System Integration & Quality Assurance23
Database Maintenance / Business System Integration & Quality Assurance24
Database Maintenance / Business System Integration & Quality AssuranceLessons Learned Creating Automation Maintenance App2...
Database Maintenance / Business System Integration & Quality Assurance26Questions?DBMaint.
Upcoming SlideShare
Loading in...5
×

Scarc joey avery_mattmole_2013

102

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
102
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Hardware & Software GIS Web, DBMS, and Security ServerArcGIS for Server Enterprise AdvancedArcSDEArcGIS Server Web ServicesUsers / Consumers of DataMultiple Editors & Viewers throughout CountyEmergency Services / Sheriff / FireAssessor /AppraisersAuditorTreasurerDelinquent TaxBoard of ElectionsEconomic DevelopmentPublic Works
  • ArcGIS Desktop Direct connection to Main GIS DatabaseDirect connection to One & Two-Way ReplicasArcGIS Engine (Network & GDB Update Extensions)Disconnected Mobile Applications Replication & ArcServerGeodata ServicesOffline MapsDisconnected In-Car Routing & NavigationDisconnected Data Collection <note here about can connect to update / sync>
  • Web ApplicationsInternal County GIS Web AppAerials, Basemaps, & Operational DataSearches: Address, TMS, OwnerFull Layer Control w/ LegendQS1 CAMA IntegrationCurrent & Active Tax Year Search & Identification Print Property RepotsIntegrated URL Links: Property Tax & Assessor WebsitesAgricultural Tax Assessment Voter RegistrationAutomobile Registration
  • Web ApplicationsInternal County GIS Web AppAerials, Basemaps, & Operational DataSearches: Address, TMS, OwnerFull Layer Control w/ LegendQS1 CAMA IntegrationCurrent & Active Tax Year Search & Identification Print Property RepotsIntegrated URL Links: Property Tax & Assessor WebsitesAgricultural Tax Assessment Voter RegistrationAutomobile Registration
  • ArcGIS Desktop Direct connection to Main GIS DatabaseDirect connection to One & Two-Way ReplicasArcGIS Engine (Network & GDB Update Extensions)Disconnected Mobile Applications Replication & ArcServerGeodata ServicesOffline MapsDisconnected In-Car Routing & NavigationDisconnected Data Collection <note here about can connect to update / sync>
  • Web ApplicationsPublic County GIS Web AppAerials, Basemaps, & Operational DataSearches: Address, TMS, OwnerModified Themed Layer ControlQS1 CAMA IntegrationCurrent & Active Tax Year Search & Identification Print Property RepotsIntegrated URL Links: Property Tax & Assessor WebsitesVoter Registration
  • Web ApplicationsPublic County GIS Web AppAerials, Basemaps, & Operational DataSearches: Address, TMS, OwnerModified Themed Layer ControlQS1 CAMA IntegrationCurrent & Active Tax Year Search & Identification Print Property RepotsIntegrated URL Links: Property Tax & Assessor WebsitesVoter Registration
  • Web ApplicationsCounty Sheriff Crime Mapping Web AppLawTrak IntegrationCrime Type – layer control and legendShow Crimes by Points or ClustersCrime Time Period FilterDaily or Custom date-range selectionFilter within currently selected time period / show animated timeline progressCrime by Area of Interest Countywide, Zipcode, City, School or Address (with distance buffer option)Export PDF MapOnline Crime Stoppers Tip Submit
  • Web ApplicationsCounty Sheriff Crime Mapping Web AppLawTrak IntegrationCrime Type – layer control and legendShow Crimes by Points or ClustersCrime Time Period FilterDaily or Custom date-range selectionFilter within currently selected time period / show animated timeline progressCrime by Area of Interest Countywide, Zipcode, City, School or Address (with distance buffer option)Export PDF MapOnline Crime Stoppers Tip Submit
  • GIS is becoming the nucleus & integration point for business systemsWe DEPEND on our GIS databases to be reliable and provide timely access to data ArcSDE Enterprise DatabasesStored in a relational database management system (RDBMS).Since many users view and edit data from these enterprise databases, regular maintenance ensures data is available to those users without any depreciation in performanceBackup Plan – protect against data lossCreate and test your plan for regular backups and recoveryDepend and Expect are probably interchangeable here.Note:For our purposes during this presentation we will be referring to Microsoft SQL Server as the RDBMS. After backup plan: Most of you have probably seen diagrams like this that diagram Enterprise GIS Architecture. One unattractive component is usually left out is “Maintaining” the Enterprise GIS.
  • What is the most important part of your vehicle?Engine I’d like to compare the similarities between a car engine and a Enterprise GIS Database.Engine  GIS DatabaseIn much the same way a engine powers a car -> GIS data / databases power your enterprise GISLike a car engine, regular maintenance and quality control keep the engine running smoothly Ignore the Maintenance and it may leave you stranded!
  • When new data is added to the database or existing data is edited…Database statistics need to be updated and indexes rebuilt.Ensures the RDBMS can efficiently query dataVersioned Multi-user Geodatabases:As a gdb is edited over time, delta tables and the number of states increase.The larger the tables and the more states, the more data that must be processed every time you display or query a version.Regular compress of the gdb is needed.Reduces the state tree and the rows in the delta tables
  • Analyze DBMS Stats and Rebuild IndexesSQL Server, ArcSDE command, or Geoprocessing Web Services – Start / Stop, Update CacheStart / Stop – AGSSOM ArcScript (10.0 and Prior), ArcGIS Server Admin Toolkit (10.1)Update Cache – GeoprocessingCompress– how often depends on amount of editsArcSDE Command, or GeoprocessingStart / Stop – AGSSOM ArcScript (10.0 and Prior), ArcGIS Server Admin Toolkit (10.1)Update Cache – GeoprocessingVersioning – reconcile / post changes, add or remove versions Geoprocessing
  • Replication – synchronize tasks for Connected or Disconnected ReplicasGeoprocessing Kill SDE Connections – remove locks and orphaned connectionsArcSDE Command, Geoprocessing (10.1)Start / Stop – AGSSOM ArcScript (10.0 and Prior), ArcGIS Server Admin Toolkit (10.1)Update Cache – GeoprocessingRebuild Locators & Network DatasetsGeoprocessingExtract / Zip/ FTP – export datasets or replica data changes to FTP site for disconnected data sharing Geoprocessing / Scripting
  • Data Calculations / AnalysisGeoprocessing / ScriptingData QA/QC Geoprocessing, Scripting, Custom Apps, etc…Data updates from integrated systemsGeoprocessing, Scripting, Custom Apps, etc…Reboot Servers
  • Performs QA/QC Calculations on GIS ParcelsGIS Acres Deed AcresDiscrepancy - % divergence between GISAC / DeedAc( Abs ( [DeedAc] – [GISAC] ) / [GISAC] ) * 100Determines if the discrepancy is w/in acceptable parametersAcceptability (Accept, Reject, Undetermined) – shows acceptability based on % divergence
  • Updates GIS with information from QS1 CAMA SystemParse real & mobile home propertyCreates QA/QC report tablesOrphan Cama – real property cama records w/ no matching Parcels in GISOrphan MH – mobile home cama records w/ no matching Parcels in GISOrphan Parcels – GIS Parcels w/ no matching records in camaOrphan SubParcels – GIS horizontal property w/no matching records in camaTMS Sum – summary list that shows duplicate TMS #’s and frequency
  • Scarc joey avery_mattmole_2013

    1. 1. Joey Avery, Laurens SC Matt Mole , URS11 February 2013SDE & Database Maintenance /Business System & Integration & QualityAssurance: Automation Not Sexy But NecessaryDBMaint.
    2. 2. Database Maintenance / Business System Integration & Quality AssuranceLaurens County Overview2• 354 Full-Time Employees• Population approximately 70,000• Area approximately 800 sq. miles• Annual budget approximately 26 mil
    3. 3. Database Maintenance / Business System Integration & Quality AssuranceLaurens County Overview3• Hardware & Software• GIS Web, DBMS, and Security Server• ArcGIS for Server Enterprise Advanced• ArcSDE• ArcGIS Server Web Services• Users / Consumers of Data• Multiple Editors & Viewers throughout County• Emergency Services / Sheriff / Fire• Assessor /Appraisers• Auditor• Treasurer• Delinquent Tax• Board of Elections• Economic Development• Public Works
    4. 4. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data4• ArcGIS Desktop• Direct connection to Main GIS Database• Direct connection to One & Two-WayReplicas• ArcGIS Engine (Network & GDB UpdateExtensions)• Disconnected Mobile Applications• Replication & ArcServer Geodata Services• Offline Maps• Disconnected In-Car Routing & Navigation• Disconnected Data Collection
    5. 5. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data5• Web Applications• Internal County GIS Web App• Aerials, Base maps, & Operational Data• Searches: Address, TMS, Owner• Full Layer Control w/ Legend• QS1 CAMA Integration• Current & Active Tax Year Search &Identification• Print Property Reports• Integrated URL Links: Property Tax &Assessor Websites• Agricultural Tax Assessment• Voter Registration• Automobile Registration
    6. 6. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data6
    7. 7. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How Employees Consume GIS Data7
    8. 8. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data8• Web Applications• Public County GIS Web App• Aerials, Basemaps, & Operational Data• Searches: Address, TMS, Owner• Modified Themed Layer Control• QS1 CAMA Integration• Current & Active Tax Year Search &Identification• Print Property Reports• Integrated URL Links: Property Tax& Assessor Websites• Voter Registration
    9. 9. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data9
    10. 10. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data10• Web Applications• County Sheriff Crime Mapping Web App• LawTrak Integration• Crime Type – layer control and legend• Show Crimes by Points or Clusters• Crime Time Period Filter• Daily or Custom date-range selection• Filter within currently selected timeperiod / show animated timeline progress• Crime by Area of Interest• Countywide, Zipcode, City, School or Address(with distance buffer option)• Export PDF Map• Online Crime Stoppers Tip Submit
    11. 11. Database Maintenance / Business System Integration & Quality AssuranceLaurens County: How General Public Consume GIS Data11
    12. 12. Database Maintenance / Business System Integration & Quality AssuranceDatabase Maintenance: Why is it Important?12 GIS is becoming the nucleus & integration pointfor business systems We DEPEND on our GIS databases to bereliable and provide timely access to data• ArcSDE Enterprise Databases• Stored in a relational database managementsystem (RDBMS).• Since many users view and edit data from theseenterprise databases, regular maintenanceensures data is available to those users withoutany depreciation in performance• Backup Plan – protect against data loss• Create and test your plan for regular backups andrecoveryEnterprise GISIntegrateUseShareMaintain
    13. 13. Database Maintenance / Business System Integration & Quality AssuranceWhat is the most important part of your vehicle? Engine Engine  GIS Database• In much the same way a engine powers a car -> GIS data /databases power your enterprise GIS• Regular maintenance and quality control keep the enginerunning smoothly• Ignore the Maintenance and it may leave you stranded!Database Maintenance: Why is it Important?13
    14. 14. Database Maintenance / Business System Integration & Quality AssuranceDatabase Maintenance: What causes the need for it?14When new data is added to the database orexisting data is edited…• Database statistics need to be updated and indexes rebuilt.• Ensures the RDBMS can efficiently query data• Versioned Multi-user Geodatabases:• As a gdb is edited over time, delta tables and the number of states increase.• The larger the tables and the more states, the more data that must be processedevery time you display or query a version.• Regular compress of the gdb is needed.• Reduces the state tree and the rows in the delta tables
    15. 15. Database Maintenance / Business System Integration & Quality AssuranceCommon Maintenance Tasks & Reoccurring Operations15• Analyze DBMS Stats and Rebuild Indexes• SQL Server, ArcSDE command, or Geoprocessing• Web Services – Start / Stop, Update Cache• Start / Stop – AGSSOM ArcScript (10.0 and Prior), ArcGIS Server Admin Toolkit (10.1)• Update Cache – Geoprocessing• Compress– how often depends on amount of edits• ArcSDE Command, or Geoprocessing• Versioning – reconcile / post changes, add or remove versions• Geoprocessing
    16. 16. Database Maintenance / Business System Integration & Quality AssuranceCommon Maintenance Tasks & Reoccurring Operations16• Replication – synchronize tasks for Connected or Disconnected Replicas• Geoprocessing• Kill SDE Connections – remove locks and orphaned connections• ArcSDE Command, Geoprocessing (10.1)• Rebuild Locators & Network Datasets• Geoprocessing• Extract / Zip/ FTP – export datasets or replica data changes to FTP site fordisconnected data sharing• Geoprocessing / Scripting
    17. 17. Database Maintenance / Business System Integration & Quality AssuranceCommon Maintenance Tasks & Reoccurring Operations17• Data Calculations / Analysis• Geoprocessing / Scripting• Data QA/QC• Geoprocessing, Scripting, Custom Apps, etc…• Data updates from integrated systems• Geoprocessing, Scripting, Custom Apps, etc…• Reboot Servers
    18. 18. Database Maintenance / Business System Integration & Quality AssuranceGIS CAMA Updater: Business System Integration & Quality Assurance18Performs QA/QC Calculations on GIS Parcels• GIS Acres• Deed Acres• Discrepancy - % divergence between GISAC / DeedAc• ( Abs ( [DeedAc] – [GISAC] ) / [GISAC] ) * 100• Determines if the discrepancy is w/in acceptable parameters• Acceptability (Accept, Reject, Undetermined) – shows acceptabilitybased on % divergenceParcel AcreageAcceptable %Divergence1.01 – 5.0 10%5.01 - 20.0 8%20.01 - 50.0 6%50.01 & above 4%
    19. 19. Database Maintenance / Business System Integration & Quality AssuranceGIS CAMA Updater: Business System Integration & Quality Assurance19Updates GIS with informationfrom QS1 CAMA SystemParse real & mobile home propertyCreates QA/QC report tables• Orphan Cama – real property cama records w/ no matching Parcels in GIS• Orphan MH – mobile home cama records w/ no matching Parcels in GIS• Orphan Parcels – GIS Parcels w/ no matching records in cama• Orphan SubParcels – GIS horizontal property w/no matching records in cama• TMS Sum – summary list that shows duplicate TMS #’s and frequency
    20. 20. Database Maintenance / Business System Integration & Quality AssuranceBlood, Sweat, & Tears: Manual Maintenance Tasks & reoccurring operations20Issues Doing it all by Hand…………• Full Compression headaches• Lock it Up! - Fighting database locks• Replica Woes – multiple synchronizations and multiple loops inanalyze statistics, indexes, and compression to obtain full databasecompression• LOCKED AGAIN ! – still fighting database locks• Watch it run – a few processes took a while to run (.5hr- 2hrs)• Working late - Most maintenance needs to beperformed during non- business hours
    21. 21. Database Maintenance / Business System Integration & Quality AssuranceBlood, Sweat, & Tears: Manual Maintenance Tasks & reoccurring operations21Issues Doing it all by Hand…………• Long way to the Top (if You want to Rock and Roll)70 manual steps = plenty of room for human error• Are We There Yet – time consuming (around 7 hours tocomplete)• Maybe next time – so time consuming it would usuallyonly get done about once or twice a month
    22. 22. Database Maintenance / Business System Integration & Quality AssuranceNo Sweat: Automated Maintenance Tasks & reoccurring operations22Automated Advantage…………• Set N’ Forget It – run automatically w/ WindowsScheduler during non-business hours.• Now maintenance runs weekly instead of monthly.• Look Ma No Hands – no user interaction required• Anything Goes – runs more than just maintenancetasks• Wake Me When We Get There – logs step progress,time, completion success or failure and sends in email• Time and Money Saved
    23. 23. Database Maintenance / Business System Integration & Quality Assurance23
    24. 24. Database Maintenance / Business System Integration & Quality Assurance24
    25. 25. Database Maintenance / Business System Integration & Quality AssuranceLessons Learned Creating Automation Maintenance App25• Efficient Design• Design the application controls for executing modules, logging, and email first. Then,build supporting modules shells.• Document each automation task and reoccurring operation that you do by hand.• Test Automation thoroughly• Test in sections / modules.• Plan for hiccups when putting it all together. It takes a few rounds of testing to get itperfect.• Continue on Error?• List processes that can be allowed to fail and still continue the next automation step.• You don’t want to hold of the rest of the automation for a meniscal task like rebuilding a locator.• List processes that have to be completed. You want automation to stop and send a failurenotice.
    26. 26. Database Maintenance / Business System Integration & Quality Assurance26Questions?DBMaint.

    ×