Metadata can sometimes be an afterthought in the data publishing process and become a cumbersome task for technology teams when deferred until the end of a project. It is however the most important step for data publishers who aspire to high data discovery and appropriate usage of the data by all consumers. During a mass data migration event, an opportunity presented itself to the city of Mesa to make it a trivial task to review and improve their metadata. Join us in Phoenix to learn more about how the city of Mesa used FME + an excel workbook + Socrata metadata API endpoint to update metadata in bulk and provide validation reports on the completeness and accuracy of their metadata across their data portal.

1. Getting Meta at Mesa
Alicia Brown
April 9, 2018

2. WHAT WORKS CITY
In 2015 developed first City Data
Governance Program with Sunlight
Foundation & Johns Hopkins
MESA, AZ
Population: 470,456
3rd largest city in AZ
MESA + SOCRATA
In 2018 partnered to review Data
Governance Policies and
Processes and relaunch data cloud
with financial transparency tools
MESA STAT
Data and Evidence based
performance management for New
Next Mesa

3. Data Migration Project
The tasks
● 100+ datasets to migrate
● 2½ weeks to copy data and vizes
● 2 teams in 2 cities (Mesa, Seattle)
● Review and Update Metadata

4. All the metadata!
Dataset Metadata
+
Fields
FME HTTPCALLER
HTTP Caller
+
AttributeManager
Metadata APIs
Robust API for
Dataset + Fields
metadata updates
data.mesaaz.gov
Public Data Cloud
+
Transparency Apps
Metadata SHOULD BE EASY
And it can be!

5. The Work
- Captured existing metadata
- Created Excel Spreadsheet on
SharePoint for easy access
- Data Governance Team reviewed and
updated names and descriptions of
datasets and fields
- New Metadata API endpoint recently
launched
- Created FME Workspace that reads
metadata properties from Excel and
then writes them to datasets via API
- Ran bulk updates daily
Team Mesa Team Socrata

6. Feature Reader to read metadata.xlsx
The Workspace

7. Attribute Manager to build Payload
And Attribute Trimmer & Replacers to tidy the metadata
The Workspace

8. HTTPCaller to Patch updates
Rather than make Put call for Full Replace of entire Metadata
The Workspace

9. Metadata Example
https://data.mesaaz.gov/api/views/metadata/v1/ipea-qa5w

10. Why do we care about Metadata?
● Discoverability of data by City Staff & Citizens
● Ensure Proper Usage: Source, Units, Updates

11. ● Standardization - Ensure that the same
metadata is being collected in the same
format
● Alignment with industry standards - Industry
practices and standards helps with data
comparability
● Improve data categorization - Helps users
locate the right datasets easily and quickly
● Improve data quality - Guidelines and
required metadata fosters a culture of high-
quality and complete metadata
● Improve Analytics - Providing reference
metadata such as methodology and
sampling, empowers user to know exactly
how the data was obtained, so that analysts
can use common methodologies
https://socrata.com/blueprint
Why have
Metadata
Standards?

12. Why FME?
● It is easy!
● It is SAFE!
● Enabled frequent updates
● Low technical barrier for future
updates by Data Stewards
● Better quality metadata with
because it could be cleaned
(trailing spaces, case, encoded
characters)
● It’s a common tool used by our
customers that requires zero code

13. Voila!
The Data Catalog
● Names
● Descriptions
● Category
● Tags

14. RESOURCES
Learn more
● Mesa Data Cloud: https://data.mesaaz.gov/
● Socrata Metadata: https://socratametadataapi.docs.apiary.io/
● Mesa What Works City: https://whatworkscities.bloomberg.org/works-city-
feature-mesa-az/
● FME HTTPCaller: https://www.safe.com/transformers/http-caller/

15. THANK YOU!
Alicia Brown
alicia.brown@socrata.com