A client collects an annual commuter travel survey which collects extremely detailed information about their staff and regular visitors. The data was presented to our team as a 250-column Excel spreadsheet with confusingly coded questions and answers, and baffling geographic data. This session will explain how we turned it into a rich, usable database with the help of FME.

1. Breathing life
into a travel survey
Roland Martin

2. Our client operates a large campus site
in California. There is a limit to the
number of vehicles that can enter their
site at any given time.

3. Every year, they carry out a travel
survey, with 20 prizes, including a grand
prize of $2,000.

4. Around 90% of site users responded.

5. The survey
1. Please select the answer that applies to you.
• I work on the main campus
• I work at an off-campus site
• None of the above
If “none of the above is selected” then skip to end of survey.
2. How do you usually commute to campus?
• Drive alone in a passenger automobile
• Bicycle
• Carpool (two or more people)
• Campus Shuttle (not as a transfer from another mode)
• Motorcycle
• Public transit (e.g. bus)
• Walk

6. Raw data

7. Required output
• A database (PostgreSQL / PostGIS)
• Access via desktop GIS (ArcGIS / QGIS)
• Possibly some kind of dashboard…
• Insight

8. Cleaning up the non-geographic data
• A lot of questions are not fully expressed in column headers
• Several column headers are repeated
• Not all questions are answered
• Some responses are “Other”, with the response in a second column
• Some questions allow multiple responses, so responses are collected
into multiple columns
• A lot of questions are not of interest for this study

9. Step 1: extract the schema
ResponseID SchemaID
ExternalDataReference q1
Affiliation q2
Staff class/Class level q3
CAC q5
Home lat q6
Home lon q7
FPC q8
Please select the answer that applies to you. q9
Although you are not eligible for this survey, we thank you for / your time and interest. q10
Where on campus is your primary work site? Please click on one location. - X q11
Where on campus is your primary work site? Please click on one location. - Y q12
Which of the following is your off-campus work / site?(Note: Consider your commute to/from / this loc... q13
Which of the following is your off-campus work / site?(Note: Consider your commute to/from / this loc...-TEXT q14
From what city and zip code do you typically begin your / commute? / If you choose to, please also p...-City q16
From what city and zip code do you typically begin your / commute? / If you choose to, please also p...-Zip q17
From what city and zip code do you typically begin your / commute? / If you choose to, please also p...-Cross street 1 q18
From what city and zip code do you typically begin your / commute? / If you choose to, please also p...-Cross street 2 q19
How do you usually commute to campus/work? (i.e., / What is the primary mode you use during your ty... q20
How do you usually commute to campus/work? (i.e., / What is the primary mode you use during your ty...-TEXT q21
Which transit system do you use for the longest distance of your / commute? q22
Which transit system do you use for the longest distance of your / commute?-TEXT q23
How do you usually get to the bus stop or train station from / your home? q24
How do you usually get to the bus stop or train station from / your home?-TEXT q25

10. Step 1: extract the schema

11. Step 2: make JSON
{
"4":"A1",
"5":"No",
"6":"No",
"7":31.2345,
"8":-120.1234,
"10":"None of the above",
"11":1,
"Zip":90210,
"City":"Santa Claus",
"State":"CA",
"Staff_Class":"A1"
}

12. Step 3: populate database

13. Step 4: set up database with more columns
alter table survey_data
alter column raw_survey_data type jsonb using raw_survey_data::jsonb
, add column survey_date date
, add column response_id text
, add column affiliation text
, add column staff_class_level text
, add column work_location_description text
, add column geom_work geometry
, add column commute_start_city text
, add column commute_start_zip text
, add column commute_mode_primary text
, add column commute_mode_transit text
, add column commute_mode_transit_access_home text
, add column commute_mode_transit_access_work text
, add column typical_station_home text
, add column typical_station_work text
, add column commute_regularity text
, add column commute_mode_primary_regularity text
, add column typical_work_time_arrive text
, add column typical_work_time_depart text

14. Step 5: populate data columns
update survey_data
set
survey_date = '2016-01-05'
, response_id = raw_survey_data ->> '1'
, external_data_reference = (raw_survey_data ->> '2')::integer
, affiliation = raw_survey_data ->> '3'
, staff_class_level = raw_survey_data ->> '4'
, work_location_description = raw_survey_data ->> '10'
, geom_work = null
, live_on_campus = (raw_survey_data ->> '16')::boolean
, commute_start_city = raw_survey_data ->> '17'
, commute_start_zip = raw_survey_data ->> '18'
, commute_mode_primary = case when
upper(raw_survey_data ->> '21')
not like 'OTHER%'
then
raw_survey_data ->> '21'
else
concat(raw_survey_data ->> '21', ' ',
raw_survey_data ->> '22')
end

15. Cleaning up the non-geographic data
A lot of questions are not fully expressed in
column headers Schema loaded separately into database
Several column headers are repeated
Not all questions are answered
PostgreSQL’s JSON capabilities used to
reduce data volumes while still retaining
raw data
Some responses are “Other”, with the
response in a second column Questions automatically and manually
assessed to ensure data integritySome questions allow multiple responses,
so responses are collected into multiple
columns
A lot of questions are not of interest for this
study
Some questions are ignored for this study

16. The story so far…
Transport Scotland: http://www.transport.gov.scot/report/j9425-10.htm

17. Two questions were very creatively
posed: where do you work on campus,
and where do you park?

18. Where do you work on campus?
Linford Nursing: http://www.linfield.edu/portland/about-portland/location/campus-map.html
580 194
502 270
41,616,668,701,171,800 3,360,666,809,082,030
504,5 3,298,699,951,171,870
563 250
4,951,040,344,238,280 3,744,800,109,863,280
486,5 3,238,699,951,171,870
507 3,296,800,231,933,590
487,5 3,748,699,951,171,870
463,5 2,688,699,951,171,870
414,5 2,808,699,951,171,870
584 196
46,261,907,958,984,300 2,562,952,575,683,590

19. Step 1: create links table for main map
Linford Nursing: http://www.linfield.edu/portland/about-portland/location/campus-map.html, map data © 2016 Google

20. Step 2: create links table for inset map
Linford Nursing: http://www.linfield.edu/portland/about-portland/location/campus-map.html
225.22 -86.66 -14960212.79 4051860.95
153.83 -205.26 -14960403.95 4051498.46
52.06 -91.52 -14960759.31 4051780.99
48.24 -95.43 -14960771.54 4051771.35
58.99 -97.74 -14960735.30 4051764.56
53.99 -101.68 -14960749.27 4051750.63
91.17 -61.12 -14960584.69 4051890.29
151.37 -213.02 -14960403.08 4051498.46
153.84 -205.21 -14960400.46 4051508.46
167.14 -152.50 -14960349.38 4051670.98
166.24 -158.44 -14960368.15 4051650.62
170.70 -146.17 -14960338.90 4051692.41
181.56 -158.92 -14960314.89 4051669.55
179.37 -161.54 -14960324.50 4051649.55
211.50 -94.75 -14960229.33 4051838.50
256.58 -53.58 -14960082.20 4051964.59
250.41 -46.34 -14960093.99 4051972.45
262.27 -48.50 -14960078.71 4051970.30
254.56 -42.21 -14960088.32 4051977.80
369.27 -29.80 -14959562.69 4052246.41
367.87 -17.80 -14959562.69 4052256.77
362.14 -24.05 -14959567.93 4052250.70
373.59 -22.93 -14959557.45 4052252.48
36.34 -247.21 -14960790.33 4051387.24
51.50 -209.42 -14960743.18 4051481.06

21. Step 3: create a mask
Linford Nursing: http://www.linfield.edu/portland/about-portland/location/campus-map.html
Main map
Inset map

22. Step 4: fun with FME

23. Step 5: fun with QGIS
Linford Nursing: http://www.linfield.edu/portland/about-portland/location/campus-map.html

24. Interactive
Dashboard

25. Thank you!
roland.martin@arup.com