In this webinar, we put all of the pieces together - showing, using the example of the Real Estate market in Los Angeles, how the CARTO platform powers a data-to-decision workflow, showcasing every step along the way. Watch it now at: https://go.carto.com/sum-parts-complete-carto-journey-webinar-recorded

1. FOLLOW @CARTO ON TWITTER

2. The Sum of Our Parts
Solutions Engineer & Customer Success Senior Data Scientist

3. The Sum of Our Parts
The Complete Journey
As an organization, we have defined 5 steps that, together, create a
holistic Location Intelligence approach.
Our goal is to empower organizations as they traverse each of these 5
steps.

4. The Sum of Our Parts
The Complete Journey
1. Data
2. Enrichment
3. Analysis
4. Solutions
5. Integration

5. The Journey - Data
PostGIS by
CARTO
Import API
SQL API Python SDK
● Spatial database with
multiple ways to connect
and manipulate your data
● Import and connect to a
wide range of data
sources
● Connect with a variety of
services to bring your data
to CARTO
Data
Management
Auth API

6. The Journey - Enrichment
Augment any data
with demographic
data from around
the globe with easeData
Observatory
Data Services
API
Routing &
Traffic
Geocoding
ETL
Develop robust ETL
processes and
update
mechanisms so
your data is always
enriched
Mastercard Human
Mobility
POI

7. The Journey - Analysis
Bring CARTO maps and data into
your data science workflows and
the Python data science
ecosystem to work with Pandas,
PySal,PyMC3, scikit-learn, etc.
CARTOFrames
Machine learning embedded in
CARTO as simple SQL calls for
clustering, outliers analysis, time
series predictions, and geospatial
weighted regression.
Crankshaft
Use the power of PostGIS and our APIs to
productionalize analysis workflows in your
CARTO platform.
PostGIS by
CARTO
SQL API Python
SDK

8. The Journey - Solutions
Builder Dashboard
CARTO.js Airship CARTO VL
Analysis tools for creating
lightweight spatial data science
dashboards and sharing insights
with your team
Develop and build custom spatial
data science applications with a full
suite of frontend libraries

9. The Journey - Integration
Using CARTO’s APIs and
SDKs, connect your
analysis into the places
that matter most for
you and your team

10. Let’s apply the journey to
a real world business
question!

11. How can we analyze and
understand real estate sales
in Los Angeles?

12. Pains
1. “Disconnected experiences to consume data - it is broken into
separate tools, teams DBs, excels.”
2. “Limited developer time in our team.”
3. “Current data science workflow doesn’t have a geo focus. and Spatial
modeling is cumbersome because I have to export results to XYZ
tool in order to visualize and test my model effectively.”
4. “Having trouble handling and visualizing big datasets.“

13. Outline the Process
1. Integrate spatial data of past home sales and property locations
in Los Angeles county
2. Enrich the data with a spatial context using a variety of relevant
resources (demographics, mastercard transactions, OSM)
3. Clean and analyze the data, and create a predictive model for
homes that have not sold
4. Present the results in a Location Intelligence solution for users
5. Integrate and deploy the model into current workflows for day
to day use

14. 1. Data

15. Integrate LA Housing Data
The Los Angeles County Assessor's office provides two different datasets
which we can use for this analysis:
● All Property Parcels in Los Angeles County for Record Year 2018
● All Property Sales from 2017 to Present

16. 2018 Parcel Data

17. 2018 Parcel Data
Import this data into CARTO
using the Sync via Import API
or the COPY command via the
SQL API for efficient streaming
of long CSV data into CARTO.

18. Past Sales Data

19. Past Sales Data
Import this data into CARTO
using the built in ArcGIS
Connector, and Sync the
results to keep them up to data
as they are added.

20. CREATE TABLE la_join AS
SELECT s.*,
p.zipcode as zipcode_p,
p.taxratearea_city,
p.ain as ain_p,
p.rollyear,
p.taxratearea,
p.assessorid,
p.propertylocation,
p.propertytype,
p.propertyusecode,
p.generalusetype,
p.specificusetype,
p.specificusedetail1,
p.specificusedetail2,
p.totbuildingdatalines,
p.yearbuilt as yearbuilt_p,
p.effectiveyearbuilt,
p.sqftmain,
p.bedrooms as bedrooms_p,
p.bathrooms as bathrooms_p,
p.units,
p.recordingdate,
p.landvalue,
p.landbaseyear,
p.improvementvalue,
p.impbaseyear,
p.the_geom as centroid
FROM sales_parcels s
LEFT JOIN assessor_parcels_data_2018 p ON s.ain::numeric = p.ain
Clean and join the data on unique
identifier using a SQL statement in Builder
or via SQL API.

21. 2. Enrichment

22. Integrate LA Housing Data
Next we want to add spatial context to our housing data to
understand more about the areas around:
● Demographics
● Mastercard (Scores and Merchants) (Nearest 5 Areas)
● Nearby Grocery Stores and Restaurants
● Proximity to Roads

23. Demographics
Add total population and median income from the US Census via the
Data Observatory in CARTOframes.

24. query_obs = '''
CREATE TABLE obs_la_half_mile AS
SELECT *,
OBS_GetMeasure(the_geom, 'us.census.acs.B01003001','predenominated','us.census.tiger.census_tract', '2011 -
2015') as total_pop_2011_2015,
OBS_GetMeasure(the_geom, 'us.census.acs.B19013001','predenominated','us.census.tiger.census_tract', '2011 -
2015') as median_income_2011_2015
FROM la_half_mile
'''
# Apply JOB query
batch_job_obs= batchSQLClient.create(query_obs)
# Check status of the Batch Job
status_obs = check_status(batch_job_obs)
# cartodbfy table when table is created
if status_obs == 'done':
cc.query("SELECT cdb_cartodbfytable('obs_la_half_mile')")

25. Mastercard
Find the merchants and sales/growth scores in the five nearest block
groups to the home via Mastercard Retail Location Insights data
and SQL API

26. (
SELECT AVG(sales_metro_score)
FROM (
SELECT sales_metro_score
FROM mc_blocks
ORDER BY la_eval_clean.the_geom <-> mc_blocks.the_geom
LIMIT 5
) a
) as sale_metro_score_knn,
(
SELECT AVG(growth_metro_score)
FROM (
SELECT growth_metro_score
FROM mc_blocks
ORDER BY la_eval_clean.the_geom <-> mc_blocks.the_geom
LIMIT 5
) a
) as growth_metro_score_knn

27. Grocery Stores/Restaurants
Find the number of grocery stores and restaurants using
OpenStreetMap Data and the SQL API.

28. (
SELECT count(restaurants_la.*)
FROM restaurants_la
WHERE ST_DWithin(
ST_Centroid(la_eval_clean.the_geom_webmercator),
restaurants_la.the_geom_webmercator,
1609 / cos(radians(ST_y(ST_Centroid(la_eval_clean.the_geom)))))
) as restaurants,
(
SELECT count(grocery_la.*)
FROM grocery_la
WHERE ST_DWithin(
ST_Centroid(la_eval_clean.the_geom_webmercator),
grocery_la.the_geom_webmercator,
1609 / cos(radians(ST_y(ST_Centroid(la_eval_clean.the_geom)))))
) as grocery_stores

29. Roads
See if a home is within one mile of a major highway or trunk highway
using the SQL API and major roads from OpenStreetMap.

30. (
SELECT CASE WHEN COUNT(la_roads.*) > 0 THEN 1 ELSE 0 END
FROM la_roads
WHERE ST_DWithin(
la_eval_clean.the_geom_webmercator,
la_roads.the_geom_webmercator,
1609 / cos(radians(ST_y(ST_Centroid(la_eval_clean.the_geom)))))
AND highway in ('motorway', 'trunk')
) as highways_in_1mile

31. 3. Analysis

32. Analysis
The analysis for this project followed the following steps:
● Moran’s I Clusters & Outliers (Exploratory Data Analysis)
● Neighbor Homes Analysis (Spatial Feature Engineering)
● Predictive Modeling & Hyperparameter Tuning (using XGBoost)

33. Moran’s I
Using Moran’s I to evaluate spatial clusters and outliers via the PySAL
package, we can see these groupings and visualize them in
CARTOframes.

34. The Sum of Our PartsThe Sum of Our Parts
Moran’s I

35. The Sum of Our Parts
Neighbor Analysis
Evaluate the attributes of
neighbor properties using
k-nearest neighbor spatial
weights in PySAL to perform
spatial feature engineering.

36. The Sum of Our PartsThe Sum of Our Parts
def evalNeighbors(data, target, neighbors):
sample_target_neighbors = neighbors[target]
if len(sample_target_neighbors) == 0:
pass
else:
d = data[data['plot_id'].isin(sample_target_neighbors)].loc[:,['median_income_2011_2015',
'merchants_in_1m', 'total_pop_2011_2015', 'restaurants', 'grocery_stores', 'highways_in_1mile',
'sale_metro_score_knn', 'growth_metro_score_knn']].mean()
d['plot_id'] = target
return d
def buildDataframe(targetlist, **args):
newdf = pd.DataFrame(columns = ['plot_id', 'median_income_2011_2015', 'merchants_in_1m',
'total_pop_2011_2015', 'restaurants', 'grocery_stores', 'highways_in_1mile', 'sale_metro_score_knn',
'growth_metro_score_knn'])
for i in targetlist:
try:
newdf = newdf.append(pd.DataFrame(evalNeighbors(neighbors, i, W.neighbors)).T)
except KeyError:
pass
return newdf

37. The Sum of Our Parts

38. The Sum of Our Parts

39. The Sum of Our Parts
Predictive Modeling
Using XGBoost we can use this data to create a regression model to
predict housing prices and push that data back to CARTO using
CARTOframes, never leaving the notebook environment.

40. The Sum of Our Parts
Predictive Modeling
After hyperparameter tuning the model, we can reduce the Mean
Average Error down to $58,179.78.

41. The Sum of Our Parts
Feature Importance

42. 4. Solutions

43. The Sum of Our Parts
Solutions
To present the data and predictive analysis, both on data from the
model that has a sales price and for homes that have not sold, we
can develop a location intelligence application to showcase these
results.

44. Application Development
Using CARTO VL we can
quickly visualize our data
layers and add a user interface
with Airship, in less than 400
lines of code.

45. Los-Angeles
Prediction
Explorer

46. 5. Integration

47. Application Development
Deploy the model via a Python
based API and sync to CARTO
data via the Python SDK to
perform on the fly predictions
for specific properties.

48. Los Angeles
Property Sales
Prediction
App

49. The Sum of Our Parts
Other Use Cases
Predicting revenue from different physical retail locations
Identify clusters and groups of specific patterns to optimize
activities such as sales outreach or site selection
Classify property types or buying patterns in a city
Review spatial feature importance for site performance, and
modify models using different spatial

50. Request a demo at CARTO.COM
Customer Success Manager // jsanchez@carto.com
Senior Data Scientist // andy@carto.com