Conference slides for "Geospatial analytics with dashDB in the cloud"

1. © 2015 IBM Corporation
Geospatial analytics with dashDB in the cloud
Session # 1823
Torsten Steinbach @torsstei

2. Please Note:
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in
making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any
material, code or functionality. Information about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a
controlled environment. The actual throughput or performance that any user will experience will vary
depending upon many factors, including considerations such as the amount of multiprogramming in the
user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated
here.
2

3. 3
dashDB

4.  area
 distance
 length
 perimeter
Multistring MultiPolygon
PolygonLineString
Multipoint
Point
Curve Surface
Geometry
Collection
Geometry
Multicurve Multisurface
Geospatial Data

5. Loading Esri Shapefiles

6. Cloudant Warehousing with GeoJSON
{GeoJSON}
Other data sourcesOther data sources

7. {GeoJSON}
GeoJSON data comes in 3 flavours:
{
"type": "LineString",
"coordinates": [
[ 2.3200, 48.8657 ],
[ 2.2951, 48.8738 ]]
}
{
"type": "LineString",
"coordinates": [
[ 2.3200, 48.8657 ],
[ 2.2951, 48.8738 ]]
}
...as „Simple“ Geometry
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"name": "Champs Elysées"},
"geometry": {
"type": "LineString",
"coordinates": [
[ 2.3200, 48.8657 ],
[ 2.2951, 48.8738 ]]
}
},
{
"type": "Feature",
"properties": {
"name" : "Notre-Dame"},
"geometry": {
"type": "Point",
"coordinates": [
2.3497, 48.8528 ]
}
}
]
}
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"name": "Champs Elysées"},
"geometry": {
"type": "LineString",
"coordinates": [
[ 2.3200, 48.8657 ],
[ 2.2951, 48.8738 ]]
}
},
{
"type": "Feature",
"properties": {
"name" : "Notre-Dame"},
"geometry": {
"type": "Point",
"coordinates": [
2.3497, 48.8528 ]
}
}
]
}
...as Feature Collection
{
"type": "Feature",
"properties": {
"name": "Champs Elysées"},
"geometry": {
"type": "LineString",
"coordinates": [
[ 2.3200, 48.8657 ],
[ 2.2951, 48.8738 ]]
}
}
{
"type": "Feature",
"properties": {
"name": "Champs Elysées"},
"geometry": {
"type": "LineString",
"coordinates": [
[ 2.3200, 48.8657 ],
[ 2.2951, 48.8738 ]]
}
}
...as Feature

8. {
"_id": "75000",
"_rev": "08066f8ecd5f780646aa1573460852c",
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"name": "Le Louvre"},
"geometry": {
"type": "Point",
"coordinates": [2.3382, 48.8605]
}
},
{
"type": "Feature",
"properties": {
"name" : "Notre-Dame"},
"geometry": {
"type": "Point",
"coordinates": [2.3500, 48.8530]
}
}
]
}
{
"_id": "75000",
"_rev": "08066f8ecd5f780646aa1573460852c",
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"name": "Le Louvre"},
"geometry": {
"type": "Point",
"coordinates": [2.3382, 48.8605]
}
},
{
"type": "Feature",
"properties": {
"name" : "Notre-Dame"},
"geometry": {
"type": "Point",
"coordinates": [2.3500, 48.8530]
}
}
]
}
_id _rev type
75000 08066f8... FeatureCollection
_id _rev type
75000 08066f8... FeatureCollection
_id array_index type properties_name geometry
75000 0 Feature Le Louvre POINT (2.34 48.86)
75000 1 Feature Notre-Dame POINT (2.35 48.85)
_id array_index type properties_name geometry
75000 0 Feature Le Louvre POINT (2.34 48.86)
75000 1 Feature Notre-Dame POINT (2.35 48.85)
Cloudant Database:
SightsInParis
DashDB Warehouse
Table: SIGHTSINPARIS
Table: SIGHTSINPARIS_FEATURES

9. GeoData & dashDB
{GeoJSON}
WKT((),())
WKB
GML

10. GeoSpatial Analytics In dashDB
• Implements OGC SFS & ISO SQL/MM part 3 standards for spatial
 See http://www.iso.org/iso/catalogue_detail.htm?csnumber=38651
• Spatial data type ST_GEOMETRY (hierarchy)
• Enables spatial operations (e.g. joins) in database through spatial
operators available as user defined functions
• Dedicated support in ESRI tools starting V 10.3
• dashDB - R support through extension to ibmdbR package

11. Spatial Functions and Predicates in dashDB
SELECT a.name, a.type
FROM highways a, floodzones b
WHERE ST_Intersects(a.location,b.location) = 1
AND b.last_flood > 1950
SELECT a.road_id, a.time, i.id,
ST_Distance(a.loc, i.loc,’METER’) as distance
FROM accidents a, intersections i
WHERE ST_Distance(a.loc,i.loc,’METER’) < 10000
AND a.weather = ‘RAIN’
- accidents near intersections
- highways in flood zones
ST_Distance(g1,g2)
?
ST_Intersects(g1,g2)
?

12. Spatial Constructor Functions
 ST_Point(x, y, srs_id) – create point at this location
 ST_Point(‘POINT (-121.5, 37.2)’, 1)
 ST_Linestring(‘LINESTRING (-121.5 37.2,-121.7 37.1)’,1)
 ST_Polygon(CAST (? AS CLOB(1M)),1)
– For host variable containing well-known text, well-known binary, or shape
representation

13. Spatial Predicates – WHERE Clause
 ST_Distance(geom1, geom2) < distance_constant or var
 ST_Contains(geom1, geom2) = 1
 ST_Within(geom1,geom2) = 1
 EnvelopesIntersect(geom1, geom2) = 1
 EnvelopesIntersect(geom1, x1, y1, x2, y2, srs_id) = 1
 ST_Area(geom) < some_value

14. Spatial Functions that Create New Spatial Values
 ST_Buffer(geom, distance)
 ST_Centroid(geom)
 ST_Intersection(geom1, geom2)
 ST_Union(geom1, geom2)

15. Functions that Return Information About a Spatial Value
 ST_Area(geom), ST_Length(geom)
 ST_MinX(geom, ST_MinY(geom), ST_MaxX(geom), ST_MaxY(geom)
 ST_IsMeasured(geom)
 ST_X(geom), ST_Y(geom)
 ST_AsText(geom)

16. And Many More …
ST_Area
ST_AsBinary
ST_AsText
ST_Boundary
ST_Buffer
ST_Centroid
ST_Contains
ST_ConvexHull
ST_CoordDim
ST_Crosses
ST_Difference
ST_Dimension
ST_Disjoint
ST_Distance
ST_Endpoint
ST_Envelope
ST_Equals
ST_ExteriorRing
ST_GeomFromWKB
ST_GeometryFromText
ST_GeometryN
ST_GeometryType
ST_InteriorRingN
ST_Intersection
ST_Intersects
ST_IsClosed
ST_IsEmpty
ST_IsRing
ST_IsSimple
ST_IsValid
ST_Length
ST_LineFromText
ST_LineFromWKB
ST_MLineFromText
ST_MLineFromWKB
ST_MPointFromText
ST_MPointFromWKB
ST_MPolyFromText
ST_MPolyFromWKB
ST_NumGeometries
ST_NumInteriorRing
ST_NumPoints
ST_OrderingEquals
ST_Overlaps
ST_Perimeter
ST_Point
ST_PointFromText
ST_PointFromWKB
ST_PointN
ST_PointOnSurface
ST_PolyFromText
ST_PolyFromWKB
ST_Polygon
ST_Relate
ST_SRID
ST_StartPoint
ST_SymmetricDiff
ST_Touches
ST_Transform
ST_Union
ST_WKBToSQL
ST_WKTToSQL
ST_Within
ST_X
ST_Y
And more…
Simplified Constructors
from
x,y
WKT
WKB
GML
shape
Linear referencing
Spatial aggregation
ST_AsGML
ST_AsShape

17. Harness the Full Power of SQL
 Outer join
 Common table expressions
 Recursive queries, sub-queries
 Aggregate functions
 Order by, group by, having clauses
 OLAP, XML, and more ...
WITH sdStores AS (SELECT * FROM stores
WHERE st_within(location, :sandiego) = 1)
SELECT s.id, s.name, AVG(h.income) FROM houseHolds h, sdStores s
WHERE st_intersects(s.zone, h.location) = 1
GROUP BY s.id, s.name
ORDER BY s.name
Example problem: Determine the average household income for the sales zone of each store in theExample problem: Determine the average household income for the sales zone of each store in the
San Diego area.San Diego area.

18. Predictive Analytics With R In dashDB

19. Backup

20. dashDB: Key Use Cases
• Minimize capital expense of DR solutionDR in the Cloud

21. We Bring Netezza Compatible Analytic Platform to the
Cloud
Analytic Extension FrameworkAnalytic Extension Framework
UDX C++ APIUDX C++ API
Canned AnalyticsCanned Analytics
Application IntegrationApplication Integration
AE FrameworkAE Framework In-DB RIn-DB R In-DB LUAIn-DB LUAIn-DB PythonIn-DB Python In-DB PerlIn-DB Perl
OLAP FunctionsOLAP Functions
ROW_NUMBERROW_NUMBER
RANKRANK
LAGLAG LEADLEAD
DENSE_RANKDENSE_RANK Linear RegressionLinear Regression
Kmeans
Clustering
Kmeans
Clustering Decision TreeDecision Tree
Association RulesAssociation Rules
Association RulesAssociation Rules
Naive BayesNaive Bayes
Spatial OperatorsSpatial Operators
ContainsContains
TouchesTouches
WithinWithin
IntersectsIntersects
CrossesCrosses
OverlapsOverlaps
R WrapperR Wrapper Watson AnalyticsWatson Analytics ESRI ArcGIS
Connector
ESRI ArcGIS
Connector ……
Analytics Applications of ISVs and CustomersAnalytics Applications of ISVs and Customers
STDDEVSTDDEV
COVARCOVAR
…………

22. Analytic Code &
Algorithms:
Analytic Data:
Data pulled out and processed in analytic
application
Analytic
Applications
This is where we start from: All analytic processing done on application side
Analytics of Warehouse Data

23. SQLs
Analytic Code &
Algorithms:
Analytic Data:
Simple data lookup & massage operations
pushed down as SQL operations
Analytic
Applications
Benefit: Acceleration with no SQL skills required
SQLs
Push Down Step 1: BLU tables only logically represented in analytic application
Accelerate Analytics for Warehouse Data

24. SQLs
Analytic Code &
Algorithms:
Analytic Data:
Call built-in functions via SQL to execute
typical algorithms inside db
Cloud Tooling
Analytic
Applications
Benefit: Bring Standard Analytics to the Data
SQLs
Canned Algorithms
Push Down Step 2: Typical and popular algorithms pushed down to canned UDFs in the db
Accelerate Analytics for Warehouse Data

25. LanguageFramework
(UDX&AE)
Analytic Code &
Algorithms:
Analytic Data:
Deploy customer code and call via special
SQL function interfaces
SQLs
SQLs
Canned Algorithms
Analytic
Applications
Benefit: Bring Custom Analytics to the Data
Push Down Step 3: Execute entire customer analytic programs inside the db
Accelerate Analytics for Warehouse Data

26. Don’t forget to submit your Insight session and speaker feedback! Your
feedback is very important to us – we use it to continually improve the
conference.
Access your surveys at insight2015survey.com to quickly submit your surveys
from your smartphone, laptop or conference kiosk.
We Value Your Feedback!
26

27. 27
Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form
without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for
accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to
update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO
EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO,
LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted
according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as
illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other
results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services
available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the
views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the
identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the
customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will
ensure that the customer is in compliance with any law.

28. 28
Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly
available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance,
compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
•IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DB2® , DOORS®, Emptoris®, Enterprise
Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM
SmartCloud®, IBM Social Business®, IMS™, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON,
OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®,
Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International
Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or
other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at:
www.ibm.com/legal/copytrade.shtml.

29. © 2015 IBM Corporation
Thank You