NoSQL and Spatial Database Capabilities using PostgreSQL

NoSQL and Spatial Database Capabilities using
PostgreSQL
May 20, 2021
9:30 am IST | 12:00 pm SGT
Mansur Shaikh | Sr. SE | EDB
May 20 2021

2021 Copyright © EnterpriseDB Corporation All Rights Reserved
What brings us here today?
• NoSQL Capabilities in PostgreSQL
• NoSQL with JSON and Hstore
• Spatial SQL Features
• Foreign Data Wrapper
• Postgres for Enterprises

Doing More with Postgres…
How to do more
with Postgres
Open source alternative to
commercial RDBMS
• Reduce cost
• Leverage in-house talent
• Flexible license model
RDBMS platform for
new developments
• Proven RDBMS
• SQL compliant
• Extremely stable
Innovative DBMS Platform
• Not only SQL (SQL + JSON/KVP)
• Foreign Data Wrappers
• PostGIS

Diversity of Use Cases Diversity of Workloads Diversity of Deployments
Why did PostgreSQL win (Multimodel)
Migration
New App Development
Replatforming to
Cloud and Containers
System of Record
System of Analysis
System of
Engagement
Public Cloud – IaaS
Public Cloud – DBaaS
Private Cloud
Virtual Machines
Containers

Postgres – ORDBMS
PostgreSQL is an Object-Relational Database
• O-R development foundation completed in 1994
• Built upon classes and inheritance
• O-R foundation continues driving innovation today
ORDBMS properties
• Highly extensible architecture (also easy)
• Data types and Indexes
• Operators, functions, casts, and aggregates
• Procedural languages
• New features easily added to original feature set
Consistency between Original and New features in:
• Design
• Implementation
• Behaviors
• Performance

NoSQL capabilities in Postgres
Foreign Data Wrapper
• Hadoop Foreign data Wrapper
• MangoFW
PostGIS
• Geospatial extension
• Considered one of the best geospatial modules in the world
HSTORE
• Key-Value store
• Excellent for handling rows with sparse data collections
JSON:
• NoSQL Document data store
• Full support for JSON document functions and operators
• Embed JSON in your SQL
• Easy bi-directional interchange of JSON and relational data
• FAST

Postgres’ Unstructured Data Capabilities
HSTORE: Key-value pairs
• Simple, fast and easy
• In Postgres – pre-dates many NoSQL solutions
• Ideal for flat data structures that are sparsely populated
JSON
• Hierarchical document model popular in web applications
JSONB
• Binary version of JSON
• Faster, more operators and even more robust

DevOps, Agile,
MicroService’s
Based Apps:
Deploying databases in RH
OpenShift Container
Platform to provide a key
database platform with
robust capabilities for :
Data management and HA
Read Scalability
Administration
Backup and
Recovery
Schema-less/
NoSQL: Best of Both
The Worlds:
Document store capabilities:
XML, JSON, JSONB, PLV8;
HStore (key-value store) :
Full Text Indexing
Performance: Postgres
Vs MongoDB
Multiple
Programming
Language Support:
Freedom for Developers to
suit their skills and needs for
new apps
Oracle’s PL/SQL
PL/pgSQL
PL/Perl
PL/Python
PL/TCL
PL/Java
GIS Support:
Best in Class
GIS/Spatial
Extension:
PostGIS: An advanced,
proven and widely used
Geospatial extension,
compared to Oracle,
Microsoft and other
proprietary or open source
options.

NoSQL Features in Postgres
EDB supercharges PostgreSQL
• Postgres - NoSQL Features
• Structured and Unstructured Data
• Spatial data with PostGIS
• Foreign data wrapper,

How NoSQL is Possible in Postgres ?
With new features and capabilities along side several long standing components and
extensions, Postgres can support virtually all of today’s data types as well as unstructured
and semi-structured data. Data has changed.
Bigger volumes, needs for faster processing and new data types mean organizations today
are facing new problems. Big Data problems.
• Postgres can power many applications written for NoSQL technologies.
• Developers can build applications in Postgres that achieve the same results as NoSQL
solutions

Most Prominent NoSQL technologies
The four most prominent NoSQL technologies most often referenced in Big Data
conversations are:
• Key Value Store
• Document Database
• Column Store
• Graph database
The technologies that address the most common kinds of challenges are key value store and document
databases
Relational technologies are advancing on the data load threshold and already can support some NoSQL
capabilities
Companies also looked to NoSQL for transactional applications that could support new data types and new
ways to store and work with them

NoSQL Database Limitations
• Lack of aggregate or data Analysis function
• Lack of powerful query language
• Not ACID Compliant
• No Query language, Query Optimizer
• No Support for Joins, referential integrity
• NoSQL technologies stores data , they don’t process data

Postgres Capabilities for NoSQL Workloads
Postgres was originally architected to be an object-relational database designed
specifically to be extensible.
There are capabilities in Postgres that enable it to achieve much of what NoSQL technologies
have been designed to do
• It supports objects and classes and custom data types and methods
• New capabilities could be developed as needs evolved and plugged into the database
seamlessly.
• Using this level of extensibility, Postgres developers were able to build new features and
capabilities as needs emerged. The perfect examples are JSON/JSONB for document
storage support and HStore for key-value support.
• With JSON/JSONB and HStore, Postgres can support applications that require a great deal
of flexibility in how data is handled.

JSON /JSONB – Document Database
Document database capabilities in Postgres advanced significantly when support for the JSON
data type was introduced
• JSON (JavaScript Object Notation) is one of the most popular data-interchanged formats on the
web
• Postgres offers robust support for JSON. Postgres has a JSON data type, which validates and
stores JSON text and provides functions for extracting elements from JSON values. And, it offers
the ability to easily encode query result sets using JSON.
• JSON-formatted data can be sent directly to the database where Postgres will not only store the
data, but properly validate it as well. With JSON functions, Postgres can read relational data
from a table and return it to the application as valid JSON formatted strings. And, the relational
data can be returned as JSON for either a single value or an entire record.
• There are also functions that convert Postgres-maintained key-value data (by means of the
Hstore contrib module–see below) to JSON- formatted data, which increases the flexibility and
scope of NoSQL-like application that can be addressed by Postgres.

JSON Examples
Creating a table with a JSONB field
CREATE TABLE json_data (data JSONB);
Simple JSON data element:
{"name": "Apple Phone", "type": "phone", "brand": "ACME",
"price": 200, "available": true, "warranty_years": 1}
Inserting this data element into the table json_data
INSERT INTO json_data (data) VALUES
(’ { "name": "Apple Phone",
"type": "phone",
"brand": "ACME",
"price": 200,
"available": true,
"warranty_years": 1
} ')

JSON data element with nesting
{“full name”: “John Joseph Carl Salinger”,
“names”:
[
{"type": "firstname", “value”: ”John”},
{“type”: “middlename”, “value”: “Joseph”},
{“type”: “middlename”, “value”: “Carl”},
{“type”: “lastname”, “value”: “Salinger”}
]
}

Extracting List of Products from JSON data
SELECT DISTINCT data->>'name' as
products FROM json_data;
products --------------------------------
Cable TV Basic Service Package
AC3 Case Black
Phone Service Basic Plan
AC3 Phone
AC3 Case Green
Phone Service Family Plan
AC3 Case Red
AC7 Phone
AC3 Series Charger
Phone Extended Warranty Phone
Service Core Plan (11 rows)

Extracting List of Products for the brand ‘ACME’
SELECT DISTINCT data->>'name' AS
"Product Name", data->>'price'
AS "price" FROM json_data WHERE
data->>'brand' = 'ACME’;
Product Name | price -----------
---+-------
AC3 Phone | 200
AC7 Phone | 320
(2 rows)

JSON data Using Select
SELECT data FROM json_data;
data
------------------------------------------
{"name": "Apple Phone", "type": "phone",
"brand": "ACME", "price": 200, "available":
true, "warranty_years": 1}
This query returns the JSON data
in its original format

Extract the price of an apple phone
SELECT data->>'price' as iphone_price FROM
json_data
WHERE data->>'name'='Apple Phone’;
iphone_price
--------------
200

Key-Value Store - Hstore
The HStore contrib module, which can store key/value pairs within a single column, enables users to
create a schema-less key-value store. But unlike pure NoSQL solutions, a key-value store created in
Postgres is ACID compliant.
• Hstore is a particularly handy tool for web developers or someone building an application that
requires the ACID properties of Postgres and NoSQL capabilities.
• HStore data type offers advanced indexing support, which makes it the solution of choice for many
applications. It is particularly useful for data with sparsely populated attributes.

HSTORE Examples
Create a table with HSTORE field
Create extension hstore;
CREATE TABLE hstore_data (data HSTORE);
Insert a record into hstore_data
INSERT INTO hstore_data (data) VALUES (’
"cost"=>"500",
"product"=>"iphone",
"provider"=>"apple"');
Select data from hstore_data
SELECT data FROM hstore_data ;
------------------------------------------
"cost"=>"500”,"product"=>"iphone”,"provider"=>"Apple"
(1 row)

Integrating JSON and HSTORE
Convert HSTORE data to JSON
SELECT hstore_to_json(data) FROM
hstore_data ;
hstore_to_json
-------------------------------------
--------------- -------
{"cost": "500", "product": "iphone",
"provider": "Apple"} (1 row)
SELECT hstore_to_json(data)->>'cost'
as price FROM hstore_data ;
Price
-------
500

A JSON and ANSI SQL – A Great Fit
JSON is naturally integrated with ANSI SQL in
Postgres
JSON and HSTORE are elegant and easy to use
extensions of the underlying object-relational
model
JSON and SQL queries use the same
language, the same planner, and the same ACID
compliant transaction framework

JSON and ANSI SQL Example
SELECT DISTINCT
product_type,
data->>'brand' as Brand,
data->>'available' as Availability
FROM json_data
JOIN products
ON (products.product_type=json_data.data->>'name')
WHERE json_data.data->>'available'=true;
product_type | brand | availability
---------------------------+-----------+--------------
AC3 Phone | ACME | true

Bridging between SQl and JSON
Simple ANSI SQL Table Definition
CREATE TABLE products (id integer, product_name text );
Select query returning standard data set
SELECT * FROM products;
id | product_name
----+--------------
1 | iPhone
2 | Samsung
3 | Nokia
Select query returning the same result as a JSON data set
SELECT ROW_TO_JSON(products) FROM products;
{"id":1,"product_name":"iPhone"}
{"id":2,"product_name":"Samsung"}
{"id":3,"product_name":"Nokia”}

EDB POSTGRES FOR POLYGLOT DATA INTEGRATION
Event Capture Legacy Systems
Data Warehouse
Big Data
Open Source ODBMS

Foreign Data Wrapper Platform
FDW implements SQL/MED ("SQL Management of External Data")
PostgreSQL - read-only support
PostgreSQL – read/write support
FDW
Uses data on other servers (or services) like native tables
Available for MongoDB, CouchDB, Cassandra, Hadoop (HDFS) MySQL, Oracle, files, services
(Twitter, LDAP, JDBC…)
EDB FDWs: https://github.com/EnterpriseDB

MangoDB FDW Example
CREATE EXTENSION mongo_fdw;
CREATE SERVER mongo_server
FOREIGN DATA WRAPPER mongo_fdw
OPTIONS (address '172.24.39.129', port '27017');
CREATE USER MAPPING FOR enterprisedb
SERVER mongo_server
OPTIONS (username 'mongo', password 'mongo');
CREATE FOREIGN TABLE mongo_data(
name text,
brand text,
type text)
SERVER mongo_server
OPTIONS (
database 'benchmark',
collection 'json_tables');

MangoDB FDW Example
SELECT * FROM mongo_data WHERE brand='ACME' limit 10;
name | brand | type
-------------+-------+-------
AC7553 Phone | ACME | phone
(10 rows)

MangoDB FDW Example
INSERT INTO mongo_data(name, brand, type)
VALUES('iphone6 phone','Apple Inc','phone');
SELECT* FROM mongo_data WHERE brand='Apple Inc';
_id | name | brand | type
--------------------------+----------------+-----------+-------
53ea4f59fe5586a15714881d | iphone6 phone | Apple Inc | phone
UPDATE mongo_data SET brand='Apple Product'
WHERE brand='Apple Inc’;
SELECT * FROM mongo_data WHERE brand='Apple Product’;
_id | name | brand | type
--------------------------+----------------+---------------+-------
53ea4f59fe5586a15714881d | iphone6 phone | Apple Product | phone

Hadoop Foreign Data Wrapper
The Hadoop Foreign Data Wrapper (hdfs_fdw) is a Postgres extension that allows you to access
data that resides on a Hadoop file system from an Advanced Server or PostgreSQL server. The
foreign data wrapper makes the Hadoop file system a read-only data source that you can use
with Postgres functions and utilities
• The Hadoop data wrapper provides an interface between a Hadoop file system and a
Postgres database. The Hadoop data wrapper transforms a Postgres SELECT statement into
a query that is understood by the HiveQL or Spark SQL interface
• The Hadoop Foreign Data Wrapper can be installed with an RPM package. You can download
an installer from the EDB website.

The following command creates a foreign server named hdfs_server that
uses the hdfs_fdw foreign data wrapper to connect to a host with an IP
address of 170.11.2.148:
CREATE SERVER hdfs_server FOREIGN DATA WRAPPER hdfs_fdw OPTIONS (host '170.11.2.148', ˓→port '10000', client_type 'hiveserver2', auth_type 'LDAP', connect_timeout '10000', ˓→query_timeout '10000');
The foreign server uses the default port (10000) for the connection to the
client on the Hadoop cluster; the connection uses an LDAP server.
Use the CREATE USER MAPPING command to define a mapping that
associates a Postgres role with a foreign server:
CREATE USER MAPPING FOR role_name SERVER server_name
[OPTIONS (option 'value' [, ...])];
ou must be the owner of the foreign server to create a user mapping for
that server.

Why Choose Between NoSQL and Relational When You Can Have Both !
• Schema-less development - fast application development
• Document store with ACID and relational capabilities
• Decades-long track record of quality and stability
• TCO cost avoidance from new systems:
• Operational support
• Duplicated storage
• Software upgrades
• Monitoring & management
• Hardware
• Training
• Additional staffing

A Multi Layer Approach for Database Security

Spatial Database
Capabilities in PostgreSQL

Geometry
Geography
Raster Topology

PostGIS Data subtypes
Point
Linestring
Polygon
Multipoint
Multilinestring
Multipolygon

Loading Spatial data into PostGIS
• Resource file :- sdb_data.zip
• Loading shapefile
• Using PostGIS shapefile and dbf exporter
• Command line :- c:programefilespostgresql10bin
• raster2pgsql , shp2pgsql
• Open Postgres shapefile import/export manager
• View connection detail-> import->add file
• Goto S DB_DATA folder -> load baea-nests.shp and bowl_habitat.shp
• Give special reference id is 4326

Making SQL Spatial
• The PostGIS Vector Geometry model
• The Spatial reference ID
• The Geometry field
• Geometry vs Geography data type
• Spatial queries-working with Geometries
• Spatial queries-Measurement
• Spatial queries – Accessing the geometry
•

Vector Geometry Model
Conforms to the OGC SFS standard
Basic data structure is a Point
• 2 dimension –X and Y values. (longitude and latitude), projection
• 3 Dimension value- z values
Line String is an array of points
Polygon is an array of closed LineString’s
• First Linestring is an exterior ring
• Each following LineString is an interior ring

Spatial Reference ID
Unique identifier for a coordinate reference system
• Coordinate System :- based in angles based on earth
• Projection
• Zone
• Datum
In PostGIS you indicate the Spatial Reference by a number
That number corresponds to well known Text
Individual Geometries can have a spatial reference
Geometry columns have a spatial reference

The Geometry Field
The geometry field is what makes a table spatial
Its just a column in the database that can store a feature geometry
It includes the SRID, the geometry type, and the actual coordinates in binary form
We can’t interpret the binary data but PostGIS provides a number of functions that can output the geometry in human
readable format
binary data could be picture, shape any things
These are functions that can output geometry as human redable text
ST_AsText(), ST_AsEWKT(), ST_AsGeoJSON, etc.
The combination of a geometry and the attribute that describes that geometry is known as features

Geometry vs Geography data types
Two different ways to store a features spatial information
• Geometry (data type). Based on plain surface
• Geography. (data type). Based on circular earth , math behind calculation is complicated
Geometry
• Based on planner surface
• Can be in a variety of coordinates reference systems defined by OGC
• Are mathematically simple
• Rich set of functions available
• OGC standard , GEOS libraries, etc.
• Accuracy declines as spatial extent increases due to cuveture

Geometry vs Geography cont..
Geography
• Based on round earth
• Can only be in geographic coordinates system (latitude/longitude)
• Are mathematically complex
• Only support a fraction of the functions that are available for geometries
• Accurate measurements
• Which should you use ??
• Scale of your data and if larger distance use geography
• Availability of functions that you need
• performance

Getting started (loading data…)
47
INSERT into events_geo(eventname,geom,date)
SELECT eventname,
st_setsrid(st_makepoint(lon,lat),4326) as geom,date
FROM events;
Events Table
eventname character varying
lat double precision
lon double precision
Date timestamp with time zone
events_geo Table
eventname character varying
lat double precision
Lon double precision
Date timestamp with time zone

Getting started (visual inspection)
48

Getting started (visual inspection)
49

Getting started (Geometry Input and
Output)
50
Example Input function
Insert into cities (name,state,geom)
Values (‘Bedford’,’MA’,
ST_GeomFromText(‘POINT(-71.248063 42.510547)’,4326));

Getting started (Power of GIS & SQL)
Create a route from a collection of waypoints captured in sequence
Write a function to tell whether a given lat/lon pair is
within a Point-Radius ring

34, 9
70, 12
Simple distance query between two points
SELECT ST_Distance
(ST_GeomFromText('POINT(34 9)’,4629),
ST_GeomFromText('POINT(70 12)’,4629));

All the powerful tools of the RDBMS are available to you

Answer powerful questions with a simple query
Example Input function

Working with Geometries
• What can we do with Geometries in PostGIS
• Get information about it
• GeometryType(geom) what type point, line or polygon
• ST_CoordDIM(geom). How many coordinate associate with each points
• ST_Dimension(geom). Like if line then it is one dimenstional, polygon is area which is two dimensional
• ST_SRID(geom). :- special reference id for geometry
• ST_IsCollection(geom) :- returns true if geometry is single point or multipoint polygon
• ST_NumGeometries(geom),ST_NuminteriorRings(geom),ST_NumPoints(geom)
• ST_IsSimple(geom),ST_IsEmpty(geom),ST_IsClosed(geom),ST_isRing(geom)
• ST_IsValid(geom), ST_IsValidReason(geom)
• Change it’s spatial reference
• ST_SetSRID(geom,SRID) changes special reference id associated with that geometry
• ST_Transform(geom,SRID) converts one coordinate reference system to another

Spatial Queries :- Measurements
• What we can do with geometry ?
• Measure it
• ST_Length(line)
Geometry- Cartesian length in SRID units
Geography-length in meters over the spheroid
• ST_3DLength(line)
- Geometry-3D cartesian length using SRID
- Geography - Not supported
• ST_LengthSpheroid(line, spheroid)
• ST_Area(polygon)
Geometry- cartesian area in SRID units
Geography-area in square meters over the spheroid
ST_Perimeter(polygon), ST_3DPerimeter(polygon)

EDB Postgres :NoSQL with
Enterprise Features

The Largest PostgreSQL Company
5,300+
lifetime customers
Over 300
dedicated PostgreSQL
technologists
28%
of Fortune 500 companies
are customers
40%
of customers
deploying in cloud
91%
customer
satisfaction rating
2
Founders of Postgres
Community

Integrations and Connectors
pgBouncer pgPool
Connection Management Integration
Data Adapters for Hadoop,
MongoDB, MySQL, CSV
Client Connections
Node.js
OCL
Python
ECPG Plus
JDBC
EDB Plus
ODBC
EDB Loader
.NET
Authentication/Authorization
LDAP
Active Directory
Kerberos
Deployment Options
Bare metal Virtualization Kubernetes Private cloud Public cloud

Postgres Enterprise Manager
GUI tool for monitoring, management and tuning databases
• Aggregates performance and status data -
Collects from DBs, OS, and jobs
• Monitors overall system health - Alerts thru
charts and dashboards, email, or SNMP
• Runs performance diagnostics - Execute SQL,
tune queries, run backups, and deploy updates
• Provides 200+ built-in alerts - Monitor bloat,
memory utilization, server status, and more

NoSQL and Spatial Database Capabilities using PostgreSQL

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to NoSQL and Spatial Database Capabilities using PostgreSQL

Similar to NoSQL and Spatial Database Capabilities using PostgreSQL (20)

More from EDB

More from EDB (20)

Recently uploaded

Recently uploaded (20)

NoSQL and Spatial Database Capabilities using PostgreSQL