Python+Unreal terrain workflow

Python+Unreal Terrain workflow
• The visualization is based on true-scale and matching texture tiles. It require
input from accurate imaging and data processing of raster+vector
• Python has support for this and can generate quality data for visualization
• Libraries – rasterio, Fiona, shapely, shpfile, pyproj, matplotlib, pysimplegui, spectral,
pylandsat, pylidar, tkinter, geopandas, numpy, pyfor, PIL, cv2
• Solution 1 – Python system-wide external full-processing and send final Grid to Game Engine
for visualization. Currently evaluating this approach, and finding the right UE4 version. [Using
Solution 1 as the Back-end server]
• Solution 2 – Embedded Python inside Unreal UE4.18, however this require minimal library
and simplest operation. It must not use complex GDAL/OSGEO and complex libraries.
[Working Demo]
• Solution 3 – Unreal OpenStreetMap plugins – limited to certain UE4 version and use Mapbox
data only. [Require Mapbox sever and Mapbox skillset]. Mapbox use and feed standard
dataset, and it is not possible toiterate the static mesh landscape.
• [Solution 4] Web-based front-end. [Using this solution as web-portal that links to a back-
end(Sol.1)+ a Raster/Image Cached-server that follows a certain grid-line system for each
partnering country]
https://www.linkedin.com/in/drzulabdullah/

System overview [Solution 4]
Find Data Creating Grid/Tile
Data processing
Back-end [Python Server]
UE4 Runtime [executable
file + data package]
Visualization
Process Query
Python [Server]
Blue Print / C++ [Unreal]
Module ‘Landscape’
Web Front-end
Make Query
Javascript Web UE4 Runtime (.exe)
Ranking
End-UserEnd-User
Process Query

User need to
select a date
[Disaster start
+range]
Year [1960 – Now]
Month [12]
Day [1 to 29-31]
AOI [Area of Interest / Tile] Disaster TypeDate / Range
Disaster type →
• Flood
• Drought
• Landslide
The Web Front-End is designed for non-tech Disaster Manager or stakeholders to start the query for dataset , where there is a
possibility for disaster situation or just any Area-of-Interest (AOI) for demonstrating the concept to the audiences.
Process QueryMake Query
Python back-endChoose Date (Range) Choose Area-of-Interest (AOI)Choose Disaster Type

Python back-end (Debug-Mode)
Python → input [Geotiff,LIDAR, Shapefile, geojson] → DEM → Masking/Binary → Result [ Python::3D surface, Shaded
Relief, Tile, Masking (features, segment), ChangeDetection] → Query by Date Range + Disaster type → to Landsat8, Sentinel
3D terrain generated from
selected tiles from [C]
[A] [B] [C]
[D] [E] [F] [G]
[B] – Hill Shade rendering of
the selected Tile from [C]
[C] – Tile Grid – Showing AOI and data which
is readily available from that area
[D] – Selected
Range of date for
disaster situation
for Python to
search from
several Providers
for dataset like
Sentinel, Landsat,
LIDAR and
shapefiles
[E,F] – Image and
post processing
looking for features
[G] – UI controller
for processing
images from [E] to
produce post-
rendered in [F]
NEON DATA PRODUCTS

Ranking system – for evaluating data quality and suitability for processing
[3] LIDAR [<1m] asc,geotiff
[2] Sentinel [30m], geotiff
[1] TandemX [90m], geotiff
Heightm
ap Data
[3] Aerial Drone [<1m]
[2] Sentinel [30m], geotiff
[1] Landsat [30-90m], geotiff
Raster
Image
[3] Sentine1 [30m], McFleeters
[2] Landsat [30-90m], NDWI
[1] Modis [>90m], NDWI
Flood
Extent
[3] Agency
[2] OSM
[1] Raster to Vector conversion
Shapefiles
[3] Within
[2] Partial [intersection]
[0] Outside
Tile
[3] Exact date
[2] Within range
[0] Out of range
Date
This Ranking system evaluate the query results (collections of satellite images, radar, shapefiles and LIDAR) and check if these
are within the acceptable quality. A Good ranking (quality) is 3 and the lesser has lower value.
If the Python search for dataset (query result) shows low ranking, the system will advise the Disaster Manager to request for
third party or agency to help get those dataset. Continuing with low ranking dataset for further processing is not advisable.

Example Tile
Variations
Resolution
Image Band
Date
Coverage
Affecting
Type of
processing
Detectable
object
Detectable
feature
THE NEON DATA PRODUCTS ARE PROVIDED "AS IS",
WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,
TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE NEON
DATA PRODUCTS BE LIABLE FOR ANY DAMAGES OR OTHER
LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE
NEON DATA PRODUCTS

Example tiles for a 505meter x 505meter
[NEON DATA PRODUCTS ]
305000,308000, 4319000,4315000
Tile numbering - left to right, goto next row
tile 11 - 305000,306009, 4319000,4317991
tile 12 - 306009, 307018,4319000,4317991
tile 13 - 307018,308027,4319000,4317991
tile 21 - 305000,306009,4317991,4316982
Tile 22 - 306009,307018, 4317991,4316982
tile 23 - 307018, 308027,4317991,4316982
tile 31 - 305000,306009,4316982,4315973
tile 32 - 306009,307018,4316982,4315973
tile 33 - 307018, 308027,4316982,4315973
tile 41 - 305000,306009,4315973,4314964
tile 42 - 306009,307018,4315973,4314964
tile 43 - 307018, 308027,4315973,4314964
Overall
size(vertices) Quads / section
Sections /
component Component size Total Components
505x505 63 4 (2x2) 126x126 16 (4x4)
505x505 63 1 63x63 64 (8x8)
https://docs.unrealengine.com/en-
us/Engine/Landscape/TechnicalGuide
Based on the Unreal documentation, and the limitation for
rendering landscape texture, it is advisable to set the tile
size as 505 by 505 pixels, which represent 505 meter by
505 meter on land.

Png raster value range 0-32767 (require rescaling from the original DN number for heightmap [intensity] , and then Normalize
to match neighbouring tiles min/max value within the range of 0-32767)
Terrain height is based on intensity range between different min-max for each tile – Require Normalization to ensure matching
neighbouring heightmap
NEON DATA PRODUCTS

Example – Final single tile of 505meter x 505meter with several overlays in QGIS.
QGIS is used as the benchmark and for experimenting with best practices for the processing workflow
before this process is duplicated in Python
NEON DATA PRODUCTS

Example Unreal Game engine rendering (http://www.epicgames.com/)
Image showing the tile (505 by 505) of a DSM (Digital Surface model dataset and Aerial Photo), OSM dataset like Building,
road, railway and river centre-line.
Credit: U.S. Geological Survey
Department of the Interior/USGS
U.S. Geological Survey/photo by Jane
Doe
NEON DATA PRODUCTS

Credit: U.S. Geological
Survey
Department of the
Interior/USGS
U.S. Geological Survey/photo
by Jane Doe
NEON DATA
PRODUCTS
Example Unreal Game engine rendering (http://www.epicgames.com/)
Image showing the tile (505 by 505) of a DSM (Digital Surface model dataset) which has been pre-processed with GDAL and
Wavelet Noise despeckle to improve the heightmap quality and remove spike noise from the point cloud dataset.

Aerial USGS 1m DSM 1m
NEON DATA
PRODUCTS
The QGIS processing for PNG file type for 0-255
data range. This need to be extended for 0-32767
data range for Unreal rendering. The process for
extending this range is shown in the next slide

USGS 1m DSM 1mAerial 1m
•‘Scale’ = 32767/(Max Elevation – Min Elevation) =
32767/(255-0) = 255
• ‘Offset’ = -1 x (Min Elevation x Scale) = -1 x (241.970 x
801.168) = -193858.620
Z-Scale’ = (Range/256) x 100 = ((282.869-241.970) /256) x 100
= 15.976
Example processing for normalizing the tiles to
improve the height match between
neighbouring tiles, since Unreal workflow has
been optimized for a small tile size of 505 by
505 pixels (meter)
NEON DATA
PRODUCTS

Image file from Python serverUnreal make a request
Unreal save the file to
specific folder
Unreal Blueprint need
to process this as an
overlay
UI for overlay
Unreal getting image from Python Server
Limitation to Blueprint
Unreal BP as the base
Add Unreal c++
??

Unreal lat long position from
screen raytrace
Unreal get screen x,y
Found Collider
(Landscape), and
return x,y,z value
Line raytrace from
Camera towards the
collider in the scene
Limitation to Blueprint
Send to Python
Process for lat long
based on the Excel
formula
Require GDAL – as
Plugin
Get x,y screen value
Python validate and
process
Unreal Raytrace Hit Test to find x,y location in Lat Long value

Workflow
Step 1- [new_value = (raw_value * scale) + offset ]
Step 2- r.rescale to 0-32767
Step 3 – save as geotiff
Step 4 - convertformat to (png) (Uint16)
Credit: U.S. Geological
Survey
Department of the
Interior/USGS
U.S. Geological Survey/photo
by Jane Doe
NEON DATA
PRODUCTS

Aerial photo – 0-255 range Credit: U.S. Geological Survey
U.S. Geological Survey/photo by Jane Doe
NEON DATA PRODUCTS

Rescaled to png 0-32767 range Credit: U.S. Geological Survey
NEON DATA PRODUCTS

DSM dataset – Rescaled to 0-32767
STATISTICS_MAXIMUM=32766
STATISTICS_MINIMUM=489
Extent
306009.0000000000000000,4315973.0000000000000000 :
306514.0000000000000000,4316478.0000000000000000
Unit meters
Width 505
Height 505
CRS EPSG:32615 - WGS 84 / UTM zone 15N - Projected
Dimensions X: 505 Y: 505 Bands: 1
Origin 306009,4.31648e+06
NEON DATA PRODUCTS

Aerial photo – tile 505 by 505, following the CRS – therefore→ expect transformation → rotation,
scale and reposition to match the CRS geometric correction from (XML, geojson file)
No Data area is clearly shown in the above Figure.
NEON DATA PRODUCTS

•‘Z-Scale’ = (Max-Min/256) x 100
•‘Z-Scale’ = (280.02-241.78/256) x 100
NEON DATA PRODUCTS

NEON DATA PRODUCTS

GIS dataset mis-alignment inside Unreal Engine, when placed on the landscape.
Possibly because of different CRS for both dataset – CRs fpor Geotiff reference for the
Aerial/DSM and CRS for the imported ESRI Shapefile
NEON DATA PRODUCTS

NEON DATA PRODUCTS
OepnStreetmap CRs match the USGS web
dataset’s CRS.
Dataset from OSM need to be scaled,
rotated and positioned properly to match
the Aerial/DSM data

NEON DATA PRODUCTSUnreal Engine runtime Change Texture to illustrate flood extent, and Ray Trace hit point on
landscape to show approximate Lat Long

Python+Unreal terrain workflow

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Python+Unreal terrain workflow

Similar to Python+Unreal terrain workflow (20)

Recently uploaded

Recently uploaded (20)

Python+Unreal terrain workflow