S.2.h Meter Data Management Service

WP4 – Integration of SUNSHINE pilot smart urban services
D4.5 Meter Daata Management service #2
CIP-ICT-PSP-2012-6 – 325161
"This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the
Competitiveness and Innovation Framework Programme by the European Community"
(http://ec.europa.eu/ict_psp).
D4.5
Meter Data
Management Service #2
WP 4 – Integration of SUNSHINE pilot smart urban services
Task 4.2 – Meter data management service
Revision: v1.0
Authors:
Stefano Pezzi (SGIS)
Luca Giovannini (SGIS)

CIP-ICT-PSP-2012-6 – 325161
Dissemination level PU (public)
Contributor(s) Stefano Pezzi, Martina Forconi, Oscar Benedetti, Enrico De
Guidi, Luca Giovannini (SGIS), Luis Torres (Meteogrid)
Reviewer(s) Federico Prandi (Graphitech)
Editor(s) Raffaele De amicis (Graphitech)
Partner in charge(s) SGIS
Due date 31/01/2015
Submission Date 31/03/2015
REVISION HISTORY AND STATEMENT OF ORIGINALITY
Revision Date Author Organisation Description
0.1 08/02/2015 Luca Giovannini SGIS Document creation
0.5 19/03/2015 Luis Torres Meteogrid Document review
1.0 20/03/2015 Luca Giovannini SGIS Document finalization
Statement of originality:
This deliverable contains original unpublished work except where clearly indicated otherwise.
Acknowledgement of previously published material and of the work of others has been made
through appropriate citation, quotation or both.
Moreover, this deliverable reflects only the author’s views. The European Community is not
liable for any use that might be made of the information contained herein.

CIP-ICT-PSP-2012-6 – 325161
Table of content
Table of content........................................................................................................................... 3
Acronyms...................................................................................................................................... 3
1 Introduction ....................................................................................................................... 5
2 Meter data management service....................................................................................... 5
2.1 Overall architecture....................................................................................................... 6
3 Hardware components ...................................................................................................... 9
3.1 Main platform servers ................................................................................................. 10
3.1.1 Backend server...................................................................................................... 10
3.1.2 FTP server ............................................................................................................. 10
3.1.3 Webserver server.................................................................................................. 10
3.1.4 Firewall server....................................................................................................... 11
4 Software components...................................................................................................... 12
4.1 Sensor DB (52° North SOS 4.0)..................................................................................... 12
4.1.1 Database structure ............................................................................................... 13
4.1.2 Database management......................................................................................... 16
4.2 FTP ingestion................................................................................................................ 18
4.3 Meter data processing................................................................................................. 19
4.4 CSV loader.................................................................................................................... 20
4.5 Weather ingestion ....................................................................................................... 21
4.6 GreenButton ingestion ................................................................................................ 23
4.7 Sensor data access....................................................................................................... 25
5 Sunshine Weather Platform............................................................................................. 27
5.1 Platform architecture .................................................................................................. 27
5.1.1 Weather Data Collecting Service .......................................................................... 28
5.1.2 Weather Data Exhibition Service - WMS – WFS ................................................... 28
5.2 Software components.................................................................................................. 29
5.2.1 Core modules........................................................................................................ 29
5.2.2 Zope Object Database........................................................................................... 30
5.2.3 Data ingestion in Sigym3 ...................................................................................... 31
5.2.4 Data Access........................................................................................................... 31
Annex I - Sensor DB schema (52° North SOS 4.0)....................................................................... 32
Annex II – Mapping weather WFS to Sensor DB........................................................................ 33
Acronyms

CIP-ICT-PSP-2012-6 – 325161
CPU
Central Processor Unit
CSV
Comma Separated Values
DB
Data Base
DMZ
DeMilitarized Zone
FTP
File Transfer Protocol
IT
Information Technology
LAN
Local Area Network
SOS
Sensor Observation Service

CIP-ICT-PSP-2012-6 – 325161
1 Introduction
This document is an annex to the final deliverable of the Meter Data Management Service of
the Sunshine project. The following chapters describe the implementation of the service with
respect to hardware components, interaction with pilot infrastructures and functional
software modules. The reader can also refer to deliverable D1.5 “SUNSHINE System
architecture – final” for an overview of this service and of its relation with the other elements
of the Sunshine platform.
2 Meter data management service
During the development of SUNSHINE’s platform the scope of Meter Data Management
Service has been extended from the management of meter data to the management of data
coming more generally from sensors. The reason being that SUNSHINE’s pilots are providing
not just consumption data from meters, but also several other types of data from other
sensors that also need to be collected and managed to support SUNSHINE’s services and
piloting activities. More specifically, the Meter Data Management service manages:
 Consumption data from energy meters in pilot buildings;
 Indoor temperature readings from indoor sensors in pilot buildings;
 Weather observation and forecasts data from a dedicated weather service from
Meteogrid;
 Status data (dimming level, hours of use, etc) from pilot lamps and light-lines.
All these data types are stored in the same Sensor DB and managed with common
procedures, which will be described in this deliverable. In order to avoid redundancies, the
detailed description of the management of status data for lamps and light-lines, which is also
part of the Remote System Management service, has not been repeated here and can be
found in deliverable D4.6 “Remote System Management Service n2”.
In brief, the purpose of the Meter Data Management Service is to:
 Gather all the data coming from meters/sensors;
 Elaborate them and generate derived data, if necessary (Section 4.6);
 Store them in a normalized format into a repository (Section 4.1) from which they can
be retrieved through a standard interface (Section 4.7).
For consumption data from meters two data flows are supported by the platform, in order to
enable both the pilots with low IT capacities and those who are able to develop software
components:

CIP-ICT-PSP-2012-6 – 325161
 Simple FTP transfer of CSV file;
 Feed from Green Button web service.
Two service endpoints correspond to these types of flows:
 FTP Ingestion service (Section 4.2);
 Green Button Ingestion service (Section 4.6).
To maximize the reuse of the developed software, the Green Button Ingestion service doesn’t
load directly the data in the DB, but produces a CSV file that is treated by a dedicated
component, the CSV loader (Section 4.4), that is the same used by the FTP Ingestion service.
The flow of indoor temperature readings is channelled through the Green Button web service
(Section 4.6) as all the pilots actually providing such data flow have implemented the needed
infrastructure.
The flow of weather data (Section 4.5) is established via a scheduling procedure that is
planned to pull data daily from the dedicated Web Feature Services (WFS) set up by partner
Meteogrid (whose dedicated platform is described in detail in Chapter 5).
The flow of lamp status data is managed by the Remote System Management service (see
deliverable D4.6 “Remote System Management Service n2”) that ensures the uniform
ingestion of such data via the ingestion API of the Sensor DB.
2.1 Overall architecture
The overall architecture of Sunshine is shown in Figure 1, we’ll now roughly describe the
whole system and then we’ll focus on the components involved in the meter data
management service. The diagram is quite complex and the different colours differentiate the
various functional sections of the entire system:
 Red modules compose the security layer;
 Green modules compose the meter data management service, overseeing the
ingestion of different data flows into Sunshine’s sensor database;
 Blue modules compose the remote system management service, managing light line
control and monitoring;
 Grey modules compose the building pre-certification service, managing building
geodata and estimating building heath needs.
Services are implemented by the boxes representing functional units of software and by the
blue circles that define the interfaces.
In the left side of the diagram there are some of the nodes that deliver information to the
Sunshine’s platform plus the Sunshine’s client application. An exception is represented by the

CIP-ICT-PSP-2012-6 – 325161
brown node on the right side, at the bottom, that is a virtual machine with City of Ferrara’s
software components that is hosted at Sinergis’ premises1
.
The server components are on the right portion of the diagram: here the red ones represent
the security layer that will be described more in detail in the D4.6 Annex on Security Layer.
The layer doesn’t add any additional functionality, but intercepts the requests that are sent to
services that need to be protected. Even the reverse proxies defined inside the web server
are in red, because this way to expose services on the internet adds also a little degree of
protection besides other advantages.
1
Physically the VM is located in the same LAN of the Sunshine’s platform, but the interactions with its
components are made via web protocols so it can be deployed to the pilot’s premises without any
problem.

CIP-ICT-PSP-2012-6 – 325161
Figure 1 - Overall deployment diagram

D4.5 Meter Data Management service #2
CIP-ICT-PSP-2012-6 – 325161
3 Hardware components
Figure 2 - Physical and virtual server
The SUNSHINE specific servers are deployed on a virtualized IT infrastructure based on
VMware vSphere software (the “Back end VM Server” in Figure 2). The underlying hardware
has the following features:
CPU Architecture x86_64
CPU Number 2 (6 core each)
CPU Clock 2300 MHz
Hard Drive Space 12TB
As SGIS plays in SUNSHINE also the role of technical partner of the Ferrara pilot, the same
infrastructure hosts as well the pilot’s server (sunshine-fe.nco.inet) used to collect the meter
data from the various buildings and to transmit them to the SUNSHINE platform. The Ferrara’s
server shares only the virtualization infrastructure and the internet domain (sinergis.it) with
the central platform, but all the communications between the SUNSHINE platform and the
pilot’s server passes through internet protocols.
Other servers that are functional to the SUNSHINE platform, but play generic roles (firewall,
web server, ftp server) are based on separate hardware in order to achieve a correct and
secure network topology and to fit in the existent SGIS internet infrastructure. In particular we

CIP-ICT-PSP-2012-6 – 325161
have the FTP server (ftp-vm) that is a VM hosted in another virtualization infrastructure
located in the DMZ. Then we have the web server (coreweb2) that is a physical server located
in the DMZ and finally the firewall. More details are given in the following paragraphs.
In the diagram of Figure 2 also a sunshine-revdb.nco.inet virtual server is shown, but its role is
descripted in D4.3.
3.1 Main platform servers
The main components are deployed on the backend server that is positioned in the internal
LAN in SGIS’ premises in Casalecchio di Reno, Bologna; the others server involved are:
 the FTP server, that is located in a separate VM server positioned in the DMZ for
security reasons;
 the web server is also a physically separated server; this server is used by SGIS
company not only for the SUNSHINE project, but for all the other company’s project
and customers.
 the firewall; also this server is a company’s asset that is functional to all projects and
that is used to guarantee security for the server that are exposed on the internet.
3.1.1 Backend server
Operating System
Ubuntu 12.04.3 LTS 64bit
Host Name
IP address
3.1.2 FTP server
Operating System
Ubuntu 12.04.3 LTS 64bit
Host Name
IP address
DNS name
3.1.3 Webserver server
Operating System
CentOS 6.4 64bit
Host Name
IP address

CIP-ICT-PSP-2012-6 – 325161
3.1.4 Firewall server
Operating System
Slackware 12.0 64bit
Host Name
IP address

CIP-ICT-PSP-2012-6 – 325161
4 Software components
The components are described in details in the sections of this chapter and their mutual
relation is described in the figure below (in green).
Figure 3 - The meter Data Management service components
4.1 Sensor DB (52° North SOS 4.0)
It is the final repository of all sensor data. A database implementation of OGC Sensor
Observation Service2
(SOS) standard protocol developed by 52° North3
has been installed and
configured on Sunshine’s platform for this purpose. It is implemented as a PostgreSQL DB and
also makes use of the spatial extension PostGIS.
2
http://www.opengeospatial.org/standards/sos
3
http://52north.org/communities/sensorweb/sos/index.html

CIP-ICT-PSP-2012-6 – 325161
4.1.1 Database structure
Annex I of this document reports the UML schema of the sensor DB. The main constructs of
the DB schema are the following:
 Codespace: univocally identifies the pilot;
 Feature of Interest: represents the geo-object who is measured by sensors. The FOI is
the means of locating (geocoding) the measuring points via their coordinates;
 Observable Property: identifies the property observed by the sensor (e.g. gas
consumption, temperature or lamp dimming level);
 Procedure: identifies a physical /virtual sensor which produces the observations
associated with the offerings;
 Offering: a logical grouping of observations that are related to each other, which are
jointly offered by a service.
The following sub-sections describe the uniform and consistent naming convention that has
been defined for these constructs.
Codespace
CodeSpaceUri (examples)
CodeSpace Pilot
http://www.sunshineproject.eu/swe/project-pilot/ROV ROV Rovereto
http://www.sunshineproject.eu/swe/project-pilot/FER FER Ferrara
http://www.sunshineproject.eu/swe/project-pilot/BAS BAS Bassano
http://www.sunshineproject.eu/swe/project-pilot/LAM LAM Lamia
http://www.sunshineproject.eu/swe/project-pilot/PAO PAO Paola
http://www.sunshineproject.eu/swe/project-pilot/TRN TRN Trentino
http://www.sunshineproject.eu/swe/project-pilot/VDN VDN Val di Non
http://www.sunshineproject.eu/swe/project-pilot/HRV HRV Croatia
CodeSpaceUri: <radix>/CodeSpace
Feature of Interest
FeatureofinterestUri (examples)
http://www.sunshineproject.eu/swe/featureOfInterest/FER/building/ScuolaACosta
http://www.sunshineproject.eu/swe/featureOfInterest/FER/building/PalazzinaMarfisaDEste
http://www.sunshineproject.eu/swe/featureOfInterest/HRV/weatherstation/06
http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lamp/101
http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lamp/102
http://www.sunshineproject.eu/swe/featureOfInterest/BAS/lightline/PiazzaGuadagnin
http://www.sunshineproject.eu/swe/featureOfInterest/LAM/building/SchoolTechApplications2
http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lightline/RotondaMarco

CIP-ICT-PSP-2012-6 – 325161
FeatureofinterestUri: <radix>/CodeSpace/<natura_della_Feature>/<Identificativo>
FoI nature
Building
Lightline
Lamp
Weatherstation
Observable property
ObservablepropertiesUri (examples)
http://www.sunshineproject.eu/swe/observableProperty/ELEC
http://www.sunshineproject.eu/swe/observableProperty/TEMP
http://www.sunshineproject.eu/swe/observableProperty/IPOW
http://www.sunshineproject.eu/swe/observableProperty/DIMM
http://www.sunshineproject.eu/swe/observableProperty/LIFE
http://www.sunshineproject.eu/swe/observableProperty/IRRA
http://www.sunshineproject.eu/swe/observableProperty/GASR
ProcedureUri: <radix>/<CodeProperty>
CodeProperty Description
ELER Electrical Energy Consumption, relative
GASR Gaseous Fuel Consumption, relative
THER Thermal Energy Consumption, relative
LIQR Liquid Fuel Consumption, relative
SLDR Solid Fuel Consumption, relative
STMR Steam Fuel consumption, relative
ELEC Electrical Energy Cost
GASC Gaseous Fuel Cost
THEC Thermal Energy Cost
LIQC Liquid Fuel Cost
SLDC Solid Fuel Cost
STMC Steam Fuel Cost
ELEA Electrical Energy Consumption, absolute
GASA Gaseous Fuel Consumption, absolute
THEA Thermal Energy Consumption, absolute
IPOW Lamp instantaneous power
DIMM Lamp dimming
LIFE Lightline usage hours

CIP-ICT-PSP-2012-6 – 325161
ONOF Lamp CB Status
NALL Lightline CB Number
N-OK Lightline CB Accreditate
SUNR Lightline sunrise time
SUNS Lightline sunset time
ICUR Lamp current
COSF Lamp Cos(phi)
IVOL Lamp voltage
SCDL Lamp scheduler
ACNT Lamp accounting
TEMP Temperature
WIND Wind Speed Module
IRRA Short Wave Irradiation
Procedure
ProcedureUri (examples)
http://www.sunshineproject.eu/swe/procedure/ROV-100
http://www.sunshineproject.eu/swe/procedure/VDN-F00
http://www.sunshineproject.eu/swe/procedure/BAS-F00
http://www.sunshineproject.eu/swe/procedure/TRN-M01
http://www.sunshineproject.eu/swe/procedure/TRN-T01
ProcedureUri: <radix>/<codespace>-<alphaNumeric>
alphaNumeric:
 Building/shelter consumption data: numeric between 000 and 999
 Lamp/line consumption and state data:
o Hundreds identify the line
o Tens and units identify lamps belonging to the same line
 Weather Forecast data: F<n> with n ∈ [0..99]
 Weather Meteorological conditions: M<n> with n ∈ [0..99]
 Indoor Temperature data: T<n> with n ∈ [0..99]
Offering
Offering (examples)
http://www.sunshineproject.eu/swe/offering/ROV-F01_TEMP_CEL_1h
http://www.sunshineproject.eu/swe/offering/LAM-M01_TEMP_CEL_1h
http://www.sunshineproject.eu/swe/offering/HRV-001_ELER_KWH_1h

CIP-ICT-PSP-2012-6 – 325161
http://www.sunshineproject.eu/swe/offering/FER-001_GASR_MCU_1d
OfferingUri: <radix>/CodeSpace-
<alphaNumeric>_<codeProperty>_<uom>_<frequenza_di_campionamento>
Sampling frequency Description
10 Every 10 minutes
15 Every 15 minutes
1h Hourly
1d Daily
1m Monthly
ir Irregular frequency
Unit of measure Description
KWH "kWh"
CEL "°C"
MCU "m3"
EUR "Euro"
WAT "W"
PRC "%"
HRS "hours"
BOO "0/1"
NUM "elements"
HMS "date"
AMP "Ampere
CDL "codelist"
KWA "kW"
KMH "m/s"
WM2 "W/m2"
4.1.2 Database management
The management of the database structure have been performed via:
 The use of the SOS APIs for the creation/deletion of new procedures (insertSensor,
deleteSensor);
 The use of custom SQL scripts for the creation/deletion of new offerings. Scripts take
care to modify all the entries related to offerings in the various DB tables.
The creation of a new offering implies the following actions on the DB:

CIP-ICT-PSP-2012-6 – 325161
DB TABLE Action
offering Create entry with identifier, e.g. “LAM-
M01_TEMP_CEL_1h”.
observableproperty Verify entry existence for “Observable Property” of the
offering, e.g. “TEMP”.
unit Verify entry existence for "Unit" of the offering, e.g.
“CEL”.
featureofinterest Verify entry existence for “Feature of Interest” of the
offering, specifying:
 featureofinteresttypeid among options in table
featureofinteresttype
 codespaceid
series Insert entry with
 featureofinterestid
 observablepropertyid
 procedureid
observationconstellation Insert entry with
 observablepropertyid
 procedureid
 observationtypeid
 offeringid
offeringallowedobservationtype Insert entry with
 offeringid
 observationtypeid
offeringallowedfeaturetype Insert entry with
 offeringid
 featureofinteresttypeid
relatedfeature Verify entry existence with
 Featureofinterestid
offeringhasrelatedfeature Insert entry with
 relatedfeatureid
 offeringid

CIP-ICT-PSP-2012-6 – 325161
DB management view
The DB management view is a fundamental tool for the set-up and maintenance of the DB
structure and for the monitoring of the correct flowing of data into the DB. At runtime the
Sensor DB has hundreds of offerings and a manual check on the correctness of all offerings
would be unfeasible. As Figure 4 shows, the view has one entry per offering and for each entry
it specifies:
 Feature of Interest identifier
 Procedure identifier
 Offering identifier
 Observable Property identifier
 Unit of measure
 Reading Frequency
 Observations count for the offering
 Timestamp of first observation
 Timestamp of last observation
Figure 4 The Db management view
4.2 FTP ingestion
It’s a scheduled program that checks if any CSV file to be loaded into the database is arrived
on the dedicated FTP folders. The files are removed from the folders and passed to the CSV
loader component that does the real job of inserting data in the DB.
Detailed guidelines have been prepared describing how to format name and content of CSV
files in order to allow the automatic ingestion procedure to take place smoothly.
Guidelines are available in the deliverable D2.4 Annex “Specifications_for_data_ingestion_via
sunshine_FTP”.

CIP-ICT-PSP-2012-6 – 325161
In order to configure the service, pilots compiled a mapping file that has the purpose of
describing the basic properties of the meter/sensor and allows to identify which pilot building
or light line does it belong to. An example of the content of a meter mapping file for the pilot
in Lamia is shown in Figure 5.
Figure 5 – Meter mapping file
4.3 Meter data processing
Some elaborations are needed before the real data ingestion. For instance, some aggregations
are pre-calculated in order to avoid this task to the client. Another kind of processing is the
interpolation for those readings that have been collected at irregular intervals of time (some
kinds of data flows have this characteristic).
This module is specific to the consumption data from sensor reading and its role is that of
guaranteeing a uniform format for its storage and therefore a cleaner server/client
interaction. Stored consumption data has the following format:
 Consumption is relative to the last reading interval;
 Readings have a regular frequency;
 Available frequencies are: every 15 minutes (10 minutes for TNET), hourly and daily.
These constraints do not apply to consumption data coming from billing documentation that
are stored in the BD as is.
In order to guarantee the above-mentioned format, three main processing tasks that have
been developed:
 Data resampler, from irregular frequency to regular frequency;
 Differential consumption, from absolute readings to relative readings (keeping track
of the last known absolute reading);
 Data aggregator, from high frequency regular relative readings to low frequency
regular relative readings (e.g. from 15 minutes frequency to hourly and daily).

CIP-ICT-PSP-2012-6 – 325161
4.4 CSV loader
This component accesses the DB for inserting the readings in the proper tables. Before doing
that, if it has been configured to do so, the files are pre-processed by the “Meter data
processing” component.
Readings are inserted in the DB using SOS insertion API, insertObservation.
What follows is an example of an insertObservation call:
<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope
xmlns:env="http://www.w3.org/2003/05/soap-envelope"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2003/05/soap-envelope
http://www.w3.org/2003/05/soap-envelope/soap-envelope.xsd">
<env:Body>
<sos:InsertObservation service="SOS" version="2.0.0"
xmlns:sos="http://www.opengis.net/sos/2.0"
xmlns:swes="http://www.opengis.net/swes/2.0"
xmlns:swe="http://www.opengis.net/swe/2.0"
xmlns:sml="http://www.opengis.net/sensorML/1.0.1"
xmlns:gml="http://www.opengis.net/gml/3.2"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:om="http://www.opengis.net/om/2.0"
xmlns:sams="http://www.opengis.net/samplingSpatial/2.0"
xmlns:sf="http://www.opengis.net/sampling/2.0"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xsi:schemaLocation="http://www.opengis.net/sos/2.0
http://schemas.opengis.net/sos/2.0/sos.xsd
http://www.opengis.net/samplingSpatial/2.0
http://schemas.opengis.net/samplingSpatial/2.0/spatialSamplingFeature.xsd">
<sos:offering>http://www.sunshineproject.eu/swe/offering/ROV-
213_ICUR_AMP_ir</sos:offering>
<sos:observation>
<om:OM_Observation gml:id="o1">
<gml:identifier
codeSpace="http://www.sunshineproject.eu/swe/project-
pilot/ROV">http://www.sunshineproject.eu/swe/observation/ROV-213_ICUR_AMP_ir/2015-03-11
11:53:24</gml:identifier>
<om:type
xlink:href="http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_Measurement"/>
<om:phenomenonTime>
<gml:TimePeriod gml:id="Ki.ObsTime.1">
<gml:beginPosition>2015-03-
11T04:50:47.000+01:00</gml:beginPosition>
<gml:endPosition>2015-03-11T05:50:47.000+01:00</gml:endPosition>
</gml:TimePeriod>
</om:phenomenonTime>
<om:resultTime>
<gml:TimeInstant gml:id="Ki.resTime.1">
<gml:timePosition>2015-03-
11T05:50:47.000+01:00</gml:timePosition>
</gml:TimeInstant>
</om:resultTime>
<om:procedure
xlink:href="http://www.sunshineproject.eu/swe/procedure/ROV-213"/>
<om:observedProperty
xlink:href="http://www.sunshineproject.eu/swe/observableProperty/ICUR"/>
<om:featureOfInterest
xlink:href="http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lamp/213"/>

CIP-ICT-PSP-2012-6 – 325161
<om:result xsi:type="gml:MeasureType" uom="A">29</om:result>
</om:OM_Observation>
</sos:observation>
</sos:InsertObservation>
</env:Body>
</env:Envelope>
4.5 Weather Data ingestion
Weather data are served by two WFS services set up by partner Meteogrid from a
dedicated platform described in detail in Chapter 5. The ingestion process queries the
WFS services, collects the data for each meteorological quantity (temperature,
irradiation, wind speed), and loads it into the Sensor DB. More in details here are the
WFS services:
 WFS forecast
A three-day forecast with hourly resolution is available daily for each pilot’s location.
 WFS observation
Measured or interpolated weather conditions are available with hourly resolution for
each pilot’s location.
From both WFSs the feature types that are read and ingested to the SOS are:
1. temperature_s: temperature in °C (Celsius Degrees)
2. wind_speed_s: wind module in m/s
3. short_wave_irradiation_s: short wave irradiation in W/m2
Each feature type has the following properties, as can be seen in Figure 6:
 msGeometry (gml:Point)
 name: name of observatory (example: ZAGREB165)
 timestamp (data are available with hourly frequency)
 value
 type
 observatory_id
<gml:featureMember>
<sigym:temperature_s fid="temperatures_2015-03-13
01:00:00+01:00,temperatura_obs_int,165">
<sigym:msGeometry>
<gml:Point srsName="EPSG:4326">
<gml:coordinates>15.965225,45.807468999999998</gml:coordinates>
</gml:Point>
</sigym:msGeometry>
<sigym:name>ZAGREB165</sigym:name>
<sigym:timestamp>2015-03-13T01:00:00+01:00</sigym:timestamp>
<sigym:value>6.0</sigym:value>
<sigym:mode/>
<sigym:type>temperatura_obs_int</sigym:type>

CIP-ICT-PSP-2012-6 – 325161
<sigym:observatory_id>165</sigym:observatory_id>
</sigym:temperature_s>
</gml:featureMember>
Figure 6 - Feature example of temperature_s from observation WFS
Each feature type of both WFS (forecast and observation) has 83 feature members
representing the observatories. The observatory name and id are the same in both WFSs.
Annex II describes the mapping between feature members of the WFS with procedures of the
SOS.
The mapping between WFS and SOS has the following main characteristics:
 a feature member of the WFS (observatory) correspond in the SOS to:
o one FeatureOfInterest;
o two Procedures (one for the forecast and one for the observation);
o six Offerings; three offering (temperature, wind, irradiation) for forecast and
for observation.
 a value of the WFS correspond to an Observation in the SOS
The Procedure name in the SOS is: <pilot>-F/M<nn> where F=forecast and M=measure
(observation)
The service that reads the WFS and ingest the data to SOS:
 is scheduled once a day at 10:10:00 AM
 the service makes 3 GetFeature calls to the Forecast WFS and 3 GetFeature calls to
the Observation WFS, one call for each feature type (temperature_s, wind_speed_s,
short_wave_irradiation_s);
 the GetFeature call response for the Forecast WFS returns the information for the
next 3 days (72 hours) for the 83 observatories. The service deletes the old forecast
and ingests the values into the SOS.
 for the Observation WFS the service calls a GetFeature using the TIME parameter to
specify a range covering one day (from the day before of current date of scheduling
to the current scheduling). The response to the GetFeature call returns the
information for 24 hours for each of the 83 observatories; the values are ingested
into the SOS (and no deletion of old values is done).
The ingestion module refers to a configuration file (importMeteoDataConfiguration.xml) that
allows configuring:
 Ingestion scheduling (hour, minute, second)
 URL of WFSs
 WFS Featuretype names
 Mapping between “WFS observatory id” and “SOS procedure code”

CIP-ICT-PSP-2012-6 – 325161
 CodeSpace Identifier of the Pilot
 FeatureOfInterest Identifier
4.6 GreenButton ingestion
A Smart Energy Grid has many different nodes (generation facility, distribution network, utility
company data storage facility, smart meters, local data concentrator, etc) and actors (energy
producer, energy deliverer, energy seller, energy customer, service provider company, etc)
and consequently many different types of communication channels, each with its own set of
possible communication protocols.
In the context of the Sunshine project, the pilot partner has both the role of energy user and
the role of data storage facility, while the Sunshine platform is the service provider company
that needs access to the consumption data stored in the storage facility in order to provide
value-adding services. This is exactly the use case typology that is covered by the GreenButton
protocol4
, the international data exchange standard for consumption data that have been
chosen to be used in SUNSHINE’s platform. The Sunshine platform does not communicate
directly with the smart meters of the pilots and is therefore not implementing any protocol
related to direct communication with smart meters.
Green Button is an initiative of the U.S. Government that aims to allow consumers to access
their own meter data about energy consumption. In the original context, the name Green
Button denotes the whole initiative, but in our project we will call GB only the suite of web
services that makes the data access possible. GreenButton protocol and API were identified as
the best option to implement the use case described above for several reasons:
 It is already in operational use;
 It is supplied with examples and demos;
 It is a standard aiming at becoming international5
So GreenButton provides Sunshine with a protocol already implemented and operational in a
real use case scenario.
This module consumes a REST service that returns an XML that is compliant with the AtomPub
standard that is the underlying standard of the Green Button. Concerning the handling of
authentication and authorization in the data communication process between the pilots and
the Sunshine platform, GreenButton protocol relies on OAuth protocol which has a
widespread use for the use cases we are concerned with (three-partner authorisation
between data owner, data custodian, data user). For the sake of truth, we must highlight that
there is not a real “service endpoint” for this kind of flow. As shown in Figure 7, the meter
data producers actually acts like server, publishing their readings and the SUNSHINE platform
4
http://energy.gov/data/green-button
5
http://www.sgip.org/PAP-20-Green-Button-ESPI-Evolution

CIP-ICT-PSP-2012-6 – 325161
acts like a consumer of the feeds. The roles are therefore switched and the GreenButton
ingestion service must be configured with the address of the pilot’s feed to read.
Detailed guidelines have been prepared describing the GreenButton standard, its security and
authentication module based on OAuth protocol6
and providing instruction on how to
implement a web service supporting the protocol. Guidelines are available in deliverable
annex “D4.5 Annex - Specifications_for_data_ingestion_via_GreenButton_v1.2”.
Figure 7 - Consuming a GreenButton service
Support was given via Skype telcos and mail exchange to pilots that decided to implement a
web service based on GreenButton protocol to expose consumption data. The most relevant
issues were related to the securization of the infrastructure:
 exchange of OAuth security token between pilot and central infrastructure;
 activation of HTTPS secure protocol;
6
http://oauth.net/

CIP-ICT-PSP-2012-6 – 325161
Moreover, a mapping was established for each pilot between the data structure of the
GreenButton DB residing in the pilot infrastructure and the data structure of the central
Sensor DB. This mapping was requested in the guidelines disseminated to pilots, where also a
template for the mapping itself was provided. Figure 8 shows a compiled example of such
mapping table.
Figure 8 – Mapping between GreenButton DB structures (usage point, meter reading,
reading type) and Sensor DB (meter identifier, last column).
4.7 Sensor data access
The last module, in logical order, is the sensor data access. Access to sensor data is
guaranteed by two services:
 Web feature service (see Figure 9):
o To publish the identifiers of the different sensors and data flows via the
linking with their geographical position.
 Sensor observation service APIs:
o GetCapabilities: returns an XML service description with information about
the interface (offered operations and endpoints) as well as the available
sensor data, such as the period for which sensor data is available, sensors that
produce the measured values, or phenomena that are observed;

CIP-ICT-PSP-2012-6 – 325161
o DescribeSensor: provides sensor metadata in SensorML. The sensor
description can contain information about the sensor in general, the identifier
and classification, position and observed phenomena, but also details such as
calibration data;
o GetObservation: allows pull-based querying of observed values, including
their metadata. The measured values and their metadata is returned in the
Observations and Measurements format (O & M);
o GetResult: provides the ability to query for sensor readings without the
metadata given consistent metadata (e.g. sensor, observed object).
Figure 9 – Sensor locations (blue dots) and identifiers (attributes in the table)
exposed via a Web Feature Service.

CIP-ICT-PSP-2012-6 – 325161
5 Sunshine Weather Platform
This chapter describes the platform implemented at Meteogrid premises that is responsible
for the provision of weather data, both observation and localized forecasts, to the central
SUSNHINE’s platform for all pilot locations.
5.1 Platform architecture
In order to support SUNSHINE’s platform services Meteogrid’s team has designed and set up
an IT platform that can generate specific weather forecasts for the SUNSHINE pilot areas,
collect information from NOAA, ECMWF and local weather centres and compute a specific
forecast downscaling for each building.
This platform consists of two dedicated servers
 MeteoGrid Weather Forecast server;
 Sigym3 MeteoServer.
MeteoGrid Weather Forecast server houses a WRF implementation. WRF - Weather Research
and Forecasting model is a numerical weather prediction system customized in much of the
public and private meteorological services worldwide.
Below major Sunshine customization implementation details are offered:
 Latitudes: 34.77º - 51.59º with resolution of 11.7 km.
 Longitudes: 3.12º - 30.08º with resolution of 11.7 km
 Temporal resolution output: 1 hour
 Temporal scope: 72 hours
 Output variables: T2, Q2, PSFC, ACSWDNB, ACLWDNB, RAINNC, rainc, U10, V10,
CLDFRA, CLT, CLL
 settings:
- Micro-physics: WRF Single-moment three-class and five-class Schemes; Hong, Song-
You, Jimy Dudhia, and Shu-Hua Chen, 2004: A revised approach to ice microphysical
Processes for the bulk parameterization of clouds and precipitation. Mon. Wea. Rev.,
132, 103-120.
- Cumulus parameterization: Kain-Fritsch Scheme; Kain, John S., 2004: The Kain-Fritsch
convective parameterization: An update. J. Appl. Meteor., 43, 170-181.
- Shortwave and Longwave: Shortwave and Longwave CAM Schemes; Collins, William
D., et al., 2004: Description of the NCAR Community Atmosphere Model (CAM 3.0).
NCAR Tech. Note NCAR / TN-464 + STR. 214 pp.
 Urban Surface: Urban Canopy Model

CIP-ICT-PSP-2012-6 – 325161
Sigym3 MeteoServer integrates a reinterpretation service of weather forecasting based on
precision procedures and statistical downscaling algorithms; a set of algorithms and models
for the generation of derived variables of interest; and a GIS platform that integrates
meteorological data, derived variables and other spatial data, considering and managing the
time axis.
In addition to the necessary adaptations to meet the needs of the project, specific services for
collection and presentation of the information managed by the system have been developed:
 the Weather Data Collecting Service
 the Weather Data Exhibition Service
5.1.1 Weather Data Collecting Service
In brief, the purpose of the Weather Data Collecting Service is to:
 Gather local observations collected from local weather services or building
meteorological stations;
 Gather forecast data from NOAA, ECMWF and MeteoGrid Weather Forecast
dedicated server;
 Elaborate them and generate derived data, geo-interpolation processes plus unit
conversion (different weather services use different units for meteorological
variables);
 Store them in a normalized format into a repository from which they can be retrieved
through a standard interface (WFS).
The flow of ECMWF and MeteoGrid forecast data are channelled through an FTP Ingestion
service. Data are received twice a day to update forecast information.
The flow of NOAA forecast data and METAR observations data are channelled through the FTP
Ingestion service. Data are gathered via a scheduling procedure that pulls data twice a day to
update forecast information and meteorological observations inside the Weather Platform.
The flow of local weather observation data is managed by the Web Scraper Ingestion and the
FTP Ingestion services that ensure the ingestion of such data every hour.
5.1.2 Weather Data Exhibition Service - WMS – WFS
Collected and processed data is exposed through WMS and WFS services. The data flows
towards Sunshine’s platform via the WFS service. SGIS has set up a scheduled procedure to
pull data daily from the dedicated Web Feature Services (WFS).

CIP-ICT-PSP-2012-6 – 325161
Sunshine-Weather specific servers are deployed on a virtualized IT infrastructure based on
KVM allowing a flexible architecture. The underlying hardware has the following features:
Sigym3 MeteoServer
CPU Architecture x86_64
CPU Number 2 (8 cores)
CPU Clock 2300 MHz
Hard Drive Space 2x2TB (RAID 1)
MeteoGrid Weather Forecast server
CPU Architecture AMD ABU DHABI
CPU Number 2 (16 cores each)
CPU Clock 2300 MHz
Hard Drive Space 8TB
5.2 Software components
All the software used in this platform is free Open-source software (OSS). All the used free
software is widely tested and used in all the software community, throughout the whole
lifecycle of the software projects, it is worldwide tested and it is used in all types of projects,
from small to large scale and, also, from small to large companies.
5.2.1 Core modules
Operating System: Ubuntu (http://www.ubuntu.com/).
Ubuntu is a Debian-based Linux operating system. Ubuntu releases updated versions
predictably – every six months – and each release receives free support for nine months
(eighteen months prior to 13.04) with security fixes, high-impact bug fixes and conservative,
substantially beneficial low-risk bug fixes.
Data Base: PostGIS (http://postgis.net/).
PostGIS is a spatial database extender for PostgreSQL object-relational database. It adds
support for geographic objects allowing location queries to be run in SQL.
Programming Language: Python (https://www.python.org/).
Python is a widely used general-purpose, high-level programming language. Its design
philosophy emphasizes code readability, and its syntax allows programmers to express
concepts in a few lines of code. The language provides constructs intended to enable clear
programs on both a small and large scale.
Apache HTTP Server (https://httpd.apache.org/).

CIP-ICT-PSP-2012-6 – 325161
Apache HTTP Server provides a secure, efficient and extensible server that provides HTTP
services in sync with the current HTTP standards.
NGINX (http://nginx.org/).
NGINX is an open source reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP
protocols, as well as a load balancer, HTTP cache, and a web server (origin server).
5.2.2 Zope Object Database
Zope Object Database (ZODB) (http://www.zodb.org/en/latest/).
ZOPE is an object-oriented database for transparently and persistently storing Python objects.
Features of the ZODB include: transactions, history/undo, transparently pluggable storage,
built-in caching, multiversion concurrency control (MVCC) and scalability across a network.
Pyramid (http://www.pylonsproject.org/projects/pyramid/about).
Pyramid is a very general open source Python web framework. Pyramid is a minimalistic,
platform-independent object publishing web framework. It also has integration with the Zope
Object Database (ZODB).
Main objects on the ZODB used for meteorological forecasts and observation gathering are
shown below.
Figure 10 – Main objects on the Zope Object database.
GEO
Projections
Bounds
Resolution
Types
Periodicity
Actual Data
- Raster
- Features
Day 1
Day 2
Day n
STORES SOURCESCONTEXTS

CIP-ICT-PSP-2012-6 – 325161
5.2.3 Data ingestion in Sigym3
Below is a detailed diagram of the FTP ingestion services and the meteorological server
modules.
Figure 11 – Data ingestion in Sigym3.
FTPRelayer, developed by Meteogrid (https://github.com/meteogrid/FTPRelayer), is a
component based on Linux’s inotify that redistributes files which arrive at a machine to
several machines, optionally processing them before sending them. It allows passive waiting
for incoming files, avoiding polling. It is freely available in the provided URL.
5.2.4 Data Access
Web Map Service (WMS) is a standard protocol for serving georeferenced map images over
the Internet that are generated by a map server using data from a GIS database.
Web Feature Service (WFS) provides an interface allowing requests for geographical features
across the web using platform-independent calls.
Proftpd
FtPRelayer
Mailprocessor
Collector
Genserver
Webserver
WMS
WFS
PostGIS
ZODB

CIP-ICT-PSP-2012-6 – 325161
Annex I - Sensor DB schema (52° North SOS 4.0)

CIP-ICT-PSP-2012-6 – 325161
Annex II – Mapping weather WFS to Sensor DB
ID WFS Name WFS Procedure SOS notes
86 CLES (VAL DI NON) VDN-F00
87 BASSANO DEL GRAPPA BAS-F00
88 PAOLA PAO-F00
89 SCHWECHAT no more in the
project
90 FERRARA FER-F00
91 ROVERETO91 ROV-F00
92 ROVERETO92 ROV-F01
93 TRENTINO93 TRN-F00

CIP-ICT-PSP-2012-6 – 325161
154 LAMIA154 LAM-F00
159 ZAGREB159 HRV-F00
163 RIJEKA HRV-F04 no buildings
166 SPLIT HRV-F07 no buildings
167 VARAZDIN HRV-F08
168 KRIZ HRV-F09

S.2.h Meter Data Management Service

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to S.2.h Meter Data Management Service

Similar to S.2.h Meter Data Management Service (20)

More from SUNSHINEProject

More from SUNSHINEProject (7)

Recently uploaded

Recently uploaded (20)

S.2.h Meter Data Management Service