EMPOWERING_D6.3

Project co-funded by the European Commission within the IEE Programme
Dissemination Level
PU Public x
CO Consortium and EACI
The sole responsibility for the content of this deliverable lies with the authors. It does not necessarily reflect the opinion of the European Union.
Neither the EACI nor the European Commission are responsible for any use that may be made of the information contained therein.
EMPOWERING customers to save
energy by informative billing
Deliverable 6.3
Publishable report on the evaluation of
EMPOWERING outcomes
Project No. IEE/12/697/S12.645843

EMPOWERING: D6.3 Publishable report on the evaluation of EMPOWERING outcomes
2
Authors
Name Organisation
Michele Pastorelli, Mariapia Martino POLITO
Giovanni Pede SINERGIE
Revision history
Revision Date Author Organisation Description
0.1 21/12/2015 Giovanni Pede SINERGIE
Summary of customer
satisfaction surveys
Initial template and customer
satisfaction chapters
0.2 18/01/2016
Marina Pirazzi,
Vittorio Gimigliano
REGGIO EMILIA
Insertion of results from the two
meetings with households held in
Reggio Emilia
Insertion of customer satisfaction
infographics
0.4 14/02/2016 Mariapia Martino POLITO
Introduction to the documents,
comments to the graphs and
conclusions

3
Table of contents
1. Introduction...........................................................................................................................4
2. Procedure for evaluation of energy saving in EMPOWERING...................................................5
2.1 Evaluation of energy savings in Empowering.............................................................................5
2.2 Evaluation of the impacts on customer satisfaction with EMPOWERING.................................7
3. Evaluation of results in each pilot site.....................................................................................9
3.1. Austria .......................................................................................................................................9
3.2. France......................................................................................................................................16
3.3. Italy..........................................................................................................................................30
3.4. Spain........................................................................................................................................41
4. Conclusions ..........................................................................................................................50
5. Bibliography.........................................................................................................................54

4
1. Introduction
This report is compiled in the frame of EMPOWERING project, work package WP6, task T6.3 “Evaluation
and performance monitoring”.
The overall objective of the evaluation work is to develop approaches that can be used for an impact
analysis of the ICT tools and solution adopted in the five pilot sites: Grenoble-France, Linz-Austria, Soller-
Spain, Torino and Reggio Emilia-Italy.
WP6 is based on the previous work packages where requirements have been defined and services have
been developed.
Fig. 1 – Simplified flow chart of major activities in EMPOWERING project.
The D6.3 represents the publishable report on the evaluation of EMPOWERING outcomes performed at
each pilot site. It will provide quantification of their impact over energy consumption and the changes in
user behaviour and satisfaction with the utilities.
The activities of “widespread service application” and “continuous evaluation of the service perception,
preference and satisfaction of customers”, are performed taking into account the methodology defined in
previous deliverables.
The evaluation procedure is carried out in compliance to the requirements defined in the annex I to the
Grant Agreement.
In the deliverable are analysed the results up to the month of December 2015, with the aim to have an
evaluation of the project results, based on the comparative behaviour of experimental and control groups.
After December 2015 four pilots (Linz, Soller, Torino and Reggio Emilia) have continued the services, with
the aim to extend EMPOWERING services to all the customers, this taking into account the specific
regulatory frameworks in each site.
The comparative analysis of different energy savings in the pilot site is carried out and general
recommendation and comments are reported in the conclusion.

5
2. Procedure for evaluation of energy saving in
EMPOWERING
Energy consumption in residential environment depends on both social and weather condition that are not
easy to model and predict.
The general approach to solve this problem is the definition of a control and experimental group, located in
the same site, where the treatment is provided to the experimental group only.
The different behavior of the two groups in the same environment is then compared, this in order to isolate
the effect of the treatment (EMPOWERING services) among the influence of several disturbances that are
observable with difficulties (weather conditions, social constraints).
2.1 Evaluation of energy savings in Empowering
As reported in D6.1, the method for the evaluation of energy saving is based on the difference in difference
algorithm. This approach is based on comparing changes in energy consumptions, at given time samples,
for the experimental and control group of users involved in EMPOWERING project.
The experimental group is composed by a group of users that accept (by opt-in choice or non-exercising an
opt-out choice) to be engaged in EMPOWERING services, while the control group consists of user with
similar characteristics but not receiving the services.
The energy consumption of the both groups are analyzed:
- prior of the start of services in order to define the baseline values;
- during the project to allow a preliminary and ongoing evaluation;
- at the end of the project for final evaluation.
Fig. 2 – General structure for the creation of the control and experimental groups.

6
The consumption data are collected by the utilities and transferred to CIMNE for processing.
The values of the daily consumption for both electrical and gas measurements are stored and processed.
Due to the large amount of data processed it is not possible to avoid outlier values and measurement
errors, thus a data cleaning algorithm is mandatory.
The data cleaning procedure is essentially based on:
- identification and removal of the quantiles related to the lower and upper values that are ties
around the core values, that really affect the mean value.
This algorithm allows the cancellation of all the measurements close to zero (i.e. users that
interrupted the consumption during the project following to transfers, relocation and/or
termination of activities) and measurement related to a reduced number of users with
consumption much larger than the average. This removal process is carried out by choosing the
values of the lower and upper quantiles to be erased from the process.
- Identification of consumers that show high differences for energy consumption acquired in the
same month and different year, this in order to take into account the different weather conditions
measurement. Energy consumption related to a specific user are processed if all the consumptions
related to a specific month are related to the minim consumption by the formula:
|ADCmax – ADCmin|/ ADCmin <= Factor
Factor = -8,5*Restrictor + 10
where:
ADC ism the average daily consumption computed on monthly base
restrictor is an input parameter, chosen by the evaluator during data processing, among a list of
available values.
factor is the maximum allowed p.u. variation
The values of the available restrictor values and the correspondent computed factor values are
reported in Table 1
Table 1 – Available restrictor parameters and correspondent factor values.
Input Parameter: Restrictor Output parameter: Factor
1 1,5
0,95 1,925
0,9 2,35
0,8 3,2
0,75 3,625
0,5 5,75
0 infinite

7
Fig. 3 - Design based on experimental and control groups.
Fig 4 - Data flow and computation.
2.2 Procedure for the evaluation of the customer satisfaction with
EMPOWERING
EMPOWERING services have been provided to customers in the five sites, with different characteristics and
offering different information according to the specific 'desiderata'.
In order to know the information requested by the customers in every site, surveys were carried out in the
initial phases of the project to collect the users' requirements in term of needed services, information, data
and data visualisation.
Those data have been used to define a preliminary version of the EMPOWERING billing/online tools that
were 'fine-tuned' after usability and intelligibility tests.

8
The preliminary versions of EMPOWERING services were used in the testing phase asking to end users and
utilities' internal staff to assess them in term of usability and intelligibility of information. An additional
testing activity has been implemented organising in March 2014 a Focus Group with experts on energy
management, infographics, sociology, graphics assessed all the services developed by all pilots. Usability
and Intelligibility questionnaires with the results from the Focus Group provided useful feedbacks to
develop the final version of the EMPOWERING services to be provided to the experimental group.
In the following chapters the results related to the impacts of EMPOWERING on utilities' customers are
inserted for each pilot.
The report is based on information provided by 1715 questionnaires collected during the whole project in
different phases, and addressed to different respondents such as customers, technicians and stakeholders.
22 key informants participate to 3 Focus Group, which provided a deep views on issues related to the
diffusion of EMPOWERING. The table below summarize the collected questionnaires and data used in the
present report.
Table 2 – Collected questionnaires and used data
Purposes Tot
Analysis of customers’ expectations for the design of the services 964
Stakeholders and external organisations involved 31
Usability assessment of services 96
Sociological analysis on end users’ behaviours in term of energy consumptions 27
Evaluation of the EMPOWERING services 542
Additional questionnaires on the evaluation of EMPOWERING collected from new
utilities
55
The information provided by the respondents is organised around the four customer service development
and improvement principles:
• Develop customer insight (investigate customer expectations).
• Design services around customers (incorporate the customer results in the services development).
• Respond to individual user needs within available resources.
• Support employees to provide excellent customer service.

9
3. Evaluation of results in each pilot site
3.1 Austria
In Austria the services are provided in the city of Linz (Upper Austria) by the utility LINZ AG. Through its
subsidiary companies Linz Strom and Linz Strom Netz, it is responsible for energy production, trading,
energy related services, telecommunications and electricity distribution. The utility supplies electricity to
approximately 242.000 customers.
LINZ AG developed the first version of its web portal and on line customer service for consumption
visualization and evaluation of CO2 emission in 2009 (projects eSESH and INTELIKON).
EMPOWERING services are offered as additional services to the already existing “Energie Informations
System” (EIS) through on line tools.
Fig. 5 - Definition of control and experimental groups in Linz.
Customers that want to access to EMPOWERING services have to voluntary subscribe to the services by a
two steps registration: first registration to EIS and additional registration to EMPOWERING
Fig. 5 shows the procedure for the definition of control and experimental group in Linz, the pilot customers
in Linz are customers already registered for EIS service, to access to EMPOWERING services they have to
additional opt-it by a voluntary choice. The availability of EMPOWERING services is shown in the EIS site
and visitors are encouraged to apply for EMPOWERING.
The open access to the services of all the customers by a voluntary registration make impossible to
randomly pre-define the control and experimental groups. The procedure adopted carries to the formation
of an experimental group composed by “always taker” and “compliance” users.
The services are on line services: the accesses in the site can be monitored and a sub-group of user, in the
experimental group, that really makes use of the service, can be defined. The active user is the user that,
after registration, voluntary access to EMPOWERING service by entering in the portal to acquire
information.
For all the user registered in the experimental group the date of registration is stored, this in order to allow
the evaluation of energy savings starting from the effective, specific user dependant, registration date.

10
Another subgroup of the experimental group is the test group, that is a group with a preferred contact
channel with the utility. This group was selected in order to have fast and effective feedback on the
supplied services.
The control group is composed by users that decided to not opt-in in EMPOWERING services that are
mainly “never taker” users, since they refused to opt-in the additional service even if informed and asked
by the portal.
All the users involved in EMPOWERING project (experimental and control group) are equipped with smart
meters.
Table 3 – Site Linz. Total number of users in the experimental and control group
Number of users in the experimental and control group
experimental group 60
control group 62.000
The services developed in the Linz site are:
- Comparison of specific user monthly consumption with average consumption of the most efficient
neighbours and with average consumption of all neighbours.
- Yearly representation of comparisons defined at the previous point.
- Comparison of specific user monthly consumption with the consumption in the same specific
month in the previous year.
- Representation of monthly consumption over an energy certification scale.
- Display of specific user consumption per flow rate zone as a stack diagram graph. Computation of
statistic with average and sum of consumption for every flow-rate zone with display of the costs.
This diagram and the statistic are available for all the days, months and years in the past since the
start of the data storing service.
- Recommendations for good practices in summer, winter or on holidays.
- Plot of consumption and average outdoor temperature vs. time.
- Alert on suddenly variation of consumption.
The definition of similar users is decided by LINZ AG, based on the customer data already stored in their
data base, giving particular relevance to the dwelling surface an location identified by postal code.

11
Table 4 – Energy Saving Valuation
Evaluation of Energy Savings
Service provided OT on line tools
Energy Electrical energy measured values
Total period analysed NOV 2013 – DEC 2015
Active date January 2015
Saving values Fixed active date - Result for the whole period
Group All contracts
Total savings results computed from Jan 15 to Dec 15 for all the users involved in the project
Estimated average daily consumption of experimental group in
absence of service [kWh/day]
8.15
Actual average daily consumption of experimental group [kWh/day] 7.71
Average daily energy savings [kWh/day] 0.44
Average daily energy savings [%] 5.39
Standard deviation [%] 3.31
Confidence interval @ 95% [%] -1.10 … 11.89
Cleaning parameters
restrictor 0.9 Lower quantile 1% Upper quantile 99%
Total measures experimental group 1178
Total measures control group 1132405
Customers experimental group 49
Customers control group 45423

12
Fig. 6 - Site: Linz. Average daily consumption computed on monthly base vs time for the control group
(blue) and the experimental group (red). Period of evaluation of energy savings January 15 – December 15.
Fig. 7 – Graphical representation of daily energy savings (kWh, %).
Scenario for evaluation of customer satisfaction
A first survey on users' requirements and expectations on the EMPOWERING services in Austria was
launched in January 2014. 31 customers answered to an online survey that aimed to investigate what kind
of information and data users already receive from their utility and what additional information they need
in order to adopt behaviours addressed to save energy.
Previously to EMPOWERING, LINZ AG customers had an additional web-based information on energy
consumption named EIS (Energy Information System); this is reflected on the answers to the first surveys in
which the majority of the respondents declared that they can easily obtain information on their
consumption in order to, for example, recognize the differences among their periodic bills or the
connection between their actual consumption and the investments they realised to save energy. The

13
respondents of the Linz pilot found their energy bill more easy to be read and understood respect the bills
of other pilots, the only dissatisfaction point was related to the cost of energy, which has been judged by
the end users as too high and not clear regarding its composition. Only the 29% of the respondents
declared that they are satisfied with their utility.
In those questionnaires, users expressed their preference according to the kind of services to be added to
EMPOWERING, the desired data visualisation tools and the request for additional services (e.g. the
possibility to download the *.csv files of their consumption).
After the processing of the data of the first survey, first versions of the Empowering services has been
realised. The first version has been sent to non-partners organisations/stakeholders with a specific
questionnaire, in order to collect from them feedbacks and possible suggestions on how to improve the
services.
In Austria, the following organisations provided feedbacks on EMPOWERING, assessing the developed
services as useful for customers and for utilities in order to increase the satisfaction of their customers:
- Econgas Gmbh;
- OÖ Energiesparverband;
- OÖ Gas-Wärme Gmbh;
- Energie AG;
- E7;
- Energie AG Vertrieb Gmbh;
- Energie AG OÖ Vertrieb Gmbh & Co KG;
- Energie AG
After the feedbacks from the above mentioned stakeholders, the developed services have been validated
by 15 employees (external to EMPOWERING) of LINZ AG and 18 end users who considered the
implemented services usable and understandable.
Perceived quality of the EMPOWERING services
In the widespread application phase of EMPOWERING, customers who received the services have been
asked to fill in a questionnaire in which they can evaluate the services received by their utility. 161
Customers filled in a questionnaire addressed to investigate the expectations and the perceptions of users
among different service dimensions. The obtained results (expressed as a 0-100 indicator for a better
readability) and the investigated dimensions are reported in the table below.
Table 5 – Obtained results
EMPOWERING service dimensions Users’ quality perception score
Presence of additional information on energy consumption 73,12
Provision of helps and suggestions on how to save energy 75,16
Provision of handy tools to understand energy consumptions 70,44
Information available at any time 75,16
Globally, Linz customers have very good perceptions on the quality of the EMPOWERING services

14
Did EMPOWERING meet users’ expectations?
After the first implementation of the services and before the widespread application phase of Empowering
to the whole experimental group, a questionnaire addressed to identify the expectations of the end users
in term of the effects/impacts that smart billing services should have has been launched. 25 users
expressed an evaluation on several dimensions that constitutes a general smart billing service, providing a
quantitative rating (on a 1-5 scale) on their importance. This survey was aimed to evaluate a starting point,
a ‘baseline’ for the final evaluation. From April to November 2015, the same questions have been asked to
the EMPOWERING users in the widespread application phase in order to assess changes in the perceived
quality and if the services met the initial expectations of end users. 161 end users answered to this survey,
using the online form available on the utility’s website or the paper-based versions at utility’s customer
care office.
A gap score (Q) have been calculated as the difference between users’ Perceptions and Expectations for
each investigated dimension. For the purpose of the present report, Q-scores have been transformed in a
0-100 scale. A score equal or major than 50 means that the services meet the expectations and the utility
create a good service. The last column shows the improvements on the Quality of EMPOWERING as
developed in Linz.
The table below shows the assessed dimensions and the Q-scores for each of them:
Table 6 - Assessed dimensions and the Q-scores
General smart billing service dimensions Q-score
(baseline)
Q-score (final) Improvements on the
Quality of services
Presence of additional information on
energy consumption
45,00 51,87 +15,28%
Provision of helps and suggestions on
how to save energy
41,25 51,25 +24,24%
Provision of handy tools to understand
energy consumptions
36,50 49,87 +36,64%
Information available at any time 41,00 50,00 +21,95%

15
For all investigated service dimension, an increase on the quality of the EMPOWERING services is
recognisable.
Changes in customer satisfaction1
A first question on the satisfaction of customers with their energy provider has been inserted in the first
survey carried out in the initial phase of the project, when the requirements of the users to develop the
EMPOWERING services were collected.
The same question was inserted in the baseline and evaluation questionnaires developed after the
implementation of EMPOWERING.
The table below shows the changes on the percentage of satisfied customers.
Table 7 - % of satisfied customers changes
Question User expectations
survey (2014)
Baseline and Evaluation surveys
(2015)
Are you satisfied with your energy provider? 29% answered 'YES' 70% answered 'YES'
1
Results on customer satisfaction are related only to services connected to informative services on energy billing.
They do not represent the satisfaction of users with their utility company.

16
3.2 France
The services in France are provided by Gas Electricity of Grenoble (GEG), one of the few independent
municipality-owned utilities in France, acting as grid operator, electricity and gas supplier, and energy
producer.
Prior to EMPOWERING project, GEG was involved in several smart grid projects focused on informative
services and demand side management, many GEG’s customers in the Grenoble are used to interact with
the on line services supplied by GEG.
GEG employ traditional energy meters and the readings are performed every 6 moth, the readings are non-
synchronous for all the customers, around 1/6 of the customers are measured per month.
Monthly consumptions are computed from the readings by proprietary algorithm developed in GEG. The
algorithm was tested and refined in years of field experience, the average daily consumptions are directly
obtained from the estimated monthly consumptions. All the customers already registered in the GEG web
portal were contacted by e-mail and were asked to connect to a separate portal where EMPOWERING
services were available (http://maconso.geg.fr).
The open access to the services of all the customers, by a voluntary registration, make impossible to
randomly pre-define the control and experimental groups. The procedure adopted carries to the formation
of an experimental group composed by “always taker” and “compliance” users.
The services are on line services, thus the accesses in the site can be monitored and an additional grouping,
among user in the experimental group, defined active user group can be identified.
For all the users registered in the experimental group the date of registration is stored, this in order to
allow the evaluation of energy savings, starting from the registration date, when EMPOWERING services
are assumed to be employed.
A subgroup of the experimental group is the test group, that is a group with a preferred contact channel
with the utility, it was selected in order to have fast and effective feedback on the supplied services.
The control group is composed by users that decided to not opt-in in EMPOWERING services that are
mainly “never taker” users, since they refused to opt-in even if informed and asked by the portal.
Using the customer data, already available in the GEG data base, the user in the experimental croup were
divided in three classes bases on the usage of electric energy and contract power (Table III). The members
of the control group were matched to the members of the experimental group following a criterion based
on the three groups identified in Table III where are defined:
group 1: low contract power and electric energy used for home appliances only,
group 2: high contract power and indoor space heating by electric energy,
group 3: low contract power but occasional use of electric energy for sanitary hot water and space heating.
Table 8 - Site: Grenoble. Grouping of the user involved in EMPOWERING program.
Energy use Number of users
Contract
power
electric
appliances
sanitary
hot water
production
electric
space
heating
control
group
experimental
group
group 1 6 kVA yes 3300 36
group 2 > 9kVA yes yes yes 720 10
group 3 6 kVA yes occasional occasional 604 8

17
The procedure of admission in EMPOWERING services, for groups 1, 2 and 3, is shown in Fig. 351: it is the
usual procedure with opt-in choice for the experimental group. After the definition of the experimental
group the control group is composed by selecting the members in order to match it on the ground of Table
III classification.
Fig. 8 - Definition of control and experimental groups in Grenoble.
The information available in the portal are:
- Actual three months consumption and comparison with the same period of the former year
(computed correction due to different weather condition available).
- Comparison with consumption of similar dwellings.
- Display of monthly consumption during the last year within the limits of maximum and minimum
consumptions of similar customers.
- Consumption forecast for the following three months.
- Display of energy scale for the ranges of consumption per unity dwelling surface and position of the
specific customer with respect to the scale.

18
Table 9 – Energy saving valuation
Energy Electrical energy estimated values
Active date Date of activation (after Oct. 14)
Saving values Date of activation – Dec 15
Group Group 1
Total savings results computed from Jan 15 to Dec 15 for group 1
6.77
Computed average daily consumption of experimental group
[kWh/day]
6.55
Confidence interval @ 95% [%] -4.32 ... 10.82
Cleaning parameters

19
Fig. 9 - Site: Grenoble – Group 1. Average daily consumption computed on monthly base vs time for the
control group (blue) and the experimental group (red). Period of plot of ADC (average energy consumption)
November 13 – October 15. Period of evaluation of energy savings from date of activation (after October
14) to December 15.
Fig. 10 - Graphical representation of daily energy savings (kWh, %).

20
Table 10 – Energy saving valuation
Group Group 2
17.25
[kWh/day]
16.18
Cleaning parameters

21
November 13 – October 15.
Period of evaluation of energy savings from date of activation (after October 14) to December 15.

22
Table 11 – Energy Saving valuation
Group Group 3
9.68
[kWh/day]
7.56
Confidence interval @ 95% [%] 5.64 ... 38.2
Cleaning parameters

23
November 13 – October 15.
Period of evaluation of energy savings from date of activation (after October 14) to December 15.

24
Gas consumption
In Grenoble EMPOWERING services operate also on gas consumption: 500 gas user were selected for the
experimental group with the procedure adopted for the electric energy, while the control group is
composed by 5000 users.
Gas consumption are read every six months and monthly consumption are computed by proprietary
algorithm.
EMPOWERING services similar to the on line tools available for electric energy have been developed for the
gas customers. In particular the following tool are available:
- Actual three months consumption and comparison with the same period of the former year. Bothe
measured consumption and corrected consumption, to take into account different weather
conditions are available.
- Consumption forecast for the following three months
Gas consumption have been evaluated in the same way of electrical consumption, the results of the
treatment are shown in the following tables and diagrams.

25
Energy Gas estimated values
Group All (gas) users
Total savings results computed from Jan 15 to Dec 15 for all gas users
18.19
[kWh/day]
17.40
Cleaning parameters

26
Fig. 15 - Site: Grenoble – Gas consumption. Average daily consumption computed on monthly base vs time
for the control group (blue) and the experimental group (red). Period of plot of ADC (average energy
consumption) November 13 – October 15. Period of evaluation of energy savings from date of activation
(after October 14) to December 15.
A first survey on users' requirements and expectations on the EMPOWERING services in France was
launched in July 2013. 150 customers answered to an online survey that aimed to investigate what kind of
information and data users already receive from their utility and what additional information they need in
order to adopt behaviours addressed to save energy.
The majority of GEG's customers (60%) declared from the beginning that they can understand their bill, but
is not so easy to obtain information on how to compare past and present consumption. Cost composition

27
appears clear, and a great part of the respondents declared that they are available to read the meter once
a month and to communicate the values to the utility in exchange of more detailed information on their
consumptions. A relevant part of respondents, found difficult to understand the amount of energy over the
period using the bill and to compare current power consumption with the past consumptions.
Analysis of the French scenario was deepen organising a focus group (June 2013) with relevant
stakeholders, such as customer representatives, trade unions, end users. The participant to focus groups
highlighted that the French energy bill lack on information on the composition of the price of energy per
consumed kWh.
In this first phase of survey, users expressed their preference according to the kind of services to be added
to EMPOWERING, the desired data visualisation tools and the request for additional services, expressing in
particular (in accordance with their comments on the capacity to understand their energy bills) the need to
receive information to compare present and past consumptions and advices and recommendations to save
energy.
After the focus group with stakeholders and the processing of the results from the survey, the developed
services have been validated by 13 employees (external to EMPOWERING) of GEG and 5 end users who
considered the implemented services usable and understandable.
Table 13 – Obtained Results
Globally, Grenoble customers have very good perceptions on the quality of the EMPOWERING services

28
quality and if the services met the initial expectations of end users. 121 end users answered to this survey
available in the websites of GEG and ALEC.
A gap score (Q) have been calculated as the difference between users’ Perceptions and Expectations for
each investigated dimension. For the purpose of the present report, Q-scores have been transformed in a
0-100 scale. A score equal or major than 50 means that the services meet the expectations and the utility
create a good service. The last column shows the improvements on the Quality of EMPOWERING as
developed in Grenoble.
The table below shows the assessed dimensions and the Q-scores for each of them.
(baseline)
Q-score
(final)
Improvements on the
Quality of services
Presence of additional information on energy
consumption
57,00 63,87 +12,06%
Provision of helps and suggestions on how to save
energy
67,00 76,62 +14,37%
Provision of handy tools to understand energy
consumptions
63,00 70,62 +12,10%
recognisable.

29
EMPOWERING services were collected. The same question was inserted in the baseline and evaluation
questionnaires developed after the implementation of EMPOWERING.
Table 15 - % of satisfied customers changes.
Question User expectations survey (2013) Baseline and Evaluation
surveys (2015)
Are you satisfied with your
energy provider?
Not assessed in a quantitative way but
there were indications on
dissatisfaction related to lack on
transparency of the bill
60,95% answered YES
2
They do not represent the satisfaction of users with their utility company

30
3.3 Italy
The services in Italy are provided by IREN Energia, which is supplier of the services for electricity and district
heating distribution in the city of Torino and for district heating distribution in the city of Reggio Emilia.
IREN Energia operates as DSO in the city of Torino. The whole city is equipped with smart meters, the
company AEM Torino Distribuzione S.p.A. (subsidiary of IREN Energia) is charged of various services related
to smart meters, in particular maintenance service and continuous reading.
Electric energy measures are available with the granularity of 15 min, thus starting from the raw data
acquired by smart meters, daily energy consumptions can be computed.
Following the Italian regulations IREN Energia is separated from the various retailer companies that
operates as final sellers on the market.
The final users do not have direct contact with IREN Energia because every final user refers to his retailer
for all the problems related to electric energy supply.
The readings acquired by the smart meters are collected by IREN Energia and then send to the retailers that
compute, following the specific contract of each final user, the amount of the invoices.
IREN Energia has the knowledge of the energy consumption but it do not have access to the final price of
energy and billing amounts.
In order to select the population for EMPOWERING project a recruitment campaign, based on newspaper
advertising, was started.
In addition to the newspaper campaign IREN Energia selected 3000 around valid e-mails from his data base,
related to electric users in Torino, and send to them a message to encourage to joint to EMPOWERING
program.
Both the e-mails and newspaper advertisements were asking to final users to volunteer register themselves
in the IREN Energia site, to give consent for EMPOWERING program and fill a questionnaire.
The questionnaire contents few simple questions to allow a class definition of the user involved in
EMPOWERING:
- number and age (adults/children) of people living in the dwelling
- dwelling surface
- information about the presence and age home appliances in the dwelling (microwaves oven,
refrigerator, dishwasher, washing machine…)
At the end of the informative campaign less than 200 customers registered themselves on the site, this
users were included in the experimental group.
In order to reach the scheduled population for the program (2000 unities), 1800 customers, selected in the
IREN Energia data base, were included in the program, without asking for previous consent, but offering
them an opt-out option.
The experimental design adopted in Torino is shown in Fig. 401, that depict the main selection channel for
the customers to be included in the project.
The selected pilot customers are random assigned to the experimental and control groups and opt-out
choice is available to whose that are assigned to the experimental group.
Due to the selection process adopted both experimental and control groups are surely composed by
“always taker”, “compliers” in addition some “never taker” users were included in the experimental group
and opt-out option is offered. Since EMPOWERING services are free of charge only “active never taker”
user exercised the opt-out option.

31
Fig. 17 - Definition of control and experimental groups in Torino.
The informative service available in the site of Torino is a monthly report about electric consumption. The
first report was related to January 2015 and the service is scheduled to operate up to the end of the
project.
The information reported in the monthly report are:
- comparison of specific user monthly consumption with the same month in the previous year.
- Comparison of specific user monthly consumption with the best users in the same class and with
average user in the class.
- Total consumption of the year (from 1st
January): value and percentage referred to the previous
year consumption.
- Plot of daily consumption vs. time for the analysed month and comparison with the same class
average user daily consumption.
The informative reports are available at the end of the month with 30 days delay. In additional to the
numeric values, an emoticon is added to give a concise and effective information.
The service is available to the customers is a billing tool, the monthly reports are send to the customers by
e-mail.
Table 16 – Number of users
experimental group
200 opt-in + registration
1600 included with opt-out option available
For a total 1800
control group 200 opt-out option available

32
The evaluation of energy saving is carried out taking into account two separate groups defined as:
group 1: consumers with contract power of 3 kW
A contract power of 3 kW is the usual contract for family use in Italy. For this contract power a particular
contract defined “enhanced protection contract” is available; this kind of contract is a contract that comes
from the electric market structured as a monopole, before the opening to competitive operators.
group 2: consumers with contract power greater than 3 kW
Both group use electric energy for home appliances and probably some users have electric boiler for hot
sanitary water production, it can be excluded the use of space electric heating except for short term
emergency use.
Air cooling systems are assumed to be included in electric appliances, both group1 and group 2 can have air
cooling systems.

33
Service provided BT Billing tools
Group Group 1
4.18
Cleaning parameters

34
Fig. 18 - Site: Torino – Group 1. Average daily consumption computed on monthly base vs time for the
control group (blue) and the experimental group (red). Period of evaluation of energy savings January 15 –
December 15.

35
Table 18 – Energy Savings valuation
Group Group 2
13.94
Average daily energy savings [kWh/day] -0.25
Average daily energy savings [%] -1.79
Cleaning parameters

36
Fig. 20 Site: Torino – Group 2. Average daily consumption computed on monthly base vs time for the
control group (blue) and the experimental group (red). Period of evaluation of energy savings January 15 –
December 15.
District heating services
EMPOWERING project acts on district heating services in the town of Torino and Reggio Emilia. The
standard heating season is from 15 October up to 15 April, the city managers, following particular cold
weather conditions, can authorise to switch on the district heating outside the standard period.
From the heating season 2012-13 up to October 2015 there was no necessity to change the standard
heating periods.
In Torino the selected buildings to be involved in EMPOWERING program are condominiums. There is only
one connection point between the district heating network and the building, where the heat exchanger and

37
the measurements units are located. The whole condominium is assumed as a single unit and only one
fiscal bill is sent to the building manager that take care of all the contacts with IREN Energia.
Following the specific rules of every condominium the invoice of IREN Energia is spitted on the flats. At
least one meeting per year has to be organised by the building manager to discuss the budget, during the
meeting the cost and consumption of district heating are analysed. Information exchange during the year
from the building manager to the users depends on the specific rules and habits of the condominium, the
contact between the building manger and the dwellers are usually carried out by printed letters or e-mails.
In Torino the experimental group is composed by 23 condominiums for a total volume of 164267 m3
, while
the control is composed by 4 condominiums for a total volume of 15770 m3
.
In Reggio Emilia the users involved in EMPOWERING project are both condominiums and single family
houses.
The condominiums are controlled by the Social Housing Agency. The Municipality of Reggio Emilia asked to
involve these buildings in EMPOWERING with the goal to improve the education and awareness about the
use of energy of the tenants. These condominiums receive only one invoice, concerning to the consumption
of the whole building, then the building manager computed the amounts related every flat.
The family houses are independent houses, distributed all over the city, the invoice is sent directly to the
owners that live in the house.
The total number of user involved in EMPOWERING project is 539 user for the experimental group, while
60 users are in the control group. The condominiums are one third around of the total users.
Number of users in the experimental and control group for the DH
Torino experimental group 23 (opt-out option available)
Torino control group 4 (opt-out option available)
Reggio Emilia experimental group 539 (opt-out option available)
Reggio Emilia control group 60 ((opt-out option available)
A first survey on users' requirements and expectations on the EMPOWERING services in Italy was launched
in July 2013. 514 customers answered to an online and paper based survey that aimed to investigate what
kind of information and data users already receive from their utility and what additional information they
need in order to adopt behaviours addressed to save energy.
The majority of customers, both in Turin and Reggio Emilia, showed dissatisfaction on the services of their
utility and considered their bill as difficult to be read and not transparent. In particular the dissatisfaction is
higher among the district heating users. Analysis of the Italian scenario was deepen organising a focus
group (June 2013) with relevant stakeholders, such as customer representatives, trade unions, end users
and with 2 meetings with customers in 2 buildings initially involved in the test group and one-to-one
interviews to almost all the involved families in Reggio Emilia. Users engagement activities and the
confrontation with stakeholders
In this first phase of survey, users expressed their preference according to the kind of services to be added
to EMPOWERING, the desired data visualisation tools and the request for additional services, expressing in
energy, in addition to the necessity to insert information on cost composition.

38
After the implementation of the prototypes of services, they have been assessed by stakeholders and non-
partner organisation, who provided feedback on the usefulness of the smart billing tools developed by
IREN. The involved non partners organisations were:
- Social services of Reggio Emilia;
- Representatives of households;
- Social housing organisations;
- Building managers;
- The CGIL trade union
services have been validated by 11 employees (external to EMPOWERING) of IREN and 15 end users who
Globally, Turin and Reggio Emilia customers have very good perceptions on the quality of the
EMPOWERING services.

39
via email or filling in a paper-based questionnaire available at IREN and municipality premises. In pilot of
Reggio Emilia, part of the questionnaires have been collected during specific meetings organised with
households involved in EMPOWERING. These meetings with households were held in November 18th
and
19th
, with the participation of Vittorio Gimigliano (RE Municipality), Ivan Arduini (ACER- Reggio Emilia),
Marina Pirazzi (Extrafondente) and Federico Boni Castagnetti (IREN). Combining the questionnaires that
they filled-in and their opinion expressed in the conversation, a few interesting comments came out.
The first and most stressed comment was that the information conveyed by the additional Empowering
info sheet was of limited utility because it just states that a household is consuming more than a
comparable household, yet it does not tell the cause of it. In other words, it spreads the alarm without
offering an explanation that could help the client to mitigate the damage.
The second collected comment was about the (apparent) inconsistence of the high consumption “red face”
alarm and the consumption of the past year, which, on the contrary, the consumers did not reckon higher
than the present year. This comment offered IREN the opportunity to check the system and possible errors.
Finally, the meetings confirmed the necessity to continue the distribution of printed info sheets (vs e-info)
in not electronically literate environments that are still widespread in Italy.
In processing the questionnaires data, a gap score (Q) have been calculated as the difference between
users’ Perceptions and Expectations for each investigated dimension. For the purpose of the present
report, Q-scores have been transformed in a 0-100 scale. A score equal or major than 50 means that the
services meet the expectations and the utility create a good service. The last column shows the
improvements on the Quality of EMPOWERING as developed in Turin and Reggio Emilia.
The table below shows the assessed dimensions and the Q-scores for each of them
(baseline)
Q-score
(final)
Improvements on the
Quality of services
consumption
19,50 26,87 +37,82%
Provision of helps and suggestions on how to
save energy
17,00 20,75 +22,06%
consumptions
18,00 19,87 +10,42%
recognisable. The low Q-scores are affected of high expectations scores. Analysing the questionnaires,

40
expectations scores exceeded the perceptions one. As showed in the paragraph before, perceptions about
the services developed in Italy are good, and great efforts have been made in order to meet the users
expectations as showed in the table above. During the project lot of improvements have been made and
this is reflected on the improving Q-score continuously monitored during the project.
The table below shows the changes on the percentage of satisfied customers. The low rate reflects the high
expectations.
Table 22 - % satisfied customers changes
Question User expectations survey
(2013)
Baseline and Evaluation surveys
(2015)
Are you satisfied with your energy
provider?
24,5% 23,37%
3

41
3.4 Spain
The services in Spain are provided at the pilot ELGAS by Grup ElGas in the city of Sóller, Mallorca. ElGas is a
pioneer energy in Mallorca which is operating as local energy distributor with the company Vall de Sóller
Energia, operates as electricity retailer through the company Electrica Sollerense and participates in
companies producing renewable energy from wind, solar or hydraulic power.
ElGas has deployed smart meters to all of its around 9000 residential customers which are all included in
the pilot. The daily consumption required for EMPOWERING project are computed from the higher
frequency measurements carried out by the smart meters.
The services at the pilot started on April 2015 by offering them on the online portal while the dispatch of
the energy reports started in May 2015 and will operate up to March 2016.
The customers are initially randomly assigned to the experimental group or to the control group, but only
the member of the experimental group have access to the services.
The services are provided to the customers in two lines: through an on line portal (on line tool) and by an
informative report attached to the energy bill (billing tool). Due to the multi-national population of the
customers in Sóller the services are offered in 3 languages (English, Spanish and Catalan).
Both lines provided the following services:
- Comparison of specific customer monthly consumption with the average consumption of the most
efficient neighbours and with the consumption of the average neighbour.
- Plot of specific customer monthly consumption, most efficient neighbours monthly consumption
and average neighbours monthly consumption vs. time (last 12 months). Multiple graphic options
available for the on line tool.
- Plot of specific customer monthly consumption for the last 12 months and comparison with the
consumption in the previous year.
- Plot of specific use consumption and average outdoor temperature.
- Alert for anomalous increment of electric consumption
Billing tool
The energy reports are sent attached to the energy bill every 2 months to all the user assigned to the
experimental group, opt-out option is available.
Since the additional information supplied by EMPOWERING are contemporary to the billing they can
immediately help to evaluate the amount of the invoice. The informative services and graphs are generated
each month, the report attached to the bill is related to the last month.
The experimental design for the billing tool is shown in Fig. 22.

42
Fig. 22 - Definition of control and experimental groups for billing tool in Soller.
Due to the selection process adopted both experimental and control groups are surely composed by
“always taker”, “compliers”. “Never taker” users are automatically included in the experimental group and
opt-out option is offered. Since EMPOWERING services are free of charge only “active never taker” user
exercised the opt-out option.
On line tool
The users are encouraged to register themselves in the on line portal in order to access to the online tools.
The active registration in the portal is equivalent to an opt-in option to the EMPOWERING on line tools, the
users assigned to the control group have no access to the on line services (the option is not active on the
screen). All the users are informed about the availability of on line tools , if a user, assigned to the control
group, asks to have access to the on line information, his access is authorized but he is contemporary
dropped out the control group.
The experimental design for the on line tool is shown in Fig. 23.

43
Fig. 23 - Definition of control and experimental groups for on line tool in Soller.
Since opt in is required to access the on line tool, the experimental group for the on line tool is composed
by “always taker” and “complier” customers.
experimental group
7362 selected for BT
364 selected for BF and registered for OT
control group 1321

44
Active date May 2015
Group All contracts
Total savings results computed from Jan 15 to Dec 15 for all the users involved in the project
11.59
Cleaning parameters

45
Fig. 24 - Site: Soller. Average daily consumption computed on monthly base vs time for the control group
(blue) and the experimental group (red). Period of evaluation of energy savings May 15 – December 15.

46
Service provided BT Billing tools + OT on line tools
Active date May 2015
Group All contracts
Total savings results computed from Jan 15 to Dec 15 for the active users in the experimental group
13.71
Cleaning parameters

47
Fig. 26 - Site: Soller. Average daily consumption computed on monthly base vs time for the control group
(blue) and the experimental group (red). Active users (OT + BT). Period of evaluation of energy savings May
15 – December 15.
A first survey on users' requirements and expectations on the EMPOWERING services in Spain was
launched in July 2013. 172 customers answered to an online and paper based survey that aimed to
investigate what kind of information and data users already receive from their utility and what additional
information they need in order to adopt behaviours addressed to save energy. The majority (56%) of
customers showed satisfaction on the services of their utility, but a relevant part did not answer. Among

48
the dissatisfied, the main comments are related to costs. In addition, more than an half of the respondents
found it difficult to understand the consumed amount of energy over the period using the bill. In this first
phase of survey, users expressed their preference according to the kind of services to be added to
EMPOWERING, the desired data visualisation tools and the request for additional services, expressing in
energy,.
After the implementation of the prototypes of services, they have been assessed by stakeholders and non-
partner organisation, who provided feedback on the usefulness of the smart billing tools developed by EL
GAS. The involved non partners organisations were:
- Government of Catalunya
- Som Energia SCCL
services have been validated by 8 employees (external to EMPOWERING) of EL GAS and 5 end users who
Presence of additional information on energy consumption 79.37
Provision of helps and suggestions on how to save energy 75.00
Provision of handy tools to understand energy consumptions 68.75
Information available at any time 79.37
Globally, Sollér customers have very good perceptions on the quality of the EMPOWERING services.

49
via email or filling in a paper-based questionnaire available at EL GAS premises.
In processing the questionnaires data, a gap score (Q) have been calculated as the difference between
users’ Perceptions and Expectations for each investigated dimension. For the purpose of the present
report, Q-scores have been transformed in a 0-100 scale. A score equal or major than 50 means that the
services meet the expectations and the utility create a good service. The last column shows the
improvements on the Quality of EMPOWERING as developed in Turin and Reggio Emilia.
The table below shows the assessed dimensions and the Q-scores for each of them.
(baseline)
Q-score
(final)
Improvements on the
Quality of services
consumption
38.00 43.00 + 13.16%
Provision of helps and suggestions on how to save
energy
27.00 42.25 + 56.48%
consumptions
26.50 37.125 + 40.09%
Information available at any time 30.50 42.62 + 37.75%
recognisable. The low Q-scores are affected of high expectations scores. Analysing the questionnaires,
expectations scores exceeded the perceptions one. As showed in the paragraph before, perceptions about

50
the services developed in Spain are very good, and great efforts have been made in order to meet the users
expectations as showed in the table above. During the project lot of improvements have been made and
this is reflected on the improving Q-score continuously monitored during the project. This is reflected also
in the results on customer satisfaction.
Table 28 - % of satisfied customers changes
Question User expectations
survey (2013)
Baseline and Evaluation
surveys (2015)
Are you satisfied with your energy provider? 56% 71.92%
4

51
4. Conclusions
The responsible use of energy is an activity that implies several aspects related to both the kind of energy
involved and to use the energy itself.
Electrical energy, that was analyzed in all the sites involved in EMPOWERING project, is a particularly
difficult area within which to promote sustainable consumption, even in targeted groups.
The electric energy usage in civil application, is involved in ICT processes, home appliances, sanitary hot
water production , control and regulation of indoor spaces to get comfortable conditions of temperature
and humidity.
A responsible use of electric energy starts from the choosing electricity from renewable or other less
environmentally detrimental sources, and then there is conscious choice of appliances, and related
duration, and, at the end, a responsible use of electric apparatus.
Final users are in general not necessary expert in the field of energy or have specific knowledge on good
practices in energy usage; in order to obtain an improvement in the use of energy the first step is the
information and the education of final users.
To get effective results the information have to be transferred taking into account that both the aspect of:
- specific information contents (i.e. daily consumption, monthly consumption ...)
- way and procedure adopted to transfer the information (i.e. mail, e-mail, web portal, ...)
EMPOWERING project operated in five different sites, producing different results in each pilot site.
The comparative analysis of the results allows a qualitative information on possible best practices that
could be adopted in the future, when informative billing services will be spread over a larger number of
customers.
The final goal of the information transfer is the improvement of final uses behavior, the specific
information transmitted is not able to grant by itself a result, but also the transfer process i.e. adopted
channel and timing is an essential component of final results. The general rule to transfer well explained
and timing information has to adapted to the specific site condition.
In EMPOWERING project similar information were transferred by different channels, the comparison of the
results can give suggestion to best practice to adopted in the future.
Final users have to involved as active parts in the process starting from the beginning. The solution to send
an e-mail containing information about energy usage is a one –way channel, it is difficult to control if the
final user has really read the mail and understood the content, a request of receipt is generally discouraged
due to the big number of customers. Moreover the user receive the mail when they are checking their mail
account, but this do not means that it is the best instant to send the information.
A more effective solution is to engage the final user by a specific web portal, where the final user can play
an active role in looking for the information, in this way the contents are transferred when the final user is
prepared to receive them, since he is looking for them.
Information on a web site are continuously available and the delay between energy consumption and
information can be forced to the minimum feasible value. In this way the link between behavior and
consumed energy and related costs (billing cost, social cost, ecologic cost) can be emphasized.
The reduction of time delay between the use of energy and the measurement of energy can help the direct
and immediate control of the effectiveness of user’s actions for energy saving, since nearly real time
feedback is available. The inclusion of information by a billing tool received together with invoice looks to
be an interesting solution because information are transferred in a key instant of the relationship between
customer and utility, when the customer is prepared to receive tips about energy and money savings.

52
The availability of a web portal together with billing tool is the most recommended solution: information
are always available in the portal and billing tools acts as periodic recommendation to register and keep
being informed by regular access to the portal. A training period look to be necessary to the customer to
really skilled to access to the portal and obtain information, better results are obtained when web portals
are available from years and the customers are used to access them.
The time required by the average user to be familiar with the service has to be taken inti account during
the experimental design. If opt-out option is made available the service is supplied to larger amount of
consumers. People not initially interested in the services have time to get in contact with it and start to
evaluate it during the time.
By the analysis of the different situations occurred in the different sites in EMPOWERING project the
transfer of information is more effective on the energy consumption when higher energy consumption are
present. This is due to the general greater interest in energy associated to the higher costs and to the fact
that energy is use to heat and cooling production. The better use of heat and cooling affects directly the
amount of primary energy employed. On the other hand high energy reduction in home appliance can be
obtained by renewing themselves, but this implies the substitution of still working appliances with newer
more efficient.
On the basis of the 442 questionnaires collected in the 4 pilots, it is possible to conclude that
EMPOWERING obtained a good acceptance among their users. This is detectable analysing globally all the
collected questionnaires. The perceived quality of the implemented services is good in all pilots. Rating the
quality perceptions of users on a scale 0 (low quality) to 100 (high quality), the evaluation of EMPOWERING
given by 442 users is 70,81.
Considering the expectations preliminary to the widespread application of EMPOWERING, the surveys
show that in all the involved countries the utilities are made improvements on the implemented services in
order to majorly meet the users’ expectations and therefore improve the quality of the provided services.
The chart below show the average increase of the gap-score Q, identifying how much the developed
services meet the users’ expectations, in the period comprised between the first implementation of
empowering in 2014 and the widespread application phase of the final part of 2015.

53
The scores reported above are reflected on the average satisfaction of users with Empowering. The 67% of
users addressed with the EMPOWERING services in the 4 pilots are satisfied with the services they received.
The creation and transfer of information force all the actors in the energy chain: production, transmission,
distribution, final use to redefine their relations in a more collaborative way.
This is a not negligible effect that EMPOWERING project produced, since long term results are more
important than actual energy savings. Sustainable energy saving, with reduction of CO2 emission, can be
achieved by a more collaborative relationship between users , utilities and DSOs that can lead to a more
responsible energy usage.

54
5. Bibliography
[1] U.S. Department of Energy Smart Grid Investment Grant Technical Advisory – Group Guidance
Document #12. DOE’s Meta Analysis Framework, 2011.
[2] Evaluation, Measurement, and Verification (EM&V) of Residential Behaviour-Based Energy
Efficiency Programs: Issues and Recommendations, May 2012.
[3] Massachusetts Three years Cross-Cutting Behavioral Program – Evaluation Integrated Report – July
2012.
[4] David Albuoy, Program Evaluation and the Difference in Difference Estimator, Economics 131,
Berkeley 2004.
[5] Bruce D. Meyer, Natural and Quasi-Experiments in Economics, Journal of Business and Economic
Statistics, 1995, Vol. 13, No 2.
[6] EPRI Report 1020855: Guidelines for Designing Effective Energy Information Feedback Pilots:
Research Protocols; 23-Apr-2010.
[7] Puget Sound Energy - Energy & Sustainability. Prepared by DNV KEMA, Madison, Wisconsin, April
20, 2012.
[8] Puget Sound Energy’s 2012-2013 - Biennial Electric Conservation Achievement.
[9] Lockart B., Wheelock C., Effective Customer Engagement Utilities Must Speak Customers’ Language,
PikeResearch, OPOWER, 2013.
[10] Cochran W. G., Sampling techniques, John Wiley & Sons, 1977.
[11] Bartlett, J. E., Kotrlik, J. W., & Higgins, C. C., Organizational Research: Determining Appropriate
Sample Size in Survey Research, Information Technology, Learning, and Performance Journal, 19(1),
2001.
[12] Parasuraman A., Zeithaml V. A., Berry L. L., Servqual, Journal of retailing, 64 (1), 12-40, 1988.

EMPOWERING_D6.3

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Viewers also liked

Viewers also liked (12)

Similar to EMPOWERING_D6.3

Similar to EMPOWERING_D6.3 (20)

More from Giovanni Pede

More from Giovanni Pede (13)

EMPOWERING_D6.3