Presentation at CBDS seminar Big Data Matters 27 sept 2017

1. Electricity smart meters: two use
cases
Maiki Ilves
Statistics Estonia
Seminar „Big Data Matters - Towards Smart Statistics“
27. September 2017, Heerlen

2. Overview
Smart meters situation in Estonia
Data hub
Use case 1: households’ electricity consumption
Use case 2: identifying vacant dwellings
Conclusions
Maiki Ilves27.09.2017

3. Smart Electricity Meter
Maiki Ilves27.09.2017
A smart meter is usually an electronic device that
records consumption of electric energy in
intervals of an hour or less and communicates
that information at least daily back to the utility
for monitoring and billing.
Smart meters enable two-way communication
between the meter and the central system.
Unlike home energy monitors, smart meters can
gather data for remote reporting.

4. Deployment of smart meters in Estonia
Maiki Ilves27.09.2017
0
100000
200000
300000
400000
500000
600000
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
2013 2014 2015
Numberofsmartmeters
No of smart meters added Total number of smart meters

5. Elering
An independent electricity and gas system operator in
Estonia
Responsible for ensuring the functioning of the energy
market.
Competence centre of energy field
Initiatives for research and development (incl. smart
software platform, e-elering)
Know-how in area of policy formulation
Maiki Ilves27.09.2017

6. Data and IT environment in NSI
2013-2016 data from Elering
Data delivery on external hard disk
Data arrives as MySQL database files, transformed into
Hadoop file system.
IT tools for processing: Hive for Hadoop built-in tool for
SQL queries, Spark and Anaconda (Python) for
visualization, aggregated data analysis in SAS.
Maiki Ilves27.09.2017

7. Smart meters data: structure
Estonian data structure: 4
main tables
Metering data – main table
with hourly consumptions
Metering points – location
Agreements – contract info
Customers – contract holder
information
Maiki Ilves27.09.2017

8. Goals of the pilot study
Combine data sources: population and building register
with data from the hub
Electricity consumption and production by households –
averages by region and by different household and
building characteristics
Identifying vacant dwellings – to be used in the register
based census to assign main living space to households
Maiki Ilves27.09.2017

9. Household’s total and mean consumption
Maiki Ilves27.09.2017

10. Consumption by month
Maiki Ilves27.09.2017

11. Consumption on a weekday
Maiki Ilves27.09.2017

12. Identifying vacant dwelling
List of the methods considered:
Using the R-function auto.arima to predict kWh usage.
Estimating the variance of the daily kWh usage.
Estimating the ratio between the mean kWh usage during day- and
night-time.
Using a volatility measure.
Using a cell wise outlier detection procedure to locate days with
'unusually' low kWh usage.
Using estimates from some of the listed methods as well as range,
mean and median of daily kWh usage in a random forest model.
Maiki Ilves27.09.2017

13. Dwellings occupancy
Occupied household Vacant household
Maiki Ilves27.09.2017

14. Vacant dwellings
61% dwellings were matched with consumption information and compared
with register based census information about the occupancy-vacancy.
Maiki Ilves27.09.2017

15. Advantages
Possibility to link it with other data sources and gain new
knowledge;
Data source can be used to validate or improve current
survey and register based statistics;
Data source could increase the quality of regional
statistics.
Maiki Ilves27.09.2017

16. Challenges
Address information is not standardized and geo-coding needs lots
of resources. Address is the key variable in the linking and the quality
of address information is crucial for the successful use of the data
source.
Lot of noise in the data (all energy flows)
One metering point can correspond to many consumers and one
needs to develop methodology to extract the consumption of the single
consumer.
Quality assessment is problematic with regard to precision and
coherence.
Maiki Ilves27.09.2017

17. Conclusions
Information about the average consumption by the household size,
type of dwelling or any other characteristics of the dwelling will be
made available to the public as experimental statistics.
Statistics Estonia plans to continue working with the electricity data and
find more ways to use vacancy indicator. In an ideal case data could
be used to not only to validate the results but to improve the main
residence address information in the register based census.
Future areas of study: production of energy by the source of renewable
energy, energy consumption as an indicator for the economic trends.
Maiki Ilves27.09.2017

18. Useful links
ESSnet project on Big Data homepage:
http://bit.ly/2weY7o5
Report about data access and data handling in case of
metering data (http://bit.ly/2kPqtyL)
Report about producing statistics with smart meters data
(http://bit.ly/2hcoqZg)
Elering: https://elering.ee/en/
27.09.2017 Maiki Ilves

19. Maiki Ilves27.09.2017
Thank you
for your
attention!
maiki.ilves@stat.ee