Open Energi's Smart Grid solutions help balance the electricity grid in real time. We collect second-by-second power consumption data from thousands of assets in the field, deeply integrating with legacy industrial controllers and building management systems. We use this data to provide our service and also to prove that we have delivered it after the fact. This data is considered sensitive by the companies who own the assets as well as the National Grid, so its flow must be tightly controlled both from the point of view of security as well as validity. I will explain how we designed an ingestion platform to meet these requirements and how we are now bringing this platform into the age of cloud computing.

1. Data Governance and the IoT
Michael Bironneau
June 2016

2. 0
5
10
15
20
25
30
35
Installed Capacity (GW)
Generation (GW)

3. Open Energi | What we do
• Industrial
• Aggregates/bitumen
• Foundries
• Education (universities)
• Retail
• Construction
• Utilities
We shift the power demand of electrical loads to help rectify
imbalances between electrical supply and demand.
Open Energi fills
peaks & troughs
Peaking power
typically coal
Base load
power typically
nuclear / gas

6. [1] infographic The Connectivist based on Cisco Data

7. • 25-40k messages processed per second
• Total size of data 500TB-800TB
• Opening up APIs to selected partners
Open Energi | Data in the coming year

8. • Precision of timestamps
• Provenance of power consumption data (hardware type, poll rates, ...)
• Data can be revealing of industrial processes and schedules
• Mandatory retention periods
• APIs – ensuring partners adhere to same standards
Open Energi | Governance Considerations

9. Example: Challenges Validating IoT Data

10. An asset reports a power reading of 100kW at 18:23.003 UTC.
{“topic”: “reading”,
“entity”: “l3234”,
“recorded_at”: “2016-06-03T18:23.003Z”
“type”: “power”,
“value”: 100
}
Open Energi | Challenges Validating IoT Data

11. An asset reports a power reading of 100kW at 18:23.003 UTC.
{“topic”: “reading”,
“entity”: “l3234”,
“recorded_at”: “2016-06-03T18:23.003Z”
“type”: “power”,
“value”: 100
}
Open Energi | Challenges Validating IoT Data

12. What's the problem?

13. A toilet fan was consuming as much power as 100 kettles.
Open Energi | Challenges Validating IoT Data

14. After-the fact, relative:
• Raise an alert if an asset reports anomalous power readings
Preventative, absolute:
• Reject data from an asset if it is not within tolerated bounds for
the asset class
Open Energi | Obvious Solutions

15. Rejected data = lost revenue

16. • Partial validation
• Flexible absolute rules that can be overridden
• Adaptive rules to detect anomalies
Open Energi | Lessons Learned

17. Open Energi | New Tooling

18. Securing Human Access to IoT Data (in 2016)

19. • Permissive API vs Restrictive API
• Big Data Platform vs Enterprise BI
• Cloud Computing vs On-Premise
• Polyglot Persistence vs Single RDBMS
Open Energi | Challenges Securing IoT Data

20. Primary
Database
Read
Replica
Application
storage
Downloaded
file from web
app
Resampled
and stored
elsewhere
Archived
in Big
Data
platform
Excel
spreadsheet
API
An asset reports a
power reading of
0.1kW at
18:23.003 UTC.
Open Energi | Challenges Securing IoT Data

21. • API Gateways
• Apache Knox and Apache Ranger
Open Energi | New Tooling

22. Thanks for listening. Any questions?