This document discusses using data streams and lakes for big data analytics in utilities. It begins by explaining how smart meter rollouts have increased meter reads from 80 million to 350 billion per year. Traditional data warehousing is challenged by the time criticality, resource demands, and scale of this unbounded data. The document then introduces data streams for real-time analytics and data lakes for flexible storage of huge amounts of structured, semi-structured, and unstructured data. It describes Valo, an open lambda architecture that uses multiple repositories and stream processing to enable both real-time and historical analysis across disparate data sources.

1. INTO THE STREAMS AND
LAKES
OUT OF THE DATA WAREHOUSES

2. BIG DATA LDN
ABOUT ME
▸ Valo Product Manager
▸ Engineering background
▸ Big data for utilities

3. BIG DATA LDN
BIG DATA FOR UTILITIES
▸ Triggered by smart meter roll-out
▸ Quarterly meter reads become half-hourly
▸ Previously 80 million reads per year
▸ Now 350 billion reads per year

4. BIG DATA LDN
BIG DATA FOR UTILITIES
▸ Triggered by smart meter roll-out
▸ Quarterly meter reads become half-hourly
▸ Previously 80 million reads per year
▸ Now 350 billion reads per year
WHAT TO DO WITH THE DATA?

5. BIG DATA LDN
TRADITIONAL DATA WAREHOUSING
Use database running batch analytics
on usage patterns:
‣ Dip in consumption due to a
holiday or fraud?
‣ How to model region or cohort
demand?
‣ How can consumption be nudged
to fit supply patterns?

6. BIG DATA LDN
BIG DATA FOR FINANCE
▸ Infrastructure monitoring: 50 metrics x 000s of machines
▸ Server and application logs
▸ FX: $3.5 trillion per year
▸ HFT

7. BIG DATA LDN
DATA WAREHOUSE ISSUES
▸ Time criticality of analytics
▸ Unbalanced resource use
▸ Unbounded data means moving goalposts
▸ Data from disparate sources

8. DATA STREAMS

9. BIG DATA LDN
DATA STREAMS
▸ Analyse unbounded data “in motion”
▸ Seriously fast
▸ Often achieved using delta & approximation functions

10. BIG DATA LDN
DATA WAREHOUSE ISSUES
▸ Time criticality of analytics
▸ Unbalanced resource use
▸ Unbounded data means moving goalposts
▸ Data from disparate sources

11. DATA LAKES

12. BIG DATA LDN
DATA LAKES
▸ Flexible storage for huge amounts of data
▸ Structured, semi-structured and unstructured
▸ Achieve speed through indexing
▸ No need to define schemas or data dictionaries up front

13. BIG DATA LDN
DATA WAREHOUSE ISSUES
▸ Unbalanced resource use
▸ Unbounded data means moving goalposts
▸ Time criticality of analytics
▸ Data from disparate sources

14. BIG DATA LDN
VALO
▸ Open lambda-style architecture
▸ Multiple repositories
▸ Realtime or historical analysis
▸ AP system
Analytics engine
TSR SSR
Stream processing

15. BIG DATA LDN
P2P CLUSTER
‣ Scale up or scale out
‣ Perform the analysis where the data is

16. SOUNDS EASY?
Query language
Data atomicity
Distributed execution engine
Distributed CRDTs
Distributed algorithms
Time semantics
Back pressure
Vector clocks
Cluster management
Transports
Distributed joins
Expression trees
Runtime code-gen
Off-heap memory
Gossip protocols
Consistent hashing
Statistical models
Actor systems
Real-time queries
Elasticity
Semi-structured repository
Query rewriting and optimisation
Data distribution
REST API
SDKs
Gap filling
Time-series repository
KV store

17. BIG DATA LDN
LAKES & STREAMS SIMPLIFIED

18. BIG DATA LDN
DATA LAKES AND STREAMS WITH VALO
▸ Streaming analytics on time-critical data
▸ Real-time and historical analysis
▸ Different repositories for different kinds of data
▸ Can ingest anything

19. BIG DATA LDN
THANK YOU
For more info come speak to us at stand 320