Analyzing the HAN Firmware Maintenance Problem

1. Analyzing the HAN Firmware
Maintenance Problem
November 2009
Marcia Martin
Director, Systems Engineering & Smart Energy Practice
303–453–8323 marcia@magpieti.com
Roxanna Podlogar
Vice President of Sales & Marketing
303–453–8380 roxy@magpieti.com
12050 N. Pecos Street, Suite 210
Denver, CO 80234
www.MagpieTI.com

2. Problem Statement
Providing the ability to remotely upgrade Home Area
Networks (HANs) is a CRITICAL NEED for the Smart Grid.
Why?
Utilities will use HANs to create 2-way communication with
customers.
HANs will reside in millions of households.
Dispatching a utility truck to upgrade HAN firmware is
cost-prohibitive.
Failures or security breaches in the energy management
infrastructure can threaten the energy grid itself.
No consensus has been established as to how solve the
problem.
www.MagpieTI.com 2

3. What is a Home Area Network (HAN)?
Low-cost, in-home network of intelligent devices
Usually wireless, mesh architecture
• ZigBee More about mesh
• WiFi networks
later on…
Alternatives
• HomePlug (a powerline mechanism)
Devices have limited or absent user interfaces
Usually no ability to access removable digital media
Originally “toys” for controlling home video, audio, lights
Now getting serious attention as a mechanism for
managing home’s energy consumption in the Smart Grid.
www.MagpieTI.com 3

4. What’s in an Energy-management HAN?
Smart Meter (advanced metering infrastructure)
Programmable Thermostat
In-home Display
DR-enabled smart plug
Range Extender
GE Smart Fridge
RE
HAN/Internet Gateway
More…
www.MagpieTI.com 4

5. HAN Connectivity Options
Advanced
Metering
Infrastructure (AMI)
[Private
Infrastructure;
Proprietary
Protocols]
RE
Internet
Accesses
utility portal
but not HAN
www.MagpieTI.com 5

6. Gateways to the HAN RF
Mesh or
ZigBee
Cellular AMI
Radio
Radio
HAN
RE
Backhaul to
Internet
Ethernet
connection
ZigBee to home
Radio DSL or
Cable Internet
Modem
www.MagpieTI.com 6

7. The Bigger Network Picture
Advanced
Metering
Infrastructure
[Private
Infrastructure;
Proprietary
Protocols]
Neighborhood Area
Concentrator
Utility
Consumer Energy
Cellular or
Portal Services
Fiber
Provider (ESP)
Backhaul
Internet
Public
Utility
www.MagpieTI.com 7

8. Observations
Two or three potential data paths from HAN to ESP
• AMI infrastructure (safest, least cost solution)
• HAN/Internet specialized gateway device (not always present)
• Internet-connected home computer (not typically part of the HAN)
Network Services provided to the HAN
• Data Collection (extension of AMI)
• Demand Response signaling and opt-in/opt-out messaging
• Consumer Rate Management
Emerging model for Energy Management Services
• Commercial Energy Service Providers (ESPs) partner with utilities
• Utilities own the data collected by the ESPs
• Consumers access data via Utility Internet Portal
• Consumers access data via HAN-connected in-home display
www.MagpieTI.com 8

9. Why HANs Need Software Distribution
Security Breaches
• Protect Consumer Privacy
• Protect the Grid
Support for New Features
• Two-way communication between Consumer & Utility in its infancy
• Demand Response
• Time of Use
Interoperability Issues
• Consumer expects to be free to introduce devices from any source
• Consumer will expect utility features to always work (!)
Software defects
• Inevitable
www.MagpieTI.com 9

10. You think it won’t happen?
2005 – Toyota recalls 75,000 Prius hybrids for a firmware
defect
2004 – Pontiac recalls Grand Prix because firmware failed
to adjust for leap years
2002 – BMW 745i had fuel pump that would shut off
when the tank became less than 1/3 full
2001 – 52,000 Jeeps recalled due to software defect that
shut down the whole instrument panel
– “Total Recall” Jack Ganssle, 2/6/2006 in embedded.com
www.MagpieTI.com 10

11. Some Wireless AMI Network Options
RF Mesh (42% of Utilities in the US favor)
• Itron
• Landis+Gyr
• Elster
• Silver Spring Networks (add-on)
• Trilliant (add-on)
Tower-based Communications
• Sensus (7 million meters under contract)
Cellular Networks
• Favored in Europe – little US adoption so far, but…
• Echelon
• SmartSynch
– Source: Advanced Metering, Jeff St. John, June 5, 2009 in
greentechmedia.com, reporting on a survey of over 100 utilities by Atlanta-
based energy research firm Chartwell.
www.MagpieTI.com 11

12. Power Line Networking
Popular in Europe
100% connectivity to meters
High bandwidth (comparable to broadband)
Downsides
• Cost
• Modulating power interferes with ham and emergency bands
Italian model – short runs over power lines from meters to
concentrators that are gateways onto internet
infrastructure
www.MagpieTI.com 12

13. Magnitude of the HAN Upgrade Problem
Example: ONCOR, Texas
700,000 Landis+Gyr Smart Meters by year end 2009
3 Million Meters deployed by end of 2012
The Gridstream™ solution is compliant with ZigBee®
Smart Energy Profile 1.0
On Sept. 14, 2009 Landis+Gyr announced a Program to
certify HAN device interoperability with its AMI Solution at
the SEP 1.0 level
www.MagpieTI.com 13

14. How Much Data Must Move?
Assume:
• 32k average firmware load per HAN device
• 6 devices per HAN
• 3 Million Households in Utility Network
Approximately 200k bytes to upgrade entire HAN
Move 600 GB to upgrade all households in the Utility
Where are the bottlenecks in the network?
What are time interval requirements for upgrade?
www.MagpieTI.com 14

15. Bandwidth Bottlenecks
Network Top Data Rate Households per Approximate
Technology (Bandwidth) Band Push Time
ZigBee 250 Kbps 1 1-3 sec/device
25 KB/sec 8-20 sec/HAN
DSL to 24 Mbps 1 .1 sec to laptop
household (max download) or HAN
2.4 MB/sec gateway
Neighborhood 100Mbps 5 to 5000
Area 10 MB/sec
Concentrator (100 bT Enet)
900 Mhz 20 Kbps 5 to 5000 16 sec/device to
RF Mesh 2 KB/sec depending on meter broadcast
(cheapest AMI) technology or single thread
www.MagpieTI.com 15

16. Bottleneck Analysis Takeaways
The ZigBee distribution time for one household is
acceptable.
Firmware updates don’t pose a problem for internet-
connected HANs.
Direct-connect to internet for every HAN is expensive
• Dedicated IP/ZigBee gateway $200-300
• USB ZigBee Dongle for home computer $60
– And, oops, computer can be turned off!
• All homes have meters – not all homes have broadband.
THE BIG QUESTION: Can the AMI network meet the
firmware distribution requirements?
www.MagpieTI.com 16

17. Is the AMI Infrastructure Good Enough?
“Normal” load on AMI Mesh
• 60KB/meter/day = about 1 character per second per household*
• Or 5KB/sec per “neighborhood”
• Actually this is typically concentrated in bursts on 15-minute intervals
• Firmware distribution should not unduly disrupt this traffic, but
• Many meters can store some interval data
Two models for firmware distribution
• Broadcast
– All households receive same download
– Houses that can’t use or don’t need the broadcast ignore it
• Connection-oriented
– Examples: HTTP or FTP download
– “Conversation” between house meter and ESP’s NOC
– Can be tailored to individual household’s device types & topology
– Greater potential for stressing the mesh’s capacity
* Implementing the Right Network for The Smart Grid. www.UtilitiesProject.com
Raj Vaswani & Eric Dresselhuys, Sliver Spring Networks
www.MagpieTI.com 17

18. Mesh Behavior of the AMI
Every meter acts as a repeater for adjacent meters
Not a problem for broadcasts
Connection-oriented streams can overrun the mesh
www.MagpieTI.com 18

19. Mesh Behavior
This connection stream pattern stresses the mesh
These repeaters
have too much
work to do!
www.MagpieTI.com 19

20. Mesh Behavior
This connection stream pattern doesn’t
Each connection has own path from concentrator to meter
www.MagpieTI.com 20

21. How long to update all 3 Million HANs?
One household at a time, 3333 days or ~9 years.
If all households were updated simultaneously by
broadcast, about 2 minutes, assuming:
• No load failures occurred
• All households contained same 6 devices (or a subset thereof)
• Neither assumption is realistic.
What assumptions are realistic?
• Would NOT broadcast to the whole service area all at the same time.
– Too big a disruption if the broadcast code load were wrong.
• Several rebroadcasts of each load will be needed, or
• A fallback method is available to deal with individual home failures.
• Interval between broadcasts allows for failure reports to come in.
www.MagpieTI.com 21

22. NEMA SG-AMI 1-2009 Standard
Establishes requirements for updating smart meter
firmware over the AMI.
• HAN gateway component shall be upgradeable independently from
AMI network component.
• Meter shall be able to store new code load while running off old code.
• Meter shall be able to detect and signal load failures.
• Meter shall defer booting new load until signaled to do so.
• Meter shall be able to revert to old code load or enter a safe state.
• Meter shall be able to authenticate trusted distribution sources.
• Standards are established for secure encryption of transmissions.
What NEMA SG-AMI 1-2009 does not do:
• Define any actual protocol for transmitting firmware loads.
• Prefer one distribution model over another.
• Require interoperability among different vendors’ products.
• Address the additional requirements and failure modes HANs
introduce.
www.MagpieTI.com 22

23. Meter Upgrade Broadcast Scenario
1. Broadcast Meter Upgrade out of 1st Neighborhood Area
Concentrator (up to 5000 homes)
2. Wait 1 hour while compiling failure rate information.
3. Re-broadcast if there are any failures.
4. Determine how many homes are still not upgraded.
5. Assuming failure rate is acceptably low, repeat steps 1-4
for additional NAC areas until all zones are upgraded.
6. Use fallback update method on failed homes.
1. Root-cause analysis on failure patterns.
2. Connection oriented update attempts.
3. Truck rolls as appropriate.
7. Issue command for meters to deploy new firmware.
www.MagpieTI.com 23

24. How long to upgrade 3 million meters?
2 hours for initial 1-zone trial.
Double zones involved for second round.
Double zones for each successful round subsequent.
Assume 600-1000 zones total.
(Each concentrator serves up to 5000 meters).
Process would complete in 18-20 hours (2 * log2(1000)).
This estimate excludes accounting for meters that failed to
upgrade after repeated broadcast attempt.
Depending on meter architecture and ESP’s firmware
management architecture, 1 hour interval between
broadcasts may be too short or too long.
www.MagpieTI.com 24

25. Why is a HAN upgrade different?
Feature upgrade to a single HAN device is not different,
providing:
• The upgrade does not affect the firmware load capability,
• The upgrade does not affect the security model,
• The upgrade does not affect the mesh networking logic.
If one or more of the above provisions don’t hold, then
usually all HAN devices, not just one, needs to be
upgraded.
Many HAN devices being deployed today lack the ability
to store a firmware load and defer rebooting it into
production.
This could impose constraints on the upgrade order for
devices in a whole-HAN upgrade.
www.MagpieTI.com 25

26. Additional Requirements for Upgrading HANS
HAN devices should adopt many of the meter requirements
described in NEMA SG-AMI 1-2009:
• Separately upgradeable “application” logic and network logic.
• Ability to accept code load (securely delivered from trusted source)
without deploying it until commanded.
• Ability to alert when code load fails and/or respond to version query.
• Ability to enter safe state (preferably still able to communicate) when an
upgrade fails or hardware failure is detected during upgrade.
HAN device topology and capabilities must be discoverable
(example in ZigBee terms)
• Is device endpoint, router, or controller capable?
• Version of running & loaded network & application code
• Distance (hops) from nearest router & controller.
Ideally, load and deployment protocols are standard (a
stronger requirement than NEMA placed on meters.
www.MagpieTI.com 26

27. Conclusions
Upgrades to single device types are manageable via
broadcast techniques even over least-capable AMI
networks.
Tailored connection-oriented whole-HAN upgrades are
practical only on HANs with broadband-class connectivity.
A real-world emergency upgrade of an entire HAN-
enabled service area including many heterogeneous
devices is not practical over least-capable AMI networks.
• “Emergency” defined as can be accomplished in one weekend.
• Experience will tell whether this capability will be needed or not.
• Deeper analysis of security and distribution failure modes is needed.
• Broadcast-only distribution model falls down if HAN devices cannot
accept a firmware load but defer deployment of it.
www.MagpieTI.com 27

28. Process Requirements
So far, we’ve discussed only physical connectivity
Firmware publication process:
• Establishing distribution repositories for binary images
• Versioning and interoperability
• Who is responsible for testing and certification?
• Who controls and tracks distribution?
Standard protocol for HAN devices to accept firmware
loads (proposals before ZigBee Alliance now)
HAN discovery (distributor must know all this stuff)
• Device ID and Level for all devices
• Topology (WAN connectivity, proximity to concentrator, and topology
of HAN inside the house)
• Distribution failure modes & remedies are a topic for another
presentation!
www.MagpieTI.com 28

29. State of the Art (in the field)
Most HAN device vendors offer some sort of network
distribution for updating device firmware
• Control4
• Tendril
• Gridpoint… and others, too
No HAN firmware distribution standard is yet approved
NEMA SG-AMI 1-2009 requirements for meters
• Does not set protocol standard
• Does not require interoperability among meter vendors
• Only touches on the HAN problem
Vendor solutions are therefore incompatible
Existing ESP solutions limit the devices the utility supports
• Support only one proprietary upgrade mechanism
• No ESP deployments really old enough to test firmware distribution
capability
• Many do not handle intra-HAN dependencies
www.MagpieTI.com 29

30. Who owns the problem?
Possible candidates:
• Consumers
• Device Vendors “The Consumer cannot be
permitted to own the
• Retail Outlets (== consumers) process of upgrading the
Energy Management HAN.”
• ESP/Utility Partnership
• Regulatory bodies
--GridWeek 2009
Panel Session on Network
Security
Roles to fill
• Set interoperability standards
• Create and publish firmware updates
• Interoperability and Replace-ability Testing
• Publish interoperability results
• Coordinate distribution to homes
• Correct distribution failures
www.MagpieTI.com 30

31. Thank you for your time and attention!
Any questions?
Want to know more?
http://www.magpieti.com/whitepapers
www.MagpieTI.com

32. Image Credits
Itron Centerpoint Smart Meter: www.centerpoint.com.
Comverge in-home display. www.comverge.com.
ecobee™ smart thermostat. www.ecobee.com.
Daintree MeshOperator Management Gateway.
www.daintree.net.
GE Smart Fridge, image from www.thatsthestuff.com.
Electrical Outlet, image from www.mindrights.com.
Sony VAIO notebook. www.sony.com.
www.MagpieTI.com 32

33. About the Author
Marcia Martin is the Director of Systems Engineering – Smart Energy, for
Magpie, a specialty software services company in Denver, CO. With over 20
years of technology experience, Marcia has dedicated her career to innovating
in emerging technology fields in the storage, telecom and health care domains.
Her role in Magpie’s Smart Energy Practice is to transfer well-accepted
technology principles, methodologies and tools that are recognized by
communications, information technology, and related disciplines into the
products of Smart Energy companies.
Marcia holds Bachelor of Science and Master of Science degrees in Computer
Science from Washington University's Sever Institute of Technology in St.
Louis, MO.
www.MagpieTI.com 33

34. About Magpie
When Your Business Depends On Software™
Magpie is a valued partner in delivering engineered software that works™. The
company specializes in solving tough technology problems for communication–enabled
applications, business support systems, interactive Internet applications, and network
protocol integration. The Magpie team is well known for its Agile, iterative, and open
software development process. With customers spanning North America, Magpie has
consistently delivered on–time, on–budget results for the telecom, health information,
and emerging smart–grid energy industries.
Magpie’s core expertise in the telecommunications arena has supported and enhanced
Magpie’s growing Smart Energy practice, as companies innovating in Smart Energy
continue to flourish and grow in Colorado’s Front Range tech corridor. We’re excited
about this emerging field!
The company warranties all its work and also has unique expertise in providing systems
architecture, automated testing, systems integration, and technology consulting
professional services. Make Magpie your choice when your business depends on
software™.
www.MagpieTI.com 34