The document summarizes a tool created by the author to remotely monitor up to 50 smart meter data collectors simultaneously. The tool logs into each collector via telnet, records debugging data, and monitors performance metrics over time. This enabled accelerated testing, diagnosis, and repair of issues causing collectors to reboot sporadically. With improved reliability, the utility saw 99.5% of smart meters reporting daily, allowing payments to resume and the company to become profitable for the first time. Management recognized the tool's value in improving product quality and supported continuing its development.

1. A Smart GridcompanyI workedatwas havingdifficultywiththe reliabilityof the “DataCollectors”,
whichwouldcollectmeteringdatafromsmartMeters intheirarea andtransmitthat data up to the
utilityheadoffice onadailybasis.The promise made tothe utilitywasthattheirsystemwouldreadand
report98+% of all installedSmartMeterseveryday.Theywere notmeetingthatgoal.
Each Data Collectorhada debugport,accessible viatelnet,thatrevealedwhatthe unitwasdoing.
Standardpractice was to use ExtraPuTTY to establishthe telnetandcapture all collecteddatatoa log
file forlaterexamination.Itwasa small manual chore toset upand examine datafromone Data
Collector,itwasa bigmanual chore to dothat withmultiple DataCollectors.Additionally,if the Data
Collectorrebootedorlockedupitstelnet,nofurtherdatawouldbe collected.
My solutionwastocreate a configurable tool thatwouldlogintomultiple DataCollectors,logtheirdata
withtimestamps,andexamine the incomingdataon-the-fly.Here isthe mainpanel of thattool:
Each horizontal line above representsone DataCollector.Notice the greenbarstopand bottom:Clicking
these scrollsthe listof collectorsupordownby tenrows.This tool will monitorup to50 Data Collectors

2. at a time,eachbeingmonitoredinitsownthread.Clickingthe “MainControl”fromOFFto ON startsthe
processand the telnetstoeveryDataCollectorthathas its“Activate”box checked.
The “Status” indicatesida connectionis active,andthe “CollectorID”isjustthe serial numberas
indicatedinthe configuration.
Clickingthe “Configure”buttononanyrow bringsupa panel like this:
The Device Type indicatesthe connectiontype,the Device IDisthe serial numberof the device,andth
IP addressesare given.If the connectionistobe anRS-232 port (supported) thenthe COMportnumber
isused.
Since I washandlingalarge numberof data collectorsscatteredall overthe building,andnotall of them
belongedtome,Ihad to keeptrackof where theywere andwhatI waspermittedtouse themforby
theirowners.Ihad myprogram track that for me,unit-by-unit.
The “TelnetsSuccessful”isthe countof successful telnetsmade tothe device.More thanone means
there wasan interruptionandthe telnetwasre-established,indictingapossiblereboot.
Clickingthe “GraphSelectionPanel”buttonbringsupthispanel:

3. Duringthe monitoringthere are several itemstrackedovertime.Since the programisnotrunningat the
time of thiswritingthere all sample sizesare zero.Aninterestinggraphisthe “ActivityGraph”,which
graphs the linesof debuginformationminute-by-minute,showingwhenthe collectorwasmore
stressed.Itrackedline-countsof upto 40,000 linesof debuginformationperminute oncome collectors.
The “BufferGraph” showsthe numberof free buffersreportedovertime,makingiteasytodetecta
bufferleakthatmightnotcause a collectorresetforweeksormonths.
The “Show Status”buttonon any rowbringsup a panel like this:

4. Thisis a summaryof statisticsandnotedproblemssee todate.The contentsof thiswindow are also
preservedinasummaryfile forthisdevice once perminute,keepingitconstantlyupdated.This
information,of course,can be viewedwithoutinterruptingthe monitoring.
The “Show Trace” buttonbringsupthis panel:

5. Whenthispanel isup onany device newlinesarrivingfromthe device are shownhere.Itletsyousee
whatis happeninginthe monitoringwithout interruptingthe monitoring.
The “Notes”buttonsimplybringsupthe contentsof a textfile thatI can make notesto. It isstoredby
serial numberandwill surviveadevice beingremovedandrestored.

6. Everytime the DebugMonitoris startedit makes a new time-stampedlogfile folder.Whenthe
monitoringisstartedafolderforthe logsfilesismade foreach device:

7. Inside the folderforadevice are the logfiles:

8. Logfile.logisthe global logfile,all debuglinesgothere.Butsome linesof debughave relationto
individualfunctionswe wanttotrack, sothose are alsostoredina special logfile forthatfunction.That
wayyou can quicklyviewwhathappenedregardingthatfunctionwithouthavingitslines“polluted”by
linesforotherfunctions.
Additionally,there are otherthingsgoingonoutside of the debugporttelnet,like anothertelnetmade
once per minute toquerythe device forstatusonsome functions.Those queriesgototheirownlog
files,likeTstBenchRead.txt.

9. I nevergoto where I wasmonitoringmore thanabouttwentycollectorsatone time,Ijustdidnot have
access to enoughunits.HoweverIfoundalot of value inmonitoringthose twenty,since the failuresI
was huntingwere sointermittent.The ease of restartingthe monitoringwaswonderful:One clickand
thenwatch ithappenonthe LEDs, as opposedtothe longmanual processof settingupall those telnets
and startingdata logging.The reliabilityof the programallowedme tocomfortablygohome fora
weekendandknowthatIwouldhave soliddatawhenIarrivedback at work onMonday morning.
My extensivepre-sortedlogfilesalsoenabledme toeasilyanalyze problemsandfile verydetailedbug
reports,whichmade iteasierforthe developerstounderstandandfix the problems.Bugturn-around
was prettygood.
The resultof all thistoolswasgreatly the acceleratedtesting,diagnosis,andrepairof causesof spurious
collectorrebootsandotherissues.Withthe improvedreliabilityof the collectorsthe numberof smart
metersbeingreadona dailybasisshotup to 99.5% andthe utilitycouldnolongerwithholdpaymentfor
an installationforlackof performance andthiscompanyexperienceditsfirstprofitable quarter.
Whenthe company’smanagementsawthe tool I was workingonand how it helpedtheirproduct
qualitytheyagreedthatcontinuingthiseffortwasmymandate andtheymade allowancestoenable me
to pursue it.I wouldsaythose allowancespaidoff.