Development and Applications of Distributed IoT Sensors for Intermittent Connectivity Environments | Kevin Claytor | US Army Laboratories

APPROVED FOR PUBLIC RELEASE
U.S. ARMY COMBAT CAPABILITIES
DEVELOPMENT COMMAND –
ARMY RESEARCH LABORATORY
Dr. Kevin Claytor
Research Physicist
CCDC-ARL / SEDD / Electric- and Magnetic-Field Sensing Team
DD MMM YYYY
Development and Applications of Distributed IoT Sensors for Intermittent Connectivity
Environments
Approved for public
release; distribution is
unlimited.

2
• Mission:
– Discover, innovate, and transition science and technology to ensure dominant strategic land power.
• Research focus:
– Concept idea to early prototype (6.1 – 6.3)
– Fundamental new technology
– Demonstrations
– Transitions to the warfighter
• About me:
– Physics
– Geophysics, optics, NMR/MRI,
electric- and magnetic-field sensing
• Why I’m here:
– Started with Influx ~1yr ago
– Enables rapid prototyping
– Easy demos
– Enables collaboration
CCDC – ARMY RESEARCH LABORATORY
THE NATION'S PREMIER LABORATORY FOR LAND FORCES.
Credit: Shaohan Hu
https://maps.google.com/
Credit: ARL

3
• Problem: Army IoT Environment
– Internet of Battlefield Things
– Contested, congested, dynamic network
– Need actionable data
• Our team: Low Frequency Electric- and Magnetic-Field Sensing
• Our technology: Noncontact Power Monitoring
– Safety Advantages
– Cost Advantages
– Driver for efficiency and resiliency
• Application areas
– Mobile Power Meter
– Environmental data
– Sensor array
– Wide area surveys
ARMY SPECIFIC CHALLENGES – A DYNAMIC NETWORK
• Influx – Edge & Cloud
– Data stack at edge and in cloud
ensures customer has data
– Replication Scheme ensures
latest points are replicated

4
COMMON SENSOR PROCESSING IN MULTIPLE FORM
FACTORS
• Ruggedized for field
experiments
• BNC inputs (voltage / current
probes)
• LEMO inputs (TEDS
compliant smart sensors)
• Integrated battery
• Designed for microgrid power
cables
• Integrated noncontact field
sensors
• Can be installed safely by
untrained operator
• Integrated battery
• Designed for substation
installation
• Supports up to 128 inputs
Common features
• 8-16 ksps raw data
• FPGA / Linux hardware
• Common processing software
• IMU / Orientation
• GPS / WiFi / Ethernet
Mobile Unattended
Ground Sensor (MUGS)
Mobile Power Meter
(MPM)
Rack Mount Chassis
(Rack)

5
MANY HARDWARE / SOFTWARE OPTIONS TO FIT CUSTOMER
5*4*3*5*4 > 1000 combinations!
5
Hardware:
Processing:
Network:
Analysis:
Mix and match sensors, processing, and analysis to suit your needs
None Wifi BluetoothEthernet Cellular 3G/4G
dLAMP
(attached PC)
C / Python
(embedded)
Transmit to
Cloud
Storage:
Local Database
(on sensor)
Nearby Database
(attached PC)
Cloud Database
(govCloud)
dLAMP
(embedded)
dLAMP
(cloud server)
dLAMP
(embedded)
Export
(binary / text)
dLAMP
(attached PC)

6
IOT DEVICE MANAGEMENT – VIPERS
(VISUALIZATION AND PROCESSING FOR EMBEDDED RESEARCH SYSTEMS)
Webserver
Sensing
Visualization
Embedded
• ARM
• FGPA

7
Webserver
NetManager
(networking)
Dataserver
(process manager)
Plot server
ControlSignals
Sensing
Visualization
Embedded
• ARM
• FGPA

8
Webserver
NetManager
(networking)
Dataserver
(process manager)
Plot server
ControlSignals
Processing Module
Sensing
Visualization
Processing Module
Processing Module
Embedded
• ARM
• FGPA

9
Webserver
NetManager
(networking)
Dataserver
(process manager)
Plot server
ControlSignals
Processing Module
Sensing
Visualization
Processing Module
Processing Module

10
• Raw data:
– 8,000 – 16,000 samples per second
– (x8) 8-16 channels of data
= 64,000 – 256,000 values / s
• IEEE “synchrophasor” Std. C37.118:
– Converts one or more cycles of sine-wave data into a
“phasor measurement”
– Reduces high data rate to more manageable size
– Loses high frequency transients
SINUSOIDAL POWER DATA IS PROCESSED INTO “PHASORS”

11
• Raw data:
– 8,000 – 16,000 samples per second
= 64,000 – 256,000 values / s
• IEEE “synchrophasor” Std. C37.118:
– Converts one or more cycles of sine-wave data into a
“phasor measurement”
– Reduces high data rate to more manageable size
– Loses high frequency transients
• Phasor data
– (10 Hz) Estimators produce data from 10 – 60 Hz
– (x5) Estimates for fundamental (60 Hz), two
harmonics, frequency, and total harmonic distortion
– (x2) Values are complex
= 800 – 9,600 values / s
– Want to store at the edge
PHASOR PROCESSING REDUCES SAMPLE RATE, BUT A HIGH
INGEST RATE IS STILL REQUIRED

12
• Well defined API’s
– Others can write software to our module
APIs
• Distribute as components
– Leave out unnecessary components
• Smaller updates
– Update only the modules needed
• Performance versions
– Rewrite single component for performance
instead of entire stack
• Can swap components based on
license
– Exchange databases
PYTHON MODULES
synchro.py
dbus inl simulator
relative.py
influx.py extras.py
replication.py
dLAMP L11 Phasors
dLAMP L11 Phasors
HTTP API
Applies:
[[channels]] and [[circuits]]
Standardizes disparate
data sources

13
• Influx is a NoSql storage
• Influx does not require a schema
– Good: Easy to get going, can rapidly shift
and experiment with schemas
– Bad: Schema may be different than
expected
– Compromise: We define the schema in
docs
• Influx does enforce type consistency
– Good: Ensure that numeric data stays
numeric
– Bad: Initial commit of an int (eg; 0) can
prevent later floats from being committed
(eg; 1.023)
– Compromise: Explicitly cast values before
writing them (eg; float(0))
DATABASE SCHEMA CONSIDERATIONS
• Local Schema
– Used on the MUGS
• Cloud Schema
– Used on influx.arlpower.net
– Databases for function / assets
– Allows for transition of assets in locations
– Allows access control by asset
• Database schemas are *identical*
– Cloud queries can be used in MUGS
– MUGS queries can be used in the Cloud
• Downsampling for long duration data
– Run CQ to downsample lower rate for a
longer period
• Backups / Restore
– We can backup to RAID

14
• replication.py
– Clones local data to the cloud
– Logs if data replication is successful
– Retries failed replications
• Algorithm
– Query for previously replicated regions
– Query for last N points avoiding these
regions
– Attempt to replicate
– If successful, mark a new replication region
REPLICATION – GET LATEST POINTS WHILE AVOIDING
PREVIOUSLY REPLICATED REGIONS
data
time
Capture last N points
Mark region replicated
data
time
Capture last N points
(avoiding region)
Pass 1:
Pass 2:

15
• replication.py
• Algorithm
– Query for previously replicated regions
– Query for last N points avoiding these
regions
– Attempt to replicate
– If successful, mark a new replication region
– Merge overlapping replication regions
(set field value = 0)
REPLICATION – GET LATEST POINTS WHILE AVOIDING
PREVIOUSLY REPLICATED REGIONS
data
time
Depreciate overlapped regions
Capture latest N points
(avoiding region)
Pass 3:

16
Time
• Every CPU cycle is valuable
• Performance testing on MUGS v2 (1st generation)
– Many modules are written with adjustable inputs (eg; data rate limiting)
– CPU consumption is total CPU consumption (eg; total system usage, influx, telegraf, etc.)
– Highlights key bottlenecks
– Performance tradeoffs
PERFORMANCE – EASY TO QUANTIFY WITH TELEGRAF
core100.py
synchro.py
influx.py (all points)
influx.py (nominal)
influx.py (low data rate)
relative.py
replication.py extras.pyinflux.py (all, med, low)
x2 circuits

17
• Every CPU cycle is valuable
• Performance testing on MUGS v2 (1st generation)
– Insert by series saves ~30% CPU
– Possibly related to lower overhead in python client
PERFORMANCE – QUANTIFYING ALLOWS IMPROVEMENTS

18
• GZIP compression on data
– Repeated tag values
OPEN SOURCE CONTRIBUTION – GZIP FEATURE
if self._gzip:
# Allow us to receive gzip'd data
# as well as write it out
headers.update({
'Accept-Encoding': 'gzip',
'Content-Encoding': 'gzip',
})
if data is not None:
# For Py 2.7 compatability use Gzipfile
compressed = io.BytesIO()
with gzip.GzipFile(
compresslevel=9,
fileobj=compressed,
mode='w'
) as f:
f.write(data)
data = compressed.getvalue()

19
DON’T TOUCH THAT WIRE! – MOBILE POWER METER
• Problem:
– Power is a resource
– Assured power needs measurement
– Measurement devices($100 - $10k)
– Electrician ($100s / hr)
– Power interruption ($X / hr)
• Solution:
– $0 power interruption
– $0 untrained solider installation
– $3-5k / device
• Impact:
– Informed energy consumption
– Saves energy, safes fuel, saves lives

20
• Calibration dependent on location around cable
• Adding inertial measurement unit (IMU) data to DB
– Identify if sensor has moved and needs re-calibration
– Measurements of field as a function of angle around cable
– Previously had to estimated cable angle assuming
“constant rotation rate”
MOBILE POWER METER – ADDING ORIENTATION DATA

21
ENVIRONMENTAL SENSING TO PROMOTE CONSERVATION
AND EFFICIENT BUILDINGS
• Goal: Improve building efficiency using
machine learning
– Relationship between power usage and
environment can indicate poor efficiency
– Inform building renovations – identify buildings that
yield highest ROI from renovations
• Proof of concept with a small box and space
heater
– Encouraging results and scaling up

22
• Student code:
– Weather sensors
– AC Control
• Joint code:
– InfluxDB-python connector
– Writes to our sensor DB
• Replication:
– Check the new field to replicate
SUPPLEMENTAL MEASUREMENTS ARE EASY TO REPLICATE
• Temperature
• Humidity
• Light
• AC Control
replication.py
+weather_station
https://opensource.com

23
ENVIRONMENTAL SENSING TO PROMOTE CONSERVATION
AND EFFICIENT BUILDINGS
• Goal: Improve building efficiency using
machine learning
– Relationship between power usage and
environment can indicate poor efficiency
– Inform building renovations – identify buildings that
yield highest ROI from renovations
• Proof of concept with a small box and space
heater
– Encouraging results and scaling up
• Sensors and control through a Raspberry Pi
– Pushes data to ARL sensor over local network
– ARL sensor replicates data into cloud
– Student summer project
– Interfaced with existing database and replication
• New shed instrumented and collecting data
• Using Chronograf to share data
Credit: Drummond, Zachary
Credit: Drummond, Zachary
MUGS / Jetpack
Raspberry Pi
AC Unit

24
MAGNETIC FIELD SENSOR ARRAY – 3 AXES SENSORS
• 4D Sensor “pixel”
– 3-Axis H-field sensor
– 1-Axis E-field sensor
– 2x pixels per processing board

25
FULL 6X8 ARRAY WITH BORDER ELEMENTS

26
• Hardware
– Supports up to 128 channels
– Array uses 4x6 x4d sensors
= 96 channels of data
• Software
– Array visualization tool
• Impact
– Test fundamental resolution
limits
IMAGING ARRAY: A LOCAL AREA “CLOUD”
Array visualization
Single Board
Computer
(i7 with CentOS)
Backplane
Signal
Conditioning
Board
ARTEMISMOBILE
Raw
SD
Phasor
Processing
InfluxDB
x6
ARTMEIS RACK
x24
Signal
Conditioning
Board
ARTEMISMOBILE
Raw
SD
Phasor
Processing
Signal
Conditioning
Board
ARTEMISMOBILE
Raw
SD
Phasor
Processing

27
• Array Designer + Streamer
– Allows design of NxM arrays
– Can be sparse arrays!
(Saves data efficiently)
• Live Visualization
– Select between harmonics, magnitude or
phase
– Dynamically connects
(handles disconnects, etc.)
– Dynamically adjust scaling
• Can even be run on the edge devices
• Dirty secret: Doesn’t use Influx
– Easier to reshape data packet than
manipulate query results
– May need more accurate solution for higher
frequency, phase sensitive measurements
STREAMING ARRAY VISUALIZATION

28
• Hardware
– 3-Axis E-field sensor
– 3-Axis H-field sensor
– 6-Axis Inertial
measurement unit (IMU)
– Integrated GPS
• Software
– Measure 60-Hz signals
– Use IMU to align to world-coordinates
– GPS to place on map
• Impact
– Background measurements facilitate sensor
placement for high-precision measurements
– Disaster recovery – Puerto Rico / Bahamas
Identify infrastructure most in need of repair
WIDE AREA ELECTRIC & MAGNETIC FIELD SURVEYS
Vector fields are
aligned to world
coordinate frame
MUGS
E-Cube
H-Cube
x
yz
Latitude
Longitude

29
• What’s a good configuration?
– Is there a way to automate configuration and “tune”
to the host system?
• Some hacks I’ve noticed
– Disable _internal logging
– Switch from inmem to tsi1?
(Memory constrained platforms)
• Hard power cycle
– Restart through power switch / dead battery /
pulling power
– Corrupts lots of files
– Recovers a few files and then fails
ANECDOTAL ISSUES
OR HOW I NOW WORRY ABOUT DELETING THE DATABASE
• Query data by series
– Fully specified series query much faster
(helper scripts scrape data and dump into csv)
• Still difficult to get pandas arrays out
• How to map data into an image like structure?
https://github.com/influxdata/influxdb/issues/10939

30
• Accelerate technology development,
demonstrations, and transitions
• Fundamental new technology
– Easy to insert new measurements
– Chronograf in cloud is nice collaboration tool
– Telegraf helps identify resource bottlenecks
and optimize code
– Helps to quantify things
• Demonstrations
– Short turn-around time for new demo
– Chronograf -> Visualization
• Transitions to the warfighter
– Actionable information: Kapacitor Alerts
• Downsides
– Hidden “gotchas”
– File corruption
TEAM, SUMMARY, AND QUESTIONS
data
time
Replication:

31
Additional Slides

32
• Working on obtaining electric- and magnetic-field map of ARL test field.
FIELD MAP OF ENVIRONMENT FACILITATES PRECISION / LOW
NOISE SENSOR STUDIES / ENVIRONMENTAL STUDIES
TODO:
Field Map

33
Problem:
• Typical Helmholtz coil has given field with
zero derivatives only in the center of the
coil and in free space.
• Presence of steel walls in laboratory
impact performance.
Solution:
• Numerically model lab and surroundings.
• Optimize the currents and spacing of the
cage coils to maximize uniform region.
H-FIELD CAGE OPTIMIZED TO PRODUCE LARGE UNIFORM
REGION
Credit: Hellwig, Ansgar
https://en.wikipedia.org/

34
• TODO: Possible slide here from IoBT folks.
ARL INTERNET OF BATTLEFIELD THINGS (IOBT) CRA
TODO: POSSIBLE IOBT SLIDE

35
WHAT IS VIPERS
Webserver
Dataserver
Network
Manager
Plot server
ControlSignals

36

37
• replication.py
• “API” allows additional replication
– Another process modifies historical data…
– Just mark the next sow point as “not-
replicated” or insert a new sow point
– Replication script gathers that data on the
next pass and replicates
REPLICATION.PY
• Influx Replication Flags
– Measurement = “replication”
– No tags!
– Field: “response”
• Code 1 = success
• Code 0 = not yet tried
• Code <0 = error
• Algorithm
– Sow Thread: “Sows” replication points with code
0 every <sow_interval> s.
– Reap Thread: Every <reap_interval> searches
for up to <reap_points> of replication
measurements with response <= 0
– Queries selected measurements within non-
replicated region(s)
– Attempts replication and stores results
Reap: Gathers data
between non-replication
points
Sow inserts new points
First pass (N = 4)
Second pass (N = 4)
success
failed
untried

38
REPLICATION.PY
Sow inserts new points
First pass (N = 4)
Second pass (N = 4)
Reap: Gathers data
between non-replication
points
success
failed
untried
Maximum extent: sow_interval
Will not replicate to -Inf

39
• The database schema defines how
data is inserted into the database.
– Impacts insert performance
– Impacts query performance
– Impacts ease-of-use
• How difficult is it to make a query
• What are the implications if a query is mal-
formed (eg; missing a GROUP BY())
DATABASE SCHEMA - CONSIDERATIONS
• Encouraged
– Encode metadata in tags
• Store common-queries in tags
• Store GROUP BY () data in tags
• Store function() data in fields
• Store non-string data in fields
– Avoid using keywords as identifier names
• Discouraged
– Don’t have too many series
• Cardinality = unique(fields) * unique(tags)
• Highly variable tags increase cardinality
• Higher cardinality = lower performance
– Don’t encode data in measurement names
• Simplifies queries
– Don’t put more than one piece of
information in a tag
• Otherwise you’ll have to use regexp to
separate

40
DATABASE SCHEMA – REJECTED
• Measurement: phasors
• Fields: channel
• Tags:
– type [current, voltage, power, frequency, thd]
– center_frequency [string, 60, 180, …]
– component [“real”, “imag”]
• Issues:
– Have to scan first for channel names
– Each query now becomes 2+ operations
• “SHOW FIELD KEYS”
• For each field, get me all the data for that channel
• Ex: what query selects only voltage measurements?
– OR: over-query and down-select
• “SELECT * FROM “db.rp.phasors”
• Overcommits CPU

41
DATABASE SCHEMA – REJECTED
• Measurement: level [synchro, relative, power]
• Fields: type [current, voltage, efield, hfield, power, frequency, thd]
• Tags:
– channel [string]
– type [current, voltage, power, frequency, thd]
• Issues:
– Each measurement may have different available fields
• Measurement = synchro supports type = [current, voltage, frequency, thd]
• Measurement = power supports type = [voltage, power, frequency, thd]
– Measurement may be missing
– > Similar queries must be re-written based on measurement type
even though the operation is similar (“yield all voltage
measurements”)

42
• Tags:
– circuit [string]
– level [string]
DATABASE SCHEMA - REJECTED
• Issues:
– Circuit and Level are very rarely used in queries
• Mostly used in post-processing graphics (and sometimes not at all)
– Complicates queries (specify two additional parameters)
– Allows for inadvertent power summation
• Eg; not specify GROUP BY “level” would sum synchro, relative, and power
measurements.

43
• Tags:
– component [“abs”, “angle”, “real”, “imag”]
DATABASE SCHEMA - REJECTED
• Issues:
– What we’re measuring still isn’t top-level
– The interesting value (absolute value power) is buried in tags.
– Allows for inadvertent power summation
• Eg; not specify GROUP BY * would sum abs, angle, real and imaginary
measurements.

44
• Measurement: type [current, voltage, efield, hfield, power, frequency, thd]
• Fields: component [“abs”, “angle”, “real”, “imag”]
• Tags:
– circuit [string]
DATABASE SCHEMA - CURRENT
What does this do:
SELECT “abs” FROM “db.rp.phasors.current” WHERE “center_frequency”=“60” GROUP BY *
• Integration with dLAMP data structure
– Channel name has to store additional information (introducing 2 reserved characters):
• . (dot) Used to separate major identifiers
• : (colon) used to separate channel from reference
• <type>.<center_frequency>.<circuit (optional)>.<channel>:<reference (optional)>
• “current.60.main.I_A:V_A”
• “power.60.main.I_A:V_A” (this is the previous multiplied by |V_A|)
• “frequency.60.V_A” [see notes]
• “thd.60.I_A” [see notes]

Development and Applications of Distributed IoT Sensors for Intermittent Connectivity Environments | Kevin Claytor | US Army Laboratories

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Development and Applications of Distributed IoT Sensors for Intermittent Connectivity Environments | Kevin Claytor | US Army Laboratories

Similar to Development and Applications of Distributed IoT Sensors for Intermittent Connectivity Environments | Kevin Claytor | US Army Laboratories (20)

More from InfluxData

More from InfluxData (20)

Recently uploaded

Recently uploaded (20)

Development and Applications of Distributed IoT Sensors for Intermittent Connectivity Environments | Kevin Claytor | US Army Laboratories

Editor's Notes