Tech Talk: Using R to Understand the Internet of Things (IoT) Carl Howe, Director of Education, RStudio Today’s home Internet of Things devices—solar panels, smart power meters, security cameras, and weather stations— are a maze of custom hardware, real-time operating systems, and proprietary smartphone apps. Worse, the costs for commercial monitoring services and devices needed to understand their operation can add up to big monthly bills. Fortunately, most of these devices can be easily understood using simple R programs and a commodity software defined radio (SDR) dongle running on a cheap Raspberry Pi. In this Tech Talk, Carl will demonstrate a simple home monitoring solution built using those tools and illustrate how R can be used for these real-time applications. The talk will conclude by describing the R packages and programming techniques needed to interact with real-time devices and how these differ from traditional R programs.

1. Carl Howe
Director of Education, RStudio
carl@rstudio.com
http://carlhowe.com
@cdhowe

2. !2
RStudio's Mission:
Equip everyone, regardless of means,
to participate in a global economy that
rewards data literacy

3. !3
RStudio Education's mission:
Train the next million R users
#NextMillionRUsers
rstd.dio/learning-r-survey

4. Using R to Understand
the Internet ofThings
Carl Howe
Director of Education
carl@rstudio.com
@cdhowe

5. 18 YEARS AGO IN 2001.....
• Google was 5 years old
• Dial-up still ruled the Internet
• E-commerce companies had just crashed and burned (e.g., pets.com)
• Facebook and Twitter hadn't been founded yet
• Nokia dominated mobile phones
• The mobile internet didn't exist

8. AGENDA
• Which Internet things are we talking about?
• Ingredients
• Software
• Tricky bits

9. Internet of Things Definition:
Semi-programmable devices
that try to talk on the Internet

11. !11

12. 2013 GRAND 5-YEAR GREEN PLAN
1.Install solar panels
2.Buy electric cars
3.Install solar batteries
4.Live more greenly

13. !13

14. DC-to-AC
Solar
Inverters
Solar
Circuilt
Breaker
#1
Solar
Power
Meter

15. Solar
Circuilt
Breaker
#2

16. Solar
Circuilt
Breaker
#3
Two-
Way
Smart
Meter

17. SOLAR POWER SYSTEM (SIMPLIFIED)
Solar Panels
Direct
Current
(DC)
Alternating
Current
(AC)
Inverters Solar Power Meter Utility Power Meter

18. TWO FUNDAMENTAL QUESTIONS
• How much power do we generate?
• How much power do we use?

19. SOLAR POWER SYSTEM (SIMPLIFIED)
Solar Panels
Direct
Current
(DC)
Alternating
Current
(AC)
Inverters Solar Power Meter Utility Power Meter
Solar
Monitoring System
Internet

20. Monitoring
Unit

23. Internet of Things Rule #1
Most IoT devices either use or respond to TCP/IP Port 80
In other words, they speak Web

24. THIS IS WHAT YOU SEE ON PORT 80!

25. WAT?
WATT???

26. Internet of Things Rule #2
If something speaks Web,
we can almost always use R and Linux to work with it

27. Our platform: Raspberry Pi B+

29. RASPBERRY PI B+ DETAILS
System on a chip: Broadcom BCM2835
CPU: ARM Cortex-A53, 700 MHz
RAM: 512MB RAM
Networking: 10/100 Ethernet, no wireless
Storage: microSD
GPIO: 40-pin header, populated
Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface
(CSI), Display Serial Interface (DSI)
Costs about $25 plus power supply and case (total about $45)

30. RASPBERRY PI SOFTWARE
Raspbian OS, which is a derivative of Debian Linux
Comes with Python pre-installed
Does NOT come with R pre-installed
WAT????

32. USEFUL TOOLS WE'LL WANT
• curl (fetch web pages)
• R
• RCurl (fetch web pages)
• XML (parse web pages)
• rvest (harvest web pages in a dplyr kind of way)
• Apache Web server

33. GOTCHAS
• A Raspberry Pi is not an Intel machine
• You may have to compile most or all of your libraries
• Some libraries and functions may not be available at all.

34. INSTALL TOOLS AND LIBRARIES YOU NEED
• apt-get is a Linux tool to install software from Internet repositories
• We'll use it to fetch tools and libraries we'll need on Raspbian
sudo apt-get -y build-dep libcurl4-gnutls-dev
sudo apt-get -y install libcurl4-gnutls-dev
sudo apt-get install libcurl4-openssl-dev libssl-dev
sudo apt-get install libxml2-dev
sudo apt-get install emacs apache2

35. NOW INSTALL THE LATEST VERSION OF R
$ time sudo apt-get install -y gfortran libreadline6-dev libx11-dev libxt-
dev
libpng-dev libjpeg-dev libcairo2-dev xvfb
libbz2-dev libzstd-dev liblzma-dev
libcurl4-openssl-dev
texinfo texlive texlive-fonts-extra
screen wget
$ cd /usr/local/src
$ sudo wget https://cran.rstudio.com/src/base/R-3/R-3.5.2.tar.gz

36. NOW INSTALL THE LATEST VERSION OF R
$ sudo su -
# cd /usr/local/src
# tar zxvf R-3.5.2.tar.gz
# cd R-3.5.2
/* Modify line 27483 in the ./configure file by changing :: to ;; */
/* This was a distribution but in 3.5.1 -- not sure if it's there in 3.5.2 */
/* But it can't hurt to check
# time ./configure
# time make
# time make install
# which R

37. AND INSTALL ALL THE R PACKAGES
# install.packages("tidyverse") fails!!! You'll have to install dplyr,
ggplot2, etc. separately
# Expect it to take an hour or two to download and install everything you
need.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> install.packages(c("dplyr", "ggplot2", "httr",
"reprex", "rvest", "RCurl", "XML2"), depends = TRUE)

38. So Where Were We?

39. ARCHITECTURE: VERSION 1
/var/www directory
Data Plotter
SolarToday.png
Apache Web
Server
Index.html2019-01-21.csv
Inverter Data
Collector

40. Read inverter monitor
Try to parse HTML
Find the data hidden in Javascript

41. The plotter is just straight ggplot2

42. OUTPUT: VERSION 1

43. HOWEVER.....
• Every few days one of the R programs just stops with funky errors
WAT????

44. Internet of Things Rule #3
Just because your IoT device speaks Web
doesn't mean it's smart or error-free

45. GOTCHA #1: DEVICES DO THEIR OWN THING
• Solar inverters turn off when the sun isn't shining
• When they are in the process of turning on or off, they don't always return results
• Both the solar monitoring and plotting program needed tweaks to deal with very
unexpected results
• Error returns
• Badly formatted HTML

46. IOT ECONOMICS
Your device was programmed by the lowest bidder
using the cheapest components
Do not expect that it has been rigorously vetted or
even rigorously tested
Treat data skeptically before trusting it

47. Inverter 1 Inverter 2
2019-01-21 10:37:52 8 195
2019-01-21 10:39:49 9 198
2019-01-21 10:41:48 4105 203
2019-01-21 10:43:48 4106 214
2019-01-21 10:45:49 10 205
2019-01-21 10:47:48 10 219
2019-01-21 10:49:48 4104 204 WAT????

48. INVERTER 1 OCCASIONALLY FLIPS THE 4096 BIT
Inverter 1 Inverter 2 Total output
2019-01-21 10:37:52 8 195 203
2019-01-21 10:39:49 9 198 207
2019-01-21 10:41:48 4105 203 219
2019-01-21 10:43:48 4106 214 213
2019-01-21 10:45:49 10 205 213
2019-01-21 10:47:48 10 219 229
2019-01-21 10:49:48 4104 204 213
After
cleaning

49. Internet of Things Rule #4
Your IoT device's environment matters

50. WAT????

52. FORTUNATELY, I HAVE A WEATHER STATION
But even a weather
station doesn't tell you
if the sun is out

53. ARCHITECTURE: VERSION 2
Data Plotter
SolarToday.png
Apache Web
Server
2019-01-21.csv
Inverter Data
Collector
SkyImage.png
Camera
Grabber
Weather
Station Monitor
WeatherToday.png Index.html
weather.gov weatherunderground.com

54. THIS IS THE VERSION RUNNING TODAY
Raspberry Pi B+
Weather Station
Ethernet Gateway

55. Click for larger image
Click for minute-by-minute .csv data
Click Here

57. BUT THE R PROGRAMS STILL CRASH
EVERY NOW AND AGAIN
WAT WAT WAT??????

58. SLEEPING LOOPS AREN'T THE BEST TOOLS....
• Difficult to have them fire at the right times (lubridate math isn't trivial)
• Difficult to sync them to wall clock time
• Code in the loops may leak memory over time

59. Internet of Things Rule #5
Run IoT things on a clock

60. LINUX HAS A SCHEDULER CALLED CRON
• Can run any program you like (including R code) based on wall clock time
• Very flexible (although the way you program it is a bit obscure)
• Allows you to quick and easy one-shot programs instead of long-running server code

61. CRONTAB EXAMPLE ON THE WEATHER PI
10 0 * * * $HOME/Solar-monitor-pi-sleepfix.R
*/5 * * * * $HOME/Plot-today-crontab.R
Run this every day at 10 minutes, 0 hours (i.e., 10 min past midnight)

62. CRONTAB EXAMPLE ON THE WEATHER PI
10 0 * * * $HOME/Solar-monitor-pi-sleepfix.R
*/5 * * * * $HOME/Plot-today-crontab.R
Run Plot-today-crontab.R every 5 minutes
• https://crontab-generator.org will generate these expressions for you
• You add them to your own cron table using "crontab -e" for editing

63. ARCHITECTURE: VERSION 3
Data Plotter
SolarToday.png
Apache Web
Server
2019-01-21.csv
Inverter Data
Collector
SkyImage.png
Camera
Grabber
Weather
Station Monitor
WeatherToday.png Index.html
weatherunderground.com
weather.gov
Cron

64. TWO FUNDAMENTAL QUESTIONS
• We now know how much power we generate
• How much power do we use?

65. HOME POWER EQUATION
Power Used Power Generated Power Bought From Utility
– =

66. HOME POWER EQUATION
Power Used Power Generated Power Bought From Utility
– =

67. ANNOYANCES
• I only get how much power I've used from the utility once a month, long after I've
used it
• Energy storage (batteries) will make this much more complicated on an hour to hour
basis

68. HOME POWER EQUATION
Power Used Power Generated Power Bought From Utility
– =
±

69. Internet of Things Rule #6
Most smart things are going wireless

70. REMEMBER THIS DEVICE?
• The power company doesn't want
to pay someone to physically read
my power sold and bought
• Instead, they simply broadcast the
information wirelessly on 900 MHz
(the Industrial and Scientific
Radio band, AKA the garbage
band) Two-
Way
Smart
Meter

71. Internet of Things Rule #7
Wireless things are pretty easy to spy on

72. ENTER THE SOFTWARE DEFINED RADIO (SDR)
• Listens to pretty much anything
between 100 KHz and 2 GHz
• USB interface
• Programmable to decode most any
radio signal
• Cheap to buy ($20-$30)

73. RASPBERRY PI V3 DETAILS FOR POWER MONITORING
System on a chip: Broadcom BCM2837
CPU: Quad-core ARM Cortex-A53, 1.2GHz
GPU: Broadcom VideoCore IV
RAM: 1GB LPDDR2 (900 MHz)
Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless built in
Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy
Storage: microSD
GPIO: 40-pin header, populated
Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display
Serial Interface (DSI)
Costs about $35 plus case and power supply (total about $55)

74. SOLAR POWER MONITOR VERSION 3
Data Plotter
SolarToday.png
Apache Web
Server
2019-01-21.csv
Inverter Data
Collector
SkyImage.png
Camera
Grabber
Weather
Station Monitor
WeatherToday.png Index.html
weatherunderground.com
weather.gov
Cron

75. ARCHITECTURE: METER READER
Data Plotter
SolarToday.png
Apache Web
Server
2019-01-21.csv
Inverter Data
Collector
Cron
power.txt
Power
Meter Reader
rtlamr
Index.html

76. MORE SOFTWARE TO LOAD
• Libraries for reading the SDR
• sudo apt-get install rtl-sdr gr-osmosdr
• Libraries for running the Go language
• sudo apt-get install golang
• Software for reading smart meters
• git clone https://github.com/bemasher/rtlamr

77. !77

78. RAW POWER METER READINGS
{Time:2019-01-22T14:02:35.477 SCM:{ID:25228408 Type: 8 Tamper:{Phy:00 Enc:01} Consumption: 57131 CRC:0x519A}}
{Time:2019-01-22T14:02:35.645 SCM:{ID:25228409 Type: 8 Tamper:{Phy:00 Enc:00} Consumption: 35975 CRC:0x6070}}
{Time:2019-01-22T14:13:47.105 SCM:{ID:25228408 Type: 8 Tamper:{Phy:00 Enc:01} Consumption: 57131 CRC:0x519A}}
{Time:2019-01-22T14:13:47.174 SCM:{ID:25228409 Type: 8 Tamper:{Phy:00 Enc:00} Consumption: 35976 CRC:0x6642}}
{Time:2019-01-22T14:22:03.393 SCM:{ID:25228408 Type: 8 Tamper:{Phy:00 Enc:01} Consumption: 57131 CRC:0x519A}}
{Time:2019-01-22T14:22:03.509 SCM:{ID:25228409 Type: 8 Tamper:{Phy:00 Enc:00} Consumption: 35976 CRC:0x6642}}
{Time:2019-01-22T14:32:15.257 SCM:{ID:25228408 Type: 8 Tamper:{Phy:00 Enc:01} Consumption: 57131 CRC:0x519A}}
{Time:2019-01-22T14:32:15.375 SCM:{ID:25228409 Type: 8 Tamper:{Phy:00 Enc:00} Consumption: 35977 CRC:0x84A0}}
{Time:2019-01-22T14:42:26.594 SCM:{ID:25228408 Type: 8 Tamper:{Phy:00 Enc:01} Consumption: 57131 CRC:0x519A}}
{Time:2019-01-22T14:42:26.711 SCM:{ID:25228409 Type: 8 Tamper:{Phy:00 Enc:00} Consumption: 35978 CRC:0xCCE5}}
{Time:2019-01-22T14:52:32.936 SCM:{ID:25228408 Type: 8 Tamper:{Phy:00 Enc:01} Consumption: 57131 CRC:0x519A}}
{Time:2019-01-22T14:52:33.054 SCM:{ID:25228409 Type: 8 Tamper:{Phy:00 Enc:00} Consumption: 35978 CRC:0xCCE5}}
Meter IDs kWh bought
kWh sold

79. COMBINED POWER DATA
Power Sales
(1 kWh each)

80. CONSOLIDATED HOUSEHOLD USE (WITH BUGS)
Bug in windowing
calculation

81. ACTUAL SCREEN DISPLAYED

82. CONCLUSION: WE'VE ANSWERED OUR QUESTIONS
• How much power do we generate?
• How much power do we use?

83. INTERNET OF THINGS RULES
1.Most IoT devices either use or respond to TCP/IP Port 80
2.If something speaks Web, we can almost always use R and Linux to work with it
3.Just because your IoT device speaks Web doesn't mean it's smart or error-free
4.Your IoT device's environment matters
5.Run IoT things on a clock
6.Most smart things are going wireless
7.Wireless things are pretty easy to spy on

84. Carl Howe
Director of Education, RStudio
carl@rstudio.com
http://carlhowe.com
@cdhowe