Using Erlang in an Embedded and
Cross-Compiled World
Frank Hunleth
Twitter: @fhunleth
ErlangDC R13B 12/7/13
I'm an embedded C/C++ programmer and Erlang/OTP
has been this nice shiny apple that's been tempting me
for a long time.
Why?
1. Software reliability (supervision and “let it
crash philosophy”)
2. Message passing
3. Simple and robust
interface to C/C++
4. Binary pattern matching
Now, I just need to introduce Erlang into a project...
I wanted to start small...
●

●

●

●

http://www.youtube.com/watch?v=QpP1hfEOO00

But Erlang/OTP didn't work
that way for my embedded
projects
It's large enough and wants
to be in control
Hard to gradually introduce
into a project
Feels like only way to
proceed is to eat the whole
apple...
Agenda
●
●

●
●

●

Baseline embedded systems of interest
Demo of a simple cross-compiled Erlang/OTP
release
Breakdown of parts involved and the tools
Using rebar and relx in the cross-compiled
environment
Items that I need some help with
What Kind of Embedded?
Limited resource/
Specific purpose

General
purpose
Microcontrollers

RTOS

Embedded Linux Android, iOS,
Ubuntu, Windows

32-bit Microprocessors
●

●

●

Focus is on purpose-built devices capable of running
Linux and Erlang
Resource limitations, cost or quantity-made drive
need to create self-contained firmware images
Developing on target is impractical
Demo Demo Demo
The Big Picture

Erlang/OTP
applications

rebar

BEAMs and crosscompiled binaries

Erlang/OTP release
relx

fwtool + scripts

Base Firmware
Image

Firmware image
End Product - The Firmware Image
Master Boot Record
Bootloaders
Root Filesystem A

Linux kernel
Erlang
Libraries
Main application

Root Filesystem B

Ping-pong location
used when upgrading
the firmware

Application Data
Partition

Majority of nonvolatile
memory for use by the
application

*This is just one of the many ways of organizing a firmware image.
Quick View of the Base Root FS
/
├──
├──
├──
├──
├──
├──
├──
│
├──
├──
│
│
│
│
│
│
│
│
│
│
│
│
└──

bin
boot
dev
etc
lib
sbin
srv
└── erlang
tmp
usr
├── bin
├── lib
│
└── erlang
│
├── bin
│
├── erts-5.10.2
│
│
├── bin
│
│
└── lib
│
├── releases
│
│
└── R16B01
│
└── usr
├── sbin
└── share
var

Erlang/OTP release
will go here

Cross-compiled ERTS
install
Building the Base Root FS
●

●

●

●

●

●

Use Buildroot
Cross-compiler built
for free
Huge selection of recipes
for cross-compiling
C/C++ libraries
Cross-compiles Erlang
“Installs” everything to a staging directory and creates an EXT4 file
system from it
Only minor customization needed for Erlang use
–

https://github.com/nerves-project/nerves-sdk
RootFS (16.8 MB)

erlangdc_demo release

Non-Erlang libraries

ext4 overhead

/usr/lib/erlang

Linux kernel

Non-Erlang binaries

Other files

/etc

0

1

2

3
MB

4

5

6
Installed Applications (5.5MB)

stdlib-1.19.2

kernel-2.16.2

erlangdc_demo-1

cowboy-0.9.0

gproc-0.2.17-1-g45a17ac

jsx-1.4.3

crypto-3.0

cowlib-0.4.0

erlang_ale-0.0.1 (gpio port only)

ranch-0.9.0
0

0.5

1

1.5
MB

2

2.5

3
System Initialization - erlinit
●

Observations
–

–

Erlang/OTP is pretty good at supervision

–
●

Main responsibilities for /sbin/init are initialization and
process supervision
Erlang/OTP releases contain initialization order

erlinit
–

Replacement for /sbin/init that starts an Erlang/OTP release

–

https://github.com/nerves-project/erlinit
Example Release Directory
/srv/erlang
├── lib
│
├── my_app-0.0.1
│
│
├── ebin
│
│
└── priv
│
├── kernel-2.16.2
│
│
└── ebin
│
└── stdlib-1.19.2
│
└── ebin
└── releases
├── 1
│
├── my_app.boot
│
├── my_app.rel
│
├── my_app.script
│
├── sys.config
│
└── vm.args
└── RELEASES

Source code, docs, etc.
trimmed

Application initialization
script (binary version)
Release specification
Application configuration
Custom VM command
line arguments
Boot Time (7.4 seconds)

SPL 0.33

u-boot (1s user timeout)

2.83

Uncompress Linux kernel

0.95

Boot kernel

1.52

erlinit

0.06

Load and start ERTS

1.00

Demo running

0.68

0

1

2

3

4
Seconds

5

6

7

8
The Big Picture

Erlang/OTP
applications

rebar

BEAMs and crosscompiled binaries

Erlang/OTP release
relx

fwtool + scripts

Base Firmware
Image

Firmware image
Back to the Demo
erlangdc_demo/
├── config
│
└── vm.args
├── priv
│
├── index.html
│
└── static
│
├── demo.css
│
├── images
│
│
└── ajax-loader.gif
│
├── jquery-1.10.2.min.js
│
├── jquery.mobile-1.3.2.min.css
│
└── jquery.mobile-1.3.2.min.js
├── Makefile
├── rebar.config
├── relhelper.sh
├── relx.config
└── src
├── erlangdc_demo_app.erl
├── erlangdc_demo.app.src
├── erlangdc_demo_sup.erl
├── hwinterface.erl
└── ws_handler.erl

https://github.com/fhunleth/erlangdc_demo

Configure sname and
cookie for remote access
(debug/demo use here)

Build scripts and
configuration

Calls to Erlang ALE
to access GPIOs
Cowboy websockets
handler
Configuration Files
rebar.config
{deps, [
{cowboy, ".*", {git, "https://github.com/extend/cowboy.git", "master"}},
{jsx, ".*", {git, "https://github.com/talentdeficit/jsx.git", "master"}},
{erlang_ale, ".*", {git, "https://github.com/fhunleth/erlang_ale.git", "rebar-bbb"}}
]}.
{post_hooks, [{"linux", compile, "./relhelper.sh"}]}.

Modified version of Erlang ALE
for the BeagleBone (C code in here)

relx.config
{include_erts, false}.
{vm_args, "./config/vm.args"}.
{release, {erlangdc_demo, "1"}, [erlangdc_demo]}.

Just calls relx. Helper
script currently needed to pass path
to cross-compiled system libraries.
Building
fhunleth@halfmarathon:~/nerves/nerves-sdk$ source ./nerves-env.sh
Nerves environment updated for this session
fhunleth@halfmarathon:~/nerves/nerves-sdk$ cd ../erlangdc_demo/
fhunleth@halfmarathon:~/nerves/erlangdc_demo$ make
rebar get-deps compile
==> cowlib (get-deps)
...
==> erlang_ale (compile)
==> erlangdc_demo (compile)
Generating a cross-compiled release
===> Starting relx build process ...
...
===> Resolved erlangdc_demo-1
===> release successfully created!
Updating base firmware image with Erlang release...
Building /home/fhunleth/nerves/erlangdc_demo/_images/bbb.fw...
Building /home/fhunleth/nerves/erlangdc_demo/_images/sdcard.img...
fhunleth@halfmarathon:~/nerves/erlangdc_demo$

Once per session to
update $PATH and other
variables

Unmodified version of
rebar
Unmodified version of
relx

Run script from nerves-sdk
to put relx output into
base Root FS and build
images
Copying to the SDCard
sdcard.img

●

Traditional Unix way
–
–

●

sudo dd if=sdcard.img of=/dev/sdc bs=1M
Bit for bit copy from sdcard.img to SDCard (/dev/sdc)

IMHO... more convenient
–

sudo mmccopy -p sdcard.img

–

Discovers the SDCard, shows progress, auto-unmounts

–

https://github.com/fhunleth/mmccopy
The Big Picture

Erlang/OTP
applications

rebar

BEAMs and crosscompiled binaries

Erlang/OTP release
relx

fwtool + scripts

Base Firmware
Image

Firmware image
Things I Don't Know How to Do Yet
●

Improve the development work flow
–

Need to shorten edit/compile/test time

–

Currently don't take advantage code replacement

–

Lots of potential here to improve embedded development

●

How to estimate and bound Erlang memory usage

●

Handle permissions/security within Erlang
–

Everything runs as root (typical of many embedded devices, but not necessarily
desirable)

–

Run multiple Erlang instances?

Probably a ton more, but I'm looking forward to more
embedded Erlang!
Using Erlang in an Embedded and
Cross-Compiled World
Frank Hunleth
fhunleth@troodon-software.com
Twitter: @fhunleth
ErlangDC R13B 12/7/13

Using Erlang in an Embedded and Cross-Compiled World

  • 1.
    Using Erlang inan Embedded and Cross-Compiled World Frank Hunleth Twitter: @fhunleth ErlangDC R13B 12/7/13
  • 2.
    I'm an embeddedC/C++ programmer and Erlang/OTP has been this nice shiny apple that's been tempting me for a long time.
  • 3.
    Why? 1. Software reliability(supervision and “let it crash philosophy”) 2. Message passing 3. Simple and robust interface to C/C++ 4. Binary pattern matching Now, I just need to introduce Erlang into a project...
  • 4.
    I wanted tostart small... ● ● ● ● http://www.youtube.com/watch?v=QpP1hfEOO00 But Erlang/OTP didn't work that way for my embedded projects It's large enough and wants to be in control Hard to gradually introduce into a project Feels like only way to proceed is to eat the whole apple...
  • 5.
    Agenda ● ● ● ● ● Baseline embedded systemsof interest Demo of a simple cross-compiled Erlang/OTP release Breakdown of parts involved and the tools Using rebar and relx in the cross-compiled environment Items that I need some help with
  • 6.
    What Kind ofEmbedded? Limited resource/ Specific purpose General purpose Microcontrollers RTOS Embedded Linux Android, iOS, Ubuntu, Windows 32-bit Microprocessors ● ● ● Focus is on purpose-built devices capable of running Linux and Erlang Resource limitations, cost or quantity-made drive need to create self-contained firmware images Developing on target is impractical
  • 7.
  • 10.
    The Big Picture Erlang/OTP applications rebar BEAMsand crosscompiled binaries Erlang/OTP release relx fwtool + scripts Base Firmware Image Firmware image
  • 11.
    End Product -The Firmware Image Master Boot Record Bootloaders Root Filesystem A Linux kernel Erlang Libraries Main application Root Filesystem B Ping-pong location used when upgrading the firmware Application Data Partition Majority of nonvolatile memory for use by the application *This is just one of the many ways of organizing a firmware image.
  • 12.
    Quick View ofthe Base Root FS / ├── ├── ├── ├── ├── ├── ├── │ ├── ├── │ │ │ │ │ │ │ │ │ │ │ │ └── bin boot dev etc lib sbin srv └── erlang tmp usr ├── bin ├── lib │ └── erlang │ ├── bin │ ├── erts-5.10.2 │ │ ├── bin │ │ └── lib │ ├── releases │ │ └── R16B01 │ └── usr ├── sbin └── share var Erlang/OTP release will go here Cross-compiled ERTS install
  • 13.
    Building the BaseRoot FS ● ● ● ● ● ● Use Buildroot Cross-compiler built for free Huge selection of recipes for cross-compiling C/C++ libraries Cross-compiles Erlang “Installs” everything to a staging directory and creates an EXT4 file system from it Only minor customization needed for Erlang use – https://github.com/nerves-project/nerves-sdk
  • 14.
    RootFS (16.8 MB) erlangdc_demorelease Non-Erlang libraries ext4 overhead /usr/lib/erlang Linux kernel Non-Erlang binaries Other files /etc 0 1 2 3 MB 4 5 6
  • 15.
  • 16.
    System Initialization -erlinit ● Observations – – Erlang/OTP is pretty good at supervision – ● Main responsibilities for /sbin/init are initialization and process supervision Erlang/OTP releases contain initialization order erlinit – Replacement for /sbin/init that starts an Erlang/OTP release – https://github.com/nerves-project/erlinit
  • 17.
    Example Release Directory /srv/erlang ├──lib │ ├── my_app-0.0.1 │ │ ├── ebin │ │ └── priv │ ├── kernel-2.16.2 │ │ └── ebin │ └── stdlib-1.19.2 │ └── ebin └── releases ├── 1 │ ├── my_app.boot │ ├── my_app.rel │ ├── my_app.script │ ├── sys.config │ └── vm.args └── RELEASES Source code, docs, etc. trimmed Application initialization script (binary version) Release specification Application configuration Custom VM command line arguments
  • 18.
    Boot Time (7.4seconds) SPL 0.33 u-boot (1s user timeout) 2.83 Uncompress Linux kernel 0.95 Boot kernel 1.52 erlinit 0.06 Load and start ERTS 1.00 Demo running 0.68 0 1 2 3 4 Seconds 5 6 7 8
  • 19.
    The Big Picture Erlang/OTP applications rebar BEAMsand crosscompiled binaries Erlang/OTP release relx fwtool + scripts Base Firmware Image Firmware image
  • 20.
    Back to theDemo erlangdc_demo/ ├── config │ └── vm.args ├── priv │ ├── index.html │ └── static │ ├── demo.css │ ├── images │ │ └── ajax-loader.gif │ ├── jquery-1.10.2.min.js │ ├── jquery.mobile-1.3.2.min.css │ └── jquery.mobile-1.3.2.min.js ├── Makefile ├── rebar.config ├── relhelper.sh ├── relx.config └── src ├── erlangdc_demo_app.erl ├── erlangdc_demo.app.src ├── erlangdc_demo_sup.erl ├── hwinterface.erl └── ws_handler.erl https://github.com/fhunleth/erlangdc_demo Configure sname and cookie for remote access (debug/demo use here) Build scripts and configuration Calls to Erlang ALE to access GPIOs Cowboy websockets handler
  • 21.
    Configuration Files rebar.config {deps, [ {cowboy,".*", {git, "https://github.com/extend/cowboy.git", "master"}}, {jsx, ".*", {git, "https://github.com/talentdeficit/jsx.git", "master"}}, {erlang_ale, ".*", {git, "https://github.com/fhunleth/erlang_ale.git", "rebar-bbb"}} ]}. {post_hooks, [{"linux", compile, "./relhelper.sh"}]}. Modified version of Erlang ALE for the BeagleBone (C code in here) relx.config {include_erts, false}. {vm_args, "./config/vm.args"}. {release, {erlangdc_demo, "1"}, [erlangdc_demo]}. Just calls relx. Helper script currently needed to pass path to cross-compiled system libraries.
  • 22.
    Building fhunleth@halfmarathon:~/nerves/nerves-sdk$ source ./nerves-env.sh Nervesenvironment updated for this session fhunleth@halfmarathon:~/nerves/nerves-sdk$ cd ../erlangdc_demo/ fhunleth@halfmarathon:~/nerves/erlangdc_demo$ make rebar get-deps compile ==> cowlib (get-deps) ... ==> erlang_ale (compile) ==> erlangdc_demo (compile) Generating a cross-compiled release ===> Starting relx build process ... ... ===> Resolved erlangdc_demo-1 ===> release successfully created! Updating base firmware image with Erlang release... Building /home/fhunleth/nerves/erlangdc_demo/_images/bbb.fw... Building /home/fhunleth/nerves/erlangdc_demo/_images/sdcard.img... fhunleth@halfmarathon:~/nerves/erlangdc_demo$ Once per session to update $PATH and other variables Unmodified version of rebar Unmodified version of relx Run script from nerves-sdk to put relx output into base Root FS and build images
  • 23.
    Copying to theSDCard sdcard.img ● Traditional Unix way – – ● sudo dd if=sdcard.img of=/dev/sdc bs=1M Bit for bit copy from sdcard.img to SDCard (/dev/sdc) IMHO... more convenient – sudo mmccopy -p sdcard.img – Discovers the SDCard, shows progress, auto-unmounts – https://github.com/fhunleth/mmccopy
  • 24.
    The Big Picture Erlang/OTP applications rebar BEAMsand crosscompiled binaries Erlang/OTP release relx fwtool + scripts Base Firmware Image Firmware image
  • 25.
    Things I Don'tKnow How to Do Yet ● Improve the development work flow – Need to shorten edit/compile/test time – Currently don't take advantage code replacement – Lots of potential here to improve embedded development ● How to estimate and bound Erlang memory usage ● Handle permissions/security within Erlang – Everything runs as root (typical of many embedded devices, but not necessarily desirable) – Run multiple Erlang instances? Probably a ton more, but I'm looking forward to more embedded Erlang!
  • 26.
    Using Erlang inan Embedded and Cross-Compiled World Frank Hunleth fhunleth@troodon-software.com Twitter: @fhunleth ErlangDC R13B 12/7/13

Editor's Notes

  • #2 Embedded systems are everywhere, but the software that runs them is still largely C and C++. Thankfully, the CPUs that run them are getting fast enough and cheap enough that it is possible to use higher level languages to shorten development times. To this end, Erlang and OTP are almost a perfect match for these systems. Developing on device can be like developing on a PC, but this isn't an option for many small to medium sized embedded projects. For these, a cross-compiled environment is needed to be practical. This talk will describe one way of embedding Erlang in a device with a small footprint and using tools like rebar and relx. And while the approach is a traditional embedded one, the use of Erlang opens up a more dynamic development environment and much simpler programming of distributed systems.
  • #7 Why Linux? In addition to general OS infrastructure, Linux provides tons of file system, network and device drivers
  • #8 The demo shows a BeagleBone Black running Erlang that presents a web page to the user. The web page communicates via WebSockets back to the BeagleBone and lets the user turn on and off the lights. It also has a real-time display of the state of the push button. The backend is running Cowboy and uses Erlang ALE to interface to the hardware. Erlang ALE includes cross-compiled C code.
  • #9 Just in case the demo gods strike...
  • #10 Lights on!!!
  • #15 These are measurements taking from the demo application firmware.
  • #16 Source code was removed Only BEAMs, compiled C code, priv directories remain The great part about making Erlang/OTP releases is that they automatically remove the Erlang applications that you don't need, so there's not much bloat.
  • #18 sys.config and vm.args found by erlinit and passed to the Erlang VM
  • #19 Boot time is currently pretty slow for the 1 GHz BeagleBone Black, but the low hanging areas to optimize are not Erlang related. U-boot time is exaggerated due to waiting for a keypress to break into the bootloader.
  • #27 Embedded systems are everywhere, but the software that runs them is still largely C and C++. Thankfully, the CPUs that run them are getting fast enough and cheap enough that it is possible to use higher level languages to shorten development times. To this end, Erlang and OTP are almost a perfect match for these systems. Developing on device can be like developing on a PC, but this isn't an option for many small to medium sized embedded projects. For these, a cross-compiled environment is needed to be practical. This talk will describe one way of embedding Erlang in a device with a small footprint and using tools like rebar and relx. And while the approach is a traditional embedded one, the use of Erlang opens up a more dynamic development environment and much simpler programming of distributed systems.