Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/22dytOc. Adam Wick takes a deep dive into a unikernel implementation of the Tor anonymity system. He discusses what aspects of Tor make it an attractive target for a unikernel, and what aspects of unikernels are particularly interesting when considering Tor. Filmed at qconsf.com. Adam Wick leads the systems software group at Galois, Inc., an R&D company.

1. Adam Wick @ QCon San Francisco, 2015
Tor in Haskell
and Other Unikernel Tricks

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/tor-haskell

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. This combination is not crazy.
(In fact, it makes a lot of sense.)

5. © 2015 Galois, Inc.3 © 2015 Galois, Inc.3
Adam Wick @ QCon San Francisco, 2015
Built the Haskell Lightweight Virtual Machine (HaLVM)
Works for
(a unikernel)
Adam Wick

6. © 2015 Galois, Inc.4 © 2015 Galois, Inc.4
Use Case #3: Nimble

7. © 2015 Galois, Inc.5 © 2015 Galois, Inc.5
Use Case #3: Nimble

8. © 2015 Galois, Inc.6 © 2015 Galois, Inc.6
Use Case #3: Nimble + Massive

9. © 2015 Galois, Inc.7 © 2015 Galois, Inc.7
Why are Unikernels and Tor such a good idea?
Unikernels are fundamentally designed to be
Lightweight
Fast
Secure
More nodes for the same price, or
the same number of nodes for
less.
Fewer schedulers means lower
latency.
A much smaller OS stack means less
code to get wrong.

10. So We Did This
(And now I’m going to tell you about it.)

11. But First: Tor

12. © 2015 Galois, Inc.10 © 2015 Galois, Inc.10
: Anonymity Through Root Vegetables
Tor is an anonymous communication layer built on TLS that
prevents people from learning who you are interacting with over
the Internet.*
* Terms and conditions may apply.

13. © 2015 Galois, Inc.11 © 2015 Galois, Inc.11
Guarantees*:
1. For any single relay node, it is impossible to tell where you are in the chain.
2. The exit node cannot tell how many hops were before it.
3. The entrance node cannot tell how many hops are in front of it.
* Terms and conditions may apply.

14. © 2015 Galois, Inc.12 © 2015 Galois, Inc.12
Terms and Conditions May Apply
The security of Tor is dependent on:
1. The correctness of the protocol design.
2. The correctness of the protocol implementation.
3. Ensuring that there are enough nodes that no single entity
owns a sufficient percentage of nodes to have a top level view.

15. And Second: Haskell

16. © 2015 Galois, Inc.14 © 2015 Galois, Inc.14
It is also known for being popular with people that know more
category theory than is good for them, which has given it a certain
… reputation.
Haskell is a functional programming language:
fact :: Integer -> Integer map :: (a -> b) -> [a] -> [b]
fact 0 = 1 map _ [] = []
fact x = x * fact (x – 1) map f (x:r) = (f x) : map f r
It is mostly known for being lazy, pure, and strongly typed.
Very HelpfulHandyMeh

17. © 2015 Galois, Inc.15 © 2015 Galois, Inc.15
do -- "To authenticate the initiator, the responder MUST check the
-- following:
-- * The CERTS cell contains exactly one CerType 3 'AUTH'
-- certificate.
let authCert = exactlyOneAuth certs Nothing
authCert' = signedObject (getSigned authCert)
-- * The CERTS cell contains exactly one CerType 2 'ID'
-- certificate
let iidCert = exactlyOneId certs Nothing
iidCert' = signedObject (getSigned iidCert)
-- * Both certificates have validAfter and validUntil dates
-- that are not expired.
when (certExpired authCert' now) $ fail "Auth certificate expired."
when (certExpired iidCert' now) $ fail "Id certificate expired."
-- * The certified key in the AUTH certificate is a 1024-bit RSA
-- key.
unless (is1024BitRSAKey authCert) $
fail "Auth certificate key is the wrong size."
-- * The certified key in the ID certificate is a 1024-bit RSA
-- key.
unless (is1024BitRSAKey iidCert) $
fail "Identity certificate key is the wrong size."
-- * The auth certificate is correctly signed with the key in the
-- ID certificate.
unless (authCert `isSignedBy` iidCert') $
fail "Auth certificate not signed by identity cert."

18. © 2015 Galois, Inc.16 © 2015 Galois, Inc.16
I’m Not Here To Sell You Haskell, But:
Type Safety + Purity + Safe Data Structures + GC
=
Fewer bugs per LOC
=
Less security advisories per LOC

19. Unikernels

20. © 2015 Galois, Inc.18 © 2015 Galois, Inc.18
Unikernels are specialised, single address space machine images
constructed using library operating systems.
- Wikipedia
or
Unikernels : Virtual Machines :: Exokernels : Physical Machines
or
Unikernels are single-process programs compiled to run directly on
(usually virtual) hardware, rather than within a full-featured OS.

21. © 2015 Galois, Inc.19 © 2015 Galois, Inc.19

22. © 2015 Galois, Inc.20 © 2015 Galois, Inc.20
Lower operating costs
Faster response to events
Smaller attack surface

23. © 2015 Galois, Inc.21 © 2015 Galois, Inc.21
Containers and Unikernels
Unikernels: The next step in separation:
1. We started with single-task code.
2. Next, we separated some of our tasks into threads.
3. Next, we separated some of our threads into processes.
4. Next, we separated some of our processes into containers.
5. Now, we separated our critical containers into unikernels.
Each level provides additional separation properties, at the cost of
increased complexity in creating and reasoning about new code.

24. So: Tor + Haskell + Unikernels?

25. © 2015 Galois, Inc.23 © 2015 Galois, Inc.23
Terms and Conditions May Apply
The security of Tor is dependent on:
1. The correctness of the protocol design.
2. The correctness of the protocol implementation.
3. Ensuring that there are enough nodes that no single entity
owns a sufficient percentage of nodes to have a top level view.
Can’t really do much for this
one. Except that more
implementations means more
eyes on the specs.
Haskell limits memory issues,
type safety helps correctness.
Unikernels limit attack surface.
Unikernels limit resource
usage, which allows a lot more
relay nodes for the same cost.

26. Cool! How Did That Work?

27. © 2015 Galois, Inc.25 © 2015 Galois, Inc.25
So You Want To Build A Unikernel
There are five steps to building a Unikernel:
1. Don’t.
2. Test & Measure.
3. Do.
4. Test (Part II)
5. Deploy.

28. © 2015 Galois, Inc.26 © 2015 Galois, Inc.26
Step #1: Don’t Build a Unikernel
Building a unikernel adds a number of complications to the
development process (which is already complicated enough).
So start by building your application as you normally would, in a
language that supports unikernels. (In this case, Haskell.)
Build using your normal tools, libraries, and techniques, but:
1. Try to avoid local storage.
2. Try to minimize the number of libraries you pull in.
3. Stay away from libraries that link against C.
With Tor, I began by implementing the core
parsing and protocol code, sufficient to
anonymously look up a hostname.
To avoid a linking problem, I also ended up
writing my own zlib decompression library.
More on that later.

29. © 2015 Galois, Inc.27 © 2015 Galois, Inc.27
Step #2: Measure & Test
What is your application’s peak memory use?
Exactly how much space do you need for configuration files, etc.?
Does this space need to be writeable?
Then TEST, TEST, TEST.
Your goal is that in later steps, any bugs you find have to do with
the unikernel translation, not your application.
In measuring my work with Tor, I discovered
that the zlib library I’d written earlier was very
… silly ... with regard to memory use.
I wish I’d found that here.

30. © 2015 Galois, Inc.28 © 2015 Galois, Inc.28
Step #3: Do it.
Once it works well, and you know about its resource usage, now
it’s time to start converting it to a unikernel.
1. Some of your libraries won’t build, and you will either need
to fix or replace these. Predicting which ones is possible, but
requires some experience and internal knowledge of the
library.
2. Disks are expensive and slow; consider using ramdisks. For
many of our uses of unikernels, and Tor can be one of them, then
the only need for a disk is to pass configuration information: small
bits of read-only data. Ramdisks are the way to go.
3. You will need to rewrite your start-up code. You will now need
to explicitly instantiate all your devices, for example.
4. Your Edit-Compile-Test loop just got more painful. Sorry.
Getting things working on the HaLVM required:
• Updates to the TLS library I was using.
• Updates to the x.509 library I was using.
And it benefited from a bunch of previous work
I’d done for other unikernels.

31. © 2015 Galois, Inc.29 © 2015 Galois, Inc.29
Step #4: Test (Part II)
Hopefully, any bugs you hit at this point are bugs in the adaptation
to the unikernel, not in your code.
But here are some more questions to verify:
1. Does your system come up cleanly, every time, even with
flakey device timings?
2. Are you flying safely under your memory bounds? (If so,
can you lower them safely?)
3. Can you run your test suite as a unikernel? (If so, do it.)
4. Optional: Does your new unikernel survive migration?
This is still in progress.
(This is always in progress.)

32. © 2015 Galois, Inc.30 © 2015 Galois, Inc.30
Step #5: Deploy
The unikernel community is working to make deployment to all the
standard clouds quicker and easier, but there are documents
online that you can use.
There are tools for distributing to EC2, but
they’re incomplete.
Different unikernels are in different places with
regard to EC2 and other clouds.

33. Let’s See Some Code

34. © 2015 Galois, Inc.32 © 2015 Galois, Inc.32
Let’s Run Some Tor
main :: IO ()
main = runDefaultMain $ flags ->
do (MkNS ns, logger) <- initializeSystem flags
let options = defaultTorOptions{ … }
tor <- startTor ns options
addrs <- torResolveName tor "www.whatismypublicip.com"
case addrs of
[] ->
putStrLn ("Could not resolve www.whatismypublicip.com!")
((addr, _ttl) : _) ->
do sock <- torConnect tor addr 80
putStrLn ("Connected to " ++ show addr)
torWrite sock (buildGet "/")
putStrLn ("Wrote GET request.")
resp <- readLoop sock
putStrLn ("Response: " ++ show resp)
torClose sock ReasonDone
Parse command line arguments,
turn them into Tor configuration
options, and start up the network.
name, based on the options
provided earlier.
Create an anonymous
connection and read the
response.
Start the various daemon threads
required to run a Tor entrance
node.
Anonymously look up a domain

35. © 2015 Galois, Inc.33 © 2015 Galois, Inc.33
Starting Means Booting
initializeSystem :: [Flag] ->
IO (SomeNetworkStack, String -> IO ())
initializeSystem _ =
do con <- initXenConsole
xs <- initXenStore
ns <- newNetworkStack
macstr <- findNIC xs
nic <- openNIC xs macstr
let mac = read macstr
addDevice ns mac (xenSend nic) (xenReceiveLoop nic)
deviceUp ns mac
ipaddr <- dhcpDiscover ns mac
return (MkNS (hansNetworkStack ns),
makeLogger ( x -> writeConsole con (x ++ "n")))
Most unikernels will get your
memory and such set up, but
devices are all up to you.
Network setup is all your
responsibility.

36. © 2015 Galois, Inc.34 © 2015 Galois, Inc.34
Abstracting Over Your Network Stack
-- |The type of a Tor-compatible network stack. The first type variable is the
-- type of a listener socket, the second the type of a standard connection
-- socket.
data TorNetworkStack lsock sock = TorNetworkStack {
connect :: String -> Word16 -> IO (Maybe sock)
-- |Lookup the given hostname and return any IP6 (Left) or IP4 (Right)
-- addresses associated with it.
, getAddress :: String -> IO [TorAddress]
, listen :: Word16 -> IO lsock
, accept :: lsock -> IO (sock, TorAddress)
, recv :: sock -> Int -> IO S.ByteString
, write :: sock -> L.ByteString -> IO ()
, flush :: sock -> IO ()
, close :: sock -> IO ()
, lclose :: lsock -> IO ()
}

37. © 2015 Galois, Inc.35 © 2015 Galois, Inc.35
QuickCheck!
tapHandshakeCheck :: Word32 -> RouterTAP -> TorRNG -> Bool
tapHandshakeCheck circId (RouterTAP myRouter priv) g0 =
let (g1, (privX, cbody)) = startTAPHandshake myRouter g0
(g2, (dcell, fenc, benc)) =
advanceTAPHandshake priv circId cbody g1
Created circIdD dbody = dcell
in case completeTAPHandshake privX dbody of
Left err ->
False
Right (fenc', benc') ->
(circId == circIdD) && (fenc == fenc') &&
(benc == benc')

38. © 2015 Galois, Inc.36 © 2015 Galois, Inc.36
QuickCheck!
TorCell Serialization:
TorAddress round-trips: [OK, passed 100 tests]
TorAddress makes sensible ByteStrings: [OK, passed 100 tests]
ExtendSpec serializes: [OK, passed 100 tests]
DestroyReason serializes (check #1): [OK, passed 100 tests]
DestroyReason serializes (check #2): [OK, passed 100 tests]
HandshakeType serializes (check #1): [OK, passed 100 tests]
HandshakeType serializes (check #2): [OK, passed 100 tests]
RelayEndReason serializes: [OK, passed 100 tests]
RelayCell serializes: [OK, passed 100 tests]
RelayCell serializes w/ digest: [OK, passed 100 tests]
RelayCell serializes w/ digest: [OK, passed 100 tests]
Tor certificates serialize: [OK, passed 100 tests]
Hybrid encryption tests:
Hybrid encryption works when forced: [OK, passed 100 tests]
Hybrid encryption works in general: [OK, passed 100 tests]
Handshakes:
TAP Handshake: [OK, passed 100 tests]
NTor Handshake: [OK, passed 100 tests]
Make sure we get data
formats like. Next
step: fuzzing to make
sure we’re sufficiently
defensive.
Test custom crypto.
Test handshakes.

39. What is it like developing unikernels?

40. © 2015 Galois, Inc.38 © 2015 Galois, Inc.38
Boring
• It’s just like normal development when you start.
• Then you have to be a little more rigorous about testing and
evaluation.
• Then you run into a wall with library support.
• Except it’s not nearly as much of a problem as it used to be.
• With Tor, this meant writing one additional library from scratch
(zlib), and then sending patches to a few other authors.
• Debugging unikernels is also now more palatable:
• printf(), gdbsx, profiling
• The big remaining gap: the big web frameworks don’t work.
• They require all the deep device and OS integration that
causes problems with unikernels.
(Mostly)

41. © 2015 Galois, Inc.39 © 2015 Galois, Inc.39
But What You Get Is
More nodes,
For less money,
With a better security posture.
Which is great for , and perhaps your next
project.

42. © 2015 Galois, Inc.40 © 2015 Galois, Inc.40
All trademarks, service marks, trade names, trade dress, product names and
logos appearing in these slides are the property of their respective owners,
including in some instances Galois, Inc.
All rights are reserved.
http://github.com/GaloisInc/haskell-tor
Adam Wick
awick@galois.com
Twitter: @acwpdx
Any questions?

43. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/tor-
haskell