Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Creating a BitTorrent Client 
with Scala and Akka 
Part I 
Dominik Gruber, @the_dom 
Scala Vienna User Group – Nov. 26, 20...
Agenda 
• Akka: Brief Introduction 
• BitTorrent: Basic Procedure 
• Bencoding 
• Tracker 
• Peer Wire Protocol 
• Softwar...
• Actor Model for the JVM 
• Scala & Java API 
• Part of the Typesafe Platform 
• http://akka.io 
Dominik scala-torrent Gr...
• Simple Concurrency & Distribution 
Asynchronous and Distributed by design. High-level abstractions like Actors, 
Futures...
Dominik scala-torrent Gruber • @the_dom
BitTorrent 
• Most popular protocol for peer-to-peer file sharing 
• Designed in 2001 
• Est. over 250 million users/month...
http://en.wikipedia.org/wiki/BitTorrent#mediaviewer/File:BitTorrent_network.svg 
Dominik scala-torrent Gruber • @the_dom
Basic Procedure 
1. Load information from a metainfo (.torrent) file 
2. Request peers from tracker 
3. Connect to peers v...
Metainfo File 
• In Bencode format 
• Includes URL of the tracker, list of files / pieces,… 
• Info hash has to be calcula...
Bencoding 
• Integers 
i<integer encoded in base ten ASCII>e 
e.g. i42e 
• Strings 
<string length>:<string data> 
e.g. 4:...
Bencoding 
• Lists 
l<bencoded values>e 
e.g. l5:scala7:torrente 
• Dictionaries 
d<bencoded string><bencoded element>e 
e...
Example: Ubuntu 14.04.1 
d8:announce39:http://torrent.ubuntu.com:6969/ 
announce13:announce-listll39:http:// 
torrent.ubun...
Parsing Bencode with Scala 
• Scala Parser Combinators 
• Since Scala 2.11 a separate library 
• Any structured format can...
Parsing Bencode with Scala 
import scala.util.parsing.combinator._ 
object BencodeParser extends RegexParsers { 
def integ...
Parsing Bencode with Scala 
object BencodeParser extends RegexParsers { 
// (…) 
def list: Parser[List[Any]] = 
"l" ~> rep...
Parsing Bencode with Scala 
object BencodeParser extends RegexParsers { 
// (…) 
def string: Parser[String] = new Parser[S...
Parsing Bencode with Scala 
object BencodeParser extends RegexParsers { 
// (…) 
def bencodeElem = 
string | integer | lis...
Tracker 
• HTTP service which holds information about a torrent 
• Communication via GET-Request if a transfer is 
started...
Tracker: Example Request 
http://torrent.ubuntu.com:6969/announce 
?event=started 
&info_hash=-%06l%94H%0A%DC%F5%2B%FD 
%1...
Tracker: Example Response 
d8:completei748e10:incompletei8e8:inter 
vali1800e5:peers300:(…)e 
Dominik scala-torrent Gruber...
Peer Wire Protocol 
• Specifies communication with peers 
• TCP 
• Basic format of every messages besides handshake: 
<len...
PWP: Handshake 
• <pstrlen><pstr><reserved><info_hash><peer_id> 
• pstr = “BitTorrent protocol” 
• reserved: Eight bytes, ...
PWP: Bitfield 
• Sent immediately after handshake (optional) 
• <len=0001+X><id=5><bitfield> 
• X: length of bitfield 
• B...
PWP: Request 
• <len=0013><id=6><index><begin><length> 
• Request a block of a piece 
• index: Piece number 
• begin / len...
PWP: Piece 
• <len=0009+X><id=7><index><begin><block> 
• X: Length of block 
• If a piece has been fully received, it is a...
PWP: Other Messages 
• keep-alive 
• choke / unchoke 
• interested / uninterested 
Dominik scala-torrent Gruber • @the_dom
scala-torrent 
• Akka/the actor model is a good fit for this project 
• The components can be clearly separated 
• Supervi...
Actors 
Coordinator 
Peer 
ConnePcetioenr 
ConnePcetieorn 
Connection 
Connection 
Handler 
Torrent 
Tracker 
Torrent Trac...
Connection Handler 
object ConnectionHandler { 
case class CreatePeerConnection(peer: PeerInformation) 
case class PeerCon...
Connection Handler 
class ConnectionHandler(endpoint: InetSocketAddress, internalPeerId: String) 
extends Actor { 
// (…) ...
Peer Connection 
class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: String, 
coordinator: ActorRef) ex...
Peer Connection 
class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: 
String, coordinator: ActorRef) ex...
Current Status 
• Bencode parsing (and encoding) 
• Communication with tracker 
• Modelling of the PWP messages 
• Handsha...
Q & A 
Dominik scala-torrent Gruber • @the_dom
Source 
• https://github.com/TheDom/scala-torrent 
• http://www.bittorrent.org/beps/bep_0003.html 
• http://jonas.nitro.dk...
Upcoming SlideShare
Loading in …5
×

2014-11-26 | Creating a BitTorrent Client with Scala and Akka, Part 1 (Vienna Scala User Group)

1,499 views

Published on

This presentation includes an overview of the BitTorrent protocol and shows my current approach and progress towards implementing a client with Scala and Akka.

Published in: Technology
  • I like this idea and happy I stumbled upon ur work Dom.. Right at the time I am looking for a pet project in scala and akka. Thank you so much for the idea and the github fork. Will get back.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

2014-11-26 | Creating a BitTorrent Client with Scala and Akka, Part 1 (Vienna Scala User Group)

  1. 1. Creating a BitTorrent Client with Scala and Akka Part I Dominik Gruber, @the_dom Scala Vienna User Group – Nov. 26, 2014
  2. 2. Agenda • Akka: Brief Introduction • BitTorrent: Basic Procedure • Bencoding • Tracker • Peer Wire Protocol • Software Architecture Dominik scala-torrent Gruber • @the_dom
  3. 3. • Actor Model for the JVM • Scala & Java API • Part of the Typesafe Platform • http://akka.io Dominik scala-torrent Gruber • @the_dom
  4. 4. • Simple Concurrency & Distribution Asynchronous and Distributed by design. High-level abstractions like Actors, Futures and STM. • Elastic & Decentralized Adaptive load balancing, routing, partitioning and configuration-driven remoting. • Resilient by Design Write systems that self-heal. Remote and/or local supervisor hierarchies. • High Performance 50 million msg/sec on a single machine. Small memory footprint; ~2.5 million actors per GB of heap. Dominik scala-torrent Gruber • @the_dom
  5. 5. Dominik scala-torrent Gruber • @the_dom
  6. 6. BitTorrent • Most popular protocol for peer-to-peer file sharing • Designed in 2001 • Est. over 250 million users/month • Currently responsible for 3.35% of all worldwide bandwidth Dominik scala-torrent Gruber • @the_dom
  7. 7. http://en.wikipedia.org/wiki/BitTorrent#mediaviewer/File:BitTorrent_network.svg Dominik scala-torrent Gruber • @the_dom
  8. 8. Basic Procedure 1. Load information from a metainfo (.torrent) file 2. Request peers from tracker 3. Connect to peers via handshake 4. Exchange data via predefined messages (Peer Wire Protocol) Dominik scala-torrent Gruber • @the_dom
  9. 9. Metainfo File • In Bencode format • Includes URL of the tracker, list of files / pieces,… • Info hash has to be calculated from the file to identify the torrent for communication with tracker and peers Dominik scala-torrent Gruber • @the_dom
  10. 10. Bencoding • Integers i<integer encoded in base ten ASCII>e e.g. i42e • Strings <string length>:<string data> e.g. 4:spam; 12:scala vienna Dominik scala-torrent Gruber • @the_dom
  11. 11. Bencoding • Lists l<bencoded values>e e.g. l5:scala7:torrente • Dictionaries d<bencoded string><bencoded element>e e.g. d5:scalal4:akka4:playe4:java15:enterprise helle Dominik scala-torrent Gruber • @the_dom
  12. 12. Example: Ubuntu 14.04.1 d8:announce39:http://torrent.ubuntu.com:6969/ announce13:announce-listll39:http:// torrent.ubuntu.com:6969/announceel44:http:// ipv6.torrent.ubuntu.com:6969/ announceee7:comment29:Ubuntu CD releases.ubuntu.com13:creation datei1406245742e4:infod6:lengthi599785472e4:n ame31:ubuntu-14.04.1-server-amd64.iso12:piece lengthi524288e6:pieces22880:(…) Dominik scala-torrent Gruber • @the_dom
  13. 13. Parsing Bencode with Scala • Scala Parser Combinators • Since Scala 2.11 a separate library • Any structured format can be defined and parsed through its custom DSL Dominik scala-torrent Gruber • @the_dom
  14. 14. Parsing Bencode with Scala import scala.util.parsing.combinator._ object BencodeParser extends RegexParsers { def integer: Parser[Int] = "i" ~> """(0|-?[1-9]d*)""".r <~ "e" ^^ (_.toInt) // (…) } Dominik scala-torrent Gruber • @the_dom
  15. 15. Parsing Bencode with Scala object BencodeParser extends RegexParsers { // (…) def list: Parser[List[Any]] = "l" ~> rep1(bencodeElem) <~ "e" def dictionary: Parser[Map[String,Any]] = "d" ~> rep1(string ~ bencodeElem) <~ "e" ^^ (_.map(x => (x._1, x._2)).toMap) // (…) } Dominik scala-torrent Gruber • @the_dom
  16. 16. Parsing Bencode with Scala object BencodeParser extends RegexParsers { // (…) def string: Parser[String] = new Parser[String] { def apply(in: Input) = { val source = in.source val offset = in.offset val start = handleWhiteSpace(source, offset) """(d+):([sS]+)""".r findPrefixMatchOf source.subSequence(start, source.length) match { case Some(matched) => val length = matched.group(1).toInt if (length <= matched.group(2).length) Success( matched.group(2).substring(0, length), in.drop(start + length.toString.length + 1 + length - offset) ) else Failure("Provided length is longer than the remaining input", in.drop(start - offset)) case None => Failure("Input is not a string", in.drop(start - offset)) } } } // (…) } Dominik scala-torrent Gruber • @the_dom
  17. 17. Parsing Bencode with Scala object BencodeParser extends RegexParsers { // (…) def bencodeElem = string | integer | list | dictionary def apply(input: String) = parseAll(bencodeElem, input) } Dominik scala-torrent Gruber • @the_dom
  18. 18. Tracker • HTTP service which holds information about a torrent • Communication via GET-Request if a transfer is started, stopped, or completed • Responds with a list of peers as a bencoded dictionary • Tracker is a single point of failure -> DHT extension Dominik scala-torrent Gruber • @the_dom
  19. 19. Tracker: Example Request http://torrent.ubuntu.com:6969/announce ?event=started &info_hash=-%06l%94H%0A%DC%F5%2B%FD %11%85%A7%5E%B4%DD%C1wvs &peer_id=-SC0001-546306326124 &port=6881 &numwant=50 &downloaded=0 &left=599785472 &uploaded=0 &compact=1 Dominik scala-torrent Gruber • @the_dom
  20. 20. Tracker: Example Response d8:completei748e10:incompletei8e8:inter vali1800e5:peers300:(…)e Dominik scala-torrent Gruber • @the_dom
  21. 21. Peer Wire Protocol • Specifies communication with peers • TCP • Basic format of every messages besides handshake: <length prefix><message ID><payload> Dominik scala-torrent Gruber • @the_dom
  22. 22. PWP: Handshake • <pstrlen><pstr><reserved><info_hash><peer_id> • pstr = “BitTorrent protocol” • reserved: Eight bytes, used to indicate supported extensions Dominik scala-torrent Gruber • @the_dom
  23. 23. PWP: Bitfield • Sent immediately after handshake (optional) • <len=0001+X><id=5><bitfield> • X: length of bitfield • Bitfield represents the pieces that have successfully been downloaded Dominik scala-torrent Gruber • @the_dom
  24. 24. PWP: Request • <len=0013><id=6><index><begin><length> • Request a block of a piece • index: Piece number • begin / length: Specifies block with the piece • Requests can be canceled via the CANCEL message Dominik scala-torrent Gruber • @the_dom
  25. 25. PWP: Piece • <len=0009+X><id=7><index><begin><block> • X: Length of block • If a piece has been fully received, it is acknowledged via the HAVE message Dominik scala-torrent Gruber • @the_dom
  26. 26. PWP: Other Messages • keep-alive • choke / unchoke • interested / uninterested Dominik scala-torrent Gruber • @the_dom
  27. 27. scala-torrent • Akka/the actor model is a good fit for this project • The components can be clearly separated • Supervision is needed • TCP communication via Akka I/O • HTTP communication via spray Dominik scala-torrent Gruber • @the_dom
  28. 28. Actors Coordinator Peer ConnePcetioenr ConnePcetieorn Connection Connection Handler Torrent Tracker Torrent Tracker Incoming Connections Dominik scala-torrent Gruber • @the_dom
  29. 29. Connection Handler object ConnectionHandler { case class CreatePeerConnection(peer: PeerInformation) case class PeerConnectionCreated(connection: ActorRef, peer: PeerInformation) } class ConnectionHandler(endpoint: InetSocketAddress, internalPeerId: String) extends Actor { import Tcp._ import context.system import ConnectionHandler._ // Torrent coordinator actor val coordinator = context.parent // Start listening to incoming connections IO(Tcp) ! Tcp.Bind(self, endpoint) // (…) } Dominik scala-torrent Gruber • @the_dom
  30. 30. Connection Handler class ConnectionHandler(endpoint: InetSocketAddress, internalPeerId: String) extends Actor { // (…) override def receive = { case CommandFailed(_: Bind) => // TODO: Handle failure case c @ Connected(remoteAddress, _) => val handler = createPeerConnectionActor(remoteAddress) sender ! Register(handler) case CreatePeerConnection(peer) => val peerConnection = createPeerConnectionActor(peer.inetSocketAddress) sender ! PeerConnectionCreated(peerConnection, peer) } private def createPeerConnectionActor(remoteAddress: InetSocketAddress) = context.actorOf(Props(classOf[PeerConnection], remoteAddress, internalPeerId, coordinator), "peer-connection-" + remoteAddress.toString.replace("/", "")) } Dominik scala-torrent Gruber • @the_dom
  31. 31. Peer Connection class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: String, coordinator: ActorRef) extends Actor { // (…) override def receive: Receive = initialStage def initialStage: Receive = { case AttachToTorrent(t, m) => torrent = Some(t) metainfo = Some(m) case SendHandshake if torrent.isDefined => if (connection.isDefined) sendHandshake() else { IO(Tcp) ! Connect(remoteAddress) context become awaitConnectionForHandshake } case Received(data) => handleHandshakeIn(data) case PeerClosed => handlePeerClosed() } // (…) } Dominik scala-torrent Gruber • @the_dom
  32. 32. Peer Connection class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: String, coordinator: ActorRef) extends Actor { // (…) def handleHandshakeIn(data: ByteString) = { Handshake.unmarshal(data.toVector) match { case Some(handshake: Handshake) => connection = Some(context.sender()) if (torrent.isDefined) context become connected else coordinator ! IncomingPeerConnection(self, handshake) case None => // TODO: Handle failure } } def sendHandshake() = { val handshake = Handshake(metainfo.get.info.infoHash, internalPeerId) connection.get ! Write(ByteString(handshake.marshal.toArray)) } } Dominik scala-torrent Gruber • @the_dom
  33. 33. Current Status • Bencode parsing (and encoding) • Communication with tracker • Modelling of the PWP messages • Handshake with clients • TODO: File exchange (= the core) Dominik scala-torrent Gruber • @the_dom
  34. 34. Q & A Dominik scala-torrent Gruber • @the_dom
  35. 35. Source • https://github.com/TheDom/scala-torrent • http://www.bittorrent.org/beps/bep_0003.html • http://jonas.nitro.dk/bittorrent/bittorrent-rfc.html • https://wiki.theory.org/BitTorrentSpecification Dominik scala-torrent Gruber • @the_dom

×