Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Creating a BitTorrent Client 
with Scala and Akka 
Part I 
Dominik Gruber, @the_dom 
Scala Vienna User Group – Nov. 26, 20...
Agenda 
• Akka: Brief Introduction 
• BitTorrent: Basic Procedure 
• Bencoding 
• Tracker 
• Peer Wire Protocol 
• Softwar...
• Actor Model for the JVM 
• Scala & Java API 
• Part of the Typesafe Platform 
• http://akka.io 
Dominik scala-torrent Gr...
• Simple Concurrency & Distribution 
Asynchronous and Distributed by design. High-level abstractions like Actors, 
Futures...
Dominik scala-torrent Gruber • @the_dom
BitTorrent 
• Most popular protocol for peer-to-peer file sharing 
• Designed in 2001 
• Est. over 250 million users/month...
http://en.wikipedia.org/wiki/BitTorrent#mediaviewer/File:BitTorrent_network.svg 
Dominik scala-torrent Gruber • @the_dom
Basic Procedure 
1. Load information from a metainfo (.torrent) file 
2. Request peers from tracker 
3. Connect to peers v...
Metainfo File 
• In Bencode format 
• Includes URL of the tracker, list of files / pieces,… 
• Info hash has to be calcula...
Bencoding 
• Integers 
i<integer encoded in base ten ASCII>e 
e.g. i42e 
• Strings 
<string length>:<string data> 
e.g. 4:...
Bencoding 
• Lists 
l<bencoded values>e 
e.g. l5:scala7:torrente 
• Dictionaries 
d<bencoded string><bencoded element>e 
e...
Example: Ubuntu 14.04.1 
d8:announce39:http://torrent.ubuntu.com:6969/ 
announce13:announce-listll39:http:// 
torrent.ubun...
Parsing Bencode with Scala 
• Scala Parser Combinators 
• Since Scala 2.11 a separate library 
• Any structured format can...
Parsing Bencode with Scala 
import scala.util.parsing.combinator._ 
object BencodeParser extends RegexParsers { 
def integ...
Parsing Bencode with Scala 
object BencodeParser extends RegexParsers { 
// (…) 
def list: Parser[List[Any]] = 
"l" ~> rep...
Parsing Bencode with Scala 
object BencodeParser extends RegexParsers { 
// (…) 
def string: Parser[String] = new Parser[S...
Parsing Bencode with Scala 
object BencodeParser extends RegexParsers { 
// (…) 
def bencodeElem = 
string | integer | lis...
Tracker 
• HTTP service which holds information about a torrent 
• Communication via GET-Request if a transfer is 
started...
Tracker: Example Request 
http://torrent.ubuntu.com:6969/announce 
?event=started 
&info_hash=-%06l%94H%0A%DC%F5%2B%FD 
%1...
Tracker: Example Response 
d8:completei748e10:incompletei8e8:inter 
vali1800e5:peers300:(…)e 
Dominik scala-torrent Gruber...
Peer Wire Protocol 
• Specifies communication with peers 
• TCP 
• Basic format of every messages besides handshake: 
<len...
PWP: Handshake 
• <pstrlen><pstr><reserved><info_hash><peer_id> 
• pstr = “BitTorrent protocol” 
• reserved: Eight bytes, ...
PWP: Bitfield 
• Sent immediately after handshake (optional) 
• <len=0001+X><id=5><bitfield> 
• X: length of bitfield 
• B...
PWP: Request 
• <len=0013><id=6><index><begin><length> 
• Request a block of a piece 
• index: Piece number 
• begin / len...
PWP: Piece 
• <len=0009+X><id=7><index><begin><block> 
• X: Length of block 
• If a piece has been fully received, it is a...
PWP: Other Messages 
• keep-alive 
• choke / unchoke 
• interested / uninterested 
Dominik scala-torrent Gruber • @the_dom
scala-torrent 
• Akka/the actor model is a good fit for this project 
• The components can be clearly separated 
• Supervi...
Actors 
Coordinator 
Peer 
ConnePcetioenr 
ConnePcetieorn 
Connection 
Connection 
Handler 
Torrent 
Tracker 
Torrent Trac...
Connection Handler 
object ConnectionHandler { 
case class CreatePeerConnection(peer: PeerInformation) 
case class PeerCon...
Connection Handler 
class ConnectionHandler(endpoint: InetSocketAddress, internalPeerId: String) 
extends Actor { 
// (…) ...
Peer Connection 
class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: String, 
coordinator: ActorRef) ex...
Peer Connection 
class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: 
String, coordinator: ActorRef) ex...
Current Status 
• Bencode parsing (and encoding) 
• Communication with tracker 
• Modelling of the PWP messages 
• Handsha...
Q & A 
Dominik scala-torrent Gruber • @the_dom
Source 
• https://github.com/TheDom/scala-torrent 
• http://www.bittorrent.org/beps/bep_0003.html 
• http://jonas.nitro.dk...
Upcoming SlideShare
Loading in …5
×

2014-11-26 | Creating a BitTorrent Client with Scala and Akka, Part 1 (Vienna Scala User Group)

1,791 views

Published on

This presentation includes an overview of the BitTorrent protocol and shows my current approach and progress towards implementing a client with Scala and Akka.

Published in: Technology

2014-11-26 | Creating a BitTorrent Client with Scala and Akka, Part 1 (Vienna Scala User Group)

  1. 1. Creating a BitTorrent Client with Scala and Akka Part I Dominik Gruber, @the_dom Scala Vienna User Group – Nov. 26, 2014
  2. 2. Agenda • Akka: Brief Introduction • BitTorrent: Basic Procedure • Bencoding • Tracker • Peer Wire Protocol • Software Architecture Dominik scala-torrent Gruber • @the_dom
  3. 3. • Actor Model for the JVM • Scala & Java API • Part of the Typesafe Platform • http://akka.io Dominik scala-torrent Gruber • @the_dom
  4. 4. • Simple Concurrency & Distribution Asynchronous and Distributed by design. High-level abstractions like Actors, Futures and STM. • Elastic & Decentralized Adaptive load balancing, routing, partitioning and configuration-driven remoting. • Resilient by Design Write systems that self-heal. Remote and/or local supervisor hierarchies. • High Performance 50 million msg/sec on a single machine. Small memory footprint; ~2.5 million actors per GB of heap. Dominik scala-torrent Gruber • @the_dom
  5. 5. Dominik scala-torrent Gruber • @the_dom
  6. 6. BitTorrent • Most popular protocol for peer-to-peer file sharing • Designed in 2001 • Est. over 250 million users/month • Currently responsible for 3.35% of all worldwide bandwidth Dominik scala-torrent Gruber • @the_dom
  7. 7. http://en.wikipedia.org/wiki/BitTorrent#mediaviewer/File:BitTorrent_network.svg Dominik scala-torrent Gruber • @the_dom
  8. 8. Basic Procedure 1. Load information from a metainfo (.torrent) file 2. Request peers from tracker 3. Connect to peers via handshake 4. Exchange data via predefined messages (Peer Wire Protocol) Dominik scala-torrent Gruber • @the_dom
  9. 9. Metainfo File • In Bencode format • Includes URL of the tracker, list of files / pieces,… • Info hash has to be calculated from the file to identify the torrent for communication with tracker and peers Dominik scala-torrent Gruber • @the_dom
  10. 10. Bencoding • Integers i<integer encoded in base ten ASCII>e e.g. i42e • Strings <string length>:<string data> e.g. 4:spam; 12:scala vienna Dominik scala-torrent Gruber • @the_dom
  11. 11. Bencoding • Lists l<bencoded values>e e.g. l5:scala7:torrente • Dictionaries d<bencoded string><bencoded element>e e.g. d5:scalal4:akka4:playe4:java15:enterprise helle Dominik scala-torrent Gruber • @the_dom
  12. 12. Example: Ubuntu 14.04.1 d8:announce39:http://torrent.ubuntu.com:6969/ announce13:announce-listll39:http:// torrent.ubuntu.com:6969/announceel44:http:// ipv6.torrent.ubuntu.com:6969/ announceee7:comment29:Ubuntu CD releases.ubuntu.com13:creation datei1406245742e4:infod6:lengthi599785472e4:n ame31:ubuntu-14.04.1-server-amd64.iso12:piece lengthi524288e6:pieces22880:(…) Dominik scala-torrent Gruber • @the_dom
  13. 13. Parsing Bencode with Scala • Scala Parser Combinators • Since Scala 2.11 a separate library • Any structured format can be defined and parsed through its custom DSL Dominik scala-torrent Gruber • @the_dom
  14. 14. Parsing Bencode with Scala import scala.util.parsing.combinator._ object BencodeParser extends RegexParsers { def integer: Parser[Int] = "i" ~> """(0|-?[1-9]d*)""".r <~ "e" ^^ (_.toInt) // (…) } Dominik scala-torrent Gruber • @the_dom
  15. 15. Parsing Bencode with Scala object BencodeParser extends RegexParsers { // (…) def list: Parser[List[Any]] = "l" ~> rep1(bencodeElem) <~ "e" def dictionary: Parser[Map[String,Any]] = "d" ~> rep1(string ~ bencodeElem) <~ "e" ^^ (_.map(x => (x._1, x._2)).toMap) // (…) } Dominik scala-torrent Gruber • @the_dom
  16. 16. Parsing Bencode with Scala object BencodeParser extends RegexParsers { // (…) def string: Parser[String] = new Parser[String] { def apply(in: Input) = { val source = in.source val offset = in.offset val start = handleWhiteSpace(source, offset) """(d+):([sS]+)""".r findPrefixMatchOf source.subSequence(start, source.length) match { case Some(matched) => val length = matched.group(1).toInt if (length <= matched.group(2).length) Success( matched.group(2).substring(0, length), in.drop(start + length.toString.length + 1 + length - offset) ) else Failure("Provided length is longer than the remaining input", in.drop(start - offset)) case None => Failure("Input is not a string", in.drop(start - offset)) } } } // (…) } Dominik scala-torrent Gruber • @the_dom
  17. 17. Parsing Bencode with Scala object BencodeParser extends RegexParsers { // (…) def bencodeElem = string | integer | list | dictionary def apply(input: String) = parseAll(bencodeElem, input) } Dominik scala-torrent Gruber • @the_dom
  18. 18. Tracker • HTTP service which holds information about a torrent • Communication via GET-Request if a transfer is started, stopped, or completed • Responds with a list of peers as a bencoded dictionary • Tracker is a single point of failure -> DHT extension Dominik scala-torrent Gruber • @the_dom
  19. 19. Tracker: Example Request http://torrent.ubuntu.com:6969/announce ?event=started &info_hash=-%06l%94H%0A%DC%F5%2B%FD %11%85%A7%5E%B4%DD%C1wvs &peer_id=-SC0001-546306326124 &port=6881 &numwant=50 &downloaded=0 &left=599785472 &uploaded=0 &compact=1 Dominik scala-torrent Gruber • @the_dom
  20. 20. Tracker: Example Response d8:completei748e10:incompletei8e8:inter vali1800e5:peers300:(…)e Dominik scala-torrent Gruber • @the_dom
  21. 21. Peer Wire Protocol • Specifies communication with peers • TCP • Basic format of every messages besides handshake: <length prefix><message ID><payload> Dominik scala-torrent Gruber • @the_dom
  22. 22. PWP: Handshake • <pstrlen><pstr><reserved><info_hash><peer_id> • pstr = “BitTorrent protocol” • reserved: Eight bytes, used to indicate supported extensions Dominik scala-torrent Gruber • @the_dom
  23. 23. PWP: Bitfield • Sent immediately after handshake (optional) • <len=0001+X><id=5><bitfield> • X: length of bitfield • Bitfield represents the pieces that have successfully been downloaded Dominik scala-torrent Gruber • @the_dom
  24. 24. PWP: Request • <len=0013><id=6><index><begin><length> • Request a block of a piece • index: Piece number • begin / length: Specifies block with the piece • Requests can be canceled via the CANCEL message Dominik scala-torrent Gruber • @the_dom
  25. 25. PWP: Piece • <len=0009+X><id=7><index><begin><block> • X: Length of block • If a piece has been fully received, it is acknowledged via the HAVE message Dominik scala-torrent Gruber • @the_dom
  26. 26. PWP: Other Messages • keep-alive • choke / unchoke • interested / uninterested Dominik scala-torrent Gruber • @the_dom
  27. 27. scala-torrent • Akka/the actor model is a good fit for this project • The components can be clearly separated • Supervision is needed • TCP communication via Akka I/O • HTTP communication via spray Dominik scala-torrent Gruber • @the_dom
  28. 28. Actors Coordinator Peer ConnePcetioenr ConnePcetieorn Connection Connection Handler Torrent Tracker Torrent Tracker Incoming Connections Dominik scala-torrent Gruber • @the_dom
  29. 29. Connection Handler object ConnectionHandler { case class CreatePeerConnection(peer: PeerInformation) case class PeerConnectionCreated(connection: ActorRef, peer: PeerInformation) } class ConnectionHandler(endpoint: InetSocketAddress, internalPeerId: String) extends Actor { import Tcp._ import context.system import ConnectionHandler._ // Torrent coordinator actor val coordinator = context.parent // Start listening to incoming connections IO(Tcp) ! Tcp.Bind(self, endpoint) // (…) } Dominik scala-torrent Gruber • @the_dom
  30. 30. Connection Handler class ConnectionHandler(endpoint: InetSocketAddress, internalPeerId: String) extends Actor { // (…) override def receive = { case CommandFailed(_: Bind) => // TODO: Handle failure case c @ Connected(remoteAddress, _) => val handler = createPeerConnectionActor(remoteAddress) sender ! Register(handler) case CreatePeerConnection(peer) => val peerConnection = createPeerConnectionActor(peer.inetSocketAddress) sender ! PeerConnectionCreated(peerConnection, peer) } private def createPeerConnectionActor(remoteAddress: InetSocketAddress) = context.actorOf(Props(classOf[PeerConnection], remoteAddress, internalPeerId, coordinator), "peer-connection-" + remoteAddress.toString.replace("/", "")) } Dominik scala-torrent Gruber • @the_dom
  31. 31. Peer Connection class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: String, coordinator: ActorRef) extends Actor { // (…) override def receive: Receive = initialStage def initialStage: Receive = { case AttachToTorrent(t, m) => torrent = Some(t) metainfo = Some(m) case SendHandshake if torrent.isDefined => if (connection.isDefined) sendHandshake() else { IO(Tcp) ! Connect(remoteAddress) context become awaitConnectionForHandshake } case Received(data) => handleHandshakeIn(data) case PeerClosed => handlePeerClosed() } // (…) } Dominik scala-torrent Gruber • @the_dom
  32. 32. Peer Connection class PeerConnection(remoteAddress: InetSocketAddress, internalPeerId: String, coordinator: ActorRef) extends Actor { // (…) def handleHandshakeIn(data: ByteString) = { Handshake.unmarshal(data.toVector) match { case Some(handshake: Handshake) => connection = Some(context.sender()) if (torrent.isDefined) context become connected else coordinator ! IncomingPeerConnection(self, handshake) case None => // TODO: Handle failure } } def sendHandshake() = { val handshake = Handshake(metainfo.get.info.infoHash, internalPeerId) connection.get ! Write(ByteString(handshake.marshal.toArray)) } } Dominik scala-torrent Gruber • @the_dom
  33. 33. Current Status • Bencode parsing (and encoding) • Communication with tracker • Modelling of the PWP messages • Handshake with clients • TODO: File exchange (= the core) Dominik scala-torrent Gruber • @the_dom
  34. 34. Q & A Dominik scala-torrent Gruber • @the_dom
  35. 35. Source • https://github.com/TheDom/scala-torrent • http://www.bittorrent.org/beps/bep_0003.html • http://jonas.nitro.dk/bittorrent/bittorrent-rfc.html • https://wiki.theory.org/BitTorrentSpecification Dominik scala-torrent Gruber • @the_dom

×