Building systems with Kamaelia

Slide 1: Kamaelia: SnapTogether Software Michael Sparks Senior Research Engineer BBC Research & Innovation Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 2: BBC R&I ... PAL, Teletext, NICAM, DVB, DAB, Freeview, TV Anytime Majority of BBC Open Source Projects inc Dirac, Kamaelia Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 3: Key Research goal? To make concurrency easy to work with Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 4: How??? Single Reader/Single Writer Communicating Sequential Things Original context: scalable servers Component Composition Written in Python Ideas are language agnostic Proof of concept in C++ Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 5: How??? No, I don't believe in “one language to rule them all” *cough*Erlang*cough* Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 6: (demo) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 7: ie this: Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 8: What I'm aiming to get through P2P Radio What? System Walkthrough Q&A Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 9: What? Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 10: What if... ? Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 11: What has it been used for? Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 12: Who's used it? Radio & Music Interactive Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 13: Who's used it? Matt Biddulph Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 14: Who's used it? Matt Biddulph Open Days “this is what I need” Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 15: Who's used it? Matt Biddulph Open Days “this is what I need” Record All Radio PVR Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 16: Who's used it? Matt Biddulph Open Days “this is what I need” Record All Radio PVR Podcasts Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 17: Who's used it? Matt Biddulph Open Days “this is what I need” Record All Radio PVR Podcasts Proof of concept Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 18: Who's used it? Matt Biddulph Open Days “this is what I need” Record All Radio PVR Podcasts Proof of concept Prototyping for experimentation for new services Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 19: Who's used it? Prototyping for experimentation for new services ... without any involvement from R&D Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 20: Who's used it? Prototyping for experimentation for new services ... without any involvement from R&D due to being open source Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 21: Who's used it? BBC Macro Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 22: Blog frontend Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/ & Meta Store Programme bbc one transcoder Kamaelia Backend bbc two transcoder DEMUXER bbc three transcoder TUNER bbc four transcoder cbbc transcoder cbeebies transcoder Kamaelia news 24 transcoder parliament transcoder

Slide 23: Who's used it? Rapid Prototyping of Collaborative Community Radio Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 24: Who's used it? DJs in the Channel Controller Community Control Streaming Networ Server k Mixer Playout Web Front End Linked by Music Store videoconf Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 25: Who's used it? DJs in the Channel Controller Community Control Streaming Networ Server k Mixer Playout Web Front End Linked by Music Store videoconf Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 26: Google's Summer of Code Student Work 2006, 2007 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 27: Bit Torrent 2006 Programmes... WEBSITE Ingest BitTorrent Metadata Split Store Torrent Maker Web Client Torrent Peer Swarm Store Torrent Peer Access Programmes... Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 28: Open GL 2006 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 29: 2006 Trusted Communications eg secure phone Decrypt ie trusted by the user Audio Decode Encrypt Decrypt BUFFER Encrypt Decrypt AudioPlayer Encrypt Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 30: 2007 Filelike Interface > (Much!) simpler usage outside Kamaelia systems IRC/AIM Components > causes creation of components > allows remote control of systems Visual Component Generation > Sub component model > aim:easier creation of components visually (rather than systems) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 31: Other Systems Reframing for mobile, remuxing DVBT interactive aps for multicast streams, Whiteboarding, VoIP testing, etc. Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 32: Walkthrough Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 33: Simple Streamer Pipeline( ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SingleServer( ), ).activate() Pipeline( TCPClient( host = \"127.0.0.1\", port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 34: Simple Streamer Pipeline( ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SingleServer( ), ).activate() Pipeline( TCPClient( host = \"127.0.0.1\", port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 35: Simple Presentation Tool Graphline( NEXT = Button(caption=\"Next\", msg=\"NEXT\", position=(72,8)), PREV = Button(caption=\"Previous\", msg=\"PREV\", position=(8,8)), FIRST = Button(caption=\"First\", msg=\"FIRST\", position=(256,8)), LAST = Button(caption=\"Last\", msg=\"LAST\", position=(320,8)), CHOOSER = Chooser(items = files), IMAGE = Image(size=(800,600), position=(8,48)), linkages = { (\"NEXT\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"PREV\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"FIRST\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"LAST\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"CHOOSER\",\"outbox\") : (\"IMAGE\",\"inbox\"), } ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 36: Simple Presentation Tool Graphline( NEXT = Button(caption=\"Next\", msg=\"NEXT\", position=(72,8)), PREV = Button(caption=\"Previous\", msg=\"PREV\", position=(8,8)), FIRST = Button(caption=\"First\", msg=\"FIRST\", position=(256,8)), LAST = Button(caption=\"Last\", msg=\"LAST\", position=(320,8)), CHOOSER = Chooser(items = files), IMAGE = Image(size=(800,600), position=(8,48)), linkages = { (\"NEXT\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"PREV\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"FIRST\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"LAST\",\"outbox\") : (\"CHOOSER\",\"inbox\"), (\"CHOOSER\",\"outbox\") : (\"IMAGE\",\"inbox\"), } ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 37: Diving Inside? from Axon.ThreadedComponent import \\ threadedcomponent class ConsoleReader(threadedcomponent): def main(self): eol = \"\\n\" while 1: # this blocks so we use a thread line = raw_input(\">>> \") line = line + eol self.send(line, \"outbox\") Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 38: Diving Inside? from Axon.Component import component class ConsoleEchoer(component): def main(self): while 1: while self.dataReady(\"inbox\"): data = self.recv(\"inbox\") _sys.stdout.write(str(data)) _sys.stdout.flush() yield 1 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 39: Used thus... Pipeline( ConsoleReader(), ConsoleEchoer(), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 40: Better Streamer def UserProtocol(): return ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SimpleServer( protocol=UserProtocol, port=1601 ).activate() Pipeline( TCPClient( host = \"127.0.0.1\", port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 41: Better Streamer def UserProtocol(): return ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SimpleServer( protocol=UserProtocol, port=1601 ).activate() Pipeline( TCPClient( host = \"127.0.0.1\", port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 42: Minimal Radio P2P Core Backplane(“RADIO”).activate() Pipeline( TCPClient( host = \"192.168.1.1\", port = 1601 ), PublishTo( “RADIO” ), ).activate() Pipeline( SubscribeTo(“RADIO”), UnixProcess(“mplayer - “), ).activate() def UserProtocol(): return SubscribeTo( “RADIO” ), SimpleServer( protocol=UserProtocol, port=1601 ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 43: Minimal Radio P2P Core Backplane(“RADIO”).activate() Pipeline( TCPClient( host = \"192.168.1.1\", port = 1601 ), PublishTo( “RADIO” ), ).activate() Pipeline( SubscribeTo(“RADIO”), UnixProcess(“mplayer - “), ).activate() def UserProtocol(): return SubscribeTo( “RADIO” ), SimpleServer( protocol=UserProtocol, port=1601 ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 44: Swarming P2P idea Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 45: Swarming P2P setup class SimpleSwarm(Axon.Component.component): clients = [] rr = 0 maxclients = 4 def __init__(self, port): super(SimpleSwarm, self).__init__() self.__class__.port = port Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 46: Swarming P2P server def main(self): while 1: while self.dataReady(\"inbox\"): data = self.recv(\"inbox\") req = data.split(\" \") if req[0] == \"REQCONNECT\": if len(self.clients) < self.maxclients: reqfrom = req[1] self.clients.append(reqfrom) self.send( \"CONNECT \" + str(self.port) , \"outbox\" ) else: reqconn = self.clients[self.__class__.rr] self.__class__.rr = (self.__class__.rr+1) % self.maxclients self.send( \"REQCONNECT \" + reqconn ) if not self.anyReady(): self.pause() yield 1 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 47: Swarming P2P client @staticmethod def clientRequest(rootip, rootport, myip, myport): servip, servport = rootip, rootport port = 0 while port == 0: l = LikeFile(TCPClient(servip, servport)) l.activate() l.send(\"REQCONNECT %s:%d\" % (myip, myport)) resp_raw = l.recv() resp = resp_raw.split(\" \") if resp[0] == \"CONNECT\": port = int(resp[1]) ip = servip elif resp[0] == \"REQCONNECT\": servip, servport = resp[1].split(\":\") servport = int(servport) return servip, port Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 48: P2P Radio – Client (was) Pipeline( TCPClient( host = \"192.168.1.1\", port = 1601 ), PublishTo( “RADIO” ), ).activate() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 49: Swarming Radio – Client Graphline(CONFIGURE= ConnectToSwarm(rootip, rootcontrolport, myip, mycontrolport), CLIENT = Carousel(mkTCPClient), PUBLISHTO = PublishTo(\"RADIO\"), linkages = { (\"CONFIGURE\",\"outbox\"):(\"CLIENT\", \"next\"), (\"CLIENT\", \"outbox\") : (\"PUBLISHTO\", \"inbox\"), } ).activate() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 50: Swarming Radio – Helpers class ConnectToSwarm(Axon.Component.component): def __init__(self, rootip, rootport, myip, myport): super(ConnectToSwarm, self).__init__() self.rootip = rootip self.rootport = rootport self.myip = myip self.myport = myport def main(self): servip, servport = SimpleSwarm.clientRequest( \\ self.rootip, self.rootport, self.myip, self.myport) yield 1 self.send( ( servip, servport ), \"outbox\") self.pause() yield 1 def mkTCPClient(args): return TCPClient(*args) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 51: Swarming Radio – Source freq = 682.166670 # Sutton Coldfields feparams = { \"inversion\" : dvb3.frontend.INVERSION_AUTO, \"constellation\" : dvb3.frontend.QAM_16, \"coderate_HP\" : dvb3.frontend.FEC_3_4, \"coderate_LP\" : dvb3.frontend.FEC_3_4, } Pipeline( DVB_Multiplex(freq, [6210], feparams), # RADIO ONE PublishTo(\"RADIO\"), ).activate() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 52: Kamaelia is already a highly capable platform that has delivered some interesting tools Currently version 0.5.0 Version reflects where we want to be Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 53: However it also shows that you can make concurrency easier to work with, even in a normal language Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 54: You can build your own version if you follow our “mini axon” tutorial using your language of choice Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 55: Suggestions: Take it, use it Build cool stuff with it Break it, Fix it Steal the ideas Clone in your own language Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slide 56: Thanks for listening :) http://kamaelia.sf.net/ michael.sparks@bbc.co.uk Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Slideshare.net (beta)