1. Sociable Software
(or “how you really already know
how to make concurrency for multicore
easy to work with”)
Michael Sparks
ms@cerenity.org
http://kamaelia.sourceforge.net/Developers/ BBC Research & Innovation

2. Note added after barcamp
This presentation aims to show people that they already know
how to deal with concurrency.
It argues that if we have the tools for large scale concurrency
(mashups) and small scale (hardware) that midrange (normal
apps) can be done in a similar way, using existing tools.
This is done by showing useful systems that have been produced
in this manner using existing tools.
ie from existing practice, not theory
http://kamaelia.sourceforge.net/Developers/ ms@cerenity.org

3. Note added after barcamp
During the actual presentation I also talked about Kamaelia
projects created by novice programmers of varying ability which
show high levels of concurrency.
These include: previewing PVR content on mobiles, multicast
island joining, as-live streaming using bit torrent, Open GL based
user interfaces & integration, seaside style webserving, speex
based secure phone, IRC/IM systems, a shakespeare script
player, and games tools.
Other systems created include Atom/RSS routing, memcached
integration, P2P whiteboarding (with audio + mixing), gesture
recognition, presentation tools, a kids development environment,
topology visualisation tools, database modelling etc.
http://kamaelia.sourceforge.net/Developers/ ms@cerenity.org

4. Sociable Software
Core idea : “building systems that
are made of many software
systems that are sociable with
each other”
(yes, this project did start at work, but I am primarily working
on it in my own time at present due to a resource crunch :)

5. Sociable Software
ie it's comprised of systems that
talk to each other.
ie are sociable.

6. Comfortable examples

7. request
Web Browser
Web Server
response

8. More sociable:
request
Web Browser
Web Server
RSS/Atom Feed

9. More sociable:
request
Web Browser
Web Server
RSS/Atom Feed
Kinda like your facebook mini-feed

10. General principle
Web Server
RSS/Atom Feed
Web Browser
Web Server
RSS/Atom Feed
Web Server
RSS/Atom Feed

11. Mash ups
(next logical step)

12. Basic client side mashup
Web Server
RSS Feed
Web Server
Some
client Atom Feed
Web Server
Web API

13. Basic client side mashup
Web Server
RSS Feed
Web Server
Some
client Atom Feed
Web Server
Example: Google Ads
Web API

14. Basic client side mashup
Web Server
RSS Feed
Web Server
Some
client Atom Feed
Web Server
Example: Google Ads
Or any similar sort of system Web API
ie using Javascript for transcludes

15. Mashup as a service
Web Server
Some RSS Feed
client
Web Server
Web Server Atom Feed
Web Server
Web API

16. Mashup as a service
Web Server
Some RSS Feed
client
Web Server
Web Server Atom Feed
Web Server
Example: gmap-pedometer
Web API

17. More generally
Web Server
Some RSS Feed
client
Web Server
Web Server Atom Feed
Service
Web Server
Web API

18. Go Crazy
(next logical step)

19. Web Server
RSS Feed
Web Server
Atom Feed
Web Server
RSS Feed
Web Server
Web API Web API
Web Server
Web Server Atom Feed
Service
Web Server
Web Server RSS Feed
Web API Web API
Web Server
Web API
Web Server
Service
` Web Server
Web API

20. Do you really think of it that way?

22. Got that?
(pretty common stuff yes?)

23. Let's look at something completely
different.

24. Formal verification of
asynchronous hardware
(Rainbow project, Manchester Uni. ~10 years old)

25. Doesn't really look
so very different
does it?

26. Even complex
examples look
similar.

27. Why?

28. They're comprised of systems that
talk to each other.

29. So what?

30. Both represent concurrent systems.
One is very high level.
One is incredibly low level.
The low level has the best tools and builds on
decades of experience of making things simpler.
They've made their lives easier

31. green Pipeline() =
Random() >> buffer(1) >>
Invert() >> buffer(1) >>
Invert() >> buffer(1) >> sink
end

32. green Substituter() =
chan thevalue,istwelve end
function test(value x:integer) : boolean =
x==12
end
Random() >> thevalue >> test() >> istwelve$
thevalue >> if istwelve then
sink$ source 0 >>
else
{}
end_if >> sink
end

33. green Substituter() =
chan thevalue,istwelve end
function test(value x:integer) : boolean =
Um?
x==12
end
Random() >> thevalue >> test() >> istwelve$
thevalue >> if istwelve then
sink$ source 0 >>
else
{}
end_if >> sink
end

34. green Substituter() =
chan thevalue,istwelve end
function test(value x:integer) : boolean =
Um?
x==12
end
Random() >> thevalue >> test() >> istwelve$
thevalue >> if istwelve then
sink$ source 0 >>
else
{}
end_if >> sink
end
That looks like software!

35. Can we write software that way
and still have it perform well,
and gain the same benefits?

36. YES!

37. Examples

38. Examples
I am only going to show you non-trivial
examples
For more trivial examples, please look at the
slides uploaded to slideshare.net/kamaelian
and also the Kamaelia cookbook.

39. Examples
Email Greylisting Server: Kamaelia Grey
Specifically written to deal with the ever growing
spam problem I was facing
Has eradicated my spam problem on my home
server.

40. Examples
Email Greylisting Server:
outbox
Selector inbox
outbox
TCPServer inbox
Greylist
Server

41. Examples
Email Greylisting Server: New Connection
outbox
Selector inbox
outbox
outbox
TCPServer inbox
outbox
Greylist
inbox
Server inbox
outbox
inbox
Connected
Socket
inbox outbox
GreyListing
Adapter outbox
inbox inbox
Policy

42. Examples
Email Greylisting Server: Many Connections
Selector
TCPServer
Greylist
Server
Connected
Socket GreyListing
Adapter Policy
Connected
Socket GreyListing
Adapter Policy
Connected
Socket GreyListing
Adapter Policy

43. Examples
Email Greylisting Core Code:
class GreylistServer(MoreComplexServer):
logfile = config[quot;greylist_logquot;]
...
port = config[quot;portquot;]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class TCPS(TCPServer):
CSA = NoActivityTimeout(ConnectedSocketAdapter,
timeout=config[quot;inactivity_timeoutquot;])
class protocol(GreyListingPolicy):
servername = config[quot;servernamequot;]
smtp_ip = config[quot;smtp_ipquot;]
smtp_port = config[quot;smtp_portquot;]
...

44. Examples Selector
Email Greylisting Core Code: TCPServer
Greylist
Server
class GreylistServer(MoreComplexServer):
logfile = config[quot;greylist_logquot;] Connected
GreyListing
Socket
... Adapter Policy
port = config[quot;portquot;]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class TCPS(TCPServer):
CSA = NoActivityTimeout(ConnectedSocketAdapter,
timeout=config[quot;inactivity_timeoutquot;])
class protocol(GreyListingPolicy):
servername = config[quot;servernamequot;]
smtp_ip = config[quot;smtp_ipquot;]
smtp_port = config[quot;smtp_portquot;]
...

45. Examples Selector
Email Greylisting Core Code: TCPServer
Greylist
Server
class GreylistServer(MoreComplexServer):
logfile = config[quot;greylist_logquot;] Connected
GreyListing
Socket
... Adapter Policy
port = config[quot;portquot;]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class TCPS(TCPServer):
CSA = NoActivityTimeout(ConnectedSocketAdapter,
timeout=config[quot;inactivity_timeoutquot;])
class protocol(GreyListingPolicy):
servername = config[quot;servernamequot;]
smtp_ip = config[quot;smtp_ipquot;]
smtp_port = config[quot;smtp_portquot;]
...

46. Examples Selector
Email Greylisting Core Code: TCPServer
Greylist
Server
class GreylistServer(MoreComplexServer):
logfile = config[quot;greylist_logquot;] Connected
GreyListing
Socket
... Adapter Policy
port = config[quot;portquot;]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class TCPS(TCPServer):
CSA = NoActivityTimeout(ConnectedSocketAdapter,
timeout=config[quot;inactivity_timeoutquot;])
class protocol(GreyListingPolicy):
servername = config[quot;servernamequot;]
smtp_ip = config[quot;smtp_ipquot;]
smtp_port = config[quot;smtp_portquot;]
...

47. Examples Selector
Email Greylisting Core Code: TCPServer
Greylist
Server
class GreylistServer(MoreComplexServer):
logfile = config[quot;greylist_logquot;] Connected
GreyListing
Socket
... Adapter Policy
port = config[quot;portquot;]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class TCPS(TCPServer):
CSA = NoActivityTimeout(ConnectedSocketAdapter,
timeout=config[quot;inactivity_timeoutquot;])
class protocol(GreyListingPolicy):
servername = config[quot;servernamequot;]
smtp_ip = config[quot;smtp_ipquot;]
smtp_port = config[quot;smtp_portquot;]
...

48. Examples Selector
Email Greylisting Core Code: TCPServer
Greylist
Server
class GreylistServer(MoreComplexServer):
logfile = config[quot;greylist_logquot;] Connected
GreyListing
Socket
... Adapter Policy
port = config[quot;portquot;]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class TCPS(TCPServer):
CSA = NoActivityTimeout(ConnectedSocketAdapter,
timeout=config[quot;inactivity_timeoutquot;])
class protocol(GreyListingPolicy):
servername = config[quot;servernamequot;]
smtp_ip = config[quot;smtp_ipquot;]
smtp_port = config[quot;smtp_portquot;]
...

49. Slightly less obvious than normal, but these
functions together form a factory.
outbox
Selector inbox
outbox
TCPServer inbox
Greylist
Server

50. Slightly less obvious than normal, but these
functions together form a factory... so as to
create new things to handle connections:
Selector
TCPServer
Greylist
Server
Connected
Socket GreyListing
Adapter Policy

51. Slightly less obvious than normal, but these
functions together form a factory... so as to
create new things to handle connections:
Selector
TCPServer
Greylist
Server
Connected
Socket GreyListing
Adapter Policy
Connected
Socket GreyListing
Adapter Policy

52. Slightly less obvious than normal, but these
functions together form a factory... so as to
create new things to handle connections:
Selector
TCPServer
Greylist
Server
Connected
Socket GreyListing
Adapter Policy
Connected
Socket GreyListing
Adapter Policy
Connected
Socket GreyListing
Adapter Policy

53. Examples
Kamaelia ER Modeller

54. Examples
Kamaelia ER Modeller
Intended for use in designing
databases. Created last
year because I had an
urgent need.
About a ½ day hack.

55. Examples
Kamaelia ER Modeller : Simple Example
entity missionagent
entity person(missionagent)
entity team(missionagent)
entity missionitem:
simpleattributes visible
entity activemission
relation participatesin(activemission,missionagent)
relation creates(missionagent,missionitem)

56. Examples
Kamaelia ER Modeller : More Complex Example
entity missionagent
entity person(missionagent):
simpleattributes name
entity team(missionagent):
simpleattributes teamname
entity mission
entity activemission
relation participatesin(activemission,missionagent)
entity missionitem:
simpleattributes visible
relation creates(missionagent,missionitem)
relation updates(missionagent,missionitem)
entity textfrag(atomicitem)
entity externalurl(atomicitem)
...

57. Examples
Kamaelia ER Modeller : Internals
Pipeline Pipeline
FileReader ConsoleEchoer
Pipeline ERParser ERVisualiser
Console Reader
ERModel2Visualiser lines_to_tokenlists
PublishTo(“TOPOLOGY”) PublishTo(“TOPOLOGY”) SubscribeTo(“TOPOLOGY”)
Backplane(“TOPOLOGY”)

58. Examples
Kamaelia ER Modeller : Internals
Pipeline Pipeline
FileReader ConsoleEchoer
Pipeline ERParser ERVisualiser
Console Reader
Backplane(quot;TOPOLOGYquot;).activate()
ERModel2Visualiser lines_to_tokenlists
PublishTo(“TOPOLOGY”) PublishTo(“TOPOLOGY”) SubscribeTo(“TOPOLOGY”)
Backplane(“TOPOLOGY”)

59. Examples
Kamaelia ER Modeller : Internals
Pipeline Pipeline
Pipeline(
FileReader ConsoleEchoer
ConsoleReader(quot;>>> quot;),
ERParser ERVisualiser
Pipeline
Console Reader
PublishTo(quot;TOPOLOGYquot;),
).activate()
ERModel2Visualiser lines_to_tokenlists
PublishTo(“TOPOLOGY”) PublishTo(“TOPOLOGY”) SubscribeTo(“TOPOLOGY”)
Backplane(“TOPOLOGY”)

60. Examples
Kamaelia ER Modeller : Internals
Pipeline( Pipeline Pipeline
ReadFileAdaptor(sys.argv[1]),
ConsoleEchoer FileReader
ERParser(), ERVisualiser ERParser
Pipeline
ERModel2Visualiser(),
Console Reader
PublishTo(quot;TOPOLOGYquot;),
lines_to_tokenlists ERModel2Visualiser
).activate()
PublishTo(“TOPOLOGY”) SubscribeTo(“TOPOLOGY”) PublishTo(“TOPOLOGY”)
Backplane(“TOPOLOGY”)

61. Examples
Kamaelia ER Modeller : Internals
Pipeline( Pipeline Pipeline
SubscribeTo(quot;TOPOLOGYquot;), FileReader ConsoleEchoer
lines_to_tokenlists(),
ERParser ERVisualiser
Pipeline
ERVisualiser(
Console Reader
screensize = (1024,768),
ERModel2Visualiser lines_to_tokenlists
fullscreen = True
PublishTo(“TOPOLOGY”) PublishTo(“TOPOLOGY”) SubscribeTo(“TOPOLOGY”)
),
ConsoleEchoer(),
).run() Backplane(“TOPOLOGY”)

62. Examples
Kamaelia ER Modeller : Internals
Pipeline Pipeline
Backplane(quot;TOPOLOGYquot;).activate() FileReader ConsoleEchoer
Pipeline ERParser ERVisualiser
Pipeline( Console Reader
ConsoleReader(quot;>>> quot;), ERModel2Visualiser lines_to_tokenlists
PublishTo(quot;TOPOLOGYquot;), PublishTo(“TOPOLOGY”) PublishTo(“TOPOLOGY”) SubscribeTo(“TOPOLOGY”)
).activate()
Pipeline( Backplane(“TOPOLOGY”)
ReadFileAdaptor(sys.argv[1]),
ERParser(),
ERModel2Visualiser(),
PublishTo(quot;TOPOLOGYquot;),
).activate()
Pipeline(
SubscribeTo(quot;TOPOLOGYquot;),
lines_to_tokenlists(),
ERVisualiser(screensize = (1024,768), fullscreen = True),
ConsoleEchoer(),
).run()

63. Examples
Kamaelia Macro

64. Examples
Kamaelia Macro
It records and transcodes
what is broadcast over DTT
for future viewing.

65. Examples
Kamaelia Macro
It records and transcodes
what is broadcast over DTT
for future viewing.
17A. ­ (1) The making in domestic premises for private and domestic use of a recording of a broadcast
solely for the purpose of enabling it to be viewed or listened to at a more convenient time does not
infringe any right conferred by Part 2 in relation to a performance or recording included in the
broadcast.
­­ Timeshifting as defined by copyright

66. Examples
Kamaelia Macro

67. Examples
Kamaelia Macro: Top Level Schematic
DVB Multiplex
outbox
inbox
BBC_ONE inbox Channel Transcoder
BBC_TWO inbox Channel Transcoder
DVB BBC_THREE inbox Channel Transcoder
Demuxer
BBC_FOUR inbox Channel Transcoder
CBEEBIES inbox Channel Transcoder
CBBC inbox Channel Transcoder

68. Examples
Kamaelia Macro: Transcode progs on a Channel
inbox
inbox outbox inbox
Splitter
Restartable
_eit_
Transcoder
transcoder_factory
inbox Prog outbox next
Change
Detector
Channel Transcoder

69. Examples
Kamaelia Macro: Finding Programme Junctions
inbox
inbox
inbox
DVB Packet Now Next
Reconstructor Parser
outbox
outbox
inbox
Now Next Now Next
inbox
Changes service filter
outbox
outbox
EIT Parsing
outbox

70. Examples
Kamaelia Macro: Programme Transcoder
inbox
control (a pipeline)
inbox control
Pipethrough
(“mencoder ­o file options...”) _stop
signal
_transcodingcomplete signal
Programme Transcoder

71. How?
What's inside the box?
The following examples are largely simplified,
simply because slides are a terrible way to
show code.
I've generally removed shutdown code.

72. How?
What's inside the box?
class ConsoleReader(threadedcomponent):
def main(self):
while 1:
line = raw_input(quot;>>quot;)
line = line + self.eol
self.send(line, quot;outboxquot;)

73. How?
What's inside the box?
class ConsoleReader(threadedcomponent):
def main(self):
while 1: See – we're saying this
line = raw_input(quot;>>quot;) component runs inside it's
line = line + self.eol own thread. This is all we have
self.send(line, quot;outboxquot;) to write for it to work.

74. How?
What's inside the box?
class ConsoleReader(threadedcomponent):
def main(self):
while 1:
line = raw_input(quot;>>quot;)
line = line + self.eol
self.send(line, quot;outboxquot;)
As you can see here, a
component communicates
with other components by
sending things to outboxes.

75. How?
What's inside the box?
class ConsoleEchoer(component):
def main(self):
while not self.shutdown():
while self.dataReady(quot;inboxquot;):
data = self.recv(quot;inboxquot;)
sys.stdout.write(str(data)) Whereas here, you can see
sys.stdout.flush() that they listen for data in
yield 1 their inboxes. and then do
stuff based on that!

76. How?
What's inside the box?
This one however is a
class ConsoleEchoer(component): generator based component.
def main(self): This runs in the same thread
while not self.shutdown(): as all other generators,
while self.dataReady(quot;inboxquot;): making this code equivalent to
data = self.recv(quot;inboxquot;) a state machine. Note the
sys.stdout.write(str(data)) baseclass does not say
sys.stdout.flush() threaded or generator.
yield 1

77. How?
What's inside the box?
class ConsoleEchoer(component):
def main(self):
while not self.shutdown():
while self.dataReady(quot;inboxquot;):
data = self.recv(quot;inboxquot;)
sys.stdout.write(str(data))
sys.stdout.flush()
yield 1
This yield keyword is what makes
this method a generator. For
now, you can assume it acts like
a “return back here when next
called” thing. It's NOT really the
same at all as a ruby yield!

78. And that's the basic idea.
On the website there's a mini axon tutorial and a
guide to writing components, and a few
magazine articles as well which should get you
started.
There are also more tutorials on slideshare
Please also ask on #kamaelia on freenode.

79. Thank you:-)