This document summarizes techniques for building reactive and real-time web applications using TurboGears and Python. It discusses how HTTP is changing to support real-time functionality through HTTP/2 and server pushing. Reactive frameworks that automatically update the web page on data changes are described. Solutions for real-time including polling, long polling, server-sent events, and websockets are covered. The document demonstrates using server-sent events and Socket.IO with TurboGears to build real-time applications. It also introduces Ractive.js and ToscaWidgets2 for adding reactivity to the client-side and integrating reactive widgets with TurboGears.

1. REACTIVE and REALTIME
Web Applications with TurboGears
Alessandro Molina
@__amol__
amol@turbogears.org

2. Who am I
● CTO @ Axant.it, mostly Python company
(with some iOS and Android)
● TurboGears2 dev team member
● Contributions to web world python libraries
○ MING MongoDB ODM
○ ToscaWidgets2
○ Formencode

3. Why Realtime
● Web Applications are moving toward
providing data on realtime as soon as it’s
available.
● This requires browser to keep an open
connection to be notified about changes
● Pushing data and Concurrency became
the primary issues

4. HTTP changing for realtime
● HTTP / 2.0 under work, based on SPDY
● Request/Response multiplexed
○ Single TCP connection with frames described by a
binary protocol (solves some concurrency issues
and speeds up on mobile networks)
● Support for server side pushing data
○ core feature for realtime applications

5. What’s Reactive
● Reactive Applications automatically
update themselves when data changes.
● Reactive frameworks make easier to
create realtime applications
● Propagating data changes becames the
primary issue.

6. Where they belong
● Realtime is usually a server side problem
○ Server is the publisher of data changes
○ It has to cope with every single client
● Reactive is usually a client side problem
○ You want to update only the portion of the web
page that changed, not regenerate everything
server side

7. Time for Mad Science

8. Solutions for Realtime
● Polling
○ While True: do you have new data?
○ Huge overhead connection setup/teardown
● Long Polling
○ Server will keep the connection open until a
timeout or it has data available.
○ When timeout or data ends client will reopen
connection to wait for next data

9. Solutions for Realtime #2
● Server Side Events
○ Covers only Server->Client
○ Automatically handles failures and recovery
○ Not supported by IE
● WebSockets
○ Bidirectional communication
○ Supported on modern browsers

10. Streaming VS Messages
● Streaming
○ Used by Long Polling & SSE
○ Keeps the connection open (blocks client read)
○ Sends data as a unique continuous stream
○ Usually implemented using a generator
● Messages
○ Used by WebSocket and SSE
○ Sends/Receives data as individual message

11. Server Side Events
# Expose text/event-stream which is the
# content type required to let the browser
# know that we are using SSE
@expose(content_type='text/event-stream')
def data(self, **kw):
# To perform streaming of data (without closing
# the connection) we just return a generator
# instead of returning a string or a dictionary.
def _generator():
while True:
# The generator simply starts fetching data
# from the queue, waiting for new tweets
# to arrive.
value = q.get(block=True)
# When a new tweet is available, format the data
# according to the SSE standard.
event = "data: %s %snn" % (value, time.time())
# Yield data to send it to the client
yield event
# Return the generator which will be streamed back to the browser
return _generator()

12. Messages with Socket.IO
● Implements messages and events
● Bidirectional
● Abstraction over the underlying available
technology (LongPoll, Flash, WebSocket)
● Provides connection errors handling
● Server side implementation for Python

13. Socket.IO with TurboGears
● tgext.socketio
○ Provides ready made support for SocketIO
○ SocketIO events are as natural as writing methods
named on_EVENTNAME
○ Based on gevent to avoid blocking concurrent
connections
○ Compatible with Gearbox and PasterServe, no
need for a custom environment

14. Ping Pong with SocketIO
<html>
<body>
<div>
<a class="ping" href="#" data-attack="ping">Ping</a>
<a class="ping" href="#" data-attack="fireball">Fireball</a>
</div>
<div id="result"></div>
<script src="/javascript/jquery.js"></script>
<script src="/javascript/socket.io.min.js"></script>
<script>
$(function(){
var socket = io.connect('/pingpong', {'resource': 'socketio'});
$('.ping').click(function(event){
socket.emit('ping', {'type': $(this).data('attack')});
});
socket.on('pong', function(data){
$('#result').append(data.sound + '<br/>');
});
});
</script>
</body>
</html>

15. Ping Pong server side
from tgext.socketio import SocketIOTGNamespace, SocketIOController
class PingPong(SocketIOTGNamespace):
def on_ping(self, attack):
if attack['type'] == 'fireball':
for i in range(10):
self.emit('pong', {'sound':'bang!'})
else:
self.emit('pong', {'sound':'pong'})
class SocketIO(SocketIOController):
pingpong = PingPong
@expose()
def page(self, **kw):
return PAGE_HTML
class RootController(BaseController):
socketio = SocketIO()

16. Speed alone is not enough
we need some help in managing the added complexity of realtime updates

17. Adding Reactivity
● Now we are being notified on real time
when something happens
● Then we need to update the web page
accordingly to the change
● Previously achieved by
○ $('#result').append(data.sound + '<br/>');

18. Ractive.JS
● Declare a Template (Mustache)
● Declare a Model (POJO)
● Join them with Ractive
● Whenever the model changes the
template gets automatically redrawn
● Written in JavaScript

19. Reactive PingPong
<html>
<body>
<div>
<a href="#" onclick="socket.emit('ping', {'type': 'ping'});">Ping</a>
<a href="#" onclick="socket.emit('ping', {'type': 'fireball'});">Fireball</a>
</div>
<div id="result"></div>
<script src="/ractive.js"></script>
<script src="/socket.io.min.js"></script>
<script id="PingPongWidget_template" type='text/ractive'>
{{#messages:idx}}
<div>{{sound}}</div>
{{/messages}}
</script>
<script>
var pingpong = new Ractive({template: '#PingPongWidget_template',
el: 'result',
data: {'messages': []}
});
var socket = io.connect('/pingpong', {'resource':'socketio'});
socket.on('pong',function(data) {
pingpong.get('messages').push(data);
});
</script>
</body>
</html>

20. Reactive and Realtime!

21. Let’s move it back the python side
● Now we ended up with MVCMVC (or MVT)
○ Python Model with Python Controller with a
Python Template on server
○ JS Model with JS Controller and JS Template on
client
● I want to keep a single stack but benefit
from ractive.
○ Let ractive handle all web page widgets

22. ToscaWidgets2
● Split your web page in a bunch of widgets
● Each widget has its data
● Reload the page to redraw the widget
● Written in Python
● Also provides data validation for the
widget

23. ToscaWidgets PingPong
from tw2.core import Widget
class PingPongWidget(Widget):
inline_engine_name = 'genshi'
template = '''
<div xmlns:py="http://genshi.edgewall.org/"
py:for="message in w.messages">
${message.sound}
</div>
'''
PingPongWidget.display(messages=[{'sound': 'pong'},
{'sound': 'bang!'}
])

24. Hey looks like Ractive!
● Both provide entities with a Template
● Both provide support for having data
within those entities
● Ractive automatically updates itself
● ToscaWidgets has data validation and
dependencies injection

25. Join them, get SuperWidgets!

26. RactiveWidget
from axf.ractive import RactiveWidget
class PingPongWidget(RactiveWidget):
ractive_params = ['messages']
ractive_template = '''
{{#messages:idx}}
<div>{{sound}}</div>
{{/messages}}
'''
● we did an experiment in AXF library for
ToscaWidgets based Ractive Widgets

27. PingPong using RactiveWidget
from tg import AppConfig, expose, TGController
from tgext.socketio import SocketIOTGNamespace, SocketIOController
class PingPong(SocketIOTGNamespace):
def on_ping(self, attack):
[...]
class SocketIO(SocketIOController):
pingpong = PingPong
@expose('genshi:page.html')
def page(self, **kw):
return dict(pingpong=PingPongWidget(id='pingpong', messages=[]))
class RootController(TGController):
socketio = SocketIO()
config = AppConfig(minimal=True, root_controller=RootController())
config.serve_static = True
config.renderers = ['genshi']
config.use_toscawidgets2 = True
config.paths['static_files'] = 'public'
print 'Serving on 8080'
from socketio.server import SocketIOServer
SocketIOServer(('0.0.0.0', 8080), config.make_wsgi_app(), resource='socketio').serve_forever()

28. PingPong Template
<html>
<head>
<script src="/socket.io.min.js"></script>
<script src="/ractive.js"></script>
</head>
<body>
<div>
<a href="#" onclick="socket.emit('ping', {'type': 'ping'});">Ping</a>
<a href="#" onclick="socket.emit('ping', {'type': 'fireball'});">Fireball</a>
</div>
${pingpong.display()}
<script>
var socket = io.connect('/pingpong', {'resource':'socketio'});
socket.on('pong',function(data) {
document.RAWidgets.pingpong.get('messages').push(data);
});
</script>
</body>
</html>

29. Benefits
● Simpler template, just display a widget and
let it update itself.
● Widget dependencies (JS, CSS and so on)
are brought in by ToscaWidgets itself, no
need to care.
● No need to encode data and pass it from
python to JS, ToscaWidget does that for us

30. Publish / Subscribe
● The real power of ractive/realtime
programming is unleashed in multiuser
environments
● tgext.socketio has built-in support for
publish/subscribe
● With multiple backends supported
(memory, Redis, mongodb, amqp)

31. Realtime chat example
● Realtime multiuser chat
● Support for multiple rooms
● Just replace PingPong namespace
● New namespace will inherit from
PubSubTGNamespace instead of
SocketIOTGNamespace

32. PubSub - Server Side
class ChatNamespace(PubSubTGNamespace):
# PubSubTGNamespace automatically calls subscribe_room
# whenever a client subscribes to a “room” channel.
def subscribe_room(self, roomid):
# Whe generate an user id for the client when it first subscribes
self.session['uid'] = str(uuid.uuid4())
# This is the channel where notifies will be published
self.session['channel'] = 'room-%s' % roomid
# Whe notify the client that his userid is the newly generated one
self.emit('userid', self.session['uid'])
# Tell tgext.socketio for which channel we subscribed
return self.session['channel']
# When the client emits e message, publish it on the
# room so it gets propagated to every subscriber.
def on_user_message(self, message):
self.publish(self.session['channel'], {'uid': self.session['uid'],
'message': message})

33. PubSub - Client Side
<body>
<div id="chat">
<div id="messages">
<div id="lines"></div>
</div>
<form id="send-message">
<input id="message"> <button>Send</button>
</form>
</div>
<script>
var socket = io.connect('/chat', {'resource':'socketio'});
var userid = null;
socket.on('connect', function () { socket.emit('subscribe', 'room', '1'); });
socket.on('userid', function (myid) { userid = myid; });
socket.on('pubblication', function (data) {
var sender = data.uid;
var msg = data.message;
$('#lines').append($('<p>').append($('<em>').text(msg)));
});
$('#send-message').submit(function () {
socket.emit('user_message', $('#message').val());
$('#message').val('').focus();
return false;
});
</script>
</body>

34. Questions?