I recently decided to use Twisted as the framework for a new project. In short, this app listens to and parses incoming events from multiple servers, as well as issues commands back to them.
Once I had Python doing its job managing the data, the next step was to expose the data in memory from the Twisted app to an existing PHP/Drupal UI. Twisted makes it pretty simple to attach an HTTP server to your app, so I did just that.
Why JSONP?
Because Apache & the PHP app was running on port 80 and my Twisted service was running on 8000, standard AJAX requests were not going to work because of XSS restrictions. My only options were either write a PHP wrapper, or use JSONP. JQuery supports JSONP very elegantly, so I chose to move forward with that option.
Why Long Polling
Standard polling from an AJAX app will typically send a request at a given interval. Even when there's no data to return from the server, the server sends an empty response and the cycle starts all over.
Long polling cuts down on the overhead a bit by keeping the request open until there is actually data to send back. Again Twisted comes through in making it easy to serve data like this.
Example
I took the relevant pieces of code to make a very generic example to provide a starting point for anyone looking to implement something similar.
Note: You will need to modify long_poll.js to point to the destination of the twisted server. Currently it is example.com:8000
index.html (HTML file served from Apache)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>Twisted - Long Polling & JSONP</title>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<script type="text/javascript" src="long_poll.js"></script>
</head>
<body>
<h1>Twisted - Long Polling & JSONP</h1>
<p>New message will appear below as they are received from the server.</p>
<hr />
<ul id="messages"></ul>
</body>
</html>
long_poll.js (JavaScript file served from Apache)
// variable to keep track of the last time we received data
// a time value will be sent back with the response in a unix timestamp format
var lastupdate = 0;
// call getData when the document has loaded
$(document).ready(function(){
getData(lastupdate);
});
// execute ajax call to server.py
var getData = function(lastupdate) {
$.ajax({
type: "GET",
// set the destination for the query
url: 'http://example.com:8000?lastupdate='+lastupdate+'&callback=?',
// define JSONP because we're using a different port and/or domain
dataType: 'jsonp',
// needs to be set to true to avoid browser loading icons
async: true,
cache: false,
// timeout after 5 minutes
timeout:300000,
// process a successful response
success: function(response) {
// append the message list with the new message
var messages = response.data.messages;
for (x in messages) {
$('<li>'+messages[x].published+' - '+messages[x].message+'</li>').appendTo('#messages');
}
// set lastupdate
lastupdate = response.timestamp;
// call again in 1 second
setTimeout('getData('+lastupdate+');', 1000);
},
// handle error
error: function(XMLHttpRequest, textStatus, errorThrown){
// try again in 10 seconds if there was a request error
setTimeout('getData('+lastupdate+');', 10000);
},
});
};
server.py (start with: $ python server.py)
from twisted.web import server
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor, task
import json
import time
# just for simulation in getData
import random
class InfoServer(Resource):
isLeaf = True
def __init__(self):
# throttle in seconds to check app for new data
self.throttle = 5
# define a list to store client requests
self.delayed_requests = []
# setup a loop to process delayed requests
loopingCall = task.LoopingCall(self.processDelayedRequests)
loopingCall.start(self.throttle, False)
# initialize parent
Resource.__init__(self)
def render(self, request):
"""
Handle a new request
"""
# set the request content type
request.setHeader('Content-Type', 'application/json')
# set args
args = request.args
# set jsonp callback handler name if it exists
if 'callback' in args:
request.jsonpcallback = args['callback'][0]
# set lastupdate if it exists
if 'lastupdate' in args:
request.lastupdate = args['lastupdate'][0]
else:
request.lastupdate = 0
# if we have data now, send it
data = self.getData(request)
if len(data) > 0:
return self.__format_response(request, 1, data)
# otherwise, put it in the delayed request list
self.delayed_requests.append(request)
# tell the client we're not done yet
return server.NOT_DONE_YET
def getData(self, request):
"""
Replace this logic with code that will actually test for
and return data your app should return.
You can use request.lastupdate here if you want to pull
data since the last time this request received data.
This is just dummy logic to make this demo work.
"""
# init data
data = {}
#simulate the chance of new data being available or not
new_data_available = bool(random.getrandbits(1))
# set some simulated data
if new_data_available:
# you can dynamically add any key/value pair here
data = {'messages':[
{
'message':'Test Message',
'published':int(time.time())
},
]
}
return data
def processDelayedRequests(self):
"""
Processes the delayed requests that did not have
any data to return last time around.
"""
# run through delayed requests
for request in self.delayed_requests:
# attempt to get data again
data = self.getData(request)
# write response and remove request from list if data is found
if len(data) > 0:
try:
request.write(self.__format_response(request, 1, data))
request.finish()
except:
# Connection was lost
print 'connection lost before complete.'
finally:
# Remove request from list
self.delayed_requests.remove(request)
def __format_response(self, request, status, data):
"""
Format responses uniformly
"""
# Set the response in a json format
response = json.dumps({'status':status,'timestamp': int(time.time()), 'data':data})
# Format with callback format if this was a jsonp request
if hasattr(request, 'jsonpcallback'):
return request.jsonpcallback+'('+response+')'
else:
return response
#############################################
if __name__ == '__main__':
resource = InfoServer()
factory = Site(resource)
reactor.listenTCP(8000, factory)
reactor.run()