I recently started using twisted a couple of weeks ago. For those who don’t know, twisted is “event-driven networking engine written in Python”. The learning curve is pretty steep if you’ve never done asynchronous programming before. During the project I was working on, I ran into a condition where I thought I needed to restart the twisted reactor. According to everything I found online, restarting the reactor is not supported. But I can be stubborn so I tried to find a way anyway.
Restarting a Twisted Reactor
Let’s start by creating a pretty standard twisted server. We’ll subclass LineReceiver which basically makes a TCP server that accepts full lines of text, although it can also do raw data too. Let’s take a look at the code:
from twisted.internet import reactor from twisted.protocols.basic import LineReceiver from twisted.internet.protocol import Factory PORT = 9000 class LineServer(LineReceiver): def connectionMade(self): """ Overridden event handler that is called when a connection to the server was made """ print "server received a connection!" def connectionLost(self, reason): """ Overridden event handler that is called when the connection between the server and the client is lost @param reason: Reason for loss of connection """ print "Connection lost" print reason def lineReceived(self, data): """ Overridden event handler for when a line of data is received from client @param data: The data received from the client """ print 'in lineReceived' print 'data => ' + data class ServerFactory(Factory): protocol = LineServer if __name__ == '__main__': factory = ServerFactory() reactor.listenTCP(PORT, factory) reactor.run()
All of the camel-cased methods are overridden twisted methods. I just stubbed them out and have them print to stdout whenever they get called. Now let’s make a client that has a reactor that we restart a few times:
import time import twisted.internet from twisted.internet import reactor, protocol from twisted.protocols.basic import LineOnlyReceiver HOST = 'localhost' PORT = 9000 class Client: """ Client class wrapper """ def __init__(self, new_user): self.new_user = new_user self.factory = MyClientFactory() def connect(self, server_address=HOST): """ Connect to the server @param server_address: The server address """ reactor.connectTCP(server_address, PORT, self.factory, timeout=30) class MyProtocol(LineOnlyReceiver): def connectionMade(self): """ Overridden event handler that is fired when a connection is made to the server """ print "client connected!" self.run_long_running_process() def lineReceived(self, data): """ Gets the data from the server @param data: The data received from the server """ print "Received data: " + data def connectionLost(self, reason): """ Connection lost event handler @param reason: The reason the client lost connection with the server """ print "Connection lost" def run_long_running_process(self): """ Run the process """ print 'running process' time.sleep(5) print "process finished!" self.transport.write('finished' + '\r\n') reactor.callLater(5, reactor.stop) class MyClientFactory(protocol.ClientFactory): protocol = MyProtocol if __name__ == '__main__': # run reactor multiple times tries = 3 while tries: client = Client(new_user=True) client.connect('localhost') try: reactor.run() tries -= 1 print "tries " + str(tries) except Exception, e: print e import sys del sys.modules['twisted.internet.reactor'] from twisted.internet import reactor from twisted.internet import default default.install()
Here we create a simple client that accepts only lines of text too (LineOnlyReceiver). The magic for restarting the reactor lies in the while loop at the end of the code. I actually found the code in the exception handler inside of twisted’s reactor.py file which gave me the idea. Basically what we’re doing is importing Python’s sys module. We then delete the reactor from sys.modules which allows us to re-port it and reinstall the default reactor. If you run the server in one terminal and the client in another, you can see the client reconnect three times.
Wrapping Up
As I mentioned at the beginning, I’m still pretty new to twisted. What you should probably do instead of restarting the reactor is run it in another thread. Or you might use one of its delayed calls or deferred threads to get around the “need” to restart the reactor. Frankly, the method in this article doesn’t even work in some conditions. I was actually trying to restart the reactor once inside a function that was decorated with contextlib’s contextmanager decorator and that somehow prevented this code to work correctly. Regardless, I thought it was an interesting way to reload a module.
Seems I found this blog post a bit late, but I felt compelled to comment all the same.
I’m not exactly an active Twisted user, but I’ve been following the project for a very long time, similarly to wxPython. One thing I’ve seen the Twisted devs repeatedly say on IRC and on mailing lists is that if you think you need to restart the reactor, you actually don’t. So don’t do that. The problem with the reactor as it’s currently designed is that it uses a lot of global state and there are no guarantees about what actually happens if you do manage to restart it.
I’m sorta curious as to why you thought you needed to restart it, but I suspect that your final paragraph might provide a clue to that. Indeed, the correct thing to do is run the reactor in its own thread and use something like reactor.callLater to run your method or function in the reactor thread much the same way you would use wx.CallAfter in wxPython for running code in the GUI thread. You would keep the reactor thread running for the entirety of your process’ lifetime.
I can’t think of the exact spelling off the top of my head, but deferToThread (I think there is a thread pool variation as well) is used for starting threads from the reactor thread and doesn’t really apply in this particular situation.
doesn’t work in my case š I got the same error: twisted.internet.error.ReactorNotRestartable
any other ideas or updates on this?
Thanks.