Incidentally, having personally written the software behind an irc network that's bigger than freenode
From what I've read in your comments in the past, I'm assuming you used Twisted for the IRC backend. I was wondering, if you were starting this project today, would you still use Twisted? (apologies for veering off topic)
Yes, I used Twisted as the non-blocking network substrate for JTV chat. Actually in the initial implementation I used their IRC protocol library too, but that has been gradually dropped and replaced as more and more problems have been found. This is my general impression of Twisted actually - as an i/o core, it's rock-solid, but the protocol libraries are much more rough and many have clearly not been tested under significant load or with real-world messy data.
As for whether I would use it again... I think I'd be really tempted to use Clojure to be honest. Python is nice enough, but I still find myself missing a real Lisp (I did a lot of CL work before I joined JTV).
Not on a single box, no - we run these machines in a cluster. Each chat machine scales up to a bit more than 80k simultaneous connections (each machine runs 8 processes, each of which can do about 10k simultaneous). There's some amount of IPC between all the chat processes in the cluster, but not too much, which means we can pretty much scale linearly just by adding hardware at this point.
Hm. Still, 80k per box is nothing to sniff at, especially with Python.
I've been using Ruby's EventMachine, I can get it up to about C20k before it blows up, but that's local only, remains to be seen what proper connections will do. Haven't tried multiple processes yet. As for IPC I just dump everything into AMQP, whose fanout exchanges seem practically designed for IRC : D
Hm, anyway, thanks for the info. Really cool to hear python stretching that far.
From what I've read in your comments in the past, I'm assuming you used Twisted for the IRC backend. I was wondering, if you were starting this project today, would you still use Twisted? (apologies for veering off topic)