-
Notifications
You must be signed in to change notification settings - Fork 0
Automatically exported from code.google.com/p/syncless
License
HanWenfang/syncless
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
README for Syncless: asynchronous client and server library using Stackless Python """""""""""""""" by pts@fazekas.hu at Sun Dec 20 22:47:13 CET 2009 -- Fri Jan 8 03:06:58 CET 2010 -- Tue Feb 9 19:02:52 CET 2010 -- Mon Apr 19 02:54:24 CEST 2010 Syncless is a non-blocking (asynchronous) concurrent client and server socket network communication library for Stackless Python 2.6 (and also for regular Python with greenlet). For high speed, Syncless uses libev (and libevent) for event notification, and parts of Syncless' code is implemented in Pyrex/Cython and C. This alone makes Syncless faster than many other non-blocking network libraries for Python. Syncless contains an asynchronous DNS resolver (using evdns) and a HTTP server capable of serving WSGI applications. Syncless aims to be a coroutine-based alternative of event-driven networking engines (such as Twisted, asyncore, pyevent, python-libevent and FriendFeed's Tornado), and it's a competitor of gevent, Eventlet and Concurrence. Features ~~~~~~~~ * handling multiple TCP connections concurrently in a single Python process, using cooperative multitasking based on coroutines, as provided by Stackless Python or greenlet (without the need for callbacks, threads, subprocesses or locking) * non-blocking DNS resolver using evdns * monkey-patchable, almost faithful non-blocking reimplementation of socket.socket, socket.gethostbyname (etc.), ssl.SSLSocket, time.sleep and select.select * compatible timeout handling on individual socket operations * I/O event detection with libevent1, libevent2 or libev (fastest) and provides a slow fallback if none of these are available * easy to convert existing single-threaded, multi-threaded or multiprocess code to Syncless coroutines (because coroutines work like lightweight threads, Syncless exposes a compatible, monkey-patchable interface for sockets, pipes and buffered I/O, and locking is not needed) * non-blocking support added by monkey-patching to built-in urllib, urllib2, smtplib, ftplib, imaplib, poplib, asyncore, popen2, subprocess etc. modules * special monkey-patching for pure Python MySQL client libraries mysql.connector and pymysql * special monkey-patching for C (Cython) MySQL client library geventmysql * I/O event detection using the fastest methods (epoll(7) on Linux, kqueue on BSD etc.) with libev * built-in (non-blocking) WSGI server, but can use CherryPy's WSGI server as well in non-blocking mode * non-blocking stdin/stdout support * built-in WSGI server capable of running not only WSGI applications, but BaseHTTPRequestHandler + BaseHTTPServer applications, CherryPy applications, web.py applications, and Google webapp applications (not supporting most other Google AppEngine technologies) as well * combination of Syncless and (Twisted, Tornado (fast), Concurrence, gevent, Eventlet, circuits and/or asyncore) in the same process * a thread pool class for wrapping blocking operations * an interactive Python console for experimenting with the Syncless library, without having to write a Python script * a remote interactive Python console (backdoor) named RemoteConsole for debugging, which accepts TCP (telnet) connections, and supports line editing (readline) if used with the supplied client * WebSocket server API in the WSGI server module * HTTP/1.1 request pipelining Trying out Syncless (the fastest way) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On Linux, it's possible to try Syncless without installation, using the StaticPython binary Python distribution, like this: $ wget -O stacklessco2.7-static http://pts-mini-gpl.googlecode.com/svn/trunk/staticpython/release/stacklessco2.7-static $ chmod +x stacklessco2.7-static $ ./stacklessco2.7-static Python 2.7.1 Stackless 3.1b3 060516 (release27-maint, Feb 1 2011, 16:57:16) [GCC 4.1.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from syncless import coio >>> coio.sleep(1.5) (sleeping for 1.5 second) <object object at 0xf7709490> >>> Limitations: * This solution requires Linux on x86 (i386) or x86_64 (amd64) architectures. See the subsequent chapters about installation on other systems. * This solution doesn't give you the newest version of Syncless. * You can't use Python extensions written in C, except for those compiled into stacklessco2.7-static. See the full list on the StaticPython home page: http://code.google.com/p/pts-mini-gpl/wiki/StaticPython . Requirements ~~~~~~~~~~~~ * A recent Unix system. Tested on Linux 2.6, should work on FreeBSD. Testers for other Unix variants are welcome. It won't work on Win32 or Win64. * Stackless Python 2.6.x (recommended, especially for production use) or normal (C)Python 2.5 or 2.6 with greenlet (slow, may leak memory, not recommended in general, but it's easy to install and try). * Python 2.6 (recommended, for epoll(7) support) or Python 2.5. * A C compiler (gcc) and the Python development package (.h files) for compilation. * Various development packages such as python2.5-dev and libssl-dev already installed. Installation (the fast and easy way) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You need a Unix system with Python 2.5 or 2.6. Stackless Python is recommended, but not necessary. To install Syncless, run this as root (without the # sign): # easy_install syncless Be patient, compilation may take up to 60 seconds. As an alterative, if you have already downloaded and extracted Syncless, you can install it with: # python setup.py install To verify that Syncless got properly installed, run this command (without the leading $): $ python -c 'from syncless.coio import sleep; sleep(1e-5); print "OK"' OK You can also try if your Syncless works using the interactive console. See http://code.google.com/p/syncless/wiki/Console for details . Example invocation: $ python -m syncless.console If you want to install Syncless to a specific Pyton version on your system, run the following instead (without python substituted properly): # python -c'from setuptools.command.easy_install import main;main()' syncless If you don't have the easy_install command installed, do this: $ wget http://peak.telecommunity.com/dist/ez_setup.py $ python ez_setup.py syncless If installation fails, or you want to reinstall Syncless to get the highest performance, please follow the next Installation section. Installation (the hard way) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ For a Python with coroutine support, you have these options (pick one): * Python 2.6 with greenlet (easy to install, slow, may leak memory, not recommended in general, but recommended if you want to try Syncless quickly without bothering to install Stackless Python) * Python 2.5 with greenlet (like Python2.6, but without SSL support) * Stackless Python >= 2.6.4 (recommended for production use) For an asynchronous event notification library with DNS support, you have these options (pick one): * minievent (very simple to install, bundled with Syncless, ideal for trying and learning Syncless, but has poor performance with >10 TCP connections) * libev >= 3.9 + minihdns (recommended, fastest, minihdns bundled with Syncless) * libev >= 3.9 + evhdns (like with minihdns, but more work to install) * libevent2 >= 2.0.4 (recommended if you don't like libev) * libevent1 >= 1.4.13 (not recommended, because it can register only a single event on the same filehandle with the same purpose at a time) Remember your picks above. 0. Install a C compiler (e.g. $ apt-get install gcc). 1. If you have picked Python 2.5 or Python 2.6 (non-Stackless), install it either from source or from binary package. When installing from binary package, please install the headers as well. For example, on Debian and Ubuntu, run one of: $ sudo apt-get install python2.5 python2.5-dev $ sudo apt-get install python2.6 python2.6-dev 2. If you have picked Stackless, most probably you have to download it and install it from source. Download and install Stackless Python 2.6.x (not 3.x) from http://stackless.com/ For convenience, rename the installed executable or make a symlink to /usr/local/bin/stackless2.6: $ sudo ln -s python2.6 stackless2.6 3. If you have picked greenlet, download and install it from source. Syncless needs a recent version of greenlet (>= 0.3.1, >= 2010-04-06). Older versions lack the `throw' method. Get it from http://pypi.python.org/pypi/greenlet (pip and easy_install also work). You can also check it out from the SVN repository with: $ svn co http://codespeak.net/svn/py/release/0.9.x/py/c-extension/greenlet Please note that http://www.undefined.org/python/#greenlet contains an old version (without greenlet.greenlet.throw). Install it as usual: $ $PYTHON setup.py build $ sudo $PYTHON setup.py install , where $PYTHON is your choice of Python above: stackless2.6, python2.5, python2.6. 4. If you have picked libev, download and install libev from http://software.schmorp.de/pkg/libev.html Syncless has been tested on Linux with libev-3.9. The version shipped with your Linux distribution will probably also be OK if it's new enough. 5. If you have picked libev, you may want to install libevhdns (That's an extra with marginal benefit, most users don't need that). Download the libevhdns sources from http://code.google.com/p/libevhdns/ , and install them. The minimum version required is 1.4.13.4. Most probably your Linux distribution doesn't have libevhdns, so you should install libevhdns from source. 6. If you have picked libevent1, download the libevent1 sources from http://www.monkey.org/~provos/libevent/ , and install them. The minimum version required is 1.4.13. Make sure you don't download version 2.x. Most probably your Linux distribution has a libevent1, but it's too old (like 1.3). If it's at least 1.4, you can try that one instead of installing from source. You don't have to install the development package, Syncless works without it. 7. If you have picked libevent2, download the libevent2 sources from http://www.monkey.org/~provos/libevent/ , and install them. The minimum version required is 2.0.4. Make sure you don't download version 1.x. Most probably your Linux distribution doesn't have libevent2, so you should install libevent2 from source. 8. Download the newest version of Syncless. Get the .tar.gz file from here: http://pypi.python.org/pypi/syncless or from here: http://code.google.com/p/syncless/ . Extract the .tar.gz file and cd into the directory. Alteratively, you may check out the trunk from the SVN repository: svn checkout http://syncless.googlecode.com/svn/trunk/ syncless-read-only 9. Compile Syncless. Make sure you are in the directory containing setup.py and syncless/patch.py . Then run $ $PYTHON setup.py build , where $PYTHON is your choice of Python above: stackless2.6, python2.5, python2.6. You may want to make sure that Syncless has picked up its right dependencies. The `setup.py build' commands displays dependency and configuration information before compilation, and it also saves that to the setup.cenv file: * COIO_USE_CO_STACKLESS: Stackless Python is used for coroutines. * COIO_USE_CO_GREENLET: greenlet is used for coroutines. * COIO_USE_LIBEVHDNS: libevhdns is used for DNS resolution. * COIO_USE_MINIHDNS: minihdns is used for DNS resolution. * COIO_USE_MINIEVENT: minievent is used for notification. * COIO_USE_LIBEV: libev is used for notification. * COIO_USE_LIBEVENT1: libevent1 is used for DNS resolution and notification. * COIO_USE_LIBEVENT2: libevent2 is used for DNS resolution and notification. Don't get confused by None, only the presence of a constant matters, e.g. if (COIO_USE_CO_STACKLESS, None) is present, than Stackless _will_ be used. Before running `setup.py build', you can set one or more of the following environment variables to force Syncless use a specific dependency instead of autodetection: * export SYNCLESS_USE_LIBEV=1 * export SYNCLESS_USE_LIBEVENT1=1 * export SYNCLESS_USE_LIBEVENT2=1 * export SYNCLESS_USE_MINIEVENT=1 * export SYNCLESS_USE_LIBEVHDNS=1 * export SYNCLESS_ALLOW_MINIEVENT=1 (is 1 by default, set to '' to disable) Please note that you don't have to run the `install' step to experiment with Syncless: after the `build' step, you can run the demos in the `examples' directory, and you can also run the tests in the `test' directory. How to use (with example code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Follow the steps in `Installation'. 2. In the Syncless directory, run $ stackless2.6 ./examples/demo.py 3. Have a look at examples/demo_*.py in the source directory to study the examples. Author ~~~~~~ with accents in UTF-8: Péter Szabó <pts@fazekas.hu> without accents: Peter Szabo <pts@fazekas.hu> License ~~~~~~~ Syncless is licensed under the Apache License, Version 2.0. If you need other licensing options, please contact the author. See also http://www.petefreitag.com/item/533.cfm . Using Syncless with web frameworks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The WSGI-capable HTTP server in Syncless can run any framework with WSGI support. Examples: * pure WSGI application, without a framework: see SimpleWsgiApp in demo.py and examples/demo_wsgiapp.py * (web.py): see examples/demo_syncless_web_py.py * CherryPy: see examples/demo_syncless_cherrypy.py * Google AppEngine webapp: see examples/demo_syncless_webapp.py * Python built-in BaseHTTPRequestHandler: see examples/demo_syncless_basehttp.py Please note that Syncless is not a web framework. Please note that Syncless is not ready yet for production (e.g. TCP communication error handling and robust recovery from exceptions are not written). Feel free to try it, however, with your web application (using any framework), and report problems. Related projects ~~~~~~~~~~~~~~~~ * Spawning (WSGI server) http://pypi.python.org/pypi/Spawning * Concurrence * gevent * node.js (in Javascript) * Eventlet (instead of Stackless Python) http://eventlet.net/ slides http://soundfarmer.com/content/slides/coroutines-nonblocking-io-eventlet-spawning/coros,%20nonblocking%20i:o,%20eventlet,%20spawning.pdf * stacklessocket http://code.google.com/p/stacklessexamples/wiki/StacklessNetworking * stacklesswsgi http://code.google.com/p/stacklessexamples/wiki/StacklessWSGI * pyevent (uses callbacks instead of coroutines) * python-libevent (uses callbacks instead of coroutines) * Twisted (uses callbacks instead of coroutines) * Tornado (uses callbacks instead of coroutines) * circuits (uses callbacks instead of coroutines) Feature design of the new event loop (syncless.coio) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ FYI This has already been implemented. My conclusion was that in order to get the fastest coroutine-based, non-blocking, line-buffering-capable I/O library for Python, I should wrap something like libevent (including event registration and I/O buffering) and Stackless and the WSGI server in hand-optimized Pyrex, manually checking the .c file Pyrex generates for inefficiencies. syncless.coio will provide the following features: * drop-in non-blocking replacements so pure Python network libraries (such as urllib2, dnspython and BaseHTTPServer) can be used unmodified: ** syncless.coio.gethostbyname for socket.gethostbyname ** syncless.coio.gethostbyaddr for socket.gethostbyaddr ** syncless.coio.getaddrinfo for socket.getaddrinfo ** syncless.coio.pipe for os.pipe ** syncless.coio.socket_compat for socket.socket ** syncless.coio.socketfile_compat for socket._fileobject ** syncless.coio.realsocket_compat for socket._realsocket ** syncless.coio.socket_fromfd for socket.fromfd ** syncless.coio.socketpair for socket.socketpair ** syncless.coio.open for open (nonblock for pipes, sockets and char devices) ** syncless.coio.file_compat for file (nonblock for pipes, sockets and char devices) ** syncless.coio.sslsocket_compat for ssl.SSLSocket ** syncless.coio.sslsocketfile_compat for ssl._fileobject ** syncless.coio.select for select.select (initial implementation works only with 1 input filehandle) ** (None for socket._socket) ** (None for ssl._ssl) ** (None for select.poll) ** (None for select.epoll) ** (TODO(pts): Which of these is high speed?) * function to monkey-patch the replacements above * syncless.coio.fastsocket without timeout (only SO_RECVTIMEO) and makefile returning self * built-in high performance WSGI server * can use CherryPy's WSGI server as well Features removed from old Syncless: * credit system for fair scheduling * edge-triggered epoll operation * ability to use greenlet instead of Stackless (may be added later) External code used ~~~~~~~~~~~~~~~~~~ Code used for syncless.coio """"""""""""""""""""""""""" Copied files from the BSD-licensed pyevent (libevent Python bindings), revision 60 from svn checkout http://pyevent.googlecode.com/svn/trunk/ pyevent-read-only * Makefile * evdns.pxi * event.pyx * setup.py * test.py Code used for the Twisted reactor """"""""""""""""""""""""""""""""" libevent.reactor v0.3 (2008-11-22), available from http://launchpad.net/python-libevent (simple BSD license). Limitations ~~~~~~~~~~~ 1. The DNS looup functions (even the emulated syncless.coio.gethostbyname) read /etc/hosts (and /etc/resolve.conf) only at startup. 2. For hostname lookups, Linux libc6 NSS mechanisms (such as /etc/nsswitch.conf and etc/host.conf) are ignored: /etc/hosts is used first, then a DNS lookup is done. 3. The reverse DNS lookup functions fail on a host with multiple PTR records: $ host 202.92.65.220 ;; Truncated, retrying in TCP mode. 220.65.92.202.in-addr.arpa domain name pointer pop.cbdcorp.com.au. 220.65.92.202.in-addr.arpa domain name pointer pop.stanicharding.com.au. 220.65.92.202.in-addr.arpa domain name pointer webmail.stanicharding.com.au. 220.65.92.202.in-addr.arpa domain name pointer webmail.cbdcorp.com.au. 220.65.92.202.in-addr.arpa domain name pointer webmail.viewhotels.com.au. 220.65.92.202.in-addr.arpa domain name pointer pop.migrationlawforum.com.au. 220.65.92.202.in-addr.arpa domain name pointer sydmail01.powertel.net.au. 220.65.92.202.in-addr.arpa domain name pointer mail.viewhotels.com.au. 220.65.92.202.in-addr.arpa domain name pointer mail.migrationlawforum.com.au. 220.65.92.202.in-addr.arpa domain name pointer webmail.migrationlawforum.com.au. 220.65.92.202.in-addr.arpa domain name pointer mail.c2cextreme.net. 220.65.92.202.in-addr.arpa domain name pointer mail.cbdcorp.com.au. 220.65.92.202.in-addr.arpa domain name pointer mail.stanicharding.com.au $ stackless2.6 -c 'import syncless.coio; print syncless.coio.dns_resolve_reverse("202.92.65.220")' Traceback (most recent call last): File "<string>", line 1, in <module> File "evdns.pxi", line 344, in syncless.coio.dns_resolve_reverse File "evdns.pxi", line 278, in syncless.coio.dns_call syncless.coio.DnsLookupError: [Errno -65] reply truncated or ill-formed 4. syncless.coio.gethostbyname_ex is slower (does 2 DNS lookups) and only approximates the answer of socket.gethostbyname_ex, because evdns doesn't support CNAME lookups. 5. Socket timeouts are enforced on file objects created by nbsocket.makefile(). The normal Python behavior is to raise IOError(errno.EAGAIN, ...) no matter what the timeout is, even if the file object was created before setting the timeout. Thus it's pretty useless. sock1, sock2 = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM, 0) f = sock1.makefile('r') assert f.fileno() != sock1.fileno() sock1.settimeout(42) f.read(1) # raises IOError(errno.EAGAIN, ...) The Syncless behavior is to copy the timeout value from the nbsocket to the new nbfile when nbsocket.makefile() or nbsocket.makefile_samefd() is called and use the copy for each read(2) and write(2) operation. There is also the nbfile.settimeout() method which lets the user change the timeout associated with the nbfile (but not with the original nbsocket). Please see syncless.util.Timeout for the recommended timeout support: you can guard a block of code with a timeout, not only individial I/O operations. FAQ ~~~ Q1. I have created quite a few tasklets, put them into the runnables list. How do I make my program run until there are no tasklets? Currently my the process exists as soon as the main tasklet returns. A1. It's impossible to do that reliably if you are using Syncless with libev, or you have done any DNS lookups (e.g. with coio.gethostbyname) not from /etc/hosts. In those cases your program will hang indefinitely even after all the tasklets have finished running. To avoid that, and make your program exit, you have to call sys.exit() explicitly from one of your tasklets before it finishes. It is still a recommended practice to call `stackless.schedule_remove()' from the main tasklet if it has nothing useful to do, because it has created other tasklets to do all the work. Calling `stackless.schedule_remove()' at the end of the main tasklet is a faster equivalent of doing `while True: coio.sleep(1000)'. Please don't do `while True: stackless.run()' or `while stackless.run(): pass' or `while True: stackless.schedule()' from any of your tasklets, because those interfere with the main loop tasklet installed by Syncless, and they will make your process use 100% CPU even if there is no work to do, because stackless.run() and stackless.schedule() would activate the main loop tasklet, which also calls stackless.schedule() in its `while loop' in this case, so the previous tasklet gets activated, and this loops forever, burning CPU. Q2. What is the `.scheduled' value for tasklets blocked on Syncless I/O? A2. The value is False, since they are not on the runnables list, and they are not waiting in a channel. After I/O is available, and the Syncless event wakeup tasklet gets scheduled, the wakeup tasklet puts back the blocked tasklet to the runnables list, so blocked_tasklet.scheduled becomes True. Q3. Is it possible to cancel (interrupt) a coio.sleep()? A3. Yes. If you have a reference to the sleeping tasklet, just insert the tasklet back to the runnables list (or send an exception to it in a stackless.bomb, and then insert it back), and the sleep will be interrupted. For example, the following program finishes immediately (not needing 5 seconds). import stackless from syncless import coio sleep_done_channel = stackless.channel() def Sleeper(): coio.sleep(5) tasklet_obj = stackless.tasklet(Sleeper)() stackless.schedule() # Start Sleeper and start sleeping. # ... tasklet_obj.insert() # Interrupt the sleep. sleep_done_channel.receive() Same example with a bomb (not recommended): import stackless from syncless import coio sleep_done_channel = stackless.channel() def Sleeper(): try: coio.sleep(5) except AssertionError: sleep_done_channel.send(None) tasklet_obj = stackless.tasklet(Sleeper)() stackless.schedule() # Start Sleeper and start sleeping. # ... tasklet_obj.tempval = stackless.bomb(AssertionError) tasklet_obj.insert() # Interrupt the sleep. sleep_done_channel.receive() Please note that interrupting can be dangerous if you are not excatly sure that the other tasklet is doing a blocking Syncless I/O operation (such as coio.sleep). Please use synchronization mechanisms (such as channels) to make sure that a sleep is going on in the tasklet. Q4. Is it possible to cancel (interrupt) a blocking Syncless I/O operation? A4. Yes, by sending an exception wrapped to a stackless.bomb to the tasklet, and reinserting it to the runnables list. See A3 for more details. (Please note that coio.sleep is one of the blocking Syncless I/O operations.) Q5. Is it possible to insert a tasklet which is blocked on a Syncless I/O operation back to the runnables list? A5. Yes, you can use tasklet_obj.insert(), tasklet_obj.run() (or any other means to insert the tasklet to the runnables list). But you are not allowed to change the tempval (None by default) of a tasklet blocked on a Syncless I/O operation. If you do so, event_del() won't be called, and you'll get spurious events later (with exceptions caused by spurious tasklet_obj.insert()). Please note, however, that for non-sleep I/O operations the tasklet would retry the I/O operation (with the timeout period restarted), so it will most probably become blocked again (and thus removed from the runnables list). To prevent that, send an exception wrapped to a stackless.bomb to the tasklet by setting its tempval before reinserting it to the runnables list. See A3 and examples/demo_multi_read_timeout.py for more examples. Q6. How do I receive on a channel with a timeout? Call coio.receive_with_timeout. But please consider that cleaning up after a timeout is cumbersome. How do you tell the other tasklets to stop generating data on the channel? You'd better create dedicated tasklets for tasks you want to time out. Q7. Can I use my existing DNS, TCP, HTTP, HTTPS, FTP, urllib, urllib2, MySQL, memcached, Redis, Tokyo Tyrant etc. client with Syncless? A7. If your client software is written in pure Python, and it uses the standard Python `socket' module to connect to the server, then you only have to call from syncless import patch patch.patch_socket() patch.patch_ssl() # Only if SSL (e.g. HTTPS) support is needed. somewhere in your script initialization. This makes all DNS queries, TCP-based and other socket-based clients non-blocking. If your client software is non-pure Python (such as a Python extension written in C, Pyrex or Cython), then non-blocking functionality will most probably not work. Porting such client software is possible, but it's usually cumbersome and needs lots of work. However, if the client is written in Pyrex or Cython, you get non-blocking functionality by calling patch.patch_socket() as early as possible. For Syncless-compatible MySQL clients, see Q8. Q8. How do I connect to a MySQL database with Syncless? A8. The fol 8000 lowing options are recommended: * MySQL Connector/Python (myconnpy). This is a pure Python MySQL client, so it is expected to be slower than gevent-MySQL. Get it from https://launchpad.net/myconnpy , use its dbapi2-compatible interface, but call patch.patch_socket() or patch.patch_mysql_connector() first. See examples/demo_mysql_client.py for an example use. * gevent-MySQL (geventmysql). This is implemented as a C (Cython) extension, so it's faster than myconnpy. It is also less flexible (like cursors don't support the iterator interface, they have to be closed explicitly, and it's not possible to fetch data from multiple cursors concurrently). Get it from http://github.com/mthurlin/gevent-MySQL . Neither gevent, nor Cython is necessary for installation. If you don't have Cython, you have to patch setup.py, like this: $ git clone http://github.com/mthurlin/gevent-MySQL.git $ cd gevent-MySQL $ python -c 's="".join([s for s in open("setup.py") if "build_ext" not in s]).replace(".pyx", ".c" );open("setup.py","w").write(s)' $ stackless2.6 setup.py build # sudo stackless2.6 setup.py install See examples/demo_mysql_client_geventmysql.py for an example use. The following options are not recommended: * pymysql. This is a pure Python MySQL client, like myconnpy, but it's a bit faster (because of less overhead), but it seems to be less maintained, has more bugs and quirks (like trivial escaping bugs and a nonfunctional encoding (UTF-8) support), so it's not recommended to use it in production until it gets mature. Get it from http://code.google.com/p/pymysql/ . See examples/demo_mysql_client_pymysql.py for an example use. * Any library that uses libmysqlclient (the official MySQL C API), because such libraries are inherently blocking. Examples: MySQL for Python (mysql-python) and oursql. See also Q7. It has not been tested if it's possible to run multiple queries (and fetch their results concurrently) in parallel in the same connection. Opening multiple connections to the MySQL server, and using 1 cursor per connection always works. A more detailed analysis: * ``MySQL Connector/Python'' (myconnpy) is a pure Python MySQL client, and it works with patch.patch_socket() and patch.patch_mysql_connector(). The disadvantage is that it's slower than C extensions (which don't work). https://launchpad.net/myconnpy * libmysqlclient is inherently blocking, so it wouldn't work. See also http://mysql-python.blogspot.com/2010/02/asynchronous-programming-and-mysqldb.html about blocking calls in the C API. * The client ``MySQL for Python'' (http://sourceforge.net/projects/mysql-python/files/) is implemented as a C extension using the official C client (libmysqlclient), so it's inherently blocking, and thus it wouldn't work concurrently with multiple Syncless coroutines. * oursql uses libmysqlclient (so it wouldn't work) * Concurrence has a non-blocking MySQL client implemented in Pyrex/Cython, but it's quite hard to abstract away the networking part to make it work with Syncless. It has been ported to gevent (as gevent-MySQL), which was easy to monkey-patch for Syncless, see above. Disadvantage: neither Concurrence, nor its MySQL client is being maintained since 2009. * There is also the pure Python client ``pymysql'' (http://code.google.com/p/pymysql/), but it seems to be less maintained than myconnpy. It also seems a bit immature and not used in production because of trivial escaping bugs and a nonfunctional encoding (UTF-8) support. pymysql seems to be a bit faster (with less overhead) than myconnpy. Q9. How do I make my SSL connections (client and server) non-blocking? A9. Use the coio.nbsslsocket class as a drop-in non-blocking replacement for ssl.SSLSocket. If you don't want to modify your source code, call this somewhere the beginning in your script: from syncless import patch patch.patch_socket() # DNS lookups and socket.socket() non-blocking. patch.patch_ssl() # Make ssl.SSLSocket non-blocking. This makes both the connecting handshake operation and the data transfers (reads and writes) non-blocking. Patching affects only Python code using the standard `ssl' Python module. Patching doesn't affect the non-standard `openssl' Python module or C extensions using libssl directly. Maybe in the future Syncless will have a patch.patch_openssl(), but C extensions will never be patched, so please make sure that you create all your SSL sockets in Python code. Q10. Does SQLite3 work with Syncless? A10. The standard `import sqlite3' works (just like any Python module with C extensions), but it blocks: so if a tasklet is busy running a long SQLite3 query, no other tasklets get scheduled at that time. This is unacceptable in most latency-sensitive situations. While it is possible to read (SELECT) from an SQLite3 database in parallel from multiple processes or threads (by connecting to it once per thread), this kind of parallelism is impossible to implement with coroutines (thus with Syncless). It is also impossible to make the database file read or write locking fcntl operation non-blocking, e.g. if tasklet A is holding a write lock (for a long-running transaction with thousands of INSERT), and tasklet B wants to acquire a read lock (for a SELECT), then it's impossible to schedule tasklet C (which doesn't need access to the SQLite3 database) while tasklet B is waiting for its lock. Threads or processing would be needed for this kind of parallelism. If you really must use SQLite3, then you may consider setting up an sqlite3.connection.set_progress_handler and call stackless.schedule() from there to improve the latency. But this can be become very tricky, because according to http://www.sqlite.org/c3ref/progress_handler.html you have to ensure that you don't use the database connection while a progress handler is running. See examples/demo_sqlite_progress_handler.py for some experiments. For real concurrency, you should create one thread per SQLite3 connection, and communicate with those threads and the tasklets in your main thread. This is complicated, tiresome and tricky to implement. Q11. Is it possible to scheduler background work with Syncless for GTK, TCL/Tk or Qt applications? A11. Not at the moment, since the main event loop of these GUI frameworks doesn't support coroutines. It would be an interesting and complicated project to make Syncless support these main event loops instead of libevent. Q12. Is it possible to use stacklessocket, asyncore, pyevent, python-libevent, Tornado, Twisted, circuits or another event-driven network library with Syncless? A12. Tornado (with its own main loop) is already supported with patch.patch_tornado(). See also examples/demo_tornado.py . Please note that the speed would be slower than native Tornado because of the select(2) emulation with coio.select. asyncore (with its own main loop) is already supported with patch.patch_asyncore(). See also examples/demo_asyncore.py . But, if possible, please don't use asyncore in production (but write your application using a modern framework like Syncless, Tornado, Twisted or Concurrence), because asyncore lacks core functionality like proper timed event handling, some developers don't like its error handling design, and it's slow. circuits (with its own main loop) is already supported with patch.patch_circuits(). See also examples/demo_circuits.py . Please note that the patch is a bit slow (it emualtes select.select()), but that seemed to be the most stable way to get it working. Twisted (with a custom reactor) is already supported with syncless.reactor. See also examples/demo_twisted.py. Adding pyevent or python-libevent may become complicated because of the multiple libevent-based I/O loops. (Most importantly: the Syncless main loop adds EVLOOP_NONBLOCK if more tasklets are in the runnables list (stackless.runcount > 1). Thus EVLOOP_NONBLOCK would always be specified with multiple main loops, which would result in busy waiting. Adding stacklessocket wouldn't give us too much benefit, because stacklessocket is not for production use. Q13. Is it possible to use gevent, Eventlet, Concurrence or another coroutine-based event communication framwork with Syncless? A13. Concurrence works with patch.patch_concurrence(). See also examples/demo_concurrence.py . The speed should be the same as with regular Concurrence, there is no Syncless-specific overhead. Please note that native Syncless is faster than native Concurrence, because Syncless provides sockets and buffered files as a C (Pyrex) extension. Eventlet works with patch.patch_eventlet(). See also examples/demo_eventlet.py . The speed should be the same as with regular Eventlet (with any of its faster hubs like eventlet.hubs.pyevent and eventlet.hubs.epoll), there is no Syncless-specific overhead. Please note that native Syncless is faster than native Eventlet, not only because Syncless provides sockets and buffered files as a C (Pyrex) extension, but also because the overhead of waiting for I/O is much smaller in Syncless. Please expect low performance if you are using Stackless Python, because Eventlet uses greenlet, so it has to be emulated with the syncless.best_greenlet emulatior, which is slow. gevent works with patch.patch_gevent() with some limitations, see the docstring of patch_gevent() for more details. See also examples/demo_gevent.py . Please expect low performance, because gevent uses greenlet, so either greenlet or Stackless has to be emulated (with syncless.best_greenlet or syncless.best_syncless) so that Syncless and gevent can work in the same process. The emulation is at least 20% slower, but it can be much slower. (Speed measurements needed.) Please note that Syncless, gevent, Eventlet and Concurrence use libevent (with Syncless being able to use libev and minievent, and Eventlet being able to support many other methods, including pure-Python implementations in eventlet.hubs.*), so techincally it's possible to unify the event loops. This has been done with Syncless + gevent (where the Syncless main loop processes both Syncless and gevent notifications). For Concurrence and Eventlet this was not needed, because the event notification abstractions of Concurrence and Eventlet were so clean that they could be efficiently emulated by Syncless (no matter libev or libevent). Q14. Is it possible to use select(2) (select.select) with Syncless? A14. Yes, either as coio.select, or as select.select after patch.patch_select(): from coio import patch patch.patch_select() ... import select print select.select(...) Limitation: Exceptional filehandles (non-empty xlist) are not supported. Please note that select(2) is inherently slow compared to libevent or the combination of Syncless and tasklets. That's because libevent uses faster polling mechanisms such as Linux epoll or BSD kqueue. If your speed-critical program uses select(2), please consider redesigning it so it would use Sycnless non-blocking communication classes and tasklets. Q15. Is it OK to close() an nbfile or an nbsocket while other tasklets are reading from it? A15. No. You must ensure that there are no other tasklets using the filehandle in any way by the time you close(). Otherwise you may get a segmentation fault. (Please note that in some systems and some libevent drivers it might be safe -- try it for yourself.) Q16. Can multiple tasklets wait on the same file descriptor at the same time? Will all of them get notified? A16. Yes and yes, this is fully supported by libev-3.9, see the testTwoReaders method in test/nbfile_test.py. Before syncless-0.06 this was not possible, because libevent-1.4.13 doesn't support this. If you attempted it with libevent, only the last tasklet would get notified and the others would be discarded (never woken up). Q17. Can I start subprocesses and communicate with them in a non-blocking way with Syncless? A17. Yes, by monkey-patching any of these methds: subprocess, popen2, os.popen. There is no support yet for more sophisticated mechanisms (such as the multiprocessing module). There is an important limitation so far: waiting for a subprocess to exit (e.g. with os.wait and os.waitpid without WNOHANG) is a blocking operation: all tasklets in the process would get blocked. Code examples: # See longer example in examples/demo_subprocess.py. import subprocess from syncless import patch patch.patch_subprocess() # or, worse, patch.patch_os() ... subprocess.popen3(...) # or any other method # See longer example in examples/demo_popen2.py . import popen2 from syncless import patch patch.patch_popen2() # or patch.patch_os() ... popen2.popen2(...) # or any other method # See longer example in examples/demo_popen.py . import os from syncless import patch patch.patch_os() # or os.popen = coio.popen ... os.popen(...) # or any other method Q18. Can I link Syncless aganst libevent instead of libev? A18. Please try Syncless with libevent2 (libevent-2.0.4 or later) by compiling it with $ SYNCLESS_USE_LIBEVENT2=true stackless2.6 ./setup.py build $ sudo stackless2.6 ./setup.py install Also the primary focus is to make Syncless work with libev, it should also work with libevent2. Please report any libevent2 issues to the author of Syncless. Currently Syncless works only partially with libevent1 (i.e. libevent-1.4 and earlier), because libevent1 has the inherent limitation that it silently starts behaving unpredictably and unreliably if multiple events are registered on the same filehandle. You can try nevertheless: $ SYNCLESS_USE_LIBEVENT1=true stackless2.6 ./setup.py build $ sudo stackless2.6 ./setup.py install Q19. Is Syncless faster than Concurrence, gevent, Eventlet, asyncore, pyevent and python-libevent? A19. It was designed to be faster, and it should be faster. But benchmarks haven't been run recently, the files in the benchmark directory are obsolete (i.e. they have been run in a faster but buggy old version of Syncless). The only exception from the rule that Syncless is the fastest may be Concurrence, because Syncless uses hard switching (i.e. C stack copying) of tasklets, which is a bit slower than the soft switching used by Concurrence. Q20. Should I use stackless.schedule() or stackless.schedule(None)? If you are not interested in stackless.current.tempval, then use stackless.schedule(None) (and stackless.schedule_remove(None)) rather than without None, to save memory. That's because stackless.schedule() is equivalent to stackless.schedule(stackless.current), which sets stackless.current.tempval = stackless.current, creating a circular reference, which can lead to memory leaks if garbage collection is disabled. Example (this raises MemoryError, reaching 200 MB): import gc import resource import stackless gc.disable() resource.setrlimit(resource.RLIMIT_AS, (200 << 20, 200 << 20)) # 200 MB while True: t = stackless.tasklet(stackless.schedule)() assert t.alive stackless.schedule() assert t.alive assert t is t.tempval # Circular reference: t --> t.tempval --> t t.remove() Q21. Can I register my signal handlers? A21. Yes, use coio.signal. Example: import signal from syncless import coio def MyHandler(signum): print 'signal %d received' % signum h = coio.signal(signal.SIGINT, MyHandler) coio.signal(signal.SIGHUP, MyHandler) ... coio.signal(signal.SIGHUP, None) # Delete it. h.delete() # Delete SIGINT. All signal handlers are persistent. Uncaught exceptions raised in signal handlers are reported (without a traceback), and then they get ignored. Q22. Does Syncles provide a way to parse the bytes read from a socket? A22. You should call nbsocket.makefile('r') to create nbfile, which has a read buffer. The basic idea for parsing is that nbfile.read_upto and nbfile.read_more can be used to read more bytes from the file descriptor to the read buffer, the methods nbfile.get_string, nbfile.get_read_buffer, nbfile.find and nbfile.rfind can be used to find out if enough data is available, and whether it is parseable, and finally nbfile.discard can be used the remove the parsed message from the beginning of the read buffer. nbfile.read_buffer_len contains the current byte size of the read buffer. You can also parse with regular expressions, e.g. re.search(r'...', nbfile_obj.get_read_buffer()), but you should make your regexp aware that the read buffer may contain only a prefix of the full message you want to parse. In that case, you should call nbfile.read_more to read more bytes, and then retry the re.search. Q23. Does Syncless work on Microsoft Windows? A23. As of now, Windows support is not implemented. Since that would be a considerably large piece of work, and it has low priority for the Syncless developers, Windows support will probably never be implemented. Currently you need a recent Unix system to run Syncless. It should work on Linux (works on 2.6), FreeBSD, NetBSD, OpenBSD, Mac OS X (works on 10.5) and Solaris. If you have a desktop or server Unix system on which Syncless doesn't compile or work properly, please let us know, so we can fix it. Q24. What are the dependencies of Syncless? A24. To run Syncless, you need a recent Unix system with Python 2.5 or 2.6. You also need Stackless Python or the greenlet (>= 0.3.1) Python package. Syncless is distributed in source form, so you should compile it before you can run it. To compile Syncless, you need the Python headers (the python-dev package) and GCC on your system. There are many optional dependencies, including libevent1, libevent2, libev and libevhdns. Syncless detects them at compilation time. Without these, Syncless is fully functional, but it's less efficient, especially at higher loads (>10 concurrent TCP connections). See the Installation section in the README for more details. Q25. Does Syncless have an interactive Python console (python -i)? A25. Yes, it's in the syncless.console module. You can start it using: $ python -m syncless.console Please don't use `python -i' directly, it doesn't cooperate well with tasklets, so if you create a tasklet there, it most probably won't ever get scheduled, because the interactive console wants to run the whole time in a blocking way. Use syncless.console instead, it solves this problem. Run the ``help'' command to find out some examples. See http://code.google.com/p/syncless/wiki/Console for more details. syncless.console is like the regular interactive Python interpreter except that it has some useful global variables preloaded (such as help, Syncless and ticker), and it supports running coroutines (tasklets) in the background while the user can issue Python commands. It's an easy-to-use environment to learn Syncless and to experiment with tasklets. The interactive console displays a prompt. At this point all tasklets are running and scheduled until you press the first key to type an interactive command. While you are typing the command (after the 1st key), other tasklets are suspended. They remain suspended until you press <Enter> (to finish the command or to start a multiline command). Please note that if an uncaught exception is raised in tasklet, the whole process exits. You can prevent that in interactive sessions by creating your tasklets with `wrap_tasklet(Function)' instead of `stackless.tasklet(Function)'. If you do so, the exception will be printed, but the interactive console resumes afterwards. IPython is not supported in syncless.console. Q26. Can I use other non-blocking network libraries in the same process which runs Syncless? A26. Syncless supports the following libraries out-of-the-box with monkey patching: gevent, Eventlet, Concurrence, Twisted, Tornado, circuits and asyncore. See Q12 and Q13 for more details how to enable them. Please note that there is no direct synchronization support between Syncless and the other library, i.e. there is no way for a tasklet managed by a library to notify a tasklet managed by another library that computation results are available. If you want to run a web framework supporting WSGI, run its WSGI application function/class directly with syncless.wsgi instead. If your pure Python network library uses select.select() for event notification, then see Q14. Otherwise, the library most probably won't work with Syncless in the same process. Most probably either Syncless I/O will make progress, and the other library's I/O will be stalled, or the other way round. Q27. Syncless is compatible with lots of other libraries. Is it possible to remove the compatibility features so my program which doesn't use them becomes faster? A27. The compatibility features are separated properly in the Syncless code base, and they are disabled with no performance impact if not used. By removing those features all you'd gain is a few milliseconds of startup time because the syncless.patch and syncless.coio modules would become smaller. Q28. Does Syncless has a thread pool for wrapping blocking operations? A28. Yes, see the coio.thread_pool class and the corresponding examples in examples/demo_thread_pool*.py. In your production code, please try to avoid a thread pool, and revert to it if there is no other feasible solution, because the thread pool needs more memory and CPU than coroutines (tasklets), so you might lose most performance advantages of Syncless (over threads) if you use the thread pool. For example, please use a non-blocking MySQL client (see Q8 in the FAQ) instead of calling the methods of libmysqlclient in a thread pool. Please note that coio.thread_pool has not been probed for performance yet. Q29. How do I connect to a PostgreSQL database with Syncless? A29. Don't use Pyscopg or PyGreSQL, because they use libpq, the standard PostgreSQL C client library, which is inherently blocking, i.e. it doesn't support non-blocking operation compatible with Syncless. So with these client libraries, all your tasklets would be blocked while one tasklet is waiting for the result of a query. Don't use py-postgresql, because it needs Python 3.x (and Syncless supports Python 2.5 and 2.6). Don't use PyPo, it's not available anymore, and the last release happened in 2003. Use any of these pure Python PostgreSQL clients: * pg8000: http://pybrary.net/pg8000/ * pg_proboscis: http://pypi.python.org/pypi/pg_proboscis/1.0.5 * py-bpgsql: http://code.google.com/p/py-bpgsql no release since June 2009 !! You will need monkey-patching with these clients (e.g. patch.patch_socket()). !! TODO(pts): Write this answer properly. !! TODO(pts): Which is the fastest? Q30. Does Syncless support SSL client and server connections? A30. Yes, with Python 2.6 (not with Python 2.5) with the builtin `ssl' module (not pyOpenSSL). The functionality is only available if the `ssl' module is also available (detected and compiled at Python compile time). See the files examples/demo_https_client.py, examples/demo_ssl_client.py, examples/demo_ssl_server.py, examples/demo_ssl_client_sslwrap_simple.py, examples/demo_ssl_client_simple.py . The syncless.coio.nbsslsocket class is a drop-in replacement for ssl.SSLSocket (please see the docstrings of methods `makefile' and `makefile_samefd' for the most important incompatibilities). The syncless.coio.ssl_wrap_socket function is a drop-in replacement for ssl.wrap_socket. No non-blocking equivalent is provided for the constructor functions ssl.sslwrap_simple and socket.ssl. Q25. Does Syncless have a remote interactive Python console (backdoor)? A25. Yes, it's in the syncless.remote_console module. You can start the server using: $ python -m syncless.remote_console 5454 Example client connection with line editing (readline): $ python -m syncless.backdoor_client 127.0.0.1 5454 Example client connection without line editing: $ telnet 127.0.0.1 5454 See more in the docstring of the syncless.remote_console module. The primary purpose of the RemoteConsole is to enable simple debugging of long-running servers based on Syncless. Please note that the Python debugger is not supported (may or may not work, most probably it won't work), the typical use case of the RemoteConsole for debugging is that the user evaluates Python statements and expression to understand and manipulate the state (memory) of a long-running server. See the docstring of the syncless.remote_console module for instructions for enabling the RemoteConsole for your server application. Q26. Does Syncless support WebSocket connections? A26. There is no-built in support WebSocket clients, but you can use e.g. patch_socket() and the pywebsocket reference implementation. The syncless.wsgi module has support built-in for WebSocket servers. See examples/demo_websocket_server.py for more information. Please note that you need a quite recent web browser as a WebSocket client to try this demo with. Read more about supported browsers in the module docstring of the .py file. See http://stackoverflow.com/questions/1253683/websocket-for-html5 for useful links and information about WebSocket. Q27. Does Syncless have unit tests? A27. Yes, they are test/*_test.py . You can run all of them them and get a summary with $ stackless2.6 ./setup.py build test Please note that test coverage is far smaller than 100%: some methods and some classes are not covered at all. Q28. Syncless is too slow! How do I make my Syncless-based application run faster? A28. Syncless is designed to be very fast compared to Python's built-in file and socket.socket, and also to other coroutine-based socket implementations. If your program is too slow, first make sure that you use Syncless with libev or libevent2. The backend libevent1 is a bit slower than libevent2, and the backend minievent is very slow (because it's optimized for portability, and not for speed). Also make sure that you are using Syncless with Stackless Python (instead of greenlet), because with greenlet Syncless emulates Stackless Python very slowly (see stackless/greenstackless.py). If your program uses Syncless with libev (or libevent2) and Stackess Python, and it's still too slow, then please check if it gets faster by using Python's built-in file and socket.socket classes (or an alternative like gevent) instead. If Syncless is slower than the others, then please report this as a bug at http://code.google.com/p/syncless/issues/entry . The performance of Syncless is a high priority for us, and we are eager to fix performance bugs. If you know of benchmark (no matter if Syncless fast or slow in it), please let the author of Syncless know. Memory leak in greenlet ~~~~~~~~~~~~~~~~~~~~~~~ PyPy’s greenlets do not suffer from the cyclic GC limitation that the CPython greenlets have: greenlets referencing each other via local variables tend to leak on top of CPython (where it is mostly impossible to do the right thing). It works correctly on top of PyPy. Links ~~~~~ * doc: about greenlet: http://codespeak.net/svn/greenlet/trunk/doc/greenlet.txt * doc: related: eventlet vs gevent: http://blog.gevent.org/2010/02/27/why-gevent/ * doc: http://www.disinterest.org/resource/stackless/2.6.4-docs-html/library/stackless/channels.html * doc: http://wiki.netbsd.se/kqueue_tutorial * doc: http://stackoverflow.com/questions/554805/stackless-python-network-performance-degrading-over-time * doc: speed benchmark: http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ * doc: gevent and gtk: http://groups.google.com/group/gevent/browse_thread/thread/36f8dd594b5e2c06 Asynchronous DNS for Python: * twisted.names.client from http://twistedmatrix.com * dnspython: http://glyphy.com/asynchronous-dns-queries-python-2008-02-09 http://www.dnspython.org/ * adns-python: http://code.google.com/p/adns/python * http://michael.susens-schurter.com/blog/2007/09/18/a-lesson-on-python-dns-and-threads/comment-page-1/ Info: In interactive stackless, repeated invocations of stackless.current may return different objects. Info: LIBEV_FLAGS=1 use select(); LIBEV_FLAGS=2 use poll() Info: gevent.backdoor doesn't support line editing with libreadline python2.5 -m gevent.backdoor 1234 * picoev: libevent/libev replacement: http://developer.cybozu.co.jp/kazuho/2009/08/picoev- 6B07 a-tiny-e.html * WSGI server meinheld http://pypi.python.org/pypi/meinheld/0.3.1 * fapws3 (very fast WSGI server in C and libev): http://github.com/william-os4y/fapws3 * nice Python extension writing in C: http://github.com/william-os4y/fapws3/blob/master/fapws/_evwsgi.c Release procedure ~~~~~~~~~~~~~~~~~ This section describes how to create a new source distribution release of Syncless. The intended audience is Syncless developers. 1. Run $ make -C coio_src coio.c 2. Make sure it works in your SVN local directory (e.g. run the tests). 3. Bump the version number in syncless/version.py . 4. Commit your changes, e.g. $ svn ci -m 'bumped version number, creating release X.YZ' 5. Run $ stackless2.6 setup.py sdist upload register Planned features ~~~~~~~~~~~~~~~~ * TODO(pts): Add Socket.IO support (https://github.com/MrJoes/tornadio and gevent-socketio) * TODO(pts): Wrap / monkey-patch the multiprocessing module (and its C code in _multiprocessing). The _multiprocessing module implemented in C doesn't seem to support non-blocking operation. * TODO(pts): Report libevent bug that evdns events are not EVLIST_INTERNAL. * TODO(pts): Document the side effect of import syncless.coio on Ctrl-<C>. * TODO(pts): HTTP client library (making urllib non-blocking?) * TODO(pts): support webob as a web framework * TODO(pts): productionization * TODO(pts): setsockopt TCP_DEFER_ACCEPT * TODO(pts): setsockopt SO_LINGER non-immediate close() for writing * TODO(pts): use SO_RCVTIMEO and SO_SNDTIMEO for timeout * TODO(pts): is it faster in Cython than in Pyrex? (it's not smaller though) * TODO(pts): Strip the coio.so files upon installation? It seems to be still importable. Some Python installations autostrip. Why not ours? * TODO(pts): Fix the AttributeError in socket.socket.close(). * TODO(pts): channel.send_exception() with a traceback. (Wait for receiver?) * TODO(pts): Handle starving (when one worker is very busy, even Ctrl-<C> is delayed) This is hard to achieve (but the main tasklet can be given priority on Ctrl-<C>, so it would be the next to be scheduled). # TODO(pts): Evaluate how fast stack copying (Stackless hard switching) is. * TODO(pts): Monkey-patch signal.signal(...). * TODO(pts): Automatic install script for Linux. * TODO(pts): Add SSL support for Python2.5 (it already has socket._ssl.ssl). * TODO(pts): Document Pyrex vs Cython * !! SUXX: why can't we connect() with O_NONBLOCK at a very high rate (just as with normal sockets?) * TODO(pts): Specify TCP socket timeout. Verify it. * TODO(pts): Specify total HTTP write timeout. * TODO(pts): Move the main loop to another tasklet (?) so async operations can work even at initialization. * TODO(pts): Implement an async DNS resolver HTTP interface. (This will demonstrate asynchronous socket creation.) * TODO(pts): Document that scheduling is not fair if there are multiple readers on the same fd. * TODO(pts): Close connection on 413 Request Entity Too Large. * TODO(pts): Prove that there is no memory leak over a long running time. * TODO(pts): Use socket.recv_into() for buffering. * TODO(pts): Handle signals (at least KeyboardInterrupt). * TODO(pts): Handle errno.EPIPE. * TODO(pts): Handle errno.EINTR. (Do we need this in Python?) * TODO(pts): /infinite 100K buffer on localhost is much faster than 10K. * TODO(pts): Consider alternative implementation with eventlet. * TODO(pts): Implement an SSL-capable HTTP proxy as a reference. * TODO(pts): doc: signal.alarm doesn't work (the SIGALRM will get ignored?) * TODO(pts): Make os.waitpid non-blocking by installing a SIGCHLD handler. * TODO(pts): Fix very small float sleep value for libev. * TODO(pts): Add more protocol parsing examples. * TODO(pts): Add proper doucmentation as .rst. * TODO(pts): Get ideas from http://code.google.com/p/gevent/wiki/ProjectsUsingGevent * TODO(pts): Support HTTP pipelining in the WSGI server. HTTP/1.1 conforming servers are required to support pipelining. This does not mean that servers are required to pipeline responses, but that they are required not to fail if a client chooses to pipeline requests. * TODO(pts): Compare speed with http://github.com/william-os4y/fapws3 only WSGI; WSGI module implemented in C; uses libev; very fast?; reads the whole POST request to memory (and makes it available with cStringIO) __EOF__
About
Automatically exported from code.google.com/p/syncless
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published