Running a Django Project with PyPy
A migration account of running Django on PyPy, including dependency compatibility fixes and observed performance gains over CPython 2.7.
I had recently been experimenting with socket programming and wrote an echo server. When I tried it with pypy, it handled several times as much traffic as python2.7, which was a startling improvement. Network programs running locally encounter almost no congestion, so this was effectively a CPU test. My guess is that PyPy reduced thread and loop overhead through some optimization.
With some spare time, I tried running a local copy of one of our company’s projects with pypy. There were inevitably a few obstacles, but working from an existing project was still much easier than starting a Django project on PyPy from scratch. The official claim was roughly seven times faster—well, supposedly.
The first problem was packages. Packages installed in the usual way lived under the Python 2 paths, so I created a .pth file in PyPy’s site-packages directory to make those packages available. First, enter PyPy’s package directory:
/usr/local/Cellar/pypy/4.0.1/libexec/site-packages
Create a file named external.pth containing these two lines:
/Library/Python/2.7/site-packages /usr/local/lib/python2.7/site-packages
This made some pure Python packages usable, but packages containing Clang-compiled extensions still failed. MySQLdb (mysql-python) was one example. Search results varied widely, while the official compatibility page said that version 1.2.4c1 and later should work. The project still failed with an import _mysql error, claiming that the package did not exist. Inspection showed only an _mysql.so file.
For comparison, when the project ran under Python 2, PyCharm generated an _mysql.py cache from _mysql.so, after which MySQLdb worked. That generation failed under pypy. PyCharm keeps a .blacklist file under Binary Skeletons that records .so files it could not parse.
I copied the _mysql.py generated by PyCharm into the Python 2 MySQLdb package so that pypy could use it directly. That removed this particular error, but other Clang-based packages still failed, so copying generated files was not a viable solution. Further research showed that packages containing C code and .so files generally cannot simply be reused. They normally need to expose a CFFI interface for PyPy, although some packages work directly. Plain C extensions without such an interface will not work. Extensions written with ctypes, on the other hand, are compatible with both CPython and PyPy.
It is therefore better to install packages for PyPy using its own tools, much as you would for CPython:
pip_pypyeasy_install_pypypypy setup.py intall
Install the packages one by one this way. Some still have issues. For example, Python 2 installed Crypto, while PyPy installed crypto, even though imports inside the package still referred to Crypto with an uppercase C.
After installing the required packages, I tested a relatively complex page. Its load time fell from nearly 1.9 seconds to under 1 second, a very noticeable improvement.
I later found another issue. Placing PyCharm’s generated _mysql.py under the Python 2 MySQLdb package allowed PyPy to run, but subsequently caused Python 2 to fail. The file has to be removed before switching back.