iw.eggproxy: a proxy for pypi

Today we have released iw.eggproxy. It's a module for apache mod_python. Its purpose is to serve as a pypi proxy.

The first motivation to make it was because we have had to work in a private network with a very very slow internet access: for example updating a plone buildout could take more than one hour (just checking eggs freshness) when it should be no more than a few minutes. This condition also prevent to run some kind of rsync against pypi. So the obvious solution was to proxy the eggs we need, on-demand.

The module is installed as a handler on a Location. When accessing this location, eggproxy will serve an index similar to pypi simple view.

This is done like this:

• we already have the information in index.html: just let apache serve the file
• or, we use setuptools to fetch index information, build index.html, and let apache serve the file
  

This allows eggproxy to serve all available eggs from pypi, without actually having to download the whole pypi + content.

Then easy_install can see a package is available on our server, and tries to fetch information on available eggs:

• the subdirectory and its index.html already exists: just let apache serve the file
• or, we use setuptools again to get package information, make the subdirectory (package name) and build index.html
  

Finally, when trying to fetch an egg we do the same:

• the file is already present and is served by apache
• or we use setuptools to get the file on the server
  

iw.eggproxy also provides an update script: "eggproxy_update". This script refreshes the main index and all proxied eggs, if their index.html is older than the interval specified in the configuration file (24h by default).

We have installed it here: http://release.ingeniweb.com/pypi.python.org-mirror

Known bugs: some packages on pypi don't have eggs, then eggproxy does not respond. This is the case with "reportlab" for example.

Enhancements:

• indexes aggregation. At ingeniweb we plan to install it on local server and agregate pypi and some private eggs indexes.
• Standalone/pluggable server. Currently we are bound to apache + mod_python, which may not suit to anyone.