use local cache of pypi

Description

When the http://builder.openwsn.org/ server runs a job, it builds the environment needed for that job. That environment usually contains Python packages such as SCons, pyserial, or pytest. With the current setup, these packages are downloaded from https://pypi.python.org each time. Besides creating a lot of networking traffic, it can fail if pypi is not available for some reason.

The goal of this improvement is to install a local mirror of pypi, containing a handful of packages, and instructing the easy_intall and pip utilities to first look in that local cache, then in the global pypi if the package is not found.

Several cache systems exist, but the simplest is probably https://pypi.python.org/pypi/pypiserver. It is a small Python program which listens on some port, and hands over packages it has in a single folder. Adding a package to the cache simply means adding the corresponding tar.gz file in the packages folder.

Both easy_install and pip can be instructed to first look in a cache at http://localhost:8080 before going to the https://pypi.python.org/pypi/pypiserver:

The following steps can be used when resolving this issue:

  • download and install pypiserver on builder.openwsn.org

  • make sure this server is started when the machine reboots. The simplest it probably to add a link to a .bat file in the startup folder in the Start Menu.

  • add the packages listed below to the pypiserver installation

  • go through the jobs currently run by Jenkins, and add the option to have easy_install and pip first use the cache.

The package which are currently used by all jobs are:

Environment

None

Gliffy Diagrams

Activity

Show:

Thomas Watteyne September 11, 2013 at 2:00 AM

The server is in place, the packages are uploaded, and the builds are configured to use the server.

Xavier Vilajosana August 27, 2013 at 12:47 AM

added packages.. need only to update jenkins jobs to use cache.

Xavier Vilajosana August 26, 2013 at 8:33 PM

installed pypiserver and created packages folder..
need to upload packages on it..
WIP:.

Fixed

Details

Assignee

Reporter

Priority

More fields

Created August 24, 2013 at 10:01 PM
Updated September 11, 2013 at 2:00 AM
Resolved September 11, 2013 at 2:00 AM