The Python module that has saved me the most time is, without a doubt, the Requests module by Kenneth Reitz. This guy never fails to produces awesomeness.

The tagline is “HTTP for Humans”, and the whole thing is very pythonic. It was missing, however, an equally pythonic way to deal with caching. Fortunately, Requests-Cache fills the void. It supports several different caching backends, as well as an extensible interface for adding new ones.

SQLite is the default, and probably works well in most cases. But it fails completely when deploying to Heroku. That’s because SQLite requires a permanent writable file system. It ultimately needs access to the POSIX fopen() and fwrite() API calls to a particular file. Heroku does not provide a permanent writable file system. It doesn’t even allow the SQLite3 module to be installed:

1

ImportError: No module named _sqlite3

Even though Requests-Cache supports in memory caching, I had to remove a couple parts of the module to get it to even import on Heroku. I removed the sqlite.py backend file requests_cache/backends/sqlite.py and delete the couple sqlite lines from requests_cache/backends/__init__.py. After that, it imported fine and is easy to set up in memory caches, like this:

1

requests_cache.configure('cache_name', 'memory')

Here’s my fork of the repo with those changes.

If you know a better solution, or have another way of caching HTTP requests easily with Python, let me know.