Looking for a high performance, powerfull process manager for a Python project I’m working on, I stumbled on Circus on this excellent benchmark blog post. After running my own benchmark tests, I agree that the Circus + Chaussette + Meinheld stack is the way to go. High concurrency, fast response time, and socket support are the features that pulled me in. I’m switching off of Supervisord, because: a) Circus integrates directly with ZeroMQ, and b) Gunicorn + Supervisord requires two levels of process management: Supervisor controls gunicorn, and gunicorn in turn watching it’s own worker processes. Circus keeps everything on the same level.

circus.readthedocs.org has excellent documentation for getting started with Circus. I did run in to a couple caveats though:

Circus doesn’t start on boot

I started testing Circus in the command line by running circusd circus.ini. I quickly switched to running it as a service in Upstart, using this etc/init/circus.conf file:

1
2

start on filesystem and net-device-up IFACE=lo
exec /usr/local/bin/circusd /etc/circus.ini

This script just waits for the file system and networking to become available, then it runs circusd with my config file in /etc/circus.ini. Easy enough.

Getting Circus workers to work with virtual environments

Although Circus workers have awesome properties like env, copy_env and copy_path (which all work great when running from a local folder), this falls apart when starting the daemon from Upstart. I looked at my $PATH variable in an active environment, and copied it into the worker config:

1

env = PATH=/path/to/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,VIRTUAL_ENV=/path/to/venv

Circus Web Console behind Nginx with Sockets

Circus has a sweet web console to manage processes and workers. By default, it runs on port 8080, and uses websockets to push stats on CPU, memory and socket reads for each running process. The web console should never be publicly available, it allows arbitrary commands to be executed on the server. The preferred way to password protect the console is to use Nginx like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

server {
    listen  8001;
  location ~/media/*(.jpg|.css|.js)$ {
      alias /usr/local/lib/python2.7/dist-packages/circus/web/;
  }

  location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_pass http://127.0.0.1:8080;
      auth_basic            "Restricted";
      auth_basic_user_file  /etc/nginx/htpasswd;
  }
}

I have the web console process listening on 8080: it’s serving both the website and the socket connection. Notice the issue? Nginx doesn’t support websockets! So I’m running Nginx on port 8001, and the web console processs on 8080. And this is where Varnish comes in: Varnish is a caching proxy, but I’ll just use it to multiplex port 8002 to two seperate backends. If the connection is a websocket, route it directly to 8080. If it’s the website, switch the backend to Nginx on port 8001:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

backend default {
    .host = "127.0.0.1";
    .port = "8001";
}

backend socket {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 1s;
    .first_byte_timeout = 2s;
    .between_bytes_timeout = 60s;
}

sub vcl_pipe {
     if (req.http.upgrade) {
         set bereq.http.upgrade = req.http.upgrade;
     }
}

sub vcl_recv {
    if (req.http.Upgrade ~ "(?i)websocket") {
        set req.backend = socket;
      return (pipe);
    }
}