WebSockets and ASGI¶
WebSockets allows CATMAID instances to push data to the client even if the client didn’t request it. This can be useful to inform the client about updates on the server. An example is the notification about new messages, created e.g. by cropping a sub-volume out of the image data.
Without WebSockets, the CATMAID web front-end will ask the server for updates every minute. Avoiding these requests especially with many clients lowers the resource requirements of CATMAID. ASGI is the protocol used for this bi-directional communication and a separate ASGI server (the asynchronous sibling of WSGI). A common choice for this server is Daphne. It is used below for an example setup.
Note that manage.py runserver
supports ASGI out of the box. This however is
not meant to be used for production setups.
Setting up an ASGI server¶
Daphne is already installed as part of CATMAID’s dependencies. To run it use the following command:
daphne -b 127.0.0.1 -p 8001 mysite.asgi:channel_layer
This will start a new Daphne server, listening on port 8001
on the localhost
network interface. Additionally, workers are required to process requests. This
can be done with:
manage.py runworker
A rule of thumb is to have as many workers as there are processors available.
Route ASGI requests to ASGI server¶
To make the ASGI server available to the client, the public facing webserver has to know about it. Of course it would be possible to replace existing WSGI setups altogether and use only Daphne for both ASGI and WSGI. There are many situations where this is impractical and could cause problems. Therefore we recommend to only route ASGI requests to the ASGI server and let everything else be handled by the regular WSGI server.
To make this easier, CATMAID makes all ASGI endpoints available under:
<CATMAID-URL>/channels/
With this we can tell Nginx (or similar in other webservers) to route all URLs
starting with /channels/
to the ASGI server. This is accomplished by the
following location
block:
location /channels/ {
proxy_pass http://127.0.0.1:8001;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
Be sure to use your sub-directory structure as a prefix, if you use any. Also,
if you see errors like “(104: Connection reset by peer) while reading response
header from upstream
” make sure to increase the available file handles on the
OS level by setting:
sysctl -w fs.file-max=65536
Make sure to persist this in /etc/sysctl.conf
. Additionally, allow Nginx
more fie handles by setting this on the most outer level in
/etc/nginx.conf
:
worker_rlimit_nofile 10000;
Use RabbitMQ as back-end¶
The WebSockets layer needs a back-end to share data between different processes.
By default, the shared memory back-end
ASGI-IPC is
used. Alternatively,
RabbitMQ
can be used, which might already be in use for the Celery
setup. To do this, install the asgi_rabbitmq
layer package into the
virtualenv:
pip install -U asgi_rabbitmq
Additionally, the folllowing has to be added to settings.py
:
CHANNEL_LAYERS["default"]["BACKEND"] = "asgi_rabbitmq.RabbitmqChannelLayer"
CHANNEL_LAYERS["default"]["CONFIG"]["url"] = "amqp://guest:guest@localhost:5672/%2F"
Of course, if you changed RabbitMQ’s default credentials or its port, the above line has to be adjusted accordingly. You can find more information on this layer here.
Process management with Supervisord¶
Supervisord is used as an example for a process management configuration in other parts of this documentation and so we use it here to show how the above ASGI configuration can be managed alongside the existing Supervisord configuration. This assumes a Supervisor process group named “catmaid” is defined in the following file:
/etc/supervisord/conf.d/catmaid.conf
Add the following lines to this file, between the last [program:<name>]
section and the [group:catmaid]
section:
[program:catmaid-daphe]
directory = /opt/catmaid/django/projects/
command = /opt/catmaid/django/env/bin/daphne -b 127.0.0.1 -p 8001 mysite.asgi:channel_layer
user = www-data
stdout_logfile = /opt/catmaid/django/projects/mysite/daphne.log
redirect_stderr = true
[program:catmaid-daphe-worker]
directory = /opt/catmaid/django/projects/
command = /opt/catmaid/django/env/bin/python manage.py runworker
user = www-data
stdout_logfile = /opt/catmaid/django/projects/mysite/daphne-worker.log
redirect_stderr = true
autorestart = true
process_name = %(program_name)s_%(process_num)02d
numprocs = <NUM-CPUS>
Replace <NUM-CPUS>
in the last line with the number of CPUs on your system.
It should however be fine to use a lower number in most cases and probably even
1 will most of the time not cause problems.