Developer setup with Vagrant

CATMAID has a lot of dependencies and is only supported on Ubuntu. To ease the setup for developers, CATMAID has a Vagrant configuration, which allows you to develop inside a virtual (“guest”) machine.

Unlike docker, data will be persisted between restarts, and you can interact with the container over SSH.

The repository directory is mounted into the container so all changes are reflected on both the host and the guest machine.

Quickstart (linux)

  • Install VirtualBox and vagrant

  • Make a fresh clone of the CATMAID repository and cd into it

  • vagrant up

  • vagrant ssh, then once inside

    • Go to code directory: cd /CATMAID

    • Finish setup: scripts/vagrant/optional.sh

    • Start the dev server: django/projects/manage.py runserver [::]:8888

  • Develop, inside or outside the container!

Vagrant

Vagrant is an easy-to-configure abstraction layer over a number of virtual machine providers (e.g. VirtualBox, VMWare, Hyper-V) implemented in ruby 2.6. Once installed, you need only vagrant up inside the repository to create a fully-fledged Ubuntu VM with PostgreSQL running, fully populated Python and Node environments, and a base R environment.

Install vagrant and the VirtualBox VM provider. You may need to enable virtualization in the BIOS. Additionally, install the vagrant-disksize plugin, with vagrant plugin install vagrant-disksize. Note also that VirtualBox may not be compatible with Docker under windows.

vagrant up

Start up the container. If the image (box) doesn’t exist, it will be downloaded. If the container doesn’t exist, it will be provisioned (i.e. dependencies will be installed).

vagrant ssh

SSH into a running container.

vagrant ssh-config

Print out configuration for SSHing into the container with the regular ssh command (see below).

vagrant suspend

Hibernate the container, saving its current state.

vagrant resume

Resume a ``suspend``ed container.

vagrant destroy

Delete the container.

See the vagrant documentation for more details.

Tools like VScode need to be able to SSH into the container themselves. They get their configuration from your user’s SSH configuration (~/.ssh/config on Linux). Copy and paste (or echo) the output of vagrant ssh-config (ignoring any warnings from ruby gems) into this file. The hostname of the container will be catmaid-vm.

Warning

suspend the container before shutting down the host!

Snapshots

You can save the state of a running VM and restore it later. This is helpful if you install additional tools or data into your VM, and is quicker to load than a full re-provision.

vagrant snapshot save <name>

Save the VM’s current state with the given name.

vagrant snapshot list

Show the available snapshots.

vagrant snapshot restore <name>

Restore the named snapshot.

Note that destroying a VM also destroys all snapshots of it.

More information is available in the vagrant documentation.

Setup

Guest machine configuration

By default, the guest machine is quite small: it may not be able to hold much data and have poor performance. You can customise the machine’s “hardware” by setting environment variables on the host machine when the guest is first set up.

The guest’s disk size is 60GB by default (but does not take up that much space until it’s full). To increase this, set the CATMAID_VM_DISK environment variable to something like "100GB". The guest’s RAM size, in MB, will be 2048: change it by setting CATMAID_VM_RAM_MB. It will use 2 CPUs, which can be set with CATMAID_VM_CPUS.

Consider using direnv or dotenv, or adding these variables to your shell rc or profile script, to ensure you always have them set when interacting with Vagrant.

Provisioning

The first time the VM is started, it is “provisioned” - i.e. CATMAID’s dependencies are installed. Subsequent startups will be much faster.

Some red messages during provisioning are expected: every line prepended with a + is just showing what command is being run.

This provisioning gets you up to step 3 in the basic installation instructions (setting up the OS-level dependencies and python environment). The database and CATMAID configuration are done separately, in case you prefer your own configuration to the recommendations in the installation instructions.

To finish off the installation according to the instructions, SSH into the VM (vagrant ssh) and bash /CATMAID/scripts/vagrant/optional.sh. If the DB_NAME, DB_USER, DB_PASSWORD, or TIMEZONE environment variables are set (on the guest), they will override the defaults (when the machine is provisioned, the host’s timezone will be added to ~/timezone, which is used as the default timezone here). This creates your local settings, applies database migrations, collects static files as symlinks, creates a CATMAID superuser (you will need to input your the username, email, and password), inserts example projects (N.B. the data for these projects is probably not accessible), and sets CATMAID’s writable directory to /CATMAID/data.

Virtual machine layout

The container runs Ubuntu Linux 18.04. In Linux, / is the root directory, and ~ is the home directory of the user (called vagrant in the container: so ~ is /home/vagrant).

  • The CATMAID repository is in /CATMAID. This is the exact same directory as lives on the host.

  • The Node environment is stored in ~/catmaid-npm-overlay/node_modules, and overlaid onto /CATMAID/node_modules. This prevents it interfering with the host’s node environment and vice versa.

  • The Python environment is stored in ~/catmaid-env, and is automatically activated when you SSH in.

  • R packages are in ~/R.

Some guest ports are forwarded to the host machine so that you can access the database, test with the dev server, and look at the generated sphinx docs.

Service

Guest port

Host port

Notes

PostgreSQL

5555

5555

Not the default port 5432

Django dev server

8888

8888

django/projects/manage.py runserver [::]:8888

Docs server

8889

8889

cd sphinx-doc && make serve

If optional.sh was used to configure the VM, and no parameters were given using environment variables:

  • The CATMAID database is called “catmaid”.

  • The database user is called “catmaid_user”.

  • The database user passwrod is “p4ssw0rd”.

  • The CATMAID time zone is the same as the host machine, if the host machine has a POSIX terminal, or defaults to America/New_York (but the guest machine is UTC).

Development

Because the development server will technically be accessed from outside of the machine it’s running on, you will need to start it with django/projects/manage.py [::]:8888

From inside the container, connect to the database with psql -U catmaid_user catmaid. From the host, add the options -h localhost -p 5555.

VSCode’s Remote - SSH extension allows you to develop in the container directly. The connection details are picked up from your ~/.ssh/config file.

PyCharm Professional has support for remote interpreters built in.

You can also install your own development toolchain inside the container - it’s just ubuntu! Alternatively, you can make your edits using the host machine, and just use the VM to test, lint, run the database, etc.

Making commits

By default, the git user is not globally configured inside the VM, and cannot make commits. You have a few options:

  • Interact with git only from the host machine

  • Configure git globally inside the VM

  • Configure your user locally in the repository (allowing its use from either the host or the guest)