Running Your Instance Properly

The previous steps demonstrate how to run the openbench using the django webserver, on localhost, with sqlite3 as the database. In practice, django is not intended to run the web-server, and instead something like Gunicorn, sitting behind a nginx reverse-proxy, is. Additionally, sqlite3 does not support concurrent database access, and is really not adequate. We prefer to use MySQL.

An important note: OpenBench performs a graceful shutdown upon receiving a SIGTERM or SIGINT. That graceful shutdown ensures that no lock files are left sitting around; and that all writes to PGN archives are completed. Sending SIGKILL can cause your archive to become corrupted quite easily. pkill -TERM gunicorn is a fairly reasonable way to shut down the server, after waiting for the processes to all exit.

Step #1: Preparing a MySQL Database

# Get the python package. You can try simply pip3 install mysqlclient, but that likely will fail
sudo apt-get install python3-dev default-libmysqlclient-dev
sudo MYSQLCLIENT_CFLAGS="-I/usr/include/mysql" MYSQLCLIENT_LDFLAGS="-L/usr/lib/x86_64-linux-gnu -lmysqlclient" pip3 install mysqlclient

# When prompted, you should disable non-localhost access to the database
sudo apt install mysql-server
sudo mysql_secure_installation
sudo systemctl start mysql
sudo systemctl enable mysql

# Set a root password, if you were not prompted during mysql_secure_installation
sudo mysql
> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '<password>';
> FLUSH PRIVILEGES;
> exit;

# Finally, create the Database
mysql -u root -p
> CREATE DATABASE openbench;
> exit;

Next, you need to modify the contents of OpenSite/settings.py. You do not want to commit these changes. If you want to be able to commit the changes, you might consider an ENV file that contains this information, that you load in the python code.

DATABASES = {
    'default': {
        'ENGINE'  : 'django.db.backends.mysql',
        'NAME'    : 'openbench',
        'USER'    : 'root',
        'PASSWORD': '<password>',
        'HOST'    : 'localhost',  # Change to your MySQL server's host, if you modified it
        'PORT'    : '3306',       # Change to your MySQL server's port, if you modified it
    }
}

# Fully prepare the database, using the migrations in the Repo
# You should never be running makemigrations as a general OpenBench instance runner

python3 manage.py migrate
python3 manage.py migrate --run-syncdb

Step 2: Setting up Nginx as a reverse Proxy

sudo apt install nginx
sudo vi /etc/nginx/sites-available/openbench

Note that this is on port 80 (http) for now. This will get updated for us after we create our certs.

server {
    listen 80;
    server_name example.com; # <--- Your domain here

    client_max_body_size 250M; # Allow uploading 250Mb Network files

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Enable the site; test the config; reload nginx
sudo ln -s /etc/nginx/sites-available/openbench /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Step 3: Running with Gunicorn, and Obtaining Certificates

# Run the server; and keep it running for the next steps
pip3 install gunicorn
gunicorn OpenSite.wsgi:application --bind 127.0.0.1:8000 --workers 3

# This will automatically update your nginx config for OpenBench
sudo certbot --nginx -d example.com

At this point, your nginx config should look something like the one below. You should now be able to access the site over https. At this point, the only thing left that you might need to do is open ports 80 or 443.

server {
    server_name example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    client_max_body_size 250M;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name example.com;
    return 404; # managed by Certbot
}

Step 4: Serving Networks and PGNs via X-Accel-Redirect

By default, Django streams Networks and PGN archives itself, tying up a Gunicorn worker for the whole transfer. With X-Accel-Redirect, Django does the permission checks and then names the file in a header, letting nginx serve the bytes. This is opt-in, as it requires the reverse proxy. In Config/config.json:

"use_x_accel_redirect"  : true,
"x_accel_redirect_root" : "/x-accel-media/",

Both keys must be present, even when disabled. The root has to match an internal location aliased to your Media/ directory. Add this to the server block from Step 3, substituting your own path -- these examples assume the repo is at /home/ubuntu/OpenBench/.

    # Only reachable via an X-Accel-Redirect from Django; never directly
    location /x-accel-media/ {
        internal;
        alias /home/ubuntu/OpenBench/Media/;
    }

Do not omit internal; it is what stops anyone from fetching /x-accel-media/<sha256> and bypassing authentication. It must be alias rather than root, and the trailing slashes must match.

sudo nginx -t
sudo systemctl reload nginx

# The config is read once at startup, so Gunicorn needs a full restart
pkill -TERM gunicorn
gunicorn OpenSite.wsgi:application --bind 127.0.0.1:8000 --workers 3

Filesystem permissions

Nginx runs as www-data, and /home/ubuntu is mode 0750 on Ubuntu, so it cannot even traverse to Media/. Without this, every download 403s. An ACL is preferred, as it grants access to www-data alone:

sudo apt install acl

# Traverse-only, so www-data can pass through but not list
sudo setfacl -m u:www-data:--x /home/ubuntu
sudo setfacl -m u:www-data:--x /home/ubuntu/OpenBench

# Read on Media and below. The -d sets a default ACL, which new
# Networks and PGN archives inherit; without it they will 403.
sudo setfacl -R    -m u:www-data:rX /home/ubuntu/OpenBench/Media
sudo setfacl -R -d -m u:www-data:rX /home/ubuntu/OpenBench/Media

Or with plain chmod, which grants access to every local user, and is not inherited by future files. Note the lack of -R on the first line; recursing o+x over a home directory would mark every file executable.

sudo chmod o+x /home/ubuntu /home/ubuntu/OpenBench
sudo chmod -R o+rX /home/ubuntu/OpenBench/Media

Verifying

# Check as www-data, before going through the browser
sudo -u www-data cat /home/ubuntu/OpenBench/Media/<some sha256> > /dev/null && echo OK

# Nginx strips the header, so query Gunicorn directly to see it
curl -sI http://127.0.0.1:8000/api/networks/Ethereal/<sha256>/ | grep -i accel

If downloads still 403, watch /var/log/nginx/error.log while reproducing. A failed (13: Permission denied) names the path that was refused. A directory index of ... is forbidden means the alias and location paths disagree.

Step 5: Going forward

At this point, you should have the server running whenever you fire up the Gunicorn command. And your clients should be connecting over HTTPs, only utilizing the HTTP redirect for simple web browsing. At some point, your certs will expire, but the certbot steps will have created a cron-job to renew them. Hopefully that all works.

As a general user now, you might just continue to merge AndyGrant/OpenBench/master into your own repo now and again. That is annoying because the Config/config.json will change, and you'll have to manually handle that. Note that use_x_accel_redirect and x_accel_redirect_root must both be present in your config, even if you are not using them -- the server will refuse to start otherwise. Aside from that, the only steps you should need to do are a python3 manage.py migrate everytime you sync with master, in case new database migrations have been created. You'll want to take the server offline to do this. It is a good idea to backup your database every now and again. That can be done very easily, by dumping the contents to a .json file via python3 manage.py dumpdata > backup.json.

Also, some general maintenance, there are a number of models in the database that don't actually matter. You are free to delete all Result, Machine, LogEvent, and PGN objects whenever you like. That can be done with the following code snippet. You can follow this up by deleting all event* files in Media/.

python3 manage.py shell
>>> from OpenBench.models import *
>>> Result.objects.all().delete()
>>> Machine.objects.all().delete()
>>> LogEvent.objects.all().delete()
>>> PGN.objects.all().delete()