Installation

Timing

A complete installation of woohoo pDNS will require about 45 minutes to complete for an experienced admin when a relational database server is already available.

This does not include making the source data available (e.g. copying log files from one machine to another or similar tasks).

Requirements

woohoo pDNS is a Python 3 project, therefore you need Python 3 to run it.

Also, a relational database is required. I use PostgreSQL but anything that SQLAlchemy can handle should do. For testing, sqlite is just fine; do not use it in production though because of limited support for timezones in datetime fields.

The RESTful API is served by Gunicorn. It is strongly suggested to have a reverse proxy (like Nginx, lighttpd, Apache, …) in front of it.

Overview

The installation will consist of the following steps:

  1. create a virtual environment (Python 3)

  2. install woohoo pDNS and dependencies

  3. configure access to the relational database

  4. set up the configuration in the reverse proxy

  5. configure Gunicorn to serve the RESTful API

  6. [OPTIONAL] configure automatic loading of new pDNS data

Installing

The virtual environment

Any way of virtualising the Python environment can be used to run woohoo pDNS. For this guide we use Python’s integrated venv method.

In case you are wondering: in production, I use Miniconda.

Caution

woohoo pDNS has pinned its dependencies! This means that the exact version is specified in requirements.txt for all dependencies. This might have undesired side effects when installing in a non-empty environment where one of the packages woohoo pDNS depends on is already installed.

So, go ahead and choose a suitable home for your installation of woohoo pDNS. For Linux/*BSD systems, something under /usr/local might make sense (e.g. /usr/local/opt/woohoo-pdns).

Once you have decided on the location and created a folder for woohoo pDNS, create a new virtual environment like this:

$ python -m venv .pdns

This will create a folder named .pdns in the current directory and this folder will hold your virtual environment of the same name.

Note: on a Mac of mine, creating the virtual environment like this failed with an error like:

Error: Command '['/Users/<username>/tmp/.pdns/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

which can be fixed by following advice found on Stackoverflow:

$ python -m venv --without-pip .pdns
$ source .pdns/bin/activate
$ curl https://bootstrap.pypa.io/get-pip.py | python
$ deactivate
$ source .pdns/bin/activate

Install woohoo pDNS and dependencies

Go ahead and activate the new environment if not already done (your shell prompt should change):

$ source .pdns/bin/activate

You should now populate this new virtual environment with woohoo pDNS and the required dependencies:

(.pdns)$ pip install woohoo-pdns

or install it from source:

(.pdns)$ git clone https://gitlab.com/scherand/woohoo-pdns
(.pdns)$ cd woohoo-pdns
(.pdns)$ python setup.py install
(.pdns)$ pip install -r requirements.txt

You should now be able to run the pdns command:

(.pdns)$ pnds -h

Create the configuration file

To properly run the pdns command, you will have to provide a config file with the following information/format (-f or --config-file CLI switch):

[DB]
conn_str = "sqlite:///demo.db"

[LOAD]
loader_class = "woohoo_pdns.load.SilkFileImporter"
data_timezone = "UTC"

The values shown here are the default values that will be used if you do not provide a config file.

Configure access to the relational database

This step depends on the database you want to use and the administrative processes you have in place for managing (relational) databases and access to them.

woohoo pDNS needs access to a database with permissions to create tables as well as read and write data.

For PostgreSQL the process is as follows:

[root@database:~]# su - postgres
$ createuser --interactive
Enter name of role to add: pdns
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
$ createdb pdns
$ psql
postgres=# ALTER USER pdns WITH ENCRYPTED PASSWORD '...';
postgres=# GRANT ALL PRIVILEGES ON DATABASE pdns to pdns;

Set up the configuration in the reverse proxy

Again, the exact steps depend on the reverse proxy software you use and the administrative processes around it. Assuming you have all the required permissions and want to use lighttpd, the configuration should look about as follows:

$HTTP["host"] =~ "^pdns.example.com$" {
    $HTTP["url"] =~ "^/api/" {
        proxy.server = ( "" => ( (
            "host" => "localhost",
            "port" => 5001
        ) ) )
    }
}

Configure Gunicorn to serve the RESTful API

The API is served by a Flask application (WSGI application) that lives in woohoo_pdns.api and is served by Gunicorn. To fire it up, you can use many different ways. For example, a startup script.

Consider using a dedicated user for Gunicorn.

You must provide the name of a config file via an environment variable called WOOHOO_PDNS_API_SETTINGS. That file should contain the following options. (In the example below, the file is called pdns_api_conf.py.) If only a filename is specified, the file is expected to be in a folder called instance in the directory you are starting flask from.

SECRET_KEY = "snakeoil"
DATABASE = "sqlite:///demo.db"
API_KEYS = [
    "IXsA7uRnxR4xek4JDEG5vk2oGjTYDSqaoKLRQLVjV2s3kw0bbv49qrgAT7Bk3g2K",
    "jLHKK0AIk1l6r3W8SAJj4Lh0v2a27JGbSSd406mr0u5FNrJn6RLWQ5m6qPYXT0d5",
]

The options shown above are the default values that are used if the file referenced in the WOOHOO_PDNS_API_SETTINGS environment variable does not set them.

You can use whatever you like for the SECRET_KEY; it is a Flask thing, see woohoo_pdns.api.config.DefaultSettings.SECRET_KEY.

The DATABASE option specifies the connection string to the relational database (this is forwarded ‘as is’ to SQLAlchemy).

The list of API_KEYS specifies all strings that will be accepted as keys for API access.

Note:

The API keys can be any string, but it is suggested to create a random character sequence using something like the following command (inspired by a gist by earthgecko):

$ cat /dev/urandom | base64 | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -1

The following outlines the FreeBSD rc.d script (/usr/local/etc/rc.d/pdns-api-gunicorn) I use for this purpose (inspired by a thread in the FreeBSD forums):

#! /bin/sh

# PROVIDE: pdns_api_gunicorn
# REQUIRE: DAEMON
# KEYWORD: shutdown

#
# Add the following lines to /etc/rc.conf to enable the woohoo pDNS API:
#
#pdns_api_gunicorn_enable="YES"

. /etc/rc.subr

name="pdns_api_gunicorn"
rcvar="${name}_enable"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
pidfile="/var/run/${name}.pid"
procname="daemon:"
gip="localhost"
gport="5001"

pdns_api_gunicorn_start(){
    chdir /usr/local/opt/woohoo-pdns
    . /root/.virtualenvs/pdns/bin/activate
    LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 FLASK_ENV=production WOOHOO_PDNS_API_SETTINGS="pdns_api_conf.py" daemon -r -S -P ${pidfile} -T pdns-api-gunicorn -u root /root/.virtualenvs/pdns/bin/gunicorn --workers 3 --bind ${gip}:${gport} "woohoo_pdns.api:create_app()"
}

pdns_api_gunicorn_stop(){
    if [ -f ${pidfile} ]; then
        echo -n "Stopping services: ${name}"
        # MUST send TERM signal (not e.g. INT) to work properly with '-P' switch
        # check daemon(8) for details
        kill -s TERM $(cat ${pidfile})
        if [ -f ${gsocket} ]; then
            rm -f ${gsocket}
        fi
        echo "."
    else
        echo "It appears ${name} is not running."
    fi
}

load_rc_config ${name}
# this sets the default 'enable' (to no)
: ${pdns_api_gunicorn_enable:="no"}
run_rc_command "$1"

Automatic loading of additional data

I run the following script every three minutes via a cron job:

*/3 * * * *  /usr/local/bin/woohoo-pdns-load.sh 2>&1 | /usr/bin/logger -t woohoo-pdns

/usr/local/bin/woohoo-pdns-load.sh:

#!/usr/local/bin/bash

. /root/.virtualenvs/pdns/bin/activate
pdns -f /usr/local/etc/woohoo-pdns/pdns.conf load -p "dns.*.txt" /var/spool/silk/dns

exit 0

New files matching the glob pattern dns.*.txt in /var/spool/silk/dns/ will be read into the database like this. After they are processed, they are renamed by appending .1 to the filename so they are not read again.

I have another ‘cron job’ (it is actually a job for FreeBSD’s periodic) that cleans out old files from /var/spool/silk/dns/ – well – periodically.

It lives in /usr/local/etc/periodic/daily/405.woohoo-pdns-cleanup and looks as follows:

#!/bin/sh

cleanup_1_files() {
    local rc

    /usr/bin/find /var/spool/silk/dns/ -name "*.1" -type f -maxdepth 1 -mmin +60 -delete

    rc=$?
    return $rc
}

cleanup_1_files

exit $rc