Installation¶
Timing¶
A complete installation of woohoo pDNS will require about 45 minutes to complete for an experienced admin when a relational database server is already available.
This does not include making the source data available (e.g. copying log files from one machine to another or similar tasks).
Requirements¶
woohoo pDNS is a Python 3 project, therefore you need Python 3 to run it.
Also, a relational database is required. I use PostgreSQL but anything that SQLAlchemy can handle should do. For testing, sqlite is just fine; do not use it in production though because of limited support for timezones in datetime fields.
The RESTful API is served by Gunicorn. It is strongly suggested to have a reverse proxy (like Nginx, lighttpd, Apache, …) in front of it.
Overview¶
The installation will consist of the following steps:
create a virtual environment (Python 3)
install woohoo pDNS and dependencies
configure access to the relational database
set up the configuration in the reverse proxy
configure Gunicorn to serve the RESTful API
[OPTIONAL] configure automatic loading of new pDNS data
Installing¶
The virtual environment¶
Any way of virtualising the Python environment can be used to run woohoo pDNS.
For this guide we use Python’s integrated venv
method.
In case you are wondering: in production, I use Miniconda.
Caution
woohoo pDNS has pinned its dependencies! This means that the exact version
is specified in requirements.txt
for all dependencies.
This might have undesired side effects when installing in a non-empty
environment where one of the packages woohoo pDNS depends on is already
installed.
So, go ahead and choose a suitable home for your installation of woohoo pDNS.
For Linux/*BSD systems, something under /usr/local
might make sense (e.g.
/usr/local/opt/woohoo-pdns
).
Once you have decided on the location and created a folder for woohoo pDNS, create a new virtual environment like this:
$ python -m venv .pdns
This will create a folder named .pdns
in the current directory and this
folder will hold your virtual environment of the same name.
Note: on a Mac of mine, creating the virtual environment like this failed with an error like:
Error: Command '['/Users/<username>/tmp/.pdns/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
which can be fixed by following advice found on Stackoverflow:
$ python -m venv --without-pip .pdns
$ source .pdns/bin/activate
$ curl https://bootstrap.pypa.io/get-pip.py | python
$ deactivate
$ source .pdns/bin/activate
Install woohoo pDNS and dependencies¶
Go ahead and activate the new environment if not already done (your shell prompt should change):
$ source .pdns/bin/activate
You should now populate this new virtual environment with woohoo pDNS and the required dependencies:
(.pdns)$ pip install woohoo-pdns
or install it from source:
(.pdns)$ git clone https://gitlab.com/scherand/woohoo-pdns
(.pdns)$ cd woohoo-pdns
(.pdns)$ python setup.py install
(.pdns)$ pip install -r requirements.txt
You should now be able to run the pdns
command:
(.pdns)$ pnds -h
Create the configuration file¶
To properly run the pdns command, you will have to provide a config file with
the following information/format (-f
or --config-file
CLI switch):
[DB]
conn_str = "sqlite:///demo.db"
[LOAD]
loader_class = "woohoo_pdns.load.SilkFileImporter"
data_timezone = "UTC"
The values shown here are the default values that will be used if you do not provide a config file.
Configure access to the relational database¶
This step depends on the database you want to use and the administrative processes you have in place for managing (relational) databases and access to them.
woohoo pDNS needs access to a database with permissions to create tables as well as read and write data.
For PostgreSQL the process is as follows:
[root@database:~]# su - postgres
$ createuser --interactive
Enter name of role to add: pdns
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
$ createdb pdns
$ psql
postgres=# ALTER USER pdns WITH ENCRYPTED PASSWORD '...';
postgres=# GRANT ALL PRIVILEGES ON DATABASE pdns to pdns;
Set up the configuration in the reverse proxy¶
Again, the exact steps depend on the reverse proxy software you use and the administrative processes around it. Assuming you have all the required permissions and want to use lighttpd, the configuration should look about as follows:
$HTTP["host"] =~ "^pdns.example.com$" {
$HTTP["url"] =~ "^/api/" {
proxy.server = ( "" => ( (
"host" => "localhost",
"port" => 5001
) ) )
}
}
Configure Gunicorn to serve the RESTful API¶
The API is served by a Flask application (WSGI application) that lives in
woohoo_pdns.api
and is served by Gunicorn. To fire it up, you can use many
different ways. For example, a startup script.
Consider using a dedicated user for Gunicorn.
You must provide the name of a config file via an environment variable
called WOOHOO_PDNS_API_SETTINGS
. That file should contain the following
options. (In the example below, the file is called pdns_api_conf.py
.) If
only a filename is specified, the file is expected to be in a folder called
instance
in the directory you are starting flask from.
SECRET_KEY = "snakeoil"
DATABASE = "sqlite:///demo.db"
API_KEYS = [
"IXsA7uRnxR4xek4JDEG5vk2oGjTYDSqaoKLRQLVjV2s3kw0bbv49qrgAT7Bk3g2K",
"jLHKK0AIk1l6r3W8SAJj4Lh0v2a27JGbSSd406mr0u5FNrJn6RLWQ5m6qPYXT0d5",
]
The options shown above are the default values that are used if the file
referenced in the WOOHOO_PDNS_API_SETTINGS
environment variable does not
set them.
You can use whatever you like for the SECRET_KEY
; it is a Flask thing, see
woohoo_pdns.api.config.DefaultSettings.SECRET_KEY
.
The DATABASE
option specifies the connection string to the relational
database (this is forwarded ‘as is’ to SQLAlchemy).
The list of API_KEYS
specifies all strings that will be accepted as keys
for API access.
- Note:
The API keys can be any string, but it is suggested to create a random character sequence using something like the following command (inspired by a gist by earthgecko):
$ cat /dev/urandom | base64 | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -1
The following outlines the FreeBSD rc.d script (/usr/local/etc/rc.d/pdns-api-gunicorn
)
I use for this purpose (inspired by a thread in the FreeBSD forums):
#! /bin/sh
# PROVIDE: pdns_api_gunicorn
# REQUIRE: DAEMON
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf to enable the woohoo pDNS API:
#
#pdns_api_gunicorn_enable="YES"
. /etc/rc.subr
name="pdns_api_gunicorn"
rcvar="${name}_enable"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
pidfile="/var/run/${name}.pid"
procname="daemon:"
gip="localhost"
gport="5001"
pdns_api_gunicorn_start(){
chdir /usr/local/opt/woohoo-pdns
. /root/.virtualenvs/pdns/bin/activate
LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 FLASK_ENV=production WOOHOO_PDNS_API_SETTINGS="pdns_api_conf.py" daemon -r -S -P ${pidfile} -T pdns-api-gunicorn -u root /root/.virtualenvs/pdns/bin/gunicorn --workers 3 --bind ${gip}:${gport} "woohoo_pdns.api:create_app()"
}
pdns_api_gunicorn_stop(){
if [ -f ${pidfile} ]; then
echo -n "Stopping services: ${name}"
# MUST send TERM signal (not e.g. INT) to work properly with '-P' switch
# check daemon(8) for details
kill -s TERM $(cat ${pidfile})
if [ -f ${gsocket} ]; then
rm -f ${gsocket}
fi
echo "."
else
echo "It appears ${name} is not running."
fi
}
load_rc_config ${name}
# this sets the default 'enable' (to no)
: ${pdns_api_gunicorn_enable:="no"}
run_rc_command "$1"
Automatic loading of additional data¶
I run the following script every three minutes via a cron job:
*/3 * * * * /usr/local/bin/woohoo-pdns-load.sh 2>&1 | /usr/bin/logger -t woohoo-pdns
/usr/local/bin/woohoo-pdns-load.sh
:
#!/usr/local/bin/bash
. /root/.virtualenvs/pdns/bin/activate
pdns -f /usr/local/etc/woohoo-pdns/pdns.conf load -p "dns.*.txt" /var/spool/silk/dns
exit 0
New files matching the glob pattern dns.*.txt
in /var/spool/silk/dns/
will be read into the database like this.
After they are processed, they are renamed by appending .1
to the filename so they are not read again.
I have another ‘cron job’ (it is actually a job for FreeBSD’s periodic
) that cleans out old files from
/var/spool/silk/dns/
– well – periodically.
It lives in /usr/local/etc/periodic/daily/405.woohoo-pdns-cleanup
and looks as follows:
#!/bin/sh
cleanup_1_files() {
local rc
/usr/bin/find /var/spool/silk/dns/ -name "*.1" -type f -maxdepth 1 -mmin +60 -delete
rc=$?
return $rc
}
cleanup_1_files
exit $rc