Problem: It is cumbersome to manage BigchainDB and Tendermint processes (#2410).

* Introduce a Monit-based setup for managing processes. Add the corresponding section to the network setup guide.

bigchaindb/config_utils.py

@@ -108,7 +108,7 @@ def file_config(filename=None):
                 'Failed to parse the JSON configuration from `{}`, {}'.format(filename, err)
             )
 
-    logger.info('Configuration loaded from `{}`'.format(filename))
+        logger.info('Configuration loaded from `{}`'.format(filename))
 
     return config
 

bigchaindb/log.py

@@ -3,10 +3,10 @@ import logging
 
 from bigchaindb.common.exceptions import ConfigurationError
 from logging.config import dictConfig as set_logging_config
-from os.path import expanduser, join
+import os
 
 
-DEFAULT_LOG_DIR = expanduser('~')
+DEFAULT_LOG_DIR = os.getcwd()
 BENCHMARK_LOG_LEVEL = 15
 
 
@@ -40,7 +40,7 @@ DEFAULT_LOGGING_CONFIG = {
         },
         'file': {
             'class': 'logging.handlers.RotatingFileHandler',
-            'filename': join(DEFAULT_LOG_DIR, 'bigchaindb.log'),
+            'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb.log'),
             'mode': 'w',
             'maxBytes':  209715200,
             'backupCount': 5,
@@ -49,7 +49,7 @@ DEFAULT_LOGGING_CONFIG = {
         },
         'errors': {
             'class': 'logging.handlers.RotatingFileHandler',
-            'filename': join(DEFAULT_LOG_DIR, 'bigchaindb-errors.log'),
+            'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb-errors.log'),
             'mode': 'w',
             'maxBytes':  209715200,
             'backupCount': 5,
@@ -58,7 +58,7 @@ DEFAULT_LOGGING_CONFIG = {
         },
         'benchmark': {
             'class': 'logging.handlers.RotatingFileHandler',
-            'filename': 'bigchaindb-benchmark.log',
+            'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb-benchmark.log'),
             'mode': 'w',
             'maxBytes':  209715200,
             'backupCount': 5,

docs/server/source/simple-network-setup.md

@@ -220,7 +220,7 @@ persistent_peers = "<Member 1 node id>@<Member 1 hostname>:26656,\
 <Member N node id>@<Member N hostname>:26656,"
 ```
 
-## Member: Start MongoDB, BigchainDB and Tendermint
+## Member: Start MongoDB
 
 If you installed MongoDB using `sudo apt install mongodb`, then MongoDB should already be running in the background. You can check using `systemctl status mongodb`.
 
@@ -228,6 +228,10 @@ If MongoDB isn't running, then you can start it using the command `mongod`, but
 
 If you installed MongoDB using `sudo apt install mongodb`, then a MongoDB startup script should already be installed (so MongoDB will start automatically when the machine is restarted). Otherwise, you should install a startup script for MongoDB.
 
+## Member: Start BigchainDB and Tendermint
+
+If you want to use a process manager, jump to the [next section](member-start-bigchaindb-and-tendermint-using-monit).
+
 To start BigchainDB, one uses the command `bigchaindb start` but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use `nohup`, `tmux`, or `screen`. For example, `nohup bigchaindb start 2>&1 > bigchaindb.log &`
 
 The _recommended_ approach is to create a startup script for BigchainDB, so it will start right after the boot of the operating system. (As mentioned earlier, MongoDB should already have a startup script.)
@@ -240,6 +244,45 @@ Note: We'll share some example startup scripts in the future. This document is a
 
 If you followed the recommended approach and created startup scripts for BigchainDB and Tendermint, then you can reboot the machine now. MongoDB, BigchainDB and Tendermint should all start.
 
+
+### Member: Start BigchainDB and Tendermint using Monit
+
+This section describes how to manage the BigchainDB and Tendermint processes using [Monit][monit] - a small open-source utility for managing and monitoring Unix processes.
+
+This section assumes that you followed the guide down to the [start MongoDB section](#member-start-mongodb) inclusive.
+
+Install Monit:
+
+```
+sudo apt install monit
+```
+
+If you installed the `bigchaindb` Python package, you should have the `bigchaindb-monit-config` script in your `PATH` now.
+
+Run the script:
+
+```
+bigchaindb-monit-config
+```
+
+The script builds a configuration file for Monit.
+
+Run Monit as a daemon, instructing it to wake up every second to check on processes:
+
+```
+monit -d 1
+```
+
+It will run the processes and restart them when they crash. If the root `bigchaindb_` process crashes, Monit will also restart the Tendermint process.
+
+Check the status by running `monit status` or `monit summary`.
+
+By default, it will collect program logs into the `~/.bigchaindb-monit/logs` folder.
+
+Consult `monit -h` or [the Monit documentation][monit-manual] to know more about the operational power you've just got the taste of.
+
+Check `bigchaindb-monit-config -h` if you want to arrange a different folder for logs or some of the Monit internal artifacts.
+
 ## How Others Can Access Your Node
 
 If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL `http://hostname:9984` (where hostname is something like `bdb7.canada.vmsareus.net` or `17.122.200.76`). That is, anyone can interact with your node using the [BigchainDB HTTP API](http-client-server-api.html) exposed at that address. The most common way to do that is to use one of the [BigchainDB Drivers](./drivers-clients/index.html).
@@ -315,3 +358,5 @@ TBD.
 [bdb:software]: https://github.com/bigchaindb/bigchaindb/
 [bdb:pypi]: https://pypi.org/project/BigchainDB/#history
 [tendermint:releases]: https://github.com/tendermint/tendermint/releases
+[monit]: https://www.mmonit.com/monit
+[monit-manual]: https://mmonit.com/monit/documentation/monit.html

pkg/scripts/bigchaindb-monit-config

@@ -0,0 +1,173 @@
+#!/bin/bash
+
+set -o nounset
+
+# Check if directory for monit logs exists
+if [ ! -d "$HOME/.bigchaindb-monit" ]; then
+  mkdir -p "$HOME/.bigchaindb-monit"
+fi
+
+monit_pid_path=${MONIT_PID_PATH:=$HOME/.bigchaindb-monit/monit_processes}
+monit_script_path=${MONIT_SCRIPT_PATH:=$HOME/.bigchaindb-monit/monit_script}
+monit_log_path=${MONIT_LOG_PATH:=$HOME/.bigchaindb-monit/logs}
+monitrc_path=${MONITRC_PATH:=$HOME/.monitrc}
+
+function usage() {
+        cat <<EOM
+
+    Usage: ${0##*/} [-h]
+
+    Configure Monit for BigchainDB and Tendermint process management.
+
+    ENV[MONIT_PID_PATH] || --monit-pid-path PATH
+
+    Absolute path to directory where the the program's pid-file will reside. 
+    The pid-file contains the ID(s) of the process(es). (default: ${monit_pid_path})
+
+    ENV[MONIT_SCRIPT_PATH] || --monit-script-path PATH
+
+    Absolute path to the directory where the executable program or
+    script is present. (default: ${monit_script_path})
+
+    ENV[MONIT_LOG_PATH] || --monit-log-path PATH
+
+    Absolute path to the directory where all the logs for processes
+    monitored by Monit are stored. (default: ${monit_log_path})
+
+    ENV[MONITRC_PATH] || --monitrc-path PATH
+
+    Absolute path to the monit control file(monitrc). (default: ${monitrc_path})
+
+    -h|--help
+        Show this help and exit.
+
+EOM
+}
+
+while [[ $# -gt 0 ]]; do
+    arg="$1"
+    case $arg in
+        --monit-pid-path)
+            monit_pid_path="$2"
+            shift
+        ;;
+        --monit-script-path)
+            monit_script_path="$2"
+            shift
+        ;;
+        --monit-log-path)
+            monit_log_path="$2"
+            shift
+        ;;
+        --monitrc-path)
+            monitrc_path="$2"
+            shift
+        ;;
+        -h|--help)
+            usage
+            exit
+        ;;
+        *)
+            echo "Unknown option: $1"
+            usage
+            exit 1
+        ;;
+    esac
+    shift
+done
+
+# Check if directory for monit logs exists
+if [ ! -d "$monit_log_path" ]; then
+  mkdir -p "$monit_log_path"
+fi
+
+# Check if directory for monit pid files exists
+if [ ! -d "$monit_pid_path" ]; then
+  mkdir -p "$monit_pid_path"
+fi
+
+cat >${monit_script_path} <<EOF
+#!/bin/bash
+case \$1 in
+
+  start_bigchaindb)
+
+    pushd \$4
+      nohup bigchaindb start >> \$3/bigchaindb.out.log 2>> \$3/bigchaindb.err.log &
+
+      echo \$! > \$2
+    popd
+
+    ;;
+
+  stop_bigchaindb)
+
+    kill -2 \`cat \$2\`
+    rm -f \$2
+
+    ;;
+
+  start_tendermint)
+
+    pushd \$4
+      nohup tendermint node --consensus.create_empty_blocks=false >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log &
+
+      echo \$! > \$2
+    popd
+
+    ;;
+
+  stop_tendermint)
+
+    kill -2 \`cat \$2\`
+    rm -f \$2
+
+    ;;
+
+esac
+exit 0
+EOF
+chmod +x ${monit_script_path}
+
+# Handling overwriting of control file interactively
+if [ -f "$monitrc_path" ]; then
+    echo "$monitrc_path already exists."
+    read -p "Overwrite[Y]? " -n 1 -r
+    echo
+    if [[ $REPLY =~ ^[Yy]$ ]]; then
+        echo "Overriding $monitrc_path/.monitrc"
+    else
+        read -p "Enter absolute path to store Monit control file: " monitrc_path
+        eval monitrc_path="$monitrc_path"
+        if [ ! -d "$(dirname $monitrc_path)" ]; then
+            echo "Failed to save monit control file '$monitrc_path': No such file or directory."
+            exit 1
+        fi
+    fi
+fi
+
+# configure monitrc
+cat >${monitrc_path} <<EOF
+set httpd
+  port 2812
+  allow localhost
+
+check process bigchaindb
+    with pidfile ${monit_pid_path}/bigchaindb.pid
+    start program "${monit_script_path} start_bigchaindb $monit_pid_path/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
+    restart program "${monit_script_path} start_bigchaindb $monit_pid_path/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
+    stop program "${monit_script_path} stop_bigchaindb $monit_pid_path/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
+
+check process tendermint
+    with pidfile ${monit_pid_path}/tendermint.pid
+    start program "${monit_script_path} start_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
+    restart program "${monit_script_path} start_bigchaindb ${monit_pid_path}/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
+    stop program "${monit_script_path} stop_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
+    depends on bigchaindb
+EOF
+
+# Setting permissions for control file
+chmod 0700 ${monitrc_path}
+
+echo -e "BigchainDB process manager configured!"
+set -o errexit

setup.py

@@ -128,6 +128,8 @@ setup(
 
     packages=find_packages(exclude=['tests*']),
 
+    scripts = ['pkg/scripts/bigchaindb-monit-config'],
+
     entry_points={
         'console_scripts': [
             'bigchaindb=bigchaindb.commands.bigchaindb:main'