Add docs for log rotation

docs/server/source/appendices/log-rotation.md

@@ -0,0 +1,46 @@
+Copyright BigchainDB GmbH and BigchainDB contributors
+SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0)
+Code is Apache-2.0 and docs are CC-BY-4.0
+--->
+
+# Log rotation for a BigchainDB node
+
+Each BigchainDB node comprises of 3 main services:
+  - BigchainDB server
+  - Tendermint
+  - MongoDB
+
+To run a BigchainDB test network/dev node, that is expected to run for relatively longer periods
+of time, we need to consider the log rotation of these services i.e. we do not want the logs taking
+up large amounts of storage and making the node unresponsive or get into bad state.
+
+## Log rotation for MongoDB
+
+Currently, we leave the log rotation of MongoDB to the BigchainDB administrator. For more notes on MongoDB log rotation
+please refer to [MongoDB docs](https://docs.mongodb.com/v3.6/tutorial/rotate-log-files/).
+
+## Log rotation for BigchainDB
+
+Log rotation is baked into BigchainDB server using the `logging` module. If you followed the [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide. BigchainDB processes are monitored using [Monit]( https://www.mmonit.com/monit). BigchainDB server logs information into the following files:
+ - `bigchaindb.log`
+ - `bigchaindb-errors.log`
+ - `bigchaindb-benchmark.log`
+
+Default location for these log files is: `$HOME/.bigchaindb-monit/logs`
+
+The logs for BigchainDB server are rotated when any of the above mentioned file exceeds `209715200 bytes`.
+
+
+## Log rotation for Tendermint
+
+Log rotation for Tendermint is handled by [Monit]( https://www.mmonit.com/monit). When we start BigchainDB and Tendermint using [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide.
+
+```bash
+$ monit -d 1
+```
+
+Monit monitors both Tendermint and BigchainDB processes as well as the Tendermint log files, `tendermint.out.log` and `tendermint.err.log`. Default location for these log files is:
+`$HOME/.bigchaindb-monit/logs`.
+
+Tendermint logs are rotated if any of the above mentioned log files exceeds `200 MB` in size.
+

docs/server/source/simple-deployment-template/network-setup.md

@@ -390,6 +390,10 @@ If you started BigchainDB in the foreground, a `Ctrl + C` or `Ctrl + Z` would sh
 
 TBD.
 
+## Log rotation
+
+Please go over the [document describing log rotation of a BigchainDB node](../appendices/log-rotation.md).
+
 
 [bdb:software]: https://github.com/bigchaindb/bigchaindb/
 [bdb:pypi]: https://pypi.org/project/BigchainDB/#history

pkg/scripts/bigchaindb-monit-config

@@ -138,7 +138,7 @@ case \$1 in
     /bin/mv \$2 \$2.\$(date +%y-%m-%d)
     /bin/tar -cvf \$2.\$(date +%Y%m%d_%H%M%S).tar.gz \$2.\$(date +%y-%m-%d)
     /bin/rm \$2.\$(date +%y-%m-%d)
-    kill -2 \$(cat \$3)
+    /bin/kill -2 \$(cat \$3)
     ;;
 
 esac
@@ -189,6 +189,7 @@ check file tendermint.out.log with path ${monit_log_path}/tendermint.out.log
 check file tendermint.err.log with path ${monit_log_path}/tendermint.err.log
     if size > 200 MB then
         exec "${monit_script_path}_logrotate rotate_tendermint_logs ${monit_log_path}/tendermint.err.log $monit_pid_path/tendermint.pid"
+
 EOF
 
 # Setting permissions for control file