mirror of
https://github.com/bigchaindb/bigchaindb.git
synced 2024-10-13 13:34:05 +00:00
Problem: No log rotation when BigchainDB and Tendermint started with Monit (#2528)
* Log rotation for tendermint and BigchainDB using Monit * Change file size from 100KB used for testing to 200 MB for test envs * Add docs for log rotation * remove --consensus.create_empty_blocks from monit start Because already taken care of in the config file * do not run bigchaindb with debug flag instead use config file * Address comments I - Rephrase log rotation to imply monit is not default - Add statement to explain where bigchaindb log files are created - Fix HTML comment - Add log rotation to index * Address comments II
This commit is contained in:
Muawia Khan
Troy McConaghy
parent
b33e3808a6
commit
5a440843b6
@ -17,3 +17,4 @@ Appendices
|
||||
ntp-notes
|
||||
licenses
|
||||
all-in-one-bigchaindb
|
||||
log-rotation
|
||||
|
||||
48
docs/server/source/appendices/log-rotation.md
Normal file
48
docs/server/source/appendices/log-rotation.md
Normal file
@ -0,0 +1,48 @@
|
||||
<!---
|
||||
Copyright BigchainDB GmbH and BigchainDB contributors
|
||||
SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0)
|
||||
Code is Apache-2.0 and docs are CC-BY-4.0
|
||||
--->
|
||||
|
||||
# Log rotation for a BigchainDB node
|
||||
|
||||
Each BigchainDB node comprises of 3 main services:
|
||||
- BigchainDB server
|
||||
- Tendermint
|
||||
- MongoDB
|
||||
|
||||
To run a BigchainDB test network/dev node, that is expected to run for relatively longer periods
|
||||
of time, we need to consider the log rotation of these services i.e. we do not want the logs taking
|
||||
up large amounts of storage and making the node unresponsive or get into bad state.
|
||||
|
||||
## Log rotation for MongoDB
|
||||
|
||||
Currently, we leave the log rotation of MongoDB to the BigchainDB administrator. For more notes on MongoDB log rotation
|
||||
please refer to [MongoDB docs](https://docs.mongodb.com/v3.6/tutorial/rotate-log-files/).
|
||||
|
||||
## Log rotation for BigchainDB
|
||||
|
||||
Log rotation is baked into BigchainDB server using the `logging` module. BigchainDB server logs information into the following files:
|
||||
- `bigchaindb.log`
|
||||
- `bigchaindb-errors.log`
|
||||
- `bigchaindb-benchmark.log`
|
||||
|
||||
These log files are created by default in the directory from where you run `bigchaindb start`, if you are using `monit`, from
|
||||
[How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide, the default directory is: `$HOME/.bigchaindb-monit/logs`
|
||||
|
||||
The logs for BigchainDB server are rotated when any of the above mentioned file exceeds `209715200 bytes (i.e. approximately 209 MB).`.
|
||||
|
||||
|
||||
## Log rotation for Tendermint
|
||||
|
||||
In order to set up log rotation of Tendermint, you will need to use the [Monit]( https://www.mmonit.com/monit) scripts provided by us. Covered in the [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide.
|
||||
|
||||
```bash
|
||||
$ monit -d 1
|
||||
```
|
||||
|
||||
Monit monitors both Tendermint and BigchainDB processes as well as the Tendermint log files, `tendermint.out.log` and `tendermint.err.log`. Default location for these log files is:
|
||||
`$HOME/.bigchaindb-monit/logs`.
|
||||
|
||||
Tendermint logs are rotated if any of the above mentioned log files exceeds `200 MB` in size.
|
||||
|
||||
@ -390,6 +390,10 @@ If you started BigchainDB in the foreground, a `Ctrl + C` or `Ctrl + Z` would sh
|
||||
|
||||
TBD.
|
||||
|
||||
## Log rotation
|
||||
|
||||
Please go over the [document describing log rotation of a BigchainDB node](../appendices/log-rotation.md).
|
||||
|
||||
|
||||
[bdb:software]: https://github.com/bigchaindb/bigchaindb/
|
||||
[bdb:pypi]: https://pypi.org/project/BigchainDB/#history
|
||||
|
||||
@ -4,7 +4,7 @@ set -o nounset
|
||||
|
||||
# Check if directory for monit logs exists
|
||||
if [ ! -d "$HOME/.bigchaindb-monit" ]; then
|
||||
mkdir -p "$HOME/.bigchaindb-monit"
|
||||
mkdir -p "$HOME/.bigchaindb-monit"
|
||||
fi
|
||||
|
||||
monit_pid_path=${MONIT_PID_PATH:=$HOME/.bigchaindb-monit/monit_processes}
|
||||
@ -13,7 +13,7 @@ monit_log_path=${MONIT_LOG_PATH:=$HOME/.bigchaindb-monit/logs}
|
||||
monitrc_path=${MONITRC_PATH:=$HOME/.monitrc}
|
||||
|
||||
function usage() {
|
||||
cat <<EOM
|
||||
cat <<EOM
|
||||
|
||||
Usage: ${0##*/} [-h]
|
||||
|
||||
@ -21,7 +21,7 @@ function usage() {
|
||||
|
||||
ENV[MONIT_PID_PATH] || --monit-pid-path PATH
|
||||
|
||||
Absolute path to directory where the the program's pid-file will reside.
|
||||
Absolute path to directory where the the program's pid-file will reside.
|
||||
The pid-file contains the ID(s) of the process(es). (default: ${monit_pid_path})
|
||||
|
||||
ENV[MONIT_SCRIPT_PATH] || --monit-script-path PATH
|
||||
@ -45,45 +45,45 @@ EOM
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
arg="$1"
|
||||
case $arg in
|
||||
--monit-pid-path)
|
||||
monit_pid_path="$2"
|
||||
shift
|
||||
;;
|
||||
--monit-script-path)
|
||||
monit_script_path="$2"
|
||||
shift
|
||||
;;
|
||||
--monit-log-path)
|
||||
monit_log_path="$2"
|
||||
shift
|
||||
;;
|
||||
--monitrc-path)
|
||||
monitrc_path="$2"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
arg="$1"
|
||||
case $arg in
|
||||
--monit-pid-path)
|
||||
monit_pid_path="$2"
|
||||
shift
|
||||
;;
|
||||
--monit-script-path)
|
||||
monit_script_path="$2"
|
||||
shift
|
||||
;;
|
||||
--monit-log-path)
|
||||
monit_log_path="$2"
|
||||
shift
|
||||
;;
|
||||
--monitrc-path)
|
||||
monitrc_path="$2"
|
||||
shift
|
||||
;;
|
||||
-h | --help)
|
||||
usage
|
||||
exit
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
# Check if directory for monit logs exists
|
||||
if [ ! -d "$monit_log_path" ]; then
|
||||
mkdir -p "$monit_log_path"
|
||||
mkdir -p "$monit_log_path"
|
||||
fi
|
||||
|
||||
# Check if directory for monit pid files exists
|
||||
if [ ! -d "$monit_pid_path" ]; then
|
||||
mkdir -p "$monit_pid_path"
|
||||
mkdir -p "$monit_pid_path"
|
||||
fi
|
||||
|
||||
cat >${monit_script_path} <<EOF
|
||||
@ -93,7 +93,7 @@ case \$1 in
|
||||
start_bigchaindb)
|
||||
|
||||
pushd \$4
|
||||
nohup bigchaindb -l DEBUG start >> \$3/bigchaindb.out.log 2>> \$3/bigchaindb.err.log &
|
||||
nohup bigchaindb start > /dev/null 2>&1 &
|
||||
|
||||
echo \$! > \$2
|
||||
popd
|
||||
@ -110,7 +110,8 @@ case \$1 in
|
||||
start_tendermint)
|
||||
|
||||
pushd \$4
|
||||
nohup tendermint node --consensus.create_empty_blocks=false >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log &
|
||||
|
||||
nohup tendermint node >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log &
|
||||
|
||||
echo \$! > \$2
|
||||
popd
|
||||
@ -129,21 +130,37 @@ exit 0
|
||||
EOF
|
||||
chmod +x ${monit_script_path}
|
||||
|
||||
cat >${monit_script_path}_logrotate <<EOF
|
||||
#!/bin/bash
|
||||
case \$1 in
|
||||
|
||||
rotate_tendermint_logs)
|
||||
/bin/cp \$2 \$2.\$(date +%y-%m-%d)
|
||||
/bin/tar -cvf \$2.\$(date +%Y%m%d_%H%M%S).tar.gz \$2.\$(date +%y-%m-%d)
|
||||
/bin/rm \$2.\$(date +%y-%m-%d)
|
||||
/bin/cp /dev/null \$2
|
||||
;;
|
||||
|
||||
esac
|
||||
exit 0
|
||||
EOF
|
||||
chmod +x ${monit_script_path}_logrotate
|
||||
|
||||
# Handling overwriting of control file interactively
|
||||
if [ -f "$monitrc_path" ]; then
|
||||
echo "$monitrc_path already exists."
|
||||
read -p "Overwrite[Y]? " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "Overriding $monitrc_path/.monitrc"
|
||||
else
|
||||
read -p "Enter absolute path to store Monit control file: " monitrc_path
|
||||
eval monitrc_path="$monitrc_path"
|
||||
if [ ! -d "$(dirname $monitrc_path)" ]; then
|
||||
echo "Failed to save monit control file '$monitrc_path': No such file or directory."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
echo "$monitrc_path already exists."
|
||||
read -p "Overwrite[Y]? " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "Overriding $monitrc_path"
|
||||
else
|
||||
read -p "Enter absolute path to store Monit control file: " monitrc_path
|
||||
eval monitrc_path="$monitrc_path"
|
||||
if [ ! -d "$(dirname $monitrc_path)" ]; then
|
||||
echo "Failed to save monit control file '$monitrc_path': No such file or directory."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# configure monitrc
|
||||
@ -161,9 +178,18 @@ check process bigchaindb
|
||||
check process tendermint
|
||||
with pidfile ${monit_pid_path}/tendermint.pid
|
||||
start program "${monit_script_path} start_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
|
||||
restart program "${monit_script_path} start_bigchaindb ${monit_pid_path}/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
|
||||
restart program "${monit_script_path} start_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
|
||||
stop program "${monit_script_path} stop_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
|
||||
depends on bigchaindb
|
||||
|
||||
check file tendermint.out.log with path ${monit_log_path}/tendermint.out.log
|
||||
if size > 200 MB then
|
||||
exec "${monit_script_path}_logrotate rotate_tendermint_logs ${monit_log_path}/tendermint.out.log $monit_pid_path/tendermint.pid"
|
||||
|
||||
check file tendermint.err.log with path ${monit_log_path}/tendermint.err.log
|
||||
if size > 200 MB then
|
||||
exec "${monit_script_path}_logrotate rotate_tendermint_logs ${monit_log_path}/tendermint.err.log $monit_pid_path/tendermint.pid"
|
||||
|
||||
EOF
|
||||
|
||||
# Setting permissions for control file
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user