Mirroristas/bigchaindb
2
0
Fork 0
mirror of https://github.com/bigchaindb/bigchaindb.git synced 2024-10-13 13:34:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into sign-tx-body

Browse Source
This commit is contained in:
Scott Sadler
2017-04-24 13:28:02 +02:00
parent bae05e80a0 4a9342ec57
commit 2c6370f42e
193 changed files with 7121 additions and 2219 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
docs/server/source/_static/Node-components.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 82 KiB

After

Width:  |  Height:  |  Size: 38 KiB

19
docs/server/source/appendices/aws-setup.md
View File

@@ -18,7 +18,7 @@ pip install awscli
## Create an AWS Access Key
The next thing you'll need is an AWS access key. If you don't have one, you can create one using the [instructions in the AWS documentation](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html). You should get an access key ID (e.g. AKIAIOSFODNN7EXAMPLE) and a secret access key (e.g. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
The next thing you'll need is AWS access keys (access key ID and secret access key). If you don't have those, see [the AWS documentation about access keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
You should also pick a default AWS region name (e.g. `eu-central-1`). That's where your cluster will run. The AWS documentation has [a list of them](http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region).
@@ -42,23 +42,10 @@ This writes two files: `~/.aws/credentials` and `~/.aws/config`. AWS tools and p
Eventually, you'll have one or more instances (virtual machines) running on AWS and you'll want to SSH to them. To do that, you need a public/private key pair. The public key will be sent to AWS, and you can tell AWS to put it in any instances you provision there. You'll keep the private key on your local workstation.
First you need to make up a key name. Some ideas:
See the [page about how to generate a key pair for SSH](generate-key-pair-for-ssh.html).
* `bcdb-troy-1`
* `bigchaindb-7`
* `bcdb-jupiter`
If you already have key pairs on AWS (Amazon EC2), you have to pick a name that's not already being used.
Below, replace every instance of `<key-name>` with your actual key name.
To generate a public/private RSA key pair with that name:
```text
ssh-keygen -t rsa -C "<key-name>" -f ~/.ssh/<key-name>
```
It will ask you for a passphrase. You can use whatever passphrase you like, but don't lose it. Two keys (files) will be created in `~/.ssh/`:
1. `~/.ssh/<key-name>.pub` is the public key
2. `~/.ssh/<key-name>` is the private key
## Send the Public Key to AWS
To send the public key to AWS, use the AWS Command-Line Interface:
```text

18
docs/server/source/appendices/commands.rst Normal file
View File

@@ -0,0 +1,18 @@
######################
Command Line Interface
######################
.. automodule:: bigchaindb.commands
:special-members: __init__
:mod:`bigchaindb.commands.bigchaindb`
-------------------------------------
.. automodule:: bigchaindb.commands.bigchaindb
:mod:`bigchaindb.commands.utils`
--------------------------------
.. automodule:: bigchaindb.commands.utils

5
docs/server/source/appendices/consensus.rst
View File

@@ -1,5 +0,0 @@
#########
Consensus
#########
.. automodule:: bigchaindb.consensus

101
docs/server/source/appendices/docker-on-mac.md Normal file
View File

@@ -0,0 +1,101 @@
# Run BigchainDB with Docker On Mac
**NOT for Production Use**
Those developing on Mac can follow this document to run BigchainDB in docker
containers for a quick dev setup.
Running BigchainDB on Mac (Docker or otherwise) is not officially supported.
Support is very much limited as there are certain things that work differently
in Docker for Mac than Docker for other platforms.
Also, we do not use mac for our development and testing. :)
This page may not be up to date with various settings and docker updates at
all the times.
These steps work as of this writing (2017.Mar.09) and might break in the
future with updates to Docker for mac.
Community contribution to make BigchainDB run on Docker for Mac will always be
welcome.
## Prerequisite
Install Docker for Mac.
## (Optional) For a clean start
1. Stop all BigchainDB and RethinkDB/MongoDB containers.
2. Delete all BigchainDB docker images.
3. Delete the ~/bigchaindb_docker folder.
## Pull the images
Pull the bigchaindb and other required docker images from docker hub.
```text
docker pull bigchaindb/bigchaindb:master
docker pull [rethinkdb:2.3|mongo:3.4.1]
```
## Create the BigchainDB configuration file on Mac
```text
docker run \
--rm \
--volume $HOME/bigchaindb_docker:/data \
bigchaindb/bigchaindb:master \
-y configure \
[mongodb|rethinkdb]
```
To ensure that BigchainDB connects to the backend database bound to the virtual
interface `172.17.0.1`, you must edit the BigchainDB configuration file
(`~/bigchaindb_docker/.bigchaindb`) and change database.host from `localhost`
to `172.17.0.1`.
## Run the backend database on Mac
From v0.9 onwards, you can run RethinkDB or MongoDB.
We use the virtual interface created by the Docker daemon to allow
communication between the BigchainDB and database containers.
It has an IP address of 172.17.0.1 by default.
You can also use docker host networking or bind to your primary (eth)
interface, if needed.
### For RethinkDB backend
```text
docker run \
--name=rethinkdb \
--publish=28015:28015 \
--publish=8080:8080 \
--restart=always \
--volume $HOME/bigchaindb_docker:/data \
rethinkdb:2.3
```
### For MongoDB backend
```text
docker run \
--name=mongodb \
--publish=27017:27017 \
--restart=always \
--volume=$HOME/bigchaindb_docker/db:/data/db \
--volume=$HOME/bigchaindb_docker/configdb:/data/configdb \
mongo:3.4.1 --replSet=bigchain-rs
```
### Run BigchainDB on Mac
```text
docker run \
--name=bigchaindb \
--publish=9984:9984 \
--restart=always \
--volume=$HOME/bigchaindb_docker:/data \
bigchaindb/bigchaindb \
start
```

25
docs/server/source/appendices/example-rethinkdb-storage-setups.md
View File

@@ -1,25 +0,0 @@
# Example RethinkDB Storage Setups
## Example Amazon EC2 Setups
We have some scripts for [deploying a _test_ BigchainDB cluster on AWS](../clusters-feds/aws-testing-cluster.html). Those scripts include command sequences to set up storage for RethinkDB.
In particular, look in the file [/deploy-cluster-aws/fabfile.py](https://github.com/bigchaindb/bigchaindb/blob/master/deploy-cluster-aws/fabfile.py), under `def prep_rethinkdb_storage(USING_EBS)`. Note that there are two cases:
1. **Using EBS ([Amazon Elastic Block Store](https://aws.amazon.com/ebs/)).** This is always an option, and for some instance types ("EBS-only"), it's the only option.
2. **Using an "instance store" volume provided with an Amazon EC2 instance.** Note that our scripts only use one of the (possibly many) volumes in the instance store.
There's some explanation of the steps in the [Amazon EC2 documentation about making an Amazon EBS volume available for use](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html).
You shouldn't use an EC2 "instance store" to store RethinkDB data for a production node, because it's not replicated and it's only intended for temporary, ephemeral data. If the associated instance crashes, is stopped, or is terminated, the data in the instance store is lost forever. Amazon EBS storage is replicated, has incremental snapshots, and is low-latency.
## Example Using Amazon EFS
TODO
## Other Examples?
TODO
Maybe RAID, ZFS, ... (over EBS volumes, i.e. a DIY Amazon EFS)

12
docs/server/source/appendices/firewall-notes.md
View File

@@ -8,9 +8,10 @@ This is a page of notes on the ports potentially used by BigchainDB nodes and th
Assuming you aren't exposing the RethinkDB web interface on port 8080 (or any other port, because [there are more secure ways to access it](https://www.rethinkdb.com/docs/security/#binding-the-web-interface-port)), there are only three ports that should expect unsolicited inbound traffic:
1. **Port 22** can expect inbound SSH (TCP) traffic from the node administrator (i.e. a small set of IP addresses).
2. **Port 9984** can expect inbound HTTP (TCP) traffic from BigchainDB clients sending transactions to the BigchainDB HTTP API.
3. If you're using RethinkDB, **Port 29015** can expect inbound TCP traffic from other RethinkDB nodes in the RethinkDB cluster (for RethinkDB intracluster communications).
4. If you're using MongoDB, **Port 27017** can expect inbound TCP traffic from other nodes.
1. **Port 9984** can expect inbound HTTP (TCP) traffic from BigchainDB clients sending transactions to the BigchainDB HTTP API.
1. **Port 9985** can expect inbount WebSocket traffic from BigchainDB clients.
1. If you're using RethinkDB, **Port 29015** can expect inbound TCP traffic from other RethinkDB nodes in the RethinkDB cluster (for RethinkDB intracluster communications).
1. If you're using MongoDB, **Port 27017** can expect inbound TCP traffic from other nodes.
All other ports should only get inbound traffic in response to specific requests from inside the node.
@@ -59,6 +60,11 @@ If Gunicorn and the reverse proxy are running on the same server, then you'll ha
You may want to have Gunicorn and the reverse proxy running on different servers, so that both can listen on port 9984. That would also help isolate the effects of a denial-of-service attack.
## Port 9985
Port 9985 is the default port for the [BigchainDB WebSocket Event Stream API](../websocket-event-stream-api.html).
## Port 28015
Port 28015 is the default port used by RethinkDB client driver connections (TCP). If your BigchainDB node is just one server, then Port 28015 only needs to listen on localhost, because all the client drivers will be running on localhost. Port 28015 doesn't need to accept inbound traffic from the outside world.

34
docs/server/source/appendices/generate-key-pair-for-ssh.md Normal file
View File

@@ -0,0 +1,34 @@
# Generate a Key Pair for SSH
This page describes how to use `ssh-keygen`
to generate a public/private RSA key pair
that can be used with SSH.
(Note: `ssh-keygen` is found on most Linux and Unix-like
operating systems; if you're using Windows,
then you'll have to use another tool,
such as PuTTYgen.)
By convention, SSH key pairs get stored in the `~/.ssh/` directory.
Check what keys you already have there:
```text
ls -1 ~/.ssh/
```
Next, make up a new key pair name (called `<name>` below).
Here are some ideas:
* `aws-bdb-2`
* `tim-bdb-azure`
* `chris-bcdb-key`
Next, generate a public/private RSA key pair with that name:
```text
ssh-keygen -t rsa -C "<name>" -f ~/.ssh/<name>
```
It will ask you for a passphrase.
You can use whatever passphrase you like, but don't lose it.
Two keys (files) will be created in `~/.ssh/`:
1. `~/.ssh/<name>.pub` is the public key
2. `~/.ssh/<name>` is the private key

7
docs/server/source/appendices/index.rst
View File

@@ -10,15 +10,18 @@ Appendices
install-os-level-deps
install-latest-pip
run-with-docker
docker-on-mac
json-serialization
cryptography
the-Bigchain-class
consensus
pipelines
backend
commands
aws-setup
generate-key-pair-for-ssh
firewall-notes
ntp-notes
example-rethinkdb-storage-setups
rethinkdb-reqs
rethinkdb-backup
licenses
install-with-lxd

2
docs/server/source/appendices/json-serialization.md
View File

@@ -24,7 +24,7 @@ deserialize(serialize(data)) == data
True
```
Since BigchainDB performs a lot of serialization we decided to use [python-rapidjson](https://github.com/kenrobbins/python-rapidjson)
Since BigchainDB performs a lot of serialization we decided to use [python-rapidjson](https://github.com/python-rapidjson/python-rapidjson)
which is a python wrapper for [rapidjson](https://github.com/miloyip/rapidjson) a fast and fully RFC complient JSON parser.
```python

8
docs/server/source/clusters-feds/backup.md → docs/server/source/appendices/rethinkdb-backup.md
View File

@@ -1,6 +1,6 @@
# Backing Up & Restoring Data
# Backing Up and Restoring Data
There are several ways to backup and restore the data in a BigchainDB cluster.
This page was written when BigchainDB only worked with RethinkDB, so its focus is on RethinkDB-based backup. BigchainDB now supports MongoDB as a backend database and we recommend that you use MongoDB in production. Nevertheless, some of the following backup ideas are still relevant regardless of the backend database being used, so we moved this page to the Appendices.
## RethinkDB's Replication as a form of Backup
@@ -64,7 +64,7 @@ In the future, it will be possible for clients to query for the blocks containin
**How could we be sure blocks and votes from a client are valid?**
All blocks and votes are signed by federation nodes. Only federation nodes can produce valid signatures because only federation nodes have the necessary private keys. A client can't produce a valid signature for a block or vote.
All blocks and votes are signed by cluster nodes (owned and operated by consortium members). Only cluster nodes can produce valid signatures because only cluster nodes have the necessary private keys. A client can't produce a valid signature for a block or vote.
**Could we restore an entire BigchainDB database using client-saved blocks and votes?**
@@ -109,7 +109,7 @@ Considerations for BigchainDB:
Although it's not advertised as such, RethinkDB's built-in replication feature is similar to continous backup, except the "backup" (i.e. the set of replica shards) is spread across all the nodes. One could take that idea a bit farther by creating a set of backup-only servers with one full backup:
* Give all the original BigchainDB nodes (RethinkDB nodes) the server tag `original`. This is the default if you used the RethinkDB config file suggested in the section titled [Configure RethinkDB Server](../dev-and-test/setup-run-node.html#configure-rethinkdb-server).
* Set up a group of servers running RethinkDB only, and give them the server tag `backup`. The `backup` servers could be geographically separated from all the `original` nodes (or not; it's up to the federation).
* Set up a group of servers running RethinkDB only, and give them the server tag `backup`. The `backup` servers could be geographically separated from all the `original` nodes (or not; it's up to the consortium to decide).
* Clients shouldn't be able to read from or write to servers in the `backup` set.
* Send a RethinkDB reconfigure command to the RethinkDB cluster to make it so that the `original` set has the same number of replicas as before (or maybe one less), and the `backup` set has one replica. Also, make sure the `primary_replica_tag='original'` so that all primary shards live on the `original` nodes.

32
docs/server/source/nodes/node-requirements.md → docs/server/source/appendices/rethinkdb-reqs.md
View File

@@ -1,20 +1,8 @@
# Production Node Requirements
# RethinkDB Requirements
Note: This section will be broken apart into several pages, e.g. NTP requirements, RethinkDB requirements, BigchainDB requirements, etc. and those pages will add more details.
[The RethinkDB documentation](https://rethinkdb.com/docs/) should be your first source of information about its requirements. This page serves mostly to document some of its more obscure requirements.
## OS Requirements
* RethinkDB Server [will run on any modern OS](https://www.rethinkdb.com/docs/install/). Note that the Fedora package isn't officially supported. Also, official support for Windows is fairly recent ([April 2016](https://rethinkdb.com/blog/2.3-release/)).
* BigchainDB Server requires Python 3.4+ and Python 3.4+ [will run on any modern OS](https://docs.python.org/3.4/using/index.html).
* BigchaindB Server uses the Python `multiprocessing` package and [some functionality in the `multiprocessing` package doesn't work on OS X](https://docs.python.org/3.4/library/multiprocessing.html#multiprocessing.Queue.qsize). You can still use Mac OS X if you use Docker or a virtual machine.
The BigchainDB core dev team uses recent LTS versions of Ubuntu and recent versions of Fedora.
We don't test BigchainDB on Windows or Mac OS X, but you can try.
* If you run into problems on Windows, then you may want to try using Vagrant. One of our community members ([@Mec-Is](https://github.com/Mec-iS)) wrote [a page about how to install BigchainDB on a VM with Vagrant](https://gist.github.com/Mec-iS/b84758397f1b21f21700).
* If you have Mac OS X and want to experiment with BigchainDB, then you could do that [using Docker](../appendices/run-with-docker.html).
RethinkDB Server [will run on any modern OS](https://www.rethinkdb.com/docs/install/). Note that the Fedora package isn't officially supported. Also, official support for Windows is fairly recent ([April 2016](https://rethinkdb.com/blog/2.3-release/)).
## Storage Requirements
@@ -28,6 +16,20 @@ For RethinkDB's failover mechanisms to work, [every RethinkDB table must have at
As for the read & write rates, what do you expect those to be for your situation? It's not enough for the storage system alone to handle those rates: the interconnects between the nodes must also be able to handle them.
**Storage Notes Specific to RethinkDB**
* The RethinkDB storage engine has a number of SSD optimizations, so you _can_ benefit from using SSDs. ([source](https://www.rethinkdb.com/docs/architecture/))
* If you have an N-node RethinkDB cluster and 1) you want to use it to store an amount of data D (unique records, before replication), 2) you want the replication factor to be R (all tables), and 3) you want N shards (all tables), then each BigchainDB node must have storage space of at least R×D/N.
* RethinkDB tables can have [at most 64 shards](https://rethinkdb.com/limitations/). What does that imply? Suppose you only have one table, with 64 shards. How big could that table be? It depends on how much data can be stored in each node. If the maximum amount of data that a node can store is d, then the biggest-possible shard is d, and the biggest-possible table size is 64 times that. (All shard replicas would have to be stored on other nodes beyond the initial 64.) If there are two tables, the second table could also have 64 shards, stored on 64 other maxed-out nodes, so the total amount of unique data in the database would be (64 shards/table)×(2 tables)×d. In general, if you have T tables, the maximum amount of unique data that can be stored in the database (i.e. the amount of data before replication) is 64×T×d.
* When you set up storage for your RethinkDB data, you may have to select a filesystem. (Sometimes, the filesystem is already decided by the choice of storage.) We recommend using a filesystem that supports direct I/O (Input/Output). Many compressed or encrypted file systems don't support direct I/O. The ext4 filesystem supports direct I/O (but be careful: if you enable the data=journal mode, then direct I/O support will be disabled; the default is data=ordered). If your chosen filesystem supports direct I/O and you're using Linux, then you don't need to do anything to request or enable direct I/O. RethinkDB does that.
<p style="background-color: lightgrey;">What is direct I/O? It allows RethinkDB to write directly to the storage device (or use its own in-memory caching mechanisms), rather than relying on the operating system's file read and write caching mechanisms. (If you're using Linux, a write-to-file normally writes to the in-memory Page Cache first; only later does that Page Cache get flushed to disk. The Page Cache is also used when reading files.)</p>
* RethinkDB stores its data in a specific directory. You can tell RethinkDB _which_ directory using the RethinkDB config file, as explained below. In this documentation, we assume the directory is `/data`. If you set up a separate device (partition, RAID array, or logical volume) to store the RethinkDB data, then mount that device on `/data`.
## Memory (RAM) Requirements

58
docs/server/source/appendices/run-with-docker.md
View File

@@ -25,7 +25,7 @@ docker run \
--interactive \
--rm \
--tty \
--volume "$HOME/bigchaindb_docker:/data" \
--volume $HOME/bigchaindb_docker:/data \
bigchaindb/bigchaindb \
-y configure \
[mongodb|rethinkdb]
@@ -45,7 +45,7 @@ Let's analyze that command:
`$HOME/bigchaindb_docker` to the container directory `/data`;
this allows us to have the data persisted on the host machine,
you can read more in the [official Docker
documentation](https://docs.docker.com/engine/tutorials/dockervolumes/#/mount-a-host-directory-as-a-data-volume)
documentation](https://docs.docker.com/engine/tutorials/dockervolumes)
* `bigchaindb/bigchaindb` the image to use. All the options after the container name are passed on to the entrypoint inside the container.
* `-y configure` execute the `configure` sub-command (of the `bigchaindb`
command) inside the container, with the `-y` option to automatically use all the default config values
@@ -75,21 +75,37 @@ docker run \
--name=rethinkdb \
--publish=172.17.0.1:28015:28015 \
--publish=172.17.0.1:58080:8080 \
--restart=always \
--volume $HOME/bigchaindb_docker:/data \
rethinkdb:2.3
```
<!-- Don't hyperlink http://172.17.0.1:58080/ because Sphinx will fail when you do "make linkcheck" -->
You can also access the RethinkDB dashboard at
[http://172.17.0.1:58080/](http://172.17.0.1:58080/)
You can also access the RethinkDB dashboard at http://172.17.0.1:58080/
#### For MongoDB
Note: MongoDB runs as user `mongodb` which had the UID `999` and GID `999`
inside the container. For the volume to be mounted properly, as user `mongodb`
in your host, you should have a `mongodb` user with UID and GID `999`.
If you have another user on the host with UID `999`, the mapped files will
be owned by this user in the host.
If there is no owner with UID 999, you can create the corresponding user and
group.
`useradd -r --uid 999 mongodb` OR `groupadd -r --gid 999 mongodb && useradd -r --uid 999 -g mongodb mongodb` should work.
```text
docker run \
--detach \
--name=mongodb \
--publish=172.17.0.1:27017:27017 \
--restart=always \
--volume=/tmp/mongodb_docker/db:/data/db \
--volume=/tmp/mongodb_docker/configdb:/data/configdb \
mongo:3.4.1 --replSet=bigchain-rs
```
@@ -100,6 +116,7 @@ docker run \
--detach \
--name=bigchaindb \
--publish=59984:9984 \
--restart=always \
--volume=$HOME/bigchaindb_docker:/data \
bigchaindb/bigchaindb \
start
@@ -123,38 +140,6 @@ machine running the Docker engine. If you are running docker-machine (e.g. on
Mac OS X) this will be the IP of the Docker machine (`docker-machine ip
machine_name`).
### Load Testing with Docker
Now that we have BigchainDB running in the Docker container named `bigchaindb`, we can
start another BigchainDB container to generate a load test for it.
First, make sure the container named `bigchaindb` is still running. You can check that using:
```text
docker ps
```
You should see a container named `bigchaindb` in the list.
You can load test the BigchainDB running in that container by running the `bigchaindb load` command in a second container:
```text
docker run \
--env BIGCHAINDB_DATABASE_HOST=bigchaindb \
--link bigchaindb \
--rm \
--volume "$HOME/bigchaindb_docker:/data" \
bigchaindb/bigchaindb \
load
```
Note the `--link` option to link to the first container (named `bigchaindb`).
Aside: The `bigchaindb load` command has several options (e.g. `-m`). You can read more about it in [the documentation about the BigchainDB command line interface](../server-reference/bigchaindb-cli.html).
If you look at the RethinkDB dashboard (in your web browser), you should see the effects of the load test. You can also see some effects in the Docker logs using:
```text
docker logs -f bigchaindb
```
## Building Your Own Image
@@ -171,3 +156,4 @@ docker build --tag local-bigchaindb .
```
Now you can use your own image to run BigchainDB containers.

178
docs/server/source/cloud-deployment-templates/add-node-on-kubernetes.rst Normal file
View File

@@ -0,0 +1,178 @@
Kubernetes Template: Add a BigchainDB Node to an Existing BigchainDB Cluster
============================================================================
This page describes how to deploy a BigchainDB node using Kubernetes,
and how to add that node to an existing BigchainDB cluster.
It assumes you already have a running Kubernetes cluster
where you can deploy the new BigchainDB node.
If you want to deploy the first BigchainDB node in a BigchainDB cluster,
or a stand-alone BigchainDB node,
then see :doc:`the page about that <node-on-kubernetes>`.
Terminology Used
----------------
``existing cluster`` will refer to one of the existing Kubernetes clusters
hosting one of the existing BigchainDB nodes.
``ctx-1`` will refer to the kubectl context of the existing cluster.
``new cluster`` will refer to the new Kubernetes cluster that will run a new
BigchainDB node (including a BigchainDB instance and a MongoDB instance).
``ctx-2`` will refer to the kubectl context of the new cluster.
``new MongoDB instance`` will refer to the MongoDB instance in the new cluster.
``existing MongoDB instance`` will refer to the MongoDB instance in the
existing cluster.
``new BigchainDB instance`` will refer to the BigchainDB instance in the new
cluster.
``existing BigchainDB instance`` will refer to the BigchainDB instance in the
existing cluster.
Step 1: Prerequisites
---------------------
* A public/private key pair for the new BigchainDB instance.
* The public key should be shared offline with the other existing BigchainDB
nodes in the existing BigchainDB cluster.
* You will need the public keys of all the existing BigchainDB nodes.
* A new Kubernetes cluster setup with kubectl configured to access it.
* Some familiarity with deploying a BigchainDB node on Kubernetes.
See our :doc:`other docs about that <node-on-kubernetes>`.
Note: If you are managing multiple Kubernetes clusters, from your local
system, you can run ``kubectl config view`` to list all the contexts that
are available for the local kubectl.
To target a specific cluster, add a ``--context`` flag to the kubectl CLI. For
example:
.. code:: bash
$ kubectl --context ctx-1 apply -f example.yaml
$ kubectl --context ctx-2 apply -f example.yaml
$ kubectl --context ctx-1 proxy --port 8001
$ kubectl --context ctx-2 proxy --port 8002
Step 2: Prepare the New Kubernetes Cluster
------------------------------------------
Follow the steps in the sections to set up Storage Classes and Persistent Volume
Claims, and to run MongoDB in the new cluster:
1. :ref:`Add Storage Classes <Step 3: Create Storage Classes>`
2. :ref:`Add Persistent Volume Claims <Step 4: Create Persistent Volume Claims>`
3. :ref:`Create the Config Map <Step 5: Create the Config Map - Optional>`
4. :ref:`Run MongoDB instance <Step 6: Run MongoDB as a StatefulSet>`
Step 3: Add the New MongoDB Instance to the Existing Replica Set
----------------------------------------------------------------
Note that by ``replica set``, we are referring to the MongoDB replica set,
not a Kubernetes' ``ReplicaSet``.
If you are not the administrator of an existing BigchainDB node, you
will have to coordinate offline with an existing administrator so that they can
add the new MongoDB instance to the replica set.
Add the new instance of MongoDB from an existing instance by accessing the
``mongo`` shell.
.. code:: bash
$ kubectl --context ctx-1 exec -it mdb-0 -c mongodb -- /bin/bash
root@mdb-0# mongo --port 27017
One can only add members to a replica set from the ``PRIMARY`` instance.
The ``mongo`` shell prompt should state that this is the primary member in the
replica set.
If not, then you can use the ``rs.status()`` command to find out who the
primary is and login to the ``mongo`` shell in the primary.
Run the ``rs.add()`` command with the FQDN and port number of the other instances:
.. code:: bash
PRIMARY> rs.add("<fqdn>:<port>")
Step 4: Verify the Replica Set Membership
-----------------------------------------
You can use the ``rs.conf()`` and the ``rs.status()`` commands available in the
mongo shell to verify the replica set membership.
The new MongoDB instance should be listed in the membership information
displayed.
Step 5: Start the New BigchainDB Instance
-----------------------------------------
Get the file ``bigchaindb-dep.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/bigchaindb/bigchaindb-dep.yaml
Note that we set the ``BIGCHAINDB_DATABASE_HOST`` to ``mdb`` which is the name
of the MongoDB service defined earlier.
Edit the ``BIGCHAINDB_KEYPAIR_PUBLIC`` with the public key of this instance,
the ``BIGCHAINDB_KEYPAIR_PRIVATE`` with the private key of this instance and
the ``BIGCHAINDB_KEYRING`` with a ``:`` delimited list of all the public keys
in the BigchainDB cluster.
Create the required Deployment using:
.. code:: bash
$ kubectl --context ctx-2 apply -f bigchaindb-dep.yaml
You can check its status using the command ``kubectl get deploy -w``
Step 6: Restart the Existing BigchainDB Instance(s)
---------------------------------------------------
Add the public key of the new BigchainDB instance to the keyring of all the
existing BigchainDB instances and update the BigchainDB instances using:
.. code:: bash
$ kubectl --context ctx-1 replace -f bigchaindb-dep.yaml
This will create a "rolling deployment" in Kubernetes where a new instance of
BigchainDB will be created, and if the health check on the new instance is
successful, the earlier one will be terminated. This ensures that there is
zero downtime during updates.
You can SSH to an existing BigchainDB instance and run the ``bigchaindb
show-config`` command to check that the keyring is updated.
Step 7: Run NGINX as a Deployment
---------------------------------
Please refer :ref:`this <Step 10: Run NGINX as a Deployment>` to
set up NGINX in your new node.
Step 8: Test Your New BigchainDB Node
-------------------------------------
Please refer to the testing steps :ref:`here <Step 11: Verify the BigchainDB
Node Setup>` to verify that your new BigchainDB node is working as expected.

454
docs/server/source/cloud-deployment-templates/first-node.rst Normal file
View File

@@ -0,0 +1,454 @@
First Node or Bootstrap Node Setup
==================================
This document is a work in progress and will evolve over time to include
security, websocket and other settings.
Step 1: Set Up the Cluster
--------------------------
.. code:: bash
az group create --name bdb-test-cluster-0 --location westeurope --debug --output json
ssh-keygen -t rsa -C "k8s-bdb-test-cluster-0" -f ~/.ssh/k8s-bdb-test-cluster-0
az acs create --name k8s-bdb-test-cluster-0 \
--resource-group bdb-test-cluster-0 \
--master-count 3 \
--agent-count 2 \
--admin-username ubuntu \
--agent-vm-size Standard_D2_v2 \
--dns-prefix k8s-bdb-test-cluster-0 \
--ssh-key-value ~/.ssh/k8s-bdb-test-cluster-0.pub \
--orchestrator-type kubernetes \
--debug --output json
az acs kubernetes get-credentials \
--resource-group bdb-test-cluster-0 \
--name k8s-bdb-test-cluster-0 \
--debug --output json
echo -e "Host k8s-bdb-test-cluster-0.westeurope.cloudapp.azure.com\n ForwardAgent yes" >> ~/.ssh/config
Step 2: Connect to the Cluster UI - (optional)
----------------------------------------------
* Get the kubectl context for this cluster using ``kubectl config view``.
* For the above commands, the context would be ``k8s-bdb-test-cluster-0``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 proxy -p 8001
Step 3. Configure the Cluster
-----------------------------
* Use the ConfigMap in ``configuration/config-map.yaml`` file for configuring
the cluster.
* Log in the the MongoDB Cloud Manager and select the group that will monitor
and backup this cluster from the dropdown box.
* Go to Settings, Group Settings and copy the ``Agent Api Key``.
* Replace the ``<api key here>`` field with this key.
* Since this is the first node of the cluster, ensure that the ``data.fqdn``
field has the value ``mdb-instance-0``.
* We only support the value ``all`` in the ``data.allowed-hosts`` field for now.
* Create the ConfigMap
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f configuration/config-map.yaml
Step 4. Start the NGINX Service
-------------------------------
* This will will give us a public IP for the cluster.
* Once you complete this step, you might need to wait up to 10 mins for the
public IP to be assigned.
* You have the option to use vanilla NGINX or an OpenResty NGINX integrated
with 3scale API Gateway.
Step 4.1. Vanilla NGINX
^^^^^^^^^^^^^^^^^^^^^^^
* This configuration is located in the file ``nginx/nginx-svc.yaml``.
* Since this is the first node, rename ``metadata.name`` and ``metadata.labels.name``
to ``ngx-instance-0``, and ``spec.selector.app`` to ``ngx-instance-0-dep``.
* Start the Kubernetes Service:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f nginx/nginx-svc.yaml
Step 4.2. OpenResty NGINX + 3scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* You have to enable HTTPS for this one and will need an HTTPS certificate
for your domain
* Assuming that the public key chain is named ``cert.pem`` and private key is
``cert.key``, run the following commands to encode the certificates into
single continuous string that can be embedded in yaml.
.. code:: bash
cat cert.pem | base64 -w 0 > cert.pem.b64
cat cert.key | base64 -w 0 > cert.key.b64
* Copy the contents of ``cert.pem.b64`` in the ``cert.pem`` field, and the
contents of ``cert.key.b64`` in the ``cert.key`` field in the file
``nginx-3scale/nginx-3scale-secret.yaml``
* Create the Kubernetes Secret:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-secret.yaml
* Since this is the first node, rename ``metadata.name`` and ``metadata.labels.name``
to ``ngx-instance-0``, and ``spec.selector.app`` to ``ngx-instance-0-dep`` in
``nginx-3scale/nginx-3scale-svc.yaml`` file.
* Start the Kubernetes Service:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-svc.yaml
Step 5. Assign DNS Name to the NGINX Public IP
----------------------------------------------
* The following command can help you find out if the nginx service strated above
has been assigned a public IP or external IP address:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 get svc -w
* Once a public IP is assigned, you can log in to the Azure portal and map it to
a DNS name.
* We usually start with bdb-test-cluster-0, bdb-test-cluster-1 and so on.
* Let us assume that we assigned the unique name of ``bdb-test-cluster-0`` here.
Step 6. Start the Mongo Kubernetes Service
------------------------------------------
* Change ``metadata.name`` and ``metadata.labels.name`` to
``mdb-instance-0``, and ``spec.selector.app`` to ``mdb-instance-0-ss``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb/mongo-svc.yaml
Step 7. Start the BigchainDB Kubernetes Service
-----------------------------------------------
* Change ``metadata.name`` and ``metadata.labels.name`` to
``bdb-instance-0``, and ``spec.selector.app`` to ``bdb-instance-0-dep``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f bigchaindb/bigchaindb-svc.yaml
Step 8. Start the NGINX Kubernetes Deployment
---------------------------------------------
* As in step 4, you have the option to use vanilla NGINX or an OpenResty NGINX
integrated with 3scale API Gateway.
Step 8.1. Vanilla NGINX
^^^^^^^^^^^^^^^^^^^^^^^
* This configuration is located in the file ``nginx/nginx-dep.yaml``.
* Since this is the first node, change the ``metadata.name`` and
``spec.template.metadata.labels.app`` to ``ngx-instance-0-dep``.
* Set ``MONGODB_BACKEND_HOST`` env var to
``mdb-instance-0.default.svc.cluster.local``.
* Set ``BIGCHAINDB_BACKEND_HOST`` env var to
``bdb-instance-0.default.svc.cluster.local``.
* Set ``MONGODB_FRONTEND_PORT`` to
``$(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_MDB_PORT)``.
* Set ``BIGCHAINDB_FRONTEND_PORT`` to
``$(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_BDB_PORT)``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f nginx/nginx-dep.yaml
Step 8.2. OpenResty NGINX + 3scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* This configuration is located in the file
``nginx-3scale/nginx-3scale-dep.yaml``.
* Since this is the first node, change the metadata.name and
spec.template.metadata.labels.app to ``ngx-instance-0-dep``.
* Set ``MONGODB_BACKEND_HOST`` env var to
``mdb-instance-0.default.svc.cluster.local``.
* Set ``BIGCHAINDB_BACKEND_HOST`` env var to
``bdb-instance-0.default.svc.cluster.local``.
* Set ``MONGODB_FRONTEND_PORT`` to
``$(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_MDB_PORT)``.
* Set ``BIGCHAINDB_FRONTEND_PORT`` to
``$(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_BDB_PORT)``.
* Also, replace the placeholder strings for the env vars with the values
obtained from 3scale. You will need the Secret Token, Service ID, Version Header
and Provider Key from 3scale.
* The ``THREESCALE_FRONTEND_API_DNS_NAME`` will be DNS name registered for your
HTTPS certificate.
* You can set the ``THREESCALE_UPSTREAM_API_PORT`` to any port other than 9984,
9985, 443, 8888 and 27017. We usually use port ``9999``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-dep.yaml
Step 9. Create a Kubernetes Storage Class for MongoDB
-----------------------------------------------------
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb/mongo-sc.yaml
Step 10. Create a Kubernetes PersistentVolumeClaim
--------------------------------------------------
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb/mongo-pvc.yaml
Step 11. Start a Kubernetes StatefulSet for MongoDB
---------------------------------------------------
* Change ``spec.serviceName`` to ``mdb-instance-0``.
* Change the ``metadata.name``, ``template.metadata.name`` and
``template.metadata.labels.app`` to ``mdb-instance-0-ss``.
* It might take up to 10 minutes for the disks to be created and attached to
the pod.
* The UI might show that the pod has errored with the
message "timeout expired waiting for volumes to attach/mount".
* Use the CLI below to check the status of the pod in this case,
instead of the UI. This happens due to a bug in Azure ACS.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb/mongo-ss.yaml
* You can check the status of the pod using the command:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 get po -w
Step 12. Start a Kubernetes Deployment for Bigchaindb
-----------------------------------------------------
* Change both ``metadata.name`` and ``spec.template.metadata.labels.app``
to ``bdb-instance-0-dep``.
* Set ``BIGCHAINDB_DATABASE_HOST`` to ``mdb-instance-0``.
* Set the appropriate ``BIGCHAINDB_KEYPAIR_PUBLIC``,
``BIGCHAINDB_KEYPAIR_PRIVATE`` values.
* One way to generate BigchainDB keypair is to run a Python shell with
the command
``from bigchaindb_driver import crypto; crypto.generate_keypair()``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f bigchaindb/bigchaindb-dep.yaml
Step 13. Start a Kubernetes Deployment for MongoDB Monitoring Agent
-------------------------------------------------------------------
* Change both metadata.name and spec.template.metadata.labels.app to
``mdb-mon-instance-0-dep``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb-monitoring-agent/mongo-mon-dep.yaml
* Get the pod name and check its logs:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 get po
kubectl --context k8s-bdb-test-cluster-0 logs -f <pod name>
Step 14. Configure MongoDB Cloud Manager for Monitoring
-------------------------------------------------------
* Open `MongoDB Cloud Manager <https://cloud.mongodb.com>`_.
* Click ``Login`` under ``MongoDB Cloud Manager`` and log in to the Cloud Manager.
* Select the group from the dropdown box on the page.
* Go to Settings, Group Settings and add a Preferred Hostnames regexp as
``^mdb-instance-[0-9]{1,2}$``. It may take up to 5 mins till this setting
is in effect. You may refresh the browser window and verify whether the changes
have been saved or not.
* Next, click the ``Deployment`` tab, and then the ``Manage Existing`` button.
* On the ``Import your deployment for monitoring`` page, enter the hostname as
``mdb-instance-0``, port number as ``27017``, with no authentication and no
TLS/SSL settings.
* Once the deployment is found, click the ``Continue`` button.
This may take about a minute or two.
* Do not add ``Automation Agent`` when given an option to add it.
* Verify on the UI that data is being by the monitoring agent.
Step 15. Start a Kubernetes Deployment for MongoDB Backup Agent
---------------------------------------------------------------
* Change both ``metadata.name`` and ``spec.template.metadata.labels.app``
to ``mdb-backup-instance-0-dep``.
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb-backup-agent/mongo-backup-dep.yaml
* Get the pod name and check its logs:
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 get po
kubectl --context k8s-bdb-test-cluster-0 logs -f <pod name>
Step 16. Configure MongoDB Cloud Manager for Backup
---------------------------------------------------
* Open `MongoDB Cloud Manager <https://cloud.mongodb.com>`_.
* Click ``Login`` under ``MongoDB Cloud Manager`` and log in to the Cloud
Manager.
* Select the group from the dropdown box on the page.
* Click ``Backup`` tab.
* Click on the ``Begin Setup``.
* Click on ``Next``, select the replica set from the dropdown menu.
* Verify the details of your MongoDB instance and click on ``Start`` again.
* It might take up to 5 minutes to start the backup process.
* Verify that data is being backed up on the UI.
Step 17. Verify that the Cluster is Correctly Set Up
----------------------------------------------------
* Start the toolbox container in the cluster
.. code:: bash
kubectl --context k8s-bdb-test-cluster-0 \
run -it toolbox \
--image bigchaindb/toolbox \
--image-pull-policy=Always \
--restart=Never --rm
* Verify MongoDB instance
.. code:: bash
nslookup mdb-instance-0
dig +noall +answer _mdb-port._tcp.mdb-instance-0.default.svc.cluster.local SRV
curl -X GET http://mdb-instance-0:27017
* Verify BigchainDB instance
.. code:: bash
nslookup bdb-instance-0
dig +noall +answer _bdb-port._tcp.bdb-instance-0.default.svc.cluster.local SRV
curl -X GET http://bdb-instance-0:9984
* Verify NGINX instance
.. code:: bash
nslookup ngx-instance-0
dig +noall +answer _ngx-public-mdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
curl -X GET http://ngx-instance-0:27017 # results in curl: (56) Recv failure: Connection reset by peer
dig +noall +answer _ngx-public-bdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
* If you have run the vanilla NGINX instance, run
.. code:: bash
curl -X GET http://ngx-instance-0:80
* If you have the OpenResty NGINX + 3scale instance, run
.. code:: bash
curl -X GET https://ngx-instance-0
* Check the MongoDB monitoring and backup agent on the MOngoDB Coud Manager portal to verify they are working fine.
* Send some transactions to BigchainDB and verify it's up and running!

6
docs/server/source/cloud-deployment-templates/index.rst
View File

@@ -5,7 +5,7 @@ We have some "templates" to deploy a basic, working, but bare-bones BigchainDB n
You don't have to use the tools we use in the templates. You can use whatever tools you prefer.
If you find the cloud deployment templates for nodes helpful, then you may also be interested in our scripts for :doc:`deploying a testing cluster on AWS <../clusters-feds/aws-testing-cluster>` (documented in the Clusters & Federations section).
If you find the cloud deployment templates for nodes helpful, then you may also be interested in our scripts for :doc:`deploying a testing cluster on AWS <../clusters-feds/aws-testing-cluster>` (documented in the Clusters section).
.. toctree::
:maxdepth: 1
@@ -15,4 +15,6 @@ If you find the cloud deployment templates for nodes helpful, then you may also
azure-quickstart-template
template-kubernetes-azure
node-on-kubernetes
add-node-on-kubernetes
upgrade-on-kubernetes
first-node

454
docs/server/source/cloud-deployment-templates/node-on-kubernetes.rst
View File

@@ -1,9 +1,13 @@
Run a BigchainDB Node in a Kubernetes Cluster
=============================================
Kubernetes Template: Deploy a Single BigchainDB Node
====================================================
Assuming you already have a `Kubernetes <https://kubernetes.io/>`_
cluster up and running, this page describes how to run a
BigchainDB node in it.
This page describes how to deploy the first BigchainDB node
in a BigchainDB cluster, or a stand-alone BigchainDB node,
using `Kubernetes <https://kubernetes.io/>`_.
It assumes you already have a running Kubernetes cluster.
If you want to add a new BigchainDB node to an existing BigchainDB cluster,
refer to :doc:`the page about that <add-node-on-kubernetes>`.
Step 1: Install kubectl
@@ -21,7 +25,7 @@ Step 2: Configure kubectl
The default location of the kubectl configuration file is ``~/.kube/config``.
If you don't have that file, then you need to get it.
If you deployed your Kubernetes cluster on Azure
**Azure.** If you deployed your Kubernetes cluster on Azure
using the Azure CLI 2.0 (as per :doc:`our template <template-kubernetes-azure>`),
then you can get the ``~/.kube/config`` file using:
@@ -31,16 +35,442 @@ then you can get the ``~/.kube/config`` file using:
--resource-group <name of resource group containing the cluster> \
--name <ACS cluster name>
If it asks for a password (to unlock the SSH key)
and you enter the correct password,
but you get an error message,
then try adding ``--ssh-key-file ~/.ssh/<name>``
to the above command (i.e. the path to the private key).
Step 3: Run a MongoDB Container
-------------------------------
To start a MongoDB Docker container in a pod on one of the cluster nodes:
Step 3: Create Storage Classes
------------------------------
MongoDB needs somewhere to store its data persistently,
outside the container where MongoDB is running.
Our MongoDB Docker container
(based on the official MongoDB Docker container)
exports two volume mounts with correct
permissions from inside the container:
* The directory where the mongod instance stores its data: ``/data/db``.
There's more explanation in the MongoDB docs about `storage.dbpath <https://docs.mongodb.com/manual/reference/configuration-options/#storage.dbPath>`_.
* The directory where the mongodb instance stores the metadata for a sharded
cluster: ``/data/configdb/``.
There's more explanation in the MongoDB docs about `sharding.configDB <https://docs.mongodb.com/manual/reference/configuration-options/#sharding.configDB>`_.
Explaining how Kubernetes handles persistent volumes,
and the associated terminology,
is beyond the scope of this documentation;
see `the Kubernetes docs about persistent volumes
<https://kubernetes.io/docs/user-guide/persistent-volumes>`_.
The first thing to do is create the Kubernetes storage classes.
**Azure.** First, you need an Azure storage account.
If you deployed your Kubernetes cluster on Azure
using the Azure CLI 2.0
(as per :doc:`our template <template-kubernetes-azure>`),
then the `az acs create` command already created two
storage accounts in the same location and resource group
as your Kubernetes cluster.
Both should have the same "storage account SKU": ``Standard_LRS``.
Standard storage is lower-cost and lower-performance.
It uses hard disk drives (HDD).
LRS means locally-redundant storage: three replicas
in the same data center.
Premium storage is higher-cost and higher-performance.
It uses solid state drives (SSD).
At the time of writing,
when we created a storage account with SKU ``Premium_LRS``
and tried to use that,
the PersistentVolumeClaim would get stuck in a "Pending" state.
For future reference, the command to create a storage account is
`az storage account create <https://docs.microsoft.com/en-us/cli/azure/storage/account#create>`_.
Get the file ``mongo-sc.yaml`` from GitHub using:
.. code:: bash
$ kubectl ?????
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/mongodb/mongo-sc.yaml
You may have to update the ``parameters.location`` field in both the files to
specify the location you are using in Azure.
Create the required storage classes using:
.. code:: bash
$ kubectl apply -f mongo-sc.yaml
Note: The BigchainDB Dashboard can be deployed
as a Docker container, like everything else.
You can check if it worked using ``kubectl get storageclasses``.
**Azure.** Note that there is no line of the form
``storageAccount: <azure storage account name>``
under ``parameters:``. When we included one
and then created a PersistentVolumeClaim based on it,
the PersistentVolumeClaim would get stuck
in a "Pending" state.
Kubernetes just looks for a storageAccount
with the specified skuName and location.
Step 4: Create Persistent Volume Claims
---------------------------------------
Next, you will create two PersistentVolumeClaim objects ``mongo-db-claim`` and
``mongo-configdb-claim``.
Get the file ``mongo-pvc.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/mongodb/mongo-pvc.yaml
Note how there's no explicit mention of Azure, AWS or whatever.
``ReadWriteOnce`` (RWO) means the volume can be mounted as
read-write by a single Kubernetes node.
(``ReadWriteOnce`` is the *only* access mode supported
by AzureDisk.)
``storage: 20Gi`` means the volume has a size of 20
`gibibytes <https://en.wikipedia.org/wiki/Gibibyte>`_.
You may want to update the ``spec.resources.requests.storage`` field in both
the files to specify a different disk size.
Create the required Persistent Volume Claims using:
.. code:: bash
$ kubectl apply -f mongo-pvc.yaml
You can check its status using: ``kubectl get pvc -w``
Initially, the status of persistent volume claims might be "Pending"
but it should become "Bound" fairly quickly.
Step 5: Create the Config Map - Optional
----------------------------------------
This step is required only if you are planning to set up multiple
`BigchainDB nodes
<https://docs.bigchaindb.com/en/latest/terminology.html>`_.
MongoDB reads the local ``/etc/hosts`` file while bootstrapping a replica set
to resolve the hostname provided to the ``rs.initiate()`` command. It needs to
ensure that the replica set is being initialized in the same instance where
the MongoDB instance is running.
To achieve this, you will create a ConfigMap with the FQDN of the MongoDB instance
and populate the ``/etc/hosts`` file with this value so that a replica set can
be created seamlessly.
Get the file ``mongo-cm.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/mongodb/mongo-cm.yaml
You may want to update the ``data.fqdn`` field in the file before creating the
ConfigMap. ``data.fqdn`` field will be the DNS name of your MongoDB instance.
This will be used by other MongoDB instances when forming a MongoDB
replica set. It should resolve to the MongoDB instance in your cluster when
you are done with the setup. This will help when you are adding more MongoDB
instances to the replica set in the future.
**Azure.**
In Kubernetes on ACS, the name you populate in the ``data.fqdn`` field
will be used to configure a DNS name for the public IP assigned to the
Kubernetes Service that is the frontend for the MongoDB instance.
We suggest using a name that will already be available in Azure.
We use ``mdb-instance-0``, ``mdb-instance-1`` and so on in this document,
which gives us ``mdb-instance-0.<azure location>.cloudapp.azure.com``,
``mdb-instance-1.<azure location>.cloudapp.azure.com``, etc. as the FQDNs.
The ``<azure location>`` is the Azure datacenter location you are using,
which can also be obtained using the ``az account list-locations`` command.
You can also try to assign a name to an Public IP in Azure before starting
the process, or use ``nslookup`` with the name you have in mind to check
if it's available for use.
You should ensure that the the name specified in the ``data.fqdn`` field is
a unique one.
**Kubernetes on bare-metal or other cloud providers.**
You need to provide the name resolution function
by other means (using DNS providers like GoDaddy, CloudFlare or your own
private DNS server). The DNS set up for other environments is currently
beyond the scope of this document.
Create the required ConfigMap using:
.. code:: bash
$ kubectl apply -f mongo-cm.yaml
You can check its status using: ``kubectl get cm``
Now you are ready to run MongoDB and BigchainDB on our Kubernetes cluster.
Step 6: Run MongoDB as a StatefulSet
------------------------------------
Get the file ``mongo-ss.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/mongodb/mongo-ss.yaml
Note how the MongoDB container uses the ``mongo-db-claim`` and the
``mongo-configdb-claim`` PersistentVolumeClaims for its ``/data/db`` and
``/data/configdb`` diretories (mount path). Note also that we use the pod's
``securityContext.capabilities.add`` specification to add the ``FOWNER``
capability to the container.
That is because MongoDB container has the user ``mongodb``, with uid ``999``
and group ``mongodb``, with gid ``999``.
When this container runs on a host with a mounted disk, the writes fail when
there is no user with uid ``999``.
To avoid this, we use the Docker feature of ``--cap-add=FOWNER``.
This bypasses the uid and gid permission checks during writes and allows data
to be persisted to disk.
Refer to the
`Docker docs <https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities>`_
for details.
As we gain more experience running MongoDB in testing and production, we will
tweak the ``resources.limits.cpu`` and ``resources.limits.memory``.
We will also stop exposing port ``27017`` globally and/or allow only certain
hosts to connect to the MongoDB instance in the future.
Create the required StatefulSet using:
.. code:: bash
$ kubectl apply -f mongo-ss.yaml
You can check its status using the commands ``kubectl get statefulsets -w``
and ``kubectl get svc -w``
You may have to wait for up to 10 minutes for the disk to be created
and attached on the first run. The pod can fail several times with the message
saying that the timeout for mounting the disk was exceeded.
Step 7: Initialize a MongoDB Replica Set - Optional
---------------------------------------------------
This step is required only if you are planning to set up multiple
`BigchainDB nodes
<https://docs.bigchaindb.com/en/latest/terminology.html>`_.
Login to the running MongoDB instance and access the mongo shell using:
.. code:: bash
$ kubectl exec -it mdb-0 -c mongodb -- /bin/bash
root@mdb-0:/# mongo --port 27017
You will initiate the replica set by using the ``rs.initiate()`` command from the
mongo shell. Its syntax is:
.. code:: bash
rs.initiate({
_id : "<replica-set-name",
members: [ {
_id : 0,
host : "<fqdn of this instance>:<port number>"
} ]
})
An example command might look like:
.. code:: bash
> rs.initiate({ _id : "bigchain-rs", members: [ { _id : 0, host :"mdb-instance-0.westeurope.cloudapp.azure.com:27017" } ] })
where ``mdb-instance-0.westeurope.cloudapp.azure.com`` is the value stored in
the ``data.fqdn`` field in the ConfigMap created using ``mongo-cm.yaml``.
You should see changes in the mongo shell prompt from ``>``
to ``bigchain-rs:OTHER>`` to ``bigchain-rs:SECONDARY>`` and finally
to ``bigchain-rs:PRIMARY>``.
You can use the ``rs.conf()`` and the ``rs.status()`` commands to check the
detailed replica set configuration now.
Step 8: Create a DNS record - Optional
--------------------------------------
This step is required only if you are planning to set up multiple
`BigchainDB nodes
<https://docs.bigchaindb.com/en/latest/terminology.html>`_.
**Azure.** Select the current Azure resource group and look for the ``Public IP``
resource. You should see at least 2 entries there - one for the Kubernetes
master and the other for the MongoDB instance. You may have to ``Refresh`` the
Azure web page listing the resources in a resource group for the latest
changes to be reflected.
Select the ``Public IP`` resource that is attached to your service (it should
have the Kubernetes cluster name along with a random string),
select ``Configuration``, add the DNS name that was added in the
ConfigMap earlier, click ``Save``, and wait for the changes to be applied.
To verify the DNS setting is operational, you can run ``nslookup <dns
name added in ConfigMap>`` from your local Linux shell.
This will ensure that when you scale the replica set later, other MongoDB
members in the replica set can reach this instance.
Step 9: Run BigchainDB as a Deployment
--------------------------------------
Get the file ``bigchaindb-dep.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/bigchaindb/bigchaindb-dep.yaml
Note that we set the ``BIGCHAINDB_DATABASE_HOST`` to ``mdb-svc`` which is the
name of the MongoDB service defined earlier.
We also hardcode the ``BIGCHAINDB_KEYPAIR_PUBLIC``,
``BIGCHAINDB_KEYPAIR_PRIVATE`` and ``BIGCHAINDB_KEYRING`` for now.
As we gain more experience running BigchainDB in testing and production, we
will tweak the ``resources.limits`` values for CPU and memory, and as richer
monitoring and probing becomes available in BigchainDB, we will tweak the
``livenessProbe`` and ``readinessProbe`` parameters.
We also plan to specify scheduling policies for the BigchainDB deployment so
that we ensure that BigchainDB and MongoDB are running in separate nodes, and
build security around the globally exposed port ``9984``.
Create the required Deployment using:
.. code:: bash
$ kubectl apply -f bigchaindb-dep.yaml
You can check its status using the command ``kubectl get deploy -w``
Step 10: Run NGINX as a Deployment
----------------------------------
NGINX is used as a proxy to both the BigchainDB and MongoDB instances in the
node.
It proxies HTTP requests on port 80 to the BigchainDB backend, and TCP
connections on port 27017 to the MongoDB backend.
You can also configure a whitelist in NGINX to allow only connections from
other instances in the MongoDB replica set to access the backend MongoDB
instance.
Get the file ``nginx-cm.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/nginx/nginx-cm.yaml
The IP address whitelist can be explicitly configured in ``nginx-cm.yaml``
file. You will need a list of the IP addresses of all the other MongoDB
instances in the cluster. If the MongoDB intances specify a hostname, then this
needs to be resolved to the corresponding IP addresses. If the IP address of
any MongoDB instance changes, we can start a 'rolling upgrade' of NGINX after
updating the corresponding ConfigMap without affecting availabilty.
Create the ConfigMap for the whitelist using:
.. code:: bash
$ kubectl apply -f nginx-cm.yaml
Get the file ``nginx-dep.yaml`` from GitHub using:
.. code:: bash
$ wget https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/k8s/nginx/nginx-dep.yaml
Create the NGINX deployment using:
.. code:: bash
$ kubectl apply -f nginx-dep.yaml
Step 11: Verify the BigchainDB Node Setup
-----------------------------------------
Step 11.1: Testing Internally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Run a container that provides utilities like ``nslookup``, ``curl`` and ``dig``
on the cluster and query the internal DNS and IP endpoints.
.. code:: bash
$ kubectl run -it toolbox -- image <docker image to run> --restart=Never --rm
There is a generic image based on alpine:3.5 with the required utilities
hosted at Docker Hub under `bigchaindb/toolbox <https://hub.docker.com/r/bigchaindb/toolbox/>`_.
The corresponding Dockerfile is in the bigchaindb/bigchaindb repository on GitHub, at `https://github.com/bigchaindb/bigchaindb/blob/master/k8s/toolbox/Dockerfile <https://github.com/bigchaindb/bigchaindb/blob/master/k8s/toolbox/Dockerfile>`_.
You can use it as below to get started immediately:
.. code:: bash
$ kubectl run -it toolbox --image bigchaindb/toolbox --restart=Never --rm
It will drop you to the shell prompt.
Now you can query for the ``mdb`` and ``bdb`` service details.
.. code:: bash
# nslookup mdb-svc
# nslookup bdb-svc
# nslookup ngx-svc
# dig +noall +answer _mdb-port._tcp.mdb-svc.default.svc.cluster.local SRV
# dig +noall +answer _bdb-port._tcp.bdb-svc.default.svc.cluster.local SRV
# dig +noall +answer _ngx-public-mdb-port._tcp.ngx-svc.default.svc.cluster.local SRV
# dig +noall +answer _ngx-public-bdb-port._tcp.ngx-svc.default.svc.cluster.local SRV
# curl -X GET http://mdb-svc:27017
# curl -X GET http://bdb-svc:9984
# curl -X GET http://ngx-svc:80
# curl -X GET http://ngx-svc:27017
The ``nslookup`` commands should output the configured IP addresses of the
services in the cluster
The ``dig`` commands should return the port numbers configured for the
various services in the cluster.
Finally, the ``curl`` commands test the availability of the services
themselves.
Step 11.2: Testing Externally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Try to access the ``<dns/ip of your exposed bigchaindb service endpoint>:80``
on your browser. You must receive a json output that shows the BigchainDB
server version among other things.
Try to access the ``<dns/ip of your exposed mongodb service endpoint>:27017``
on your browser. If your IP is in the whitelist, you will receive a message
from the MongoDB instance stating that it doesn't allow HTTP connections to
the port anymore. If your IP is not in the whitelist, your access will be
blocked and you will not see any response from the MongoDB instance.

2
docs/server/source/cloud-deployment-templates/template-ansible.md
View File

@@ -81,4 +81,4 @@ where, as before, `<key-name>` must be replaced.
## Next Steps
You could make changes to the Ansible playbook (and the resources it uses) to make the node more production-worthy. See [the section on production node assumptions, components and requirements](../nodes/index.html).
You could make changes to the Ansible playbook (and the resources it uses) to make the node more production-worthy. See [the section on production node assumptions, components and requirements](../production-nodes/index.html).

114
docs/server/source/cloud-deployment-templates/template-kubernetes-azure.rst
View File

@@ -18,7 +18,20 @@ You may find that you have to sign up for a Free Trial subscription first.
That's okay: you can have many subscriptions.
Step 2: Deploy an Azure Container Service (ACS)
Step 2: Create an SSH Key Pair
------------------------------
You'll want an SSH key pair so you'll be able to SSH
to the virtual machines that you'll deploy in the next step.
(If you already have an SSH key pair, you *could* reuse it,
but it's probably a good idea to make a new SSH key pair
for your Kubernetes VMs and nothing else.)
See the
:ref:`page about how to generate a key pair for SSH <Generate a Key Pair for SSH>`.
Step 3: Deploy an Azure Container Service (ACS)
-----------------------------------------------
It's *possible* to deploy an Azure Container Service (ACS)
@@ -26,16 +39,18 @@ from the `Azure Portal <https://portal.azure.com>`_
(i.e. online in your web browser)
but it's actually easier to do it using the Azure
Command-Line Interface (CLI).
(The Azure Portal will ask you for a public SSH key
and a "service principal," and you'll have to create those
first if they don't exist. The CLI will create them
for you if necessary.)
Microsoft has `instructions to install the Azure CLI 2.0
on most common operating systems
<https://docs.microsoft.com/en-us/cli/azure/install-az-cli2>`_.
Do that.
First, update the Azure CLI to the latest version:
.. code:: bash
$ az component update
Next, login to your account using:
.. code:: bash
@@ -79,11 +94,12 @@ Finally, you can deploy an ACS using something like:
$ az acs create --name <a made-up cluster name> \
--resource-group <name of resource group created earlier> \
--master-count 3 \
--agent-count 3 \
--admin-username ubuntu \
--agent-vm-size Standard_D2_v2 \
--dns-prefix <make up a name> \
--generate-ssh-keys \
--location <same location as the resource group> \
--ssh-key-value ~/.ssh/<name>.pub \
--orchestrator-type kubernetes
There are more options. For help understanding all the options, use the built-in help:
@@ -99,5 +115,87 @@ go to **Resource groups** (with the blue cube icon)
and click on the one you created
to see all the resources in it.
Optional: SSH to Your New Kubernetes Cluster Nodes
--------------------------------------------------
You can SSH to one of the just-deployed Kubernetes "master" nodes
(virtual machines) using:
.. code:: bash
$ ssh -i ~/.ssh/<name>.pub ubuntu@<master-ip-address-or-hostname>
where you can get the IP address or hostname
of a master node from the Azure Portal. For example:
.. code:: bash
$ ssh -i ~/.ssh/mykey123.pub ubuntu@mydnsprefix.westeurope.cloudapp.azure.com
.. note::
All the master nodes should have the *same* IP address and hostname
(also called the Master FQDN).
The "agent" nodes shouldn't get public IP addresses or hostnames,
so you can't SSH to them *directly*,
but you can first SSH to the master
and then SSH to an agent from there.
To do that, you could
copy your SSH key pair to the master (a bad idea),
or use SSH agent forwarding (better).
To do the latter, do the following on the machine you used
to SSH to the master:
.. code:: bash
$ echo -e "Host <FQDN of the cluster from Azure Portal>\n ForwardAgent yes" >> ~/.ssh/config
To verify that SSH agent forwarding works properly,
SSH to the one of the master nodes and do:
.. code:: bash
$ echo "$SSH_AUTH_SOCK"
If you get an empty response,
then SSH agent forwarding hasn't been set up correctly.
If you get a non-empty response,
then SSH agent forwarding should work fine
and you can SSH to one of the agent nodes (from a master)
using something like:
.. code:: bash
$ ssh ubuntu@k8s-agent-4AC80E97-0
where ``k8s-agent-4AC80E97-0`` is the name
of a Kubernetes agent node in your Kubernetes cluster.
You will have to replace it by the name
of an agent node in your cluster.
Optional: Delete the Kubernetes Cluster
---------------------------------------
.. code:: bash
$ az acs delete \
--name <ACS cluster name> \
--resource-group <name of resource group containing the cluster>
Optional: Delete the Resource Group
-----------------------------------
CAUTION: You might end up deleting resources other than the ACS cluster.
.. code:: bash
$ az group delete \
--name <name of resource group containing the cluster>
Next, you can :doc:`run a BigchainDB node on your new
Kubernetes cluster <node-on-kubernetes>`.
Kubernetes cluster <node-on-kubernetes>`.

105
docs/server/source/cloud-deployment-templates/upgrade-on-kubernetes.rst Normal file
View File

@@ -0,0 +1,105 @@
Kubernetes Template: Upgrade all Software in a BigchainDB Node
==============================================================
This page outlines how to upgrade all the software associated
with a BigchainDB node running on Kubernetes,
including host operating systems, Docker, Kubernetes,
and BigchainDB-related software.
Upgrade Host OS, Docker and Kubernetes
--------------------------------------
Some Kubernetes installation & management systems
can do full or partial upgrades of host OSes, Docker,
or Kubernetes, e.g.
`Tectonic <https://coreos.com/tectonic/>`_,
`Rancher <https://docs.rancher.com/rancher/v1.5/en/>`_,
and
`Kubo <https://pivotal.io/kubo>`_.
Consult the documentation for your system.
**Azure Container Service (ACS).**
On Dec. 15, 2016, a Microsoft employee
`wrote <https://github.com/colemickens/azure-kubernetes-status/issues/15#issuecomment-267453251>`_:
"In the coming months we [the Azure Kubernetes team] will be building managed updates in the ACS service."
At the time of writing, managed updates were not yet available,
but you should check the latest
`ACS documentation <https://docs.microsoft.com/en-us/azure/container-service/>`_
to see what's available now.
Also at the time of writing, ACS only supported Ubuntu
as the host (master and agent) operating system.
You can upgrade Ubuntu and Docker on Azure
by SSHing into each of the hosts,
as documented on
:ref:`another page <Optional: SSH to Your New Kubernetes Cluster Nodes>`.
In general, you can SSH to each host in your Kubernetes Cluster
to update the OS and Docker.
.. note::
Once you are in an SSH session with a host,
the ``docker info`` command is a handy way to detemine the
host OS (including version) and the Docker version.
When you want to upgrade the software on a Kubernetes node,
you should "drain" the node first,
i.e. tell Kubernetes to gracefully terminate all pods
on the node and mark it as unscheduleable
(so no new pods get put on the node during its downtime).
.. code::
kubectl drain $NODENAME
There are `more details in the Kubernetes docs <https://kubernetes.io/docs/concepts/cluster-administration/cluster-management/#maintenance-on-a-node>`_,
including instructions to make the node scheduleable again.
To manually upgrade the host OS,
see the docs for that OS.
To manually upgrade Docker, see
`the Docker docs <https://docs.docker.com/>`_.
To manually upgrade all Kubernetes software in your Kubernetes cluster, see
`the Kubernetes docs <https://kubernetes.io/docs/admin/cluster-management/>`_.
Upgrade BigchainDB-Related Software
-----------------------------------
We use Kubernetes "Deployments" for NGINX, BigchainDB,
and most other BigchainDB-related software.
The only exception is MongoDB; we use a Kubernetes
StatefulSet for that.
The nice thing about Kubernetes Deployments
is that Kubernetes can manage most of the upgrade process.
A typical upgrade workflow for a single Deployment would be:
.. code::
$ KUBE_EDITOR=nano kubectl edit deployment/<name of Deployment>
The ``kubectl edit`` command
opens the specified editor (nano in the above example),
allowing you to edit the specified Deployment *in the Kubernetes cluster*.
You can change the version tag on the Docker image, for example.
Don't forget to save your edits before exiting the editor.
The Kubernetes docs have more information about
`Deployments <https://kubernetes.io/docs/concepts/workloads/controllers/deployment/>`_ (including updating them).
The upgrade story for the MongoDB StatefulSet is *different*.
(This is because MongoDB has persistent state,
which is stored in some storage associated with a PersistentVolumeClaim.)
At the time of writing, StatefulSets were still in beta,
and they did not support automated image upgrade (Docker image tag upgrade).
We expect that to change.
Rather than trying to keep these docs up-to-date,
we advise you to check out the current
`Kubernetes docs about updating containers in StatefulSets
<https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-containers>`_.

5
docs/server/source/clusters-feds/aws-testing-cluster.md
View File

@@ -1,6 +1,6 @@
# Deploy a Testing Cluster on AWS
# Deploy a RethinkDB-Based Testing Cluster on AWS
This section explains a way to deploy a cluster of BigchainDB nodes on Amazon Web Services (AWS) for testing purposes.
This section explains a way to deploy a _RethinkDB-based_ cluster of BigchainDB nodes on Amazon Web Services (AWS) for testing purposes.
## Why?
@@ -86,7 +86,6 @@ Step 2 is to make an AWS deployment configuration file, if necessary. There's an
```text
NUM_NODES=3
BRANCH="master"
WHAT_TO_DEPLOY="servers"
SSH_KEY_NAME="not-set-yet"
USE_KEYPAIRS_FILE=False
IMAGE_ID="ami-8504fdea"

7
docs/server/source/clusters-feds/index.rst
View File

@@ -1,10 +1,9 @@
Clusters & Federations
======================
Clusters
========
.. toctree::
:maxdepth: 1
set-up-a-federation
backup
set-up-a-cluster
aws-testing-cluster

28
docs/server/source/clusters-feds/set-up-a-cluster.md Normal file
View File

@@ -0,0 +1,28 @@
# Set Up a Cluster
This section is about how to set up a BigchainDB cluster where each node is operated by a different operator. If you want to set up and run a testing cluster on AWS (where all nodes are operated by you), then see [the section about that](aws-testing-cluster.html).
## Initial Questions
There are many questions that must be answered before setting up a BigchainDB cluster. For example:
* Do you have a governance process for making consortium-level decisions, such as how to admit new members?
* What will you store in creation transactions (data payload)? Is there a data schema?
* Will you use transfer transactions? Will they include a non-empty data payload?
* Who will be allowed to submit transactions? Who will be allowed to read or query transactions? How will you enforce the access rules?
## Set Up the Initial Cluster
The consortium must decide some things before setting up the initial cluster (initial set of BigchainDB nodes):
1. Who will operate each node in the initial cluster?
2. What will the replication factor be? (It should be 3 or more.)
3. Who will deploy the first node, second node, etc.?
Once those things have been decided, the cluster deployment process can begin. The process for deploying a production node is outlined in [the section on production nodes](../production-nodes/index.html).
Every time a new BigchainDB node is added, every other node must update their [BigchainDB keyring](../server-reference/configuration.html#keyring) (one of the BigchainDB configuration settings): they must add the public key of the new node.
To secure communications between BigchainDB nodes, each BigchainDB node can use a firewall or similar, and doing that will require additional coordination.

28
docs/server/source/clusters-feds/set-up-a-federation.md
View File

@@ -1,28 +0,0 @@
# Set Up a Federation
This section is about how to set up a BigchainDB _federation_, where each node is operated by a different operator. If you want to set up and run a testing cluster on AWS (where all nodes are operated by you), then see [the section about that](aws-testing-cluster.html).
## Initial Checklist
* Do you have a governance process for making federation-level decisions, such as how to admit new members?
* What will you store in creation transactions (data payload)? Is there a data schema?
* Will you use transfer transactions? Will they include a non-empty data payload?
* Who will be allowed to submit transactions? Who will be allowed to read or query transactions? How will you enforce the access rules?
## Set Up the Initial Cluster
The federation must decide some things before setting up the initial cluster (initial set of BigchainDB nodes):
1. Who will operate a node in the initial cluster?
2. What will the replication factor be? (It must be 3 or more for [RethinkDB failover](https://rethinkdb.com/docs/failover/) to work.)
3. Which node will be responsible for sending the commands to configure the RethinkDB database?
Once those things have been decided, each node operator can begin setting up their BigchainDB (production) node.
Each node operator will eventually need two pieces of information from all other nodes in the federation:
1. Their RethinkDB hostname, e.g. `rdb.farm2.organization.org`
2. Their BigchainDB public key, e.g. `Eky3nkbxDTMgkmiJC8i5hKyVFiAQNmPP4a2G4JdDxJCK`

2
docs/server/source/conf.py
View File

@@ -82,7 +82,7 @@ master_doc = 'index'
# General information about the project.
project = 'BigchainDB Server'
copyright = '2016'
copyright = '2017, BigchainDB Contributors'
author = 'BigchainDB Contributors'
# The version info for the project you're documenting, acts as replacement for

8
docs/server/source/data-models/block-model.rst
View File

@@ -11,7 +11,7 @@ A block has the following structure:
"timestamp": "<block-creation timestamp>",
"transactions": ["<list of transactions>"],
"node_pubkey": "<public key of the node creating the block>",
"voters": ["<list of federation nodes public keys>"]
"voters": ["<list of public keys of all nodes in the cluster>"]
},
"signature": "<signature of block>"
}
@@ -23,9 +23,9 @@ A block has the following structure:
- ``timestamp``: The Unix time when the block was created. It's provided by the node that created the block.
- ``transactions``: A list of the transactions included in the block.
- ``node_pubkey``: The public key of the node that created the block.
- ``voters``: A list of the public keys of federation nodes at the time the block was created.
It's the list of federation nodes which can cast a vote on this block.
This list can change from block to block, as nodes join and leave the federation.
- ``voters``: A list of the public keys of all cluster nodes at the time the block was created.
It's the list of nodes which can cast a vote on this block.
This list can change from block to block, as nodes join and leave the cluster.
- ``signature``: :ref:`Cryptographic signature <Signature Algorithm and Keys>` of the block by the node that created the block (i.e. the node with public key ``node_pubkey``). To generate the signature, the node signs the serialized inner ``block`` (the same thing that was hashed to determine the ``id``) using the private key corresponding to ``node_pubkey``.

14
docs/server/source/data-models/inputs-outputs.rst
View File

@@ -22,7 +22,19 @@ One can also put different weights on the inputs to a threshold condition, along
The (single) output of a threshold condition can be used as one of the inputs of other threshold conditions. This means that one can combine threshold conditions to build complex logical expressions, e.g. (x OR y) AND (u OR v).
When one creates a condition, one can calculate its fulfillment length (e.g. 96). The more complex the condition, the larger its fulfillment length will be. A BigchainDB federation can put an upper limit on the allowed fulfillment length, as a way of capping the complexity of conditions (and the computing time required to validate them).
When one creates a condition, one can calculate its fulfillment length (e.g.
96). The more complex the condition, the larger its fulfillment length will be.
A BigchainDB federation can put an upper limit on the complexity of the
conditions, either directly by setting a maximum allowed fulfillment length,
or
`indirectly <https://github.com/bigchaindb/bigchaindb/issues/356#issuecomment-288085251>`_
by :ref:`setting a maximum allowed transaction size <Enforcing a Max Transaction Size>`
which would limit
the overall complexity accross all inputs and outputs of a transaction.
Note: At the time of writing, there was no configuration setting
to set a maximum allowed fulfillment length,
so the only real option was to
:ref:`set a maximum allowed transaction size <Enforcing a Max Transaction Size>`.
If someone tries to make a condition where the output of a threshold condition feeds into the input of another “earlier” threshold condition (i.e. in a closed logical circuit), then their computer will take forever to calculate the (infinite) “condition URI”, at least in theory. In practice, their computer will run out of memory or their client software will timeout after a while.

20
docs/server/source/dev-and-test/setup-run-node.md
View File

@@ -23,7 +23,9 @@ Start RethinkDB using:
$ rethinkdb
```
You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at [http://localhost:8080/](http://localhost:8080/).
You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at http://localhost:8080/
<!-- Don't hyperlink http://localhost:8080/ because Sphinx will fail when you do "make linkcheck" -->
To run BigchainDB Server, do:
```text
@@ -87,28 +89,28 @@ Start RethinkDB:
docker-compose up -d rdb
```
The RethinkDB web interface should be accessible at <http://localhost:58080/>.
The RethinkDB web interface should be accessible at http://localhost:58080/.
Depending on which platform, and/or how you are running docker, you may need
to change `localhost` for the `ip` of the machine that is running docker. As a
dummy example, if the `ip` of that machine was `0.0.0.0`, you would access the
web interface at: <http://0.0.0.0:58080/>.
web interface at: http://0.0.0.0:58080/.
Start a BigchainDB node:
```bash
docker-compose up -d bdb
docker-compose up -d bdb-rdb
```
You can monitor the logs:
```bash
docker-compose logs -f bdb
docker-compose logs -f bdb-rdb
```
If you wish to run the tests:
```bash
docker-compose run --rm bdb py.test -v -n auto
docker-compose run --rm bdb-rdb py.test -v -n auto
```
### Docker with MongoDB
@@ -128,19 +130,19 @@ $ docker-compose port mdb 27017
Start a BigchainDB node:
```bash
docker-compose up -d bdb-mdb
docker-compose up -d bdb
```
You can monitor the logs:
```bash
docker-compose logs -f bdb-mdb
docker-compose logs -f bdb
```
If you wish to run the tests:
```bash
docker-compose run --rm bdb-mdb py.test -v --database-backend=mongodb
docker-compose run --rm bdb py.test -v --database-backend=mongodb
```
### Accessing the HTTP API

6
docs/server/source/drivers-clients/index.rst
View File

@@ -14,18 +14,16 @@ community projects listed below.
.. toctree::
:maxdepth: 1
http-client-server-api
websocket-event-stream-api
The Python Driver <https://docs.bigchaindb.com/projects/py-driver/en/latest/index.html>
Transaction CLI <https://docs.bigchaindb.com/projects/cli/en/latest/>
Community Driven Libraries and Tools
Community-Driven Libraries and Tools
------------------------------------
Please note that some of these projects may be work in progress, but may
nevertheless be very useful.
* `Javascript transaction builder <https://github.com/sohkai/js-bigchaindb-quickstart>`_
* `Haskell transaction builder <https://github.com/libscott/bigchaindb-hs>`_
* `Haskell transaction builder <https://github.com/bigchaindb/bigchaindb-hs>`_
* `Go driver <https://github.com/zbo14/envoke/blob/master/bigchain/bigchain.go>`_
* `Java driver <https://github.com/mgrand/bigchaindb-java-driver>`_

48
docs/server/source/drivers-clients/http-client-server-api.rst → docs/server/source/http-client-server-api.rst
View File

@@ -22,7 +22,7 @@ or ``https://example.com:9984``
then you should get an HTTP response
with something like the following in the body:
.. literalinclude:: samples/index-response.http
.. literalinclude:: http-samples/index-response.http
:language: http
@@ -35,7 +35,7 @@ or ``https://example.com:9984/api/v1/``,
then you should get an HTTP response
that allows you to discover the BigchainDB API endpoints:
.. literalinclude:: samples/api-index-response.http
.. literalinclude:: http-samples/api-index-response.http
:language: http
@@ -46,20 +46,24 @@ Transactions
Get the transaction with the ID ``tx_id``.
This endpoint returns a transaction only if a ``VALID`` block on
``bigchain`` exists.
This endpoint returns a transaction if it was included in a ``VALID`` block,
if it is still waiting to be processed (``BACKLOG``) or is still in an
undecided block (``UNDECIDED``). All instances of a transaction in invalid
blocks are ignored and treated as if they don't exist. If a request is made
for a transaction and instances of that transaction are found only in
invalid blocks, then the response will be ``404 Not Found``.
:param tx_id: transaction ID
:type tx_id: hex string
**Example request**:
.. literalinclude:: samples/get-tx-id-request.http
.. literalinclude:: http-samples/get-tx-id-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-tx-id-response.http
.. literalinclude:: http-samples/get-tx-id-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -106,12 +110,12 @@ Transactions
**Example request**:
.. literalinclude:: samples/get-tx-by-asset-request.http
.. literalinclude:: http-samples/get-tx-by-asset-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-tx-by-asset-response.http
.. literalinclude:: http-samples/get-tx-by-asset-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -135,12 +139,12 @@ Transactions
**Example request**:
.. literalinclude:: samples/post-tx-request.http
.. literalinclude:: http-samples/post-tx-request.http
:language: http
**Example response**:
.. literalinclude:: samples/post-tx-response.http
.. literalinclude:: http-samples/post-tx-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -223,12 +227,12 @@ Statuses
**Example request**:
.. literalinclude:: samples/get-statuses-tx-request.http
.. literalinclude:: http-samples/get-statuses-tx-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-statuses-tx-valid-response.http
.. literalinclude:: http-samples/get-statuses-tx-valid-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -246,17 +250,17 @@ Statuses
**Example request**:
.. literalinclude:: samples/get-statuses-block-request.http
.. literalinclude:: http-samples/get-statuses-block-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-statuses-block-invalid-response.http
.. literalinclude:: http-samples/get-statuses-block-invalid-response.http
:language: http
**Example response**:
.. literalinclude:: samples/get-statuses-block-valid-response.http
.. literalinclude:: http-samples/get-statuses-block-valid-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -294,12 +298,12 @@ Blocks
**Example request**:
.. literalinclude:: samples/get-block-request.http
.. literalinclude:: http-samples/get-block-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-block-response.http
.. literalinclude:: http-samples/get-block-response.http
:language: http
@@ -349,12 +353,12 @@ Blocks
**Example request**:
.. literalinclude:: samples/get-block-txid-request.http
.. literalinclude:: http-samples/get-block-txid-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-block-txid-response.http
.. literalinclude:: http-samples/get-block-txid-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -380,12 +384,12 @@ Votes
**Example request**:
.. literalinclude:: samples/get-vote-request.http
.. literalinclude:: http-samples/get-vote-request.http
:language: http
**Example response**:
.. literalinclude:: samples/get-vote-response.http
.. literalinclude:: http-samples/get-vote-response.http
:language: http
:resheader Content-Type: ``application/json``
@@ -402,7 +406,7 @@ Determining the API Root URL
When you start BigchainDB Server using ``bigchaindb start``,
an HTTP API is exposed at some address. The default is:
`http://localhost:9984/api/v1/ <http://localhost:9984/api/v1/>`_
``http://localhost:9984/api/v1/``
It's bound to ``localhost``,
so you can access it from the same machine,

5
docs/server/source/index.rst
View File

@@ -8,12 +8,13 @@ BigchainDB Server Documentation
introduction
quickstart
cloud-deployment-templates/index
nodes/index
production-nodes/index
dev-and-test/index
server-reference/index
http-client-server-api
websocket-event-stream-api
drivers-clients/index
clusters-feds/index
topic-guides/index
data-models/index
schema/transaction
schema/vote

4
docs/server/source/introduction.md
View File

@@ -10,7 +10,7 @@ Note that there are a few kinds of nodes:
- A **bare-bones node** is a node deployed in the cloud, either as part of a testing cluster or as a starting point before upgrading the node to be production-ready. Our cloud deployment templates deploy a bare-bones node, as do our scripts for deploying a testing cluster on AWS.
- A **production node** is a node that is part of a federation's BigchainDB cluster. A production node has the most components and requirements.
- A **production node** is a node that is part of a consortium's BigchainDB cluster. A production node has the most components and requirements.
## Setup Instructions for Various Cases
@@ -19,7 +19,7 @@ Note that there are a few kinds of nodes:
* [Set up and run a bare-bones node in the cloud](cloud-deployment-templates/index.html)
* [Set up and run a local dev/test node for developing and testing BigchainDB Server](dev-and-test/setup-run-node.html)
* [Deploy a testing cluster on AWS](clusters-feds/aws-testing-cluster.html)
* [Set up and run a federation (including production nodes)](clusters-feds/set-up-a-federation.html)
* [Set up and run a cluster (including production nodes)](clusters-feds/set-up-a-cluster.html)
Instructions for setting up a client will be provided once there's a public test net.

10
docs/server/source/nodes/index.rst
View File

@@ -1,10 +0,0 @@
Production Node Assumptions, Components & Requirements
======================================================
.. toctree::
:maxdepth: 1
node-assumptions
node-components
node-requirements
setup-run-node

13
docs/server/source/nodes/node-assumptions.md
View File

@@ -1,13 +0,0 @@
# Production Node Assumptions
If you're not sure what we mean by a BigchainDB *node*, *cluster*, *federation*, or *production node*, then see [the section in the Introduction where we defined those terms](../introduction.html#some-basic-vocabulary).
We make some assumptions about production nodes:
1. **Each production node is set up and managed by an experienced professional system administrator (or a team of them).**
2. Each production node in a federation's cluster is managed by a different person or team.
Because of the first assumption, we don't provide a detailed cookbook explaining how to secure a server, or other things that a sysadmin should know. (We do provide some [templates](../cloud-deployment-templates/index.html), but those are just a starting point.)

23
docs/server/source/nodes/node-components.md
View File

@@ -1,23 +0,0 @@
# Production Node Components
A BigchainDB node must include, at least:
* BigchainDB Server and
* RethinkDB Server.
When doing development and testing, it's common to install both on the same machine, but in a production environment, it may make more sense to install them on separate machines.
In a production environment, a BigchainDB node should have several other components, including:
* nginx or similar, as a reverse proxy and/or load balancer for the Gunicorn server(s) inside the node
* An NTP daemon running on all machines running BigchainDB code, and possibly other machines
* A RethinkDB proxy server
* A RethinkDB "wire protocol firewall" (in the future: this component doesn't exist yet)
* Scalable storage for RethinkDB (e.g. using RAID)
* Monitoring software, to monitor all the machines in the node
* Configuration management agents (if you're using a configuration managment system that uses agents)
* Maybe more
The relationship between these components is illustrated below.
![Components of a node](../_static/Node-components.png)

193
docs/server/source/nodes/setup-run-node.md
View File

@@ -1,193 +0,0 @@
# Set Up and Run a Cluster Node
This is a page of general guidelines for setting up a production node. It says nothing about how to upgrade software, storage, processing, etc. or other details of node management. It will be expanded more in the future.
## Get a Server
The first step is to get a server (or equivalent) which meets [the requirements for a BigchainDB node](node-requirements.html).
## Secure Your Server
The steps that you must take to secure your server depend on your server OS and where your server is physically located. There are many articles and books about how to secure a server. Here we just cover special considerations when securing a BigchainDB node.
There are some [notes on BigchainDB-specific firewall setup](../appendices/firewall-notes.html) in the Appendices.
## Sync Your System Clock
A BigchainDB node uses its system clock to generate timestamps for blocks and votes, so that clock should be kept in sync with some standard clock(s). The standard way to do that is to run an NTP daemon (Network Time Protocol daemon) on the node. (You could also use tlsdate, which uses TLS timestamps rather than NTP, but don't: it's not very accurate and it will break with TLS 1.3, which removes the timestamp.)
NTP is a standard protocol. There are many NTP daemons implementing it. We don't recommend a particular one. On the contrary, we recommend that different nodes in a federation run different NTP daemons, so that a problem with one daemon won't affect all nodes.
Please see the [notes on NTP daemon setup](../appendices/ntp-notes.html) in the Appendices.
## Set Up Storage for RethinkDB Data
Below are some things to consider when setting up storage for the RethinkDB data. The Appendices have a [section with concrete examples](../appendices/example-rethinkdb-storage-setups.html).
We suggest you set up a separate storage "device" (partition, RAID array, or logical volume) to store the RethinkDB data. Here are some questions to ask:
* How easy will it be to add storage in the future? Will I have to shut down my server?
* How big can the storage get? (Remember that [RAID](https://en.wikipedia.org/wiki/RAID) can be used to make several physical drives look like one.)
* How fast can it read & write data? How many input/output operations per second (IOPS)?
* How does IOPS scale as more physical hard drives are added?
* What's the latency?
* What's the reliability? Is there replication?
* What's in the Service Level Agreement (SLA), if applicable?
* What's the cost?
There are many options and tradeoffs. Don't forget to look into Amazon Elastic Block Store (EBS) and Amazon Elastic File System (EFS), or their equivalents from other providers.
**Storage Notes Specific to RethinkDB**
* The RethinkDB storage engine has a number of SSD optimizations, so you _can_ benefit from using SSDs. ([source](https://www.rethinkdb.com/docs/architecture/))
* If you want a RethinkDB cluster to store an amount of data D, with a replication factor of R (on every table), and the cluster has N nodes, then each node will need to be able to store R×D/N data.
* RethinkDB tables can have [at most 64 shards](https://rethinkdb.com/limitations/). For example, if you have only one table and more than 64 nodes, some nodes won't have the primary of any shard, i.e. they will have replicas only. In other words, once you pass 64 nodes, adding more nodes won't provide more storage space for new data. If the biggest single-node storage available is d, then the most you can store in a RethinkDB cluster is < 64×d: accomplished by putting one primary shard in each of 64 nodes, with all replica shards on other nodes. (This is assuming one table. If there are T tables, then the most you can store is < 64×d×T.)
* When you set up storage for your RethinkDB data, you may have to select a filesystem. (Sometimes, the filesystem is already decided by the choice of storage.) We recommend using a filesystem that supports direct I/O (Input/Output). Many compressed or encrypted file systems don't support direct I/O. The ext4 filesystem supports direct I/O (but be careful: if you enable the data=journal mode, then direct I/O support will be disabled; the default is data=ordered). If your chosen filesystem supports direct I/O and you're using Linux, then you don't need to do anything to request or enable direct I/O. RethinkDB does that.
<p style="background-color: lightgrey;">What is direct I/O? It allows RethinkDB to write directly to the storage device (or use its own in-memory caching mechanisms), rather than relying on the operating system's file read and write caching mechanisms. (If you're using Linux, a write-to-file normally writes to the in-memory Page Cache first; only later does that Page Cache get flushed to disk. The Page Cache is also used when reading files.)</p>
* RethinkDB stores its data in a specific directory. You can tell RethinkDB _which_ directory using the RethinkDB config file, as explained below. In this documentation, we assume the directory is `/data`. If you set up a separate device (partition, RAID array, or logical volume) to store the RethinkDB data, then mount that device on `/data`.
## Install RethinkDB Server
If you don't already have RethinkDB Server installed, you must install it. The RethinkDB documentation has instructions for [how to install RethinkDB Server on a variety of operating systems](https://rethinkdb.com/docs/install/).
## Configure RethinkDB Server
Create a RethinkDB configuration file (text file) named `instance1.conf` with the following contents (explained below):
```text
directory=/data
bind=all
direct-io
# Replace node?_hostname with actual node hostnames below, e.g. rdb.examples.com
join=node0_hostname:29015
join=node1_hostname:29015
join=node2_hostname:29015
# continue until there's a join= line for each node in the federation
```
* `directory=/data` tells the RethinkDB node to store its share of the database data in `/data`.
* `bind=all` binds RethinkDB to all local network interfaces (e.g. loopback, Ethernet, wireless, whatever is available), so it can communicate with the outside world. (The default is to bind only to local interfaces.)
* `direct-io` tells RethinkDB to use direct I/O (explained earlier). Only include this line if your file system supports direct I/O.
* `join=hostname:29015` lines: A cluster node needs to find out the hostnames of all the other nodes somehow. You _could_ designate one node to be the one that every other node asks, and put that node's hostname in the config file, but that wouldn't be very decentralized. Instead, we include _every_ node in the list of nodes-to-ask.
If you're curious about the RethinkDB config file, there's [a RethinkDB documentation page about it](https://www.rethinkdb.com/docs/config-file/). The [explanations of the RethinkDB command-line options](https://rethinkdb.com/docs/cli-options/) are another useful reference.
See the [RethinkDB documentation on securing your cluster](https://rethinkdb.com/docs/security/).
## Install Python 3.4+
If you don't already have it, then you should [install Python 3.4+](https://www.python.org/downloads/).
If you're testing or developing BigchainDB on a stand-alone node, then you should probably create a Python 3.4+ virtual environment and activate it (e.g. using virtualenv or conda). Later we will install several Python packages and you probably only want those installed in the virtual environment.
## Install BigchainDB Server
First, [install the OS-level dependencies of BigchainDB Server (link)](../appendices/install-os-level-deps.html).
With OS-level dependencies installed, you can install BigchainDB Server with `pip` or from source.
### How to Install BigchainDB with pip
BigchainDB (i.e. both the Server and the officially-supported drivers) is distributed as a Python package on PyPI so you can install it using `pip`. First, make sure you have an up-to-date Python 3.4+ version of `pip` installed:
```text
pip -V
```
If it says that `pip` isn't installed, or it says `pip` is associated with a Python version less than 3.4, then you must install a `pip` version associated with Python 3.4+. In the following instructions, we call it `pip3` but you may be able to use `pip` if that refers to the same thing. See [the `pip` installation instructions](https://pip.pypa.io/en/stable/installing/).
On Ubuntu 16.04, we found that this works:
```text
sudo apt-get install python3-pip
```
That should install a Python 3 version of `pip` named `pip3`. If that didn't work, then another way to get `pip3` is to do `sudo apt-get install python3-setuptools` followed by `sudo easy_install3 pip`.
You can upgrade `pip` (`pip3`) and `setuptools` to the latest versions using:
```text
pip3 install --upgrade pip setuptools
pip3 -V
```
Now you can install BigchainDB Server (and officially-supported BigchainDB drivers) using:
```text
pip3 install bigchaindb
```
(If you're not in a virtualenv and you want to install bigchaindb system-wide, then put `sudo` in front.)
Note: You can use `pip3` to upgrade the `bigchaindb` package to the latest version using `pip3 install --upgrade bigchaindb`.
### How to Install BigchainDB from Source
If you want to install BitchainDB from source because you want to use the very latest bleeding-edge code, clone the public repository:
```text
git clone git@github.com:bigchaindb/bigchaindb.git
python setup.py install
```
## Configure BigchainDB Server
Start by creating a default BigchainDB config file:
```text
bigchaindb -y configure rethinkdb
```
(There's documentation for the `bigchaindb` command is in the section on [the BigchainDB Command Line Interface (CLI)](bigchaindb-cli.html).)
Edit the created config file:
* Open `$HOME/.bigchaindb` (the created config file) in your text editor.
* Change `"server": {"bind": "localhost:9984", ... }` to `"server": {"bind": "0.0.0.0:9984", ... }`. This makes it so traffic can come from any IP address to port 9984 (the HTTP Client-Server API port).
* Change `"keyring": []` to `"keyring": ["public_key_of_other_node_A", "public_key_of_other_node_B", "..."]` i.e. a list of the public keys of all the other nodes in the federation. The keyring should _not_ include your node's public key.
For more information about the BigchainDB config file, see [Configuring a BigchainDB Node](configuration.html).
## Run RethinkDB Server
Start RethinkDB using:
```text
rethinkdb --config-file path/to/instance1.conf
```
except replace the path with the actual path to `instance1.conf`.
Note: It's possible to [make RethinkDB start at system startup](https://www.rethinkdb.com/docs/start-on-startup/).
You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at `http://rethinkdb-hostname:8080/`. If you're running RethinkDB on localhost, that would be [http://localhost:8080/](http://localhost:8080/).
## Run BigchainDB Server
After all node operators have started RethinkDB, but before they start BigchainDB, one designated node operator must configure the RethinkDB database by running the following commands:
```text
bigchaindb init
bigchaindb set-shards numshards
bigchaindb set-replicas numreplicas
```
where:
* `bigchaindb init` creates the database within RethinkDB, the tables, the indexes, and the genesis block.
* `numshards` should be set to the number of nodes in the initial cluster.
* `numreplicas` should be set to the database replication factor decided by the federation. It must be 3 or more for [RethinkDB failover](https://rethinkdb.com/docs/failover/) to work.
Once the RethinkDB database is configured, every node operator can start BigchainDB using:
```text
bigchaindb start
```

12
docs/server/source/production-nodes/index.rst Normal file
View File

@@ -0,0 +1,12 @@
Production Nodes
================
.. toctree::
:maxdepth: 1
node-assumptions
node-components
node-requirements
setup-run-node
reverse-proxy-notes

16
docs/server/source/production-nodes/node-assumptions.md Normal file
View File

@@ -0,0 +1,16 @@
# Production Node Assumptions
Be sure you know the key BigchainDB terminology:
* [BigchainDB node, BigchainDB cluster and BigchainDB consortum](https://docs.bigchaindb.com/en/latest/terminology.html)
* [dev/test node, bare-bones node and production node](../introduction.html)
We make some assumptions about production nodes:
1. Production nodes use MongoDB, not RethinkDB.
1. Each production node is set up and managed by an experienced professional system administrator or a team of them.
1. Each production node in a cluster is managed by a different person or team.
You can use RethinkDB when building prototypes, but we don't advise or support using it in production.
We don't provide a detailed cookbook explaining how to secure a server, or other things that a sysadmin should know. (We do provide some [templates](../cloud-deployment-templates/index.html), but those are just a starting point.)

22
docs/server/source/production-nodes/node-components.md Normal file
View File

@@ -0,0 +1,22 @@
# Production Node Components
A production BigchainDB node must include:
* BigchainDB Server
* MongoDB Server 3.4+ (mongod)
* Scalable storage for MongoDB
It could also include several other components, including:
* NGINX or similar, to provide authentication, rate limiting, etc.
* An NTP daemon running on all machines running BigchainDB Server or mongod, and possibly other machines
* **Not** MongoDB Automation Agent. It's for automating the deployment of an entire MongoDB cluster, not just one MongoDB node within a cluster.
* MongoDB Monitoring Agent
* MongoDB Backup Agent
* Log aggregation software
* Monitoring software
* Maybe more
The relationship between the main components is illustrated below. Note that BigchainDB Server must be able to communicate with the _primary_ MongoDB instance, and any of the MongoDB instances might be the primary, so BigchainDB Server must be able to communicate with all the MongoDB instances. Also, all MongoDB instances must be able to communicate with each other.
![Components of a production node](../_static/Node-components.png)

17
docs/server/source/production-nodes/node-requirements.md Normal file
View File

@@ -0,0 +1,17 @@
# Production Node Requirements
**This page is about the requirements of BigchainDB Server.** You can find the requirements of MongoDB, NGINX, your NTP daemon, your monitoring software, and other [production node components](node-components.html) in the documentation for that software.
## OS Requirements
BigchainDB Server requires Python 3.4+ and Python 3.4+ [will run on any modern OS](https://docs.python.org/3.4/using/index.html), but we recommend using an LTS version of [Ubuntu Server](https://www.ubuntu.com/server) or a similarly server-grade Linux distribution.
_Don't use macOS_ (formerly OS X, formerly Mac OS X), because it's not a server-grade operating system. Also, BigchaindB Server uses the Python multiprocessing package and [some functionality in the multiprocessing package doesn't work on Mac OS X](https://docs.python.org/3.4/library/multiprocessing.html#multiprocessing.Queue.qsize).
## General Considerations
BigchainDB Server runs many concurrent processes, so more RAM and more CPU cores is better.
As mentioned on the page about [production node components](node-components.html), every machine running BigchainDB Server should be running an NTP daemon.

72
docs/server/source/production-nodes/reverse-proxy-notes.md Normal file
View File

@@ -0,0 +1,72 @@
# Using a Reverse Proxy
You may want to:
* rate limit inbound HTTP requests,
* authenticate/authorize inbound HTTP requests,
* block requests with an HTTP request body that's too large, or
* enable HTTPS (TLS) between your users and your node.
While we could have built all that into BigchainDB Server,
we didn't, because you can do all that (and more)
using a reverse proxy such as NGINX or HAProxy.
(You would put it in front of your BigchainDB Server,
so that all inbound HTTP requests would arrive
at the reverse proxy before *maybe* being proxied
onwards to your BigchainDB Server.)
For detailed instructions, see the documentation
for your reverse proxy.
Below, we note how a reverse proxy can be used
to do some BigchainDB-specific things.
You may also be interested in
[our NGINX configuration file template](https://github.com/bigchaindb/nginx_3scale/blob/master/nginx.conf.template)
(open source, on GitHub).
## Enforcing a Max Transaction Size
The BigchainDB HTTP API has several endpoints,
but only one of them, the `POST /transactions` endpoint,
expects a non-empty HTTP request body:
the transaction (JSON) being submitted by the user.
If you want to enforce a maximum-allowed transaction size
(discarding any that are larger),
then you can do so by configuring a maximum request body size
in your reverse proxy.
For example, NGINX has the `client_max_body_size`
configuration setting. You could set it to 15 kB
with the following line in your NGINX config file:
```text
client_max_body_size 15k;
```
For more information, see
[the NGINX docs about client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size).
Note: By enforcing a maximum transaction size, you
[indirectly enforce a maximum crypto-conditions complexity](https://github.com/bigchaindb/bigchaindb/issues/356#issuecomment-288085251).
**Aside: Why 15 kB?**
Both [RethinkDB](https://rethinkdb.com/limitations/) and
[MongoDB have a maximum document size of 16 MB](https://docs.mongodb.com/manual/reference/limits/#limit-bson-document-size).
In BigchainDB, the biggest documents are the blocks.
A BigchainDB block can contain up to 1000 transactions,
plus some other data (e.g. the timestamp).
If we ignore the other data as negligible relative to all the transactions,
then a block of size 16 MB
will have an average transaction size of (16 MB)/1000 = 16 kB.
Therefore by limiting the max transaction size to 15 kB,
you can be fairly sure that no blocks will ever be
bigger than 16 MB.
Note: Technically, the documents that MongoDB stores aren't the JSON
that BigchainDB users think of; they're JSON converted to BSON.
Moreover, [one can use GridFS with MongoDB to store larger documents](https://docs.mongodb.com/manual/core/gridfs/).
Therefore the above calculation shoud be seen as a rough guide,
not the last word.

137
docs/server/source/production-nodes/setup-run-node.md Normal file
View File

@@ -0,0 +1,137 @@
# Set Up and Run a Cluster Node
This is a page of general guidelines for setting up a production BigchainDB node. Before continuing, make sure you've read the pages about production node [assumptions](node-assumptions.html), [components](node-components.html) and [requirements](node-requirements.html).
Note: These are just guidelines. You can modify them to suit your needs. For example, if you want to initialize the MongoDB replica set before installing BigchainDB, you _can_ do that. If you'd prefer to use Docker and Kubernetes, you can (and [we have a template](../cloud-deployment-templates/node-on-kubernetes.html)). We don't cover all possible setup procedures here.
## Security Guidelines
There are many articles, websites and books about securing servers, virtual machines, networks, etc. Consult those.
There are some [notes on BigchainDB-specific firewall setup](../appendices/firewall-notes.html) in the Appendices.
## Sync Your System Clock
A BigchainDB node uses its system clock to generate timestamps for blocks and votes, so that clock should be kept in sync with some standard clock(s). The standard way to do that is to run an NTP daemon (Network Time Protocol daemon) on the node.
MongoDB also recommends having an NTP daemon running on all MongoDB nodes.
NTP is a standard protocol. There are many NTP daemons implementing it. We don't recommend a particular one. On the contrary, we recommend that different nodes in a cluster run different NTP daemons, so that a problem with one daemon won't affect all nodes.
Please see the [notes on NTP daemon setup](../appendices/ntp-notes.html) in the Appendices.
## Set Up Storage for MongoDB
We suggest you set up a separate storage device (partition, RAID array, or logical volume) to store the data in the MongoDB database. Here are some questions to ask:
* How easy will it be to add storage in the future? Will I have to shut down my server?
* How big can the storage get? (Remember that [RAID](https://en.wikipedia.org/wiki/RAID) can be used to make several physical drives look like one.)
* How fast can it read & write data? How many input/output operations per second (IOPS)?
* How does IOPS scale as more physical hard drives are added?
* What's the latency?
* What's the reliability? Is there replication?
* What's in the Service Level Agreement (SLA), if applicable?
* What's the cost?
There are many options and tradeoffs.
Consult the MongoDB documentation for its recommendations regarding storage hardware, software and settings, e.g. in the [MongoDB Production Notes](https://docs.mongodb.com/manual/administration/production-notes/).
## Install and Run MongoDB
* [Install MongoDB 3.4+](https://docs.mongodb.com/manual/installation/). (BigchainDB only works with MongoDB 3.4+.)
* [Run MongoDB (mongod)](https://docs.mongodb.com/manual/reference/program/mongod/)
## Install BigchainDB Server
### Install BigchainDB Server Dependencies
Before you can install BigchainDB Server, you must [install its OS-level dependencies](../appendices/install-os-level-deps.html) and you may have to [install Python 3.4+](https://www.python.org/downloads/).
### How to Install BigchainDB Server with pip
BigchainDB is distributed as a Python package on PyPI so you can install it using `pip`. First, make sure you have an up-to-date Python 3.4+ version of `pip` installed:
```text
pip -V
```
If it says that `pip` isn't installed, or it says `pip` is associated with a Python version less than 3.4, then you must install a `pip` version associated with Python 3.4+. In the following instructions, we call it `pip3` but you may be able to use `pip` if that refers to the same thing. See [the `pip` installation instructions](https://pip.pypa.io/en/stable/installing/).
On Ubuntu 16.04, we found that this works:
```text
sudo apt-get install python3-pip
```
That should install a Python 3 version of `pip` named `pip3`. If that didn't work, then another way to get `pip3` is to do `sudo apt-get install python3-setuptools` followed by `sudo easy_install3 pip`.
You can upgrade `pip` (`pip3`) and `setuptools` to the latest versions using:
```text
pip3 install --upgrade pip setuptools
pip3 -V
```
Now you can install BigchainDB Server using:
```text
pip3 install bigchaindb
```
(If you're not in a virtualenv and you want to install bigchaindb system-wide, then put `sudo` in front.)
Note: You can use `pip3` to upgrade the `bigchaindb` package to the latest version using `pip3 install --upgrade bigchaindb`.
### How to Install BigchainDB Server from Source
If you want to install BitchainDB from source because you want to use the very latest bleeding-edge code, clone the public repository:
```text
git clone git@github.com:bigchaindb/bigchaindb.git
cd bigchaindb
python setup.py install
```
## Configure BigchainDB Server
Start by creating a default BigchainDB config file for a MongoDB backend:
```text
bigchaindb -y configure mongodb
```
(There's documentation for the `bigchaindb` command is in the section on [the BigchainDB Command Line Interface (CLI)](../server-reference/bigchaindb-cli.html).)
Edit the created config file by opening `$HOME/.bigchaindb` (the created config file) in your text editor:
* Change `"server": {"bind": "localhost:9984", ... }` to `"server": {"bind": "0.0.0.0:9984", ... }`. This makes it so traffic can come from any IP address to port 9984 (the HTTP Client-Server API port).
* Change `"keyring": []` to `"keyring": ["public_key_of_other_node_A", "public_key_of_other_node_B", "..."]` i.e. a list of the public keys of all the other nodes in the cluster. The keyring should _not_ include your node's public key.
* Ensure that `database.host` and `database.port` are set to the hostname and port of your MongoDB instance. (The port is usually 27017, unless you changed it.)
For more information about the BigchainDB config file, see the page about the [BigchainDB configuration settings](../server-reference/configuration.html).
## Get All Other Nodes to Update Their Keyring
All other BigchainDB nodes in the cluster must add your new node's public key to their BigchainDB keyring. Currently, the only way to get BigchainDB Server to "notice" a changed keyring is to shut it down and start it back up again (with the new keyring).
## Maybe Update the MongoDB Replica Set
**If this isn't the first node in the BigchainDB cluster**, then someone with an existing BigchainDB node (not you) must add your MongoDB instance to the MongoDB replica set. They can do so (on their node) using:
```text
bigchaindb add-replicas your-mongod-hostname:27017
```
where they must replace `your-mongod-hostname` with the actual hostname of your MongoDB instance, and they may have to replace `27017` with the actual port.
## Start BigchainDB
**Warning: If you're not deploying the first node in the BigchainDB cluster, then don't start BigchainDB before your MongoDB instance has been added to the MongoDB replica set (as outlined above).**
```text
# See warning above
bigchaindb start
```

18
docs/server/source/server-reference/bigchaindb-cli.md
View File

@@ -68,16 +68,22 @@ You can also use the `--dev-start-rethinkdb` command line option to automaticall
e.g. `bigchaindb --dev-start-rethinkdb start`. Note that this will also shutdown rethinkdb when the bigchaindb process stops.
The option `--dev-allow-temp-keypair` will generate a keypair on the fly if no keypair is found, this is useful when you want to run a temporary instance of BigchainDB in a Docker container, for example.
### Options
The log level for the console can be set via the option `--log-level` or its
abbreviation `-l`. Example:
## bigchaindb load
Write transactions to the backlog (for benchmarking tests). You can learn more about it using:
```text
$ bigchaindb load -h
```bash
$ bigchaindb --log-level INFO start
```
Note: This command uses the Python Server API to write transactions to the database. It _doesn't_ use the HTTP API or a driver that wraps the HTTP API.
The allowed levels are `DEBUG`, `INFO` , `WARNING`, `ERROR`, and `CRITICAL`.
For an explanation regarding these levels please consult the
[Logging Levels](https://docs.python.org/3.6/library/logging.html#levels)
section of Python's documentation.
For a more fine-grained control over the logging configuration you can use the
configuration file as documented under
[Configuration Settings](configuration.html).
## bigchaindb set-shards

323
docs/server/source/server-reference/configuration.md
View File

@@ -16,12 +16,26 @@ For convenience, here's a list of all the relevant environment variables (docume
`BIGCHAINDB_DATABASE_PORT`<br>
`BIGCHAINDB_DATABASE_NAME`<br>
`BIGCHAINDB_DATABASE_REPLICASET`<br>
`BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT`<br>
`BIGCHAINDB_DATABASE_MAX_TRIES`<br>
`BIGCHAINDB_SERVER_BIND`<br>
`BIGCHAINDB_SERVER_LOGLEVEL`<br>
`BIGCHAINDB_SERVER_WORKERS`<br>
`BIGCHAINDB_SERVER_THREADS`<br>
`BIGCHAINDB_WSSERVER_HOST`<br>
`BIGCHAINDB_WSSERVER_PORT`<br>
`BIGCHAINDB_CONFIG_PATH`<br>
`BIGCHAINDB_BACKLOG_REASSIGN_DELAY`<br>
`BIGCHAINDB_CONSENSUS_PLUGIN`<br>
`BIGCHAINDB_LOG`<br>
`BIGCHAINDB_LOG_FILE`<br>
`BIGCHAINDB_LOG_ERROR_FILE`<br>
`BIGCHAINDB_LOG_LEVEL_CONSOLE`<br>
`BIGCHAINDB_LOG_LEVEL_LOGFILE`<br>
`BIGCHAINDB_LOG_DATEFMT_CONSOLE`<br>
`BIGCHAINDB_LOG_DATEFMT_LOGFILE`<br>
`BIGCHAINDB_LOG_FMT_CONSOLE`<br>
`BIGCHAINDB_LOG_FMT_LOGFILE`<br>
`BIGCHAINDB_LOG_GRANULAR_LEVELS`<br>
The local config file is `$HOME/.bigchaindb` by default (a file which might not even exist), but you can tell BigchainDB to use a different file by using the `-c` command-line option, e.g. `bigchaindb -c path/to/config_file.json start`
or using the `BIGCHAINDB_CONFIG_PATH` environment variable, e.g. `BIGHAINDB_CONFIG_PATH=.my_bigchaindb_config bigchaindb start`.
@@ -76,9 +90,18 @@ Note how the keys in the list are separated by colons.
```
## database.backend, database.host, database.port, database.name & database.replicaset
## database.*
The database backend to use (`rethinkdb` or `mongodb`) and its hostname, port and name. If the database backend is `mongodb`, then there's a fifth setting: the name of the replica set. If the database backend is `rethinkdb`, you *can* set the name of the replica set, but it won't be used for anything.
The settings with names of the form `database.*` are for the database backend
(currently either RethinkDB or MongoDB). They are:
* `database.backend` is either `rethinkdb` or `mongodb`.
* `database.host` is the hostname (FQDN) of the backend database.
* `database.port` is self-explanatory.
* `database.name` is a user-chosen name for the database inside RethinkDB or MongoDB, e.g. `bigchain`.
* `database.replicaset` is only relevant if using MongoDB; it's the name of the MongoDB replica set, e.g. `bigchain-rs`.
* `database.connection_timeout` is the maximum number of milliseconds that BigchainDB will wait before giving up on one attempt to connect to the database backend. Note: At the time of writing, this setting was only used by MongoDB; there was an open [issue to make RethinkDB use it as well](https://github.com/bigchaindb/bigchaindb/issues/1337).
* `database.max_tries` is the maximum number of times that BigchainDB will try to establish a connection with the database backend. If 0, then it will try forever.
**Example using environment variables**
```text
@@ -87,6 +110,8 @@ export BIGCHAINDB_DATABASE_HOST=localhost
export BIGCHAINDB_DATABASE_PORT=27017
export BIGCHAINDB_DATABASE_NAME=bigchain
export BIGCHAINDB_DATABASE_REPLICASET=bigchain-rs
export BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT=5000
export BIGCHAINDB_DATABASE_MAX_TRIES=3
```
**Default values**
@@ -96,8 +121,10 @@ If (no environment variables were set and there's no local config file), or you
"database": {
"backend": "rethinkdb",
"host": "localhost",
"port": 28015,
"name": "bigchain",
"port": 28015
"connection_timeout": 5000,
"max_tries": 3
}
```
@@ -106,24 +133,31 @@ If you used `bigchaindb -y configure mongodb` to create a default local config f
"database": {
"backend": "mongodb",
"host": "localhost",
"name": "bigchain",
"port": 27017,
"replicaset": "bigchain-rs"
"name": "bigchain",
"replicaset": "bigchain-rs",
"connection_timeout": 5000,
"max_tries": 3
}
```
## server.bind, server.workers & server.threads
## server.bind, server.loglevel, server.workers & server.threads
These settings are for the [Gunicorn HTTP server](http://gunicorn.org/), which is used to serve the [HTTP client-server API](../drivers-clients/http-client-server-api.html).
These settings are for the [Gunicorn HTTP server](http://gunicorn.org/), which is used to serve the [HTTP client-server API](../http-client-server-api.html).
`server.bind` is where to bind the Gunicorn HTTP server socket. It's a string. It can be any valid value for [Gunicorn's bind setting](http://docs.gunicorn.org/en/stable/settings.html#bind). If you want to allow IPv4 connections from anyone, on port 9984, use '0.0.0.0:9984'. In a production setting, we recommend you use Gunicorn behind a reverse proxy server. If Gunicorn and the reverse proxy are running on the same machine, then use 'localhost:PORT' where PORT is _not_ 9984 (because the reverse proxy needs to listen on port 9984). Maybe use PORT=9983 in that case because we know 9983 isn't used. If Gunicorn and the reverse proxy are running on different machines, then use 'A.B.C.D:9984' where A.B.C.D is the IP address of the reverse proxy. There's [more information about deploying behind a reverse proxy in the Gunicorn documentation](http://docs.gunicorn.org/en/stable/deploy.html). (They call it a proxy.)
`server.loglevel` sets the log level of Gunicorn's Error log outputs. See
[Gunicorn's documentation](http://docs.gunicorn.org/en/latest/settings.html#loglevel)
for more information.
`server.workers` is [the number of worker processes](http://docs.gunicorn.org/en/stable/settings.html#workers) for handling requests. If `None` (the default), the value will be (cpu_count * 2 + 1). `server.threads` is [the number of threads-per-worker](http://docs.gunicorn.org/en/stable/settings.html#threads) for handling requests. If `None` (the default), the value will be (cpu_count * 2 + 1). The HTTP server will be able to handle `server.workers` * `server.threads` requests simultaneously.
**Example using environment variables**
```text
export BIGCHAINDB_SERVER_BIND=0.0.0.0:9984
export BIGCHAINDB_SERVER_LOGLEVEL=debug
export BIGCHAINDB_SERVER_WORKERS=5
export BIGCHAINDB_SERVER_THREADS=5
```
@@ -132,6 +166,7 @@ export BIGCHAINDB_SERVER_THREADS=5
```js
"server": {
"bind": "0.0.0.0:9984",
"loglevel": "debug",
"workers": 5,
"threads": 5
}
@@ -141,11 +176,46 @@ export BIGCHAINDB_SERVER_THREADS=5
```js
"server": {
"bind": "localhost:9984",
"loglevel": "info",
"workers": null,
"threads": null
}
```
## wsserver.host and wsserver.port
These settings are for the
[aiohttp server](https://aiohttp.readthedocs.io/en/stable/index.html),
which is used to serve the
[WebSocket Event Stream API](../websocket-event-stream-api.html).
`wsserver.host` is where to bind the aiohttp server socket and
`wsserver.port` is the corresponding port.
If you want to allow connections from anyone, on port 9985,
set `wsserver.host` to 0.0.0.0 and `wsserver.port` to 9985.
**Example using environment variables**
```text
export BIGCHAINDB_WSSERVER_HOST=0.0.0.0
export BIGCHAINDB_WSSERVER_PORT=9985
```
**Example config file snippet**
```js
"wsserver": {
"host": "0.0.0.0",
"port": 65000
}
```
**Default values (from a config file)**
```js
"wsserver": {
"host": "localhost",
"port": 9985
}
```
## backlog_reassign_delay
Specifies how long, in seconds, transactions can remain in the backlog before being reassigned. Long-waiting transactions must be reassigned because the assigned node may no longer be responsive. The default duration is 120 seconds.
@@ -160,16 +230,237 @@ export BIGCHAINDB_BACKLOG_REASSIGN_DELAY=30
"backlog_reassign_delay": 120
```
## consensus_plugin
The [consensus plugin](../appendices/consensus.html) to use.
## log
**Example using an environment variable**
```text
export BIGCHAINDB_CONSENSUS_PLUGIN=default
The `log` key is expected to point to a mapping (set of key/value pairs)
holding the logging configuration.
**Example**:
```
{
"log": {
"file": "/var/log/bigchaindb.log",
"error_file": "/var/log/bigchaindb-errors.log",
"level_console": "info",
"level_logfile": "info",
"datefmt_console": "%Y-%m-%d %H:%M:%S",
"datefmt_logfile": "%Y-%m-%d %H:%M:%S",
"fmt_console": "%(asctime)s [%(levelname)s] (%(name)s) %(message)s",
"fmt_logfile": "%(asctime)s [%(levelname)s] (%(name)s) %(message)s",
"granular_levels": {
"bichaindb.backend": "info",
"bichaindb.core": "info"
}
}
```
**Example config file snippet: the default**
```js
"consensus_plugin": "default"
**Defaults to**:
```
{
"log": {
"file": "~/bigchaindb.log",
"error_file": "~/bigchaindb-errors.log",
"level_console": "info",
"level_logfile": "info",
"datefmt_console": "%Y-%m-%d %H:%M:%S",
"datefmt_logfile": "%Y-%m-%d %H:%M:%S",
"fmt_logfile": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
"fmt_console": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
"granular_levels": {}
}
```
The next subsections explain each field of the `log` configuration.
### log.file & log.error_file
The full paths to the files where logs and error logs should be written to.
**Example**:
```
{
"log": {
"file": "/var/log/bigchaindb/bigchaindb.log"
"error_file": "/var/log/bigchaindb/bigchaindb-errors.log"
}
}
```
**Defaults to**:
* `"~/bigchaindb.log"`
* `"~/bigchaindb-errors.log"`
Please note that the user running `bigchaindb` must have write access to the
locations.
#### Log rotation
Log files have a size limit of 200 MB and will be rotated up to five times.
For example if we consider the log file setting:
```
{
"log": {
"file": "~/bigchain.log"
}
}
```
logs would always be written to `bigchain.log`. Each time the file
`bigchain.log` reaches 200 MB it would be closed and renamed
`bigchain.log.1`. If `bigchain.log.1` and `bigchain.log.2` already exist they
would be renamed `bigchain.log.2` and `bigchain.log.3`. This pattern would be
applied up to `bigchain.log.5` after which `bigchain.log.5` would be
overwritten by `bigchain.log.4`, thus ending the rotation cycle of whatever
logs were in `bigchain.log.5`.
### log.level_console
The log level used to log to the console. Possible allowed values are the ones
defined by [Python](https://docs.python.org/3.6/library/logging.html#levels),
but case insensitive for convenience's sake:
```
"critical", "error", "warning", "info", "debug", "notset"
```
**Example**:
```
{
"log": {
"level_console": "info"
}
}
```
**Defaults to**: `"info"`.
### log.level_logfile
The log level used to log to the log file. Possible allowed values are the ones
defined by [Python](https://docs.python.org/3.6/library/logging.html#levels),
but case insensitive for convenience's sake:
```
"critical", "error", "warning", "info", "debug", "notset"
```
**Example**:
```
{
"log": {
"level_file": "info"
}
}
```
**Defaults to**: `"info"`.
### log.datefmt_console
The format string for the date/time portion of a message, when logged to the
console.
**Example**:
```
{
"log": {
"datefmt_console": "%x %X %Z"
}
}
```
**Defaults to**: `"%Y-%m-%d %H:%M:%S"`.
For more information on how to construct the format string please consult the
table under Python's documentation of
[`time.strftime(format[, t])`](https://docs.python.org/3.6/library/time.html#time.strftime)
### log.datefmt_logfile
The format string for the date/time portion of a message, when logged to a log
file.
**Example**:
```
{
"log": {
"datefmt_logfile": "%c %z"
}
}
```
**Defaults to**: `"%Y-%m-%d %H:%M:%S"`.
For more information on how to construct the format string please consult the
table under Python's documentation of
[`time.strftime(format[, t])`](https://docs.python.org/3.6/library/time.html#time.strftime)
### log.fmt_console
A string used to format the log messages when logged to the console.
**Example**:
```
{
"log": {
"fmt_console": "%(asctime)s [%(levelname)s] %(message)s %(process)d"
}
}
```
**Defaults to**: `"[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)"`
For more information on possible formatting options please consult Python's
documentation on
[LogRecord attributes](https://docs.python.org/3.6/library/logging.html#logrecord-attributes)
### log.fmt_logfile
A string used to format the log messages when logged to a log file.
**Example**:
```
{
"log": {
"fmt_logfile": "%(asctime)s [%(levelname)s] %(message)s %(process)d"
}
}
```
**Defaults to**: `"[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)"`
For more information on possible formatting options please consult Python's
documentation on
[LogRecord attributes](https://docs.python.org/3.6/library/logging.html#logrecord-attributes)
### log.granular_levels
Log levels for BigchainDB's modules. This can be useful to control the log
level of specific parts of the application. As an example, if you wanted the
logging of the `core.py` module to be more verbose, you would set the
configuration shown in the example below.
**Example**:
```
{
"log": {
"granular_levels": {
"bichaindb.core": "debug"
}
}
```
**Defaults to**: `"{}"`

12
docs/server/source/topic-guides/index.rst
View File

@@ -1,12 +0,0 @@
Topic Guides
============
.. note::
Most of the Topic Guides have been moved over to `the root BigchainDB project docs <https://docs.bigchaindb.com/en/latest/index.html>`_.
.. toctree::
:maxdepth: 1
models

6
docs/server/source/topic-guides/models.md
View File

@@ -1,6 +0,0 @@
# The Transaction, Block and Vote Models
This page about transaction concepts and data models was getting too big, so it was split into smaller pages. It will be deleted eventually, so update your links. Here's where you can find the new pages:
* [Transaction Concepts](https://docs.bigchaindb.com/en/latest/transaction-concepts.html)
* [Data Models (all of them)](../data-models/index.html)

12
docs/server/source/drivers-clients/websocket-event-stream-api.rst → docs/server/source/websocket-event-stream-api.rst
View File

@@ -2,7 +2,9 @@ The WebSocket Event Stream API
==============================
.. important::
This is currently scheduled to be implemented in BigchainDB Server 0.10.
The WebSocket Event Stream runs on a different port than the Web API. The
default port for the Web API is `9984`, while the one for the Event Stream
is `9985`.
BigchainDB provides real-time event streams over the WebSocket protocol with
the Event Stream API.
@@ -28,7 +30,7 @@ response contains a ``streams_<version>`` property in ``_links``::
{
"_links": {
"streams_v1": "ws://example.com:9984/api/v1/streams/"
"streams_v1": "ws://example.com:9985/api/v1/streams/"
}
}
@@ -80,9 +82,9 @@ the transaction's ID, associated asset ID, and containing block's ID.
Example message::
{
"txid": "<sha3-256 hash>",
"assetid": "<sha3-256 hash>",
"blockid": "<sha3-256 hash>"
"tx_id": "<sha3-256 hash>",
"asset_id": "<sha3-256 hash>",
"block_id": "<sha3-256 hash>"
}