Merge branch 'master' into docs/1170/move-http-api-docs-to-top-level-section

docs/server/source/data-models/inputs-outputs.rst

@@ -25,9 +25,16 @@ The (single) output of a threshold condition can be used as one of the inputs of
 When one creates a condition, one can calculate its fulfillment length (e.g.
 96). The more complex the condition, the larger its fulfillment length will be.
 A BigchainDB federation can put an upper limit on the complexity of the
-conditions, either directly by setting an allowed maximum fulfillment length,
-or indirectly by setting a maximum allowed transaction size which would limit
+conditions, either directly by setting a maximum allowed fulfillment length,
+or
+`indirectly <https://github.com/bigchaindb/bigchaindb/issues/356#issuecomment-288085251>`_
+by :ref:`setting a maximum allowed transaction size <Enforcing a Max Transaction Size>`
+which would limit
 the overall complexity accross all inputs and outputs of a transaction.
+Note: At the time of writing, there was no configuration setting
+to set a maximum allowed fulfillment length,
+so the only real option was to
+:ref:`set a maximum allowed transaction size <Enforcing a Max Transaction Size>`.
 
 If someone tries to make a condition where the output of a threshold condition feeds into the input of another “earlier” threshold condition (i.e. in a closed logical circuit), then their computer will take forever to calculate the (infinite) “condition URI”, at least in theory. In practice, their computer will run out of memory or their client software will timeout after a while.
 

docs/server/source/drivers-clients/index.rst

@@ -14,7 +14,6 @@ community projects listed below.
 .. toctree::
    :maxdepth: 1
 
-   websocket-event-stream-api
    The Python Driver <https://docs.bigchaindb.com/projects/py-driver/en/latest/index.html>
    Transaction CLI <https://docs.bigchaindb.com/projects/cli/en/latest/>
 

docs/server/source/index.rst

@@ -12,6 +12,7 @@ BigchainDB Server Documentation
    dev-and-test/index
    server-reference/index
    http-client-server-api
+   websocket-event-stream-api
    drivers-clients/index
    clusters-feds/index
    data-models/index

docs/server/source/production-nodes/index.rst

@@ -8,3 +8,5 @@ Production Nodes
    node-components
    node-requirements
    setup-run-node
+   reverse-proxy-notes
+   

docs/server/source/production-nodes/reverse-proxy-notes.md

@@ -0,0 +1,72 @@
+# Using a Reverse Proxy
+
+You may want to:
+
+* rate limit inbound HTTP requests,
+* authenticate/authorize inbound HTTP requests,
+* block requests with an HTTP request body that's too large, or
+* enable HTTPS (TLS) between your users and your node.
+
+While we could have built all that into BigchainDB Server,
+we didn't, because you can do all that (and more)
+using a reverse proxy such as NGINX or HAProxy.
+(You would put it in front of your BigchainDB Server,
+so that all inbound HTTP requests would arrive
+at the reverse proxy before *maybe* being proxied
+onwards to your BigchainDB Server.)
+For detailed instructions, see the documentation
+for your reverse proxy.
+
+Below, we note how a reverse proxy can be used
+to do some BigchainDB-specific things.
+
+You may also be interested in
+[our NGINX configuration file template](https://github.com/bigchaindb/nginx_3scale/blob/master/nginx.conf.template)
+(open source, on GitHub).
+
+
+## Enforcing a Max Transaction Size
+
+The BigchainDB HTTP API has several endpoints,
+but only one of them, the `POST /transactions` endpoint,
+expects a non-empty HTTP request body:
+the transaction (JSON) being submitted by the user.
+
+If you want to enforce a maximum-allowed transaction size
+(discarding any that are larger),
+then you can do so by configuring a maximum request body size
+in your reverse proxy.
+For example, NGINX has the `client_max_body_size`
+configuration setting. You could set it to 15 kB
+with the following line in your NGINX config file:
+
+```text
+client_max_body_size 15k;
+```
+
+For more information, see
+[the NGINX docs about client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size).
+
+Note: By enforcing a maximum transaction size, you
+[indirectly enforce a maximum crypto-conditions complexity](https://github.com/bigchaindb/bigchaindb/issues/356#issuecomment-288085251).
+
+
+**Aside: Why 15 kB?**
+
+Both [RethinkDB](https://rethinkdb.com/limitations/) and
+[MongoDB have a maximum document size of 16 MB](https://docs.mongodb.com/manual/reference/limits/#limit-bson-document-size).
+In BigchainDB, the biggest documents are the blocks.
+A BigchainDB block can contain up to 1000 transactions,
+plus some other data (e.g. the timestamp).
+If we ignore the other data as negligible relative to all the transactions,
+then a block of size 16 MB
+will have an average transaction size of (16 MB)/1000 = 16 kB.
+Therefore by limiting the max transaction size to 15 kB,
+you can be fairly sure that no blocks will ever be
+bigger than 16 MB.
+
+Note: Technically, the documents that MongoDB stores aren't the JSON
+that BigchainDB users think of; they're JSON converted to BSON.
+Moreover, [one can use GridFS with MongoDB to store larger documents](https://docs.mongodb.com/manual/core/gridfs/).
+Therefore the above calculation shoud be seen as a rough guide,
+not the last word.

docs/server/source/server-reference/configuration.md

@@ -16,7 +16,10 @@ For convenience, here's a list of all the relevant environment variables (docume
 `BIGCHAINDB_DATABASE_PORT`<br>
 `BIGCHAINDB_DATABASE_NAME`<br>
 `BIGCHAINDB_DATABASE_REPLICASET`<br>
+`BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT`<br>
+`BIGCHAINDB_DATABASE_MAX_TRIES`<br>
 `BIGCHAINDB_SERVER_BIND`<br>
+`BIGCHAINDB_SERVER_LOGLEVEL`<br>
 `BIGCHAINDB_SERVER_WORKERS`<br>
 `BIGCHAINDB_SERVER_THREADS`<br>
 `BIGCHAINDB_CONFIG_PATH`<br>
@@ -84,9 +87,18 @@ Note how the keys in the list are separated by colons.
 ```
 
 
-## database.backend, database.host, database.port, database.name & database.replicaset
+## database.*
 
-The database backend to use (`rethinkdb` or `mongodb`) and its hostname, port and name. If the database backend is `mongodb`, then there's a fifth setting: the name of the replica set. If the database backend is `rethinkdb`, you *can* set the name of the replica set, but it won't be used for anything.
+The settings with names of the form `database.*` are for the database backend
+(currently either RethinkDB or MongoDB). They are:
+
+* `database.backend` is either `rethinkdb` or `mongodb`.
+* `database.host` is the hostname (FQDN) of the backend database.
+* `database.port` is self-explanatory.
+* `database.name` is a user-chosen name for the database inside RethinkDB or MongoDB, e.g. `bigchain`.
+* `database.replicaset` is only relevant if using MongoDB; it's the name of the MongoDB replica set, e.g. `bigchain-rs`.
+* `database.connection_timeout` is the maximum number of milliseconds that BigchainDB will wait before giving up on one attempt to connect to the database backend. Note: At the time of writing, this setting was only used by MongoDB; there was an open [issue to make RethinkDB use it as well](https://github.com/bigchaindb/bigchaindb/issues/1337).
+* `database.max_tries` is the maximum number of times that BigchainDB will try to establish a connection with the database backend. If 0, then it will try forever.
 
 **Example using environment variables**
 ```text
@@ -95,6 +107,8 @@ export BIGCHAINDB_DATABASE_HOST=localhost
 export BIGCHAINDB_DATABASE_PORT=27017
 export BIGCHAINDB_DATABASE_NAME=bigchain
 export BIGCHAINDB_DATABASE_REPLICASET=bigchain-rs
+export BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT=5000
+export BIGCHAINDB_DATABASE_MAX_TRIES=3
 ```
 
 **Default values**
@@ -104,8 +118,10 @@ If (no environment variables were set and there's no local config file), or you
 "database": {
     "backend": "rethinkdb",
     "host": "localhost",
+    "port": 28015,
     "name": "bigchain",
-    "port": 28015
+    "connection_timeout": 5000,
+    "max_tries": 3
 }
 ```
 
@@ -114,24 +130,31 @@ If you used `bigchaindb -y configure mongodb` to create a default local config f
 "database": {
     "backend": "mongodb",
     "host": "localhost",
-    "name": "bigchain",
     "port": 27017,
-    "replicaset": "bigchain-rs"
+    "name": "bigchain",
+    "replicaset": "bigchain-rs",
+    "connection_timeout": 5000,
+    "max_tries": 3
 }
 ```
 
 
-## server.bind, server.workers & server.threads
+## server.bind, server.loglevel, server.workers & server.threads
 
 These settings are for the [Gunicorn HTTP server](http://gunicorn.org/), which is used to serve the [HTTP client-server API](../http-client-server-api.html).
 
 `server.bind` is where to bind the Gunicorn HTTP server socket. It's a string. It can be any valid value for [Gunicorn's bind setting](http://docs.gunicorn.org/en/stable/settings.html#bind). If you want to allow IPv4 connections from anyone, on port 9984, use '0.0.0.0:9984'. In a production setting, we recommend you use Gunicorn behind a reverse proxy server. If Gunicorn and the reverse proxy are running on the same machine, then use 'localhost:PORT' where PORT is _not_ 9984 (because the reverse proxy needs to listen on port 9984). Maybe use PORT=9983 in that case because we know 9983 isn't used. If Gunicorn and the reverse proxy are running on different machines, then use 'A.B.C.D:9984' where A.B.C.D is the IP address of the reverse proxy. There's [more information about deploying behind a reverse proxy in the Gunicorn documentation](http://docs.gunicorn.org/en/stable/deploy.html). (They call it a proxy.)
 
+`server.loglevel` sets the log level of Gunicorn's Error log outputs. See
+[Gunicorn's documentation](http://docs.gunicorn.org/en/latest/settings.html#loglevel)
+for more information.
+
 `server.workers` is [the number of worker processes](http://docs.gunicorn.org/en/stable/settings.html#workers) for handling requests. If `None` (the default), the value will be (cpu_count * 2 + 1). `server.threads` is [the number of threads-per-worker](http://docs.gunicorn.org/en/stable/settings.html#threads) for handling requests. If `None` (the default), the value will be (cpu_count * 2 + 1). The HTTP server will be able to handle `server.workers` * `server.threads` requests simultaneously.
 
 **Example using environment variables**
 ```text
 export BIGCHAINDB_SERVER_BIND=0.0.0.0:9984
+export BIGCHAINDB_SERVER_LOGLEVEL=debug
 export BIGCHAINDB_SERVER_WORKERS=5
 export BIGCHAINDB_SERVER_THREADS=5
 ```
@@ -140,6 +163,7 @@ export BIGCHAINDB_SERVER_THREADS=5
 ```js
 "server": {
     "bind": "0.0.0.0:9984",
+    "loglevel": "debug",
     "workers": 5,
     "threads": 5
 }
@@ -149,6 +173,7 @@ export BIGCHAINDB_SERVER_THREADS=5
 ```js
 "server": {
     "bind": "localhost:9984",
+    "loglevel": "info",
     "workers": null,
     "threads": null
 }

docs/server/source/websocket-event-stream-api.rst

@@ -2,7 +2,9 @@ The WebSocket Event Stream API
 ==============================
 
 .. important::
-    This is currently scheduled to be implemented in BigchainDB Server 0.10.
+    The WebSocket Event Stream runs on a different port than the Web API. The
+    default port for the Web API is `9984`, while the one for the Event Stream
+    is `9985`.
 
 BigchainDB provides real-time event streams over the WebSocket protocol with
 the Event Stream API.
@@ -28,7 +30,7 @@ response contains a ``streams_<version>`` property in ``_links``::
 
     {
       "_links": {
-         "streams_v1": "ws://example.com:9984/api/v1/streams/"
+         "streams_v1": "ws://example.com:9985/api/v1/streams/"
       }
     }
 
@@ -80,9 +82,9 @@ the transaction's ID, associated asset ID, and containing block's ID.
 Example message::
 
     {
-        "txid": "<sha3-256 hash>",
-        "assetid": "<sha3-256 hash>",
-        "blockid": "<sha3-256 hash>"
+        "tx_id": "<sha3-256 hash>",
+        "asset_id": "<sha3-256 hash>",
+        "block_id": "<sha3-256 hash>"
     }