From 218780cde3cf59846b63d0863e70ac1b3ee23805 Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 9 Jun 2016 10:24:18 +0200 Subject: [PATCH 01/27] Removed who-for and 'Please don't use' stuff in Intro --- docs/source/introduction.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/docs/source/introduction.md b/docs/source/introduction.md index 1d8f9bb9..1f029fbc 100644 --- a/docs/source/introduction.md +++ b/docs/source/introduction.md @@ -5,19 +5,9 @@ BigchainDB is a scalable blockchain database. That is, it's a "big data" databas You can read about the motivations, goals and high-level architecture in the [BigchainDB whitepaper](https://www.bigchaindb.com/whitepaper/). -## Who this Documentation for? - -The BigchainDB Documentation is intended primarily for: - -1. Developers of BigchainDB server software. -2. People deploying and managing BigchainDB clusters. -3. Developers of BigchainDB driver software (SDKs used to develop client software). -4. App developers who are developing client apps to talk to one or more live, operational BigchainDB clusters. They would use one of the BigchainDB drivers. - - ## Is BigchainDB Production-Ready? -No, BigchainDB is not production-ready. You can use it to build a prototype or proof-of-concept (POC); many people are already doing that. Please don't use it for something mission-critical. +No, BigchainDB is not production-ready. You can use it to build a prototype or proof-of-concept (POC); many people are already doing that. BigchainDB is currently in version 0.X. ([The Releases page on GitHub](https://github.com/bigchaindb/bigchaindb/releases) has the exact version number.) Once we believe that BigchainDB is production-ready, we'll release version 1.0. From f7233c46f6de1a07c153b1252733a2e8810d0cf6 Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 9 Jun 2016 12:04:37 +0200 Subject: [PATCH 02/27] Docs: Added section on why BCBB good for assets --- docs/source/introduction.md | 2 +- docs/source/topic-guides/assets.md | 13 +++++++++++++ docs/source/topic-guides/index.rst | 1 + 3 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 docs/source/topic-guides/assets.md diff --git a/docs/source/introduction.md b/docs/source/introduction.md index 1f029fbc..c78a7358 100644 --- a/docs/source/introduction.md +++ b/docs/source/introduction.md @@ -1,6 +1,6 @@ # Introduction -BigchainDB is a scalable blockchain database. That is, it's a "big data" database with some blockchain characteristics, including decentralization, immutability and native support for assets. +BigchainDB is a scalable blockchain database. That is, it's a "big data" database with some blockchain characteristics, including [decentralization](topic-guides/decentralized.html), [immutability](topic-guides/immutable.html) and [native support for assets](topic-guides/assets.html). You can read about the motivations, goals and high-level architecture in the [BigchainDB whitepaper](https://www.bigchaindb.com/whitepaper/). diff --git a/docs/source/topic-guides/assets.md b/docs/source/topic-guides/assets.md new file mode 100644 index 00000000..dc736ad2 --- /dev/null +++ b/docs/source/topic-guides/assets.md @@ -0,0 +1,13 @@ +# How BigchainDB is Good for Asset Registrations & Transfers + +BigchainDB can store data of any kind (within reason), but it's designed to be particularly good for storing asset registrations and transfers: + +* The fundamental thing that one submits to a BigchainDB federation to be checked and stored (if valid) is a _transaction_, and there are two kinds: creation transactions and transfer transactions +* A creation transaction can be use to register any kind of indivisible asset, along with arbitrary metadata +* An asset can have zero, one, or several owners +* The owners of an asset can specify (crypto-)conditions which must be satisified by anyone wishing transfer the asset to new owners (typically at least m signatures from the n current owners) +* BigchainDB verifies that the conditions have been satisified as part of checking the validity of transfer transactions. (Moreover, anyone can check that they were satisfied.) +* BigchainDB prevents double-spending of an asset +* Validated transactions are strongly tamper-resistant; see [the section about immutability / tamper-resistance](immutable.html) + +You can read more about the details of BigchainDB transactions in [the section on Transaction, Block and Vote Models (data models)](models.html). diff --git a/docs/source/topic-guides/index.rst b/docs/source/topic-guides/index.rst index 274cfe04..309620e2 100644 --- a/docs/source/topic-guides/index.rst +++ b/docs/source/topic-guides/index.rst @@ -12,4 +12,5 @@ Topic guides give background and explain concepts at a high level. decentralized diversity immutable + assets models From f2eb6b23868a7db55699d884d5202ff1697764e9 Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 9 Jun 2016 12:56:20 +0200 Subject: [PATCH 03/27] Started section on decentralized fed. setup --- docs/source/servers/federation-set-up.md | 16 ++++++++++++++++ docs/source/servers/index.rst | 1 + 2 files changed, 17 insertions(+) create mode 100644 docs/source/servers/federation-set-up.md diff --git a/docs/source/servers/federation-set-up.md b/docs/source/servers/federation-set-up.md new file mode 100644 index 00000000..faff16f2 --- /dev/null +++ b/docs/source/servers/federation-set-up.md @@ -0,0 +1,16 @@ +# How to Set Up a Decentralized BigchainDB Federation + +This section outlines the steps to _set up_ a decentralized BigchainDB federation. + +If you want to deploy a centrally-controlled BigchainDB cluster on AWS (e.g. for testing), then see [the section about that](deploy-on-aws.html). + + +## INSERT SUBSECTION TITLE HERE + +What kinds of information will the federation store? BigchainDB is well-suited to storing asset registrations and ownership transfers, but it can handle many other kinds of data as well. + +How will assets be described? Do you have a data model (schema)? + +Who (or what organizations) will run the nodes? + + diff --git a/docs/source/servers/index.rst b/docs/source/servers/index.rst index 2b85a609..2c535a6e 100644 --- a/docs/source/servers/index.rst +++ b/docs/source/servers/index.rst @@ -12,5 +12,6 @@ BigchainDB Servers (Nodes) & Clusters configuration bigchaindb-cli python-server-api-examples + federation-set-up.md deploy-on-aws monitoring From b13d47b79028485f83723eb9d637660e2aab1103 Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 9 Jun 2016 15:08:00 +0200 Subject: [PATCH 04/27] Removed Glossary & added section on terminology in /servers --- docs/source/appendices/glossary.md | 17 ----------------- docs/source/servers/index.rst | 5 +++-- docs/source/servers/node-cluster-federation.md | 7 +++++++ 3 files changed, 10 insertions(+), 19 deletions(-) delete mode 100644 docs/source/appendices/glossary.md create mode 100644 docs/source/servers/node-cluster-federation.md diff --git a/docs/source/appendices/glossary.md b/docs/source/appendices/glossary.md deleted file mode 100644 index 6111282e..00000000 --- a/docs/source/appendices/glossary.md +++ /dev/null @@ -1,17 +0,0 @@ -# Glossary of Terms - -**BigchaindB client.** A computer that can communicate with a BigchainDB cluster via the BigchainDB client-server API. - -**BigchainDB cluster.** A collection of servers running BigchainDB Server software and communicating with each other via RethinkDB protocols. a BigchainDB cluster contains one logical RethinkDB datastore. - -**Block.** A collection of up to 1000 transactions, plus other things. They get chained together by votes. See [the section on the block model](../topic-guides/models.html#the-block-model). - -**Federation.** 1. An organization with members and some kind of governance structure. 2. The BigchainDB cluster owned and operated by a Federation's members. - -**Federation node.** A server running BigchainDB Server software, with permission to communicate with other nodes in a federation. - -**Node.** See _Federation node_. - -**Transaction.** The basic informational unit. A transaction can represent the creation or transfer of a digital asset. See [the section on the transaction model](../topic-guides/models.html#the-transaction-model). - -**Vote.** Each federation node in a federation is required to vote on the validity of every block (i.e. whether the block valid or not). A node's vote on a block also includes the id of the block it considers to be the previous block. See [the section on the vote model](../topic-guides/models.html#the-vote-model). diff --git a/docs/source/servers/index.rst b/docs/source/servers/index.rst index 2c535a6e..4142d119 100644 --- a/docs/source/servers/index.rst +++ b/docs/source/servers/index.rst @@ -1,12 +1,13 @@ .. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -BigchainDB Servers (Nodes) & Clusters -===================================== +BigchainDB Nodes & Clusters +=========================== .. toctree:: :maxdepth: 1 + node-cluster-federation installing-server running-unit-tests configuration diff --git a/docs/source/servers/node-cluster-federation.md b/docs/source/servers/node-cluster-federation.md new file mode 100644 index 00000000..a0b21c4f --- /dev/null +++ b/docs/source/servers/node-cluster-federation.md @@ -0,0 +1,7 @@ +# Terminology: Nodes, Clusters & Federations + +A **BigchainDB node** is a server or set of closely-linked servers running RethinkDB Server, BigchainDB Server, and other BigchainDB-related software. Each node is controlled by one person or organization. + +A set of BigchainDB nodes can connect to each other to form a **cluster**. Each node in the cluster runs the same software. A cluster contains one logical RethinkDB datastore. A cluster may have additional servers to do things such as cluster monitoring. + +The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization, although it may be quite informal). A federation might be an organization in its own right, with a governance structure and executives. Confusingly, sometimes we refer to a cluster as a federation. You can probably tell what we mean from context. \ No newline at end of file From 69a8e7de108951601f85d6b6bddd84565804f88d Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 9 Jun 2016 15:08:27 +0200 Subject: [PATCH 05/27] Renamed Monitoring to Cluster Monitoring --- docs/source/servers/monitoring.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/servers/monitoring.md b/docs/source/servers/monitoring.md index 6f633daa..1da58e4d 100644 --- a/docs/source/servers/monitoring.md +++ b/docs/source/servers/monitoring.md @@ -1,6 +1,6 @@ -# Monitoring +# Cluster Monitoring -BigchainDB uses [StatsD](https://github.com/etsy/statsd) for monitoring. We require some additional infrastructure to take full advantage of its functionality: +BigchainDB uses [StatsD](https://github.com/etsy/statsd) for cluster monitoring. We require some additional infrastructure to take full advantage of its functionality: * an agent to listen for metrics: [Telegraf](https://github.com/influxdata/telegraf), * a time-series database: [InfluxDB](https://influxdata.com/time-series-platform/influxdb/), and From 7b146fb5a9084fb62e1457c354a77fade2e48890 Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 9 Jun 2016 15:15:21 +0200 Subject: [PATCH 06/27] Clarify some points about federations --- docs/source/servers/node-cluster-federation.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/servers/node-cluster-federation.md b/docs/source/servers/node-cluster-federation.md index a0b21c4f..57e336cf 100644 --- a/docs/source/servers/node-cluster-federation.md +++ b/docs/source/servers/node-cluster-federation.md @@ -4,4 +4,6 @@ A **BigchainDB node** is a server or set of closely-linked servers running Rethi A set of BigchainDB nodes can connect to each other to form a **cluster**. Each node in the cluster runs the same software. A cluster contains one logical RethinkDB datastore. A cluster may have additional servers to do things such as cluster monitoring. -The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization, although it may be quite informal). A federation might be an organization in its own right, with a governance structure and executives. Confusingly, sometimes we refer to a cluster as a federation. You can probably tell what we mean from context. \ No newline at end of file +The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization). A federation must have some sort of governance structure to make decisions. If a cluster is run by a single company, then the federation is just that company. + +Confusingly, sometimes we refer to a cluster as a federation. You can probably tell what we mean from context. From f6cd88338bc33c6d468593d99aae352f75c0e2d3 Mon Sep 17 00:00:00 2001 From: troymc Date: Fri, 10 Jun 2016 09:44:23 +0200 Subject: [PATCH 07/27] Removed glossary from Appendices TOC --- docs/source/appendices/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/appendices/index.rst b/docs/source/appendices/index.rst index 204b0742..353e7909 100644 --- a/docs/source/appendices/index.rst +++ b/docs/source/appendices/index.rst @@ -7,7 +7,6 @@ Appendices .. toctree:: :maxdepth: 1 - glossary json-serialization cryptography the-Bigchain-class From e1b744a1f805668e3caea49a835742ef017665ce Mon Sep 17 00:00:00 2001 From: troymc Date: Fri, 10 Jun 2016 11:05:43 +0200 Subject: [PATCH 08/27] Clarified the crypto-condition example in assets.md --- docs/source/topic-guides/assets.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/topic-guides/assets.md b/docs/source/topic-guides/assets.md index dc736ad2..0f964114 100644 --- a/docs/source/topic-guides/assets.md +++ b/docs/source/topic-guides/assets.md @@ -2,12 +2,12 @@ BigchainDB can store data of any kind (within reason), but it's designed to be particularly good for storing asset registrations and transfers: -* The fundamental thing that one submits to a BigchainDB federation to be checked and stored (if valid) is a _transaction_, and there are two kinds: creation transactions and transfer transactions -* A creation transaction can be use to register any kind of indivisible asset, along with arbitrary metadata -* An asset can have zero, one, or several owners -* The owners of an asset can specify (crypto-)conditions which must be satisified by anyone wishing transfer the asset to new owners (typically at least m signatures from the n current owners) +* The fundamental thing that one submits to a BigchainDB federation to be checked and stored (if valid) is a _transaction_, and there are two kinds: creation transactions and transfer transactions. +* A creation transaction can be use to register any kind of indivisible asset, along with arbitrary metadata. +* An asset can have zero, one, or several owners. +* The owners of an asset can specify (crypto-)conditions which must be satisified by anyone wishing transfer the asset to new owners. For example, a condition might be that at least 3 of the 5 current owners must cryptographically sign a transfer transaction. * BigchainDB verifies that the conditions have been satisified as part of checking the validity of transfer transactions. (Moreover, anyone can check that they were satisfied.) -* BigchainDB prevents double-spending of an asset -* Validated transactions are strongly tamper-resistant; see [the section about immutability / tamper-resistance](immutable.html) +* BigchainDB prevents double-spending of an asset. +* Validated transactions are strongly tamper-resistant; see [the section about immutability / tamper-resistance](immutable.html). You can read more about the details of BigchainDB transactions in [the section on Transaction, Block and Vote Models (data models)](models.html). From 95bb499c03da80e48f4c2690fc7ec6f79a49fd3c Mon Sep 17 00:00:00 2001 From: troymc Date: Fri, 10 Jun 2016 15:15:21 +0200 Subject: [PATCH 09/27] Renamed docs/source/servers docs/source/nodes --- docs/source/appendices/local-rethinkdb-cluster.md | 2 +- docs/source/index.rst | 2 +- docs/source/{servers => nodes}/bigchaindb-cli.md | 0 docs/source/{servers => nodes}/configuration.md | 0 docs/source/{servers => nodes}/deploy-on-aws.md | 0 docs/source/{servers => nodes}/federation-set-up.md | 0 docs/source/{servers => nodes}/index.rst | 0 docs/source/{servers => nodes}/installing-server.md | 0 docs/source/{servers => nodes}/monitoring.md | 0 docs/source/{servers => nodes}/node-cluster-federation.md | 0 docs/source/{servers => nodes}/python-server-api-examples.md | 0 docs/source/{servers => nodes}/running-unit-tests.md | 0 12 files changed, 2 insertions(+), 2 deletions(-) rename docs/source/{servers => nodes}/bigchaindb-cli.md (100%) rename docs/source/{servers => nodes}/configuration.md (100%) rename docs/source/{servers => nodes}/deploy-on-aws.md (100%) rename docs/source/{servers => nodes}/federation-set-up.md (100%) rename docs/source/{servers => nodes}/index.rst (100%) rename docs/source/{servers => nodes}/installing-server.md (100%) rename docs/source/{servers => nodes}/monitoring.md (100%) rename docs/source/{servers => nodes}/node-cluster-federation.md (100%) rename docs/source/{servers => nodes}/python-server-api-examples.md (100%) rename docs/source/{servers => nodes}/running-unit-tests.md (100%) diff --git a/docs/source/appendices/local-rethinkdb-cluster.md b/docs/source/appendices/local-rethinkdb-cluster.md index e13b935e..518fd261 100644 --- a/docs/source/appendices/local-rethinkdb-cluster.md +++ b/docs/source/appendices/local-rethinkdb-cluster.md @@ -4,7 +4,7 @@ This section explains one way to deploy a multi-node RethinkDB cluster on one ma ## Launching More RethinkDB Nodes -Assuming you've already [installed RethinkDB](../servers/installing-server.html#install-and-run-rethinkdb-server) and have one RethinkDB node running, here's how you can launch two more nodes on the same machine. First, prepare two additional nodes. Note that the user who starts RethinkDB must have write access to the created directories: +Assuming you've already [installed RethinkDB](../nodes/installing-server.html#install-and-run-rethinkdb-server) and have one RethinkDB node running, here's how you can launch two more nodes on the same machine. First, prepare two additional nodes. Note that the user who starts RethinkDB must have write access to the created directories: mkdir -p /path/to/node2 mkdir -p /path/to/node3 diff --git a/docs/source/index.rst b/docs/source/index.rst index cadc9b12..2ef1a8fc 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,7 +12,7 @@ Table of Contents introduction topic-guides/index - servers/index + nodes/index drivers-clients/index release-notes appendices/index diff --git a/docs/source/servers/bigchaindb-cli.md b/docs/source/nodes/bigchaindb-cli.md similarity index 100% rename from docs/source/servers/bigchaindb-cli.md rename to docs/source/nodes/bigchaindb-cli.md diff --git a/docs/source/servers/configuration.md b/docs/source/nodes/configuration.md similarity index 100% rename from docs/source/servers/configuration.md rename to docs/source/nodes/configuration.md diff --git a/docs/source/servers/deploy-on-aws.md b/docs/source/nodes/deploy-on-aws.md similarity index 100% rename from docs/source/servers/deploy-on-aws.md rename to docs/source/nodes/deploy-on-aws.md diff --git a/docs/source/servers/federation-set-up.md b/docs/source/nodes/federation-set-up.md similarity index 100% rename from docs/source/servers/federation-set-up.md rename to docs/source/nodes/federation-set-up.md diff --git a/docs/source/servers/index.rst b/docs/source/nodes/index.rst similarity index 100% rename from docs/source/servers/index.rst rename to docs/source/nodes/index.rst diff --git a/docs/source/servers/installing-server.md b/docs/source/nodes/installing-server.md similarity index 100% rename from docs/source/servers/installing-server.md rename to docs/source/nodes/installing-server.md diff --git a/docs/source/servers/monitoring.md b/docs/source/nodes/monitoring.md similarity index 100% rename from docs/source/servers/monitoring.md rename to docs/source/nodes/monitoring.md diff --git a/docs/source/servers/node-cluster-federation.md b/docs/source/nodes/node-cluster-federation.md similarity index 100% rename from docs/source/servers/node-cluster-federation.md rename to docs/source/nodes/node-cluster-federation.md diff --git a/docs/source/servers/python-server-api-examples.md b/docs/source/nodes/python-server-api-examples.md similarity index 100% rename from docs/source/servers/python-server-api-examples.md rename to docs/source/nodes/python-server-api-examples.md diff --git a/docs/source/servers/running-unit-tests.md b/docs/source/nodes/running-unit-tests.md similarity index 100% rename from docs/source/servers/running-unit-tests.md rename to docs/source/nodes/running-unit-tests.md From 6eca80344a63264ca385ddc268083617e654800f Mon Sep 17 00:00:00 2001 From: troymc Date: Fri, 10 Jun 2016 15:42:27 +0200 Subject: [PATCH 10/27] In README.md, changed /servers/ to /nodes/ --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 14edff0f..979239f7 100644 --- a/README.md +++ b/README.md @@ -13,9 +13,9 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit ## Quick Start -### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.io/en/latest/servers/installing-server.html) +### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.io/en/latest/nodes/installing-server.html) ### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.io/en/latest/installing-server.html#run-bigchaindb-with-docker) -### [The Python Server API by Example](http://bigchaindb.readthedocs.io/en/latest/servers/python-server-api-examples.html) +### [The Python Server API by Example](http://bigchaindb.readthedocs.io/en/latest/nodes/python-server-api-examples.html) ### [The Python Driver API by Example](http://bigchaindb.readthedocs.io/en/latest/drivers-clients/python-driver-api-examples.html) ## Links for Everyone From d2ad9663ad364ba96e62ee00dec2e08644ad1414 Mon Sep 17 00:00:00 2001 From: troymc Date: Mon, 13 Jun 2016 10:46:05 +0200 Subject: [PATCH 11/27] Moved nodes/clusters/federations terminology section inside nodes/index.rst --- docs/source/nodes/index.rst | 18 +++++++++++++++--- docs/source/nodes/node-cluster-federation.md | 9 --------- 2 files changed, 15 insertions(+), 12 deletions(-) delete mode 100644 docs/source/nodes/node-cluster-federation.md diff --git a/docs/source/nodes/index.rst b/docs/source/nodes/index.rst index 4142d119..abfd966a 100644 --- a/docs/source/nodes/index.rst +++ b/docs/source/nodes/index.rst @@ -1,13 +1,25 @@ .. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -BigchainDB Nodes & Clusters -=========================== +BigchainDB Nodes, Clusters & Federations +======================================== + +A **BigchainDB node** is a server or set of closely-linked servers running RethinkDB Server, BigchainDB Server, and other BigchainDB-related software. Each node is controlled by one person or organization. + +A set of BigchainDB nodes can connect to each other to form a **cluster**. Each node in the cluster runs the same software. A cluster contains one logical RethinkDB datastore. A cluster may have additional servers to do things such as cluster monitoring. + +The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization). A federation must have some sort of governance structure to make decisions. If a cluster is run by a single company, then the federation is just that company. + +**What's the Difference Between a Cluster and a Federation?** + +A cluster is just a bunch of connected nodes (computers). A cluster might be operated by just one person. A federation is an organization which has a cluster, and where each node in the cluster has a different operator. + +Confusingly, we sometimes call a federation's cluster its "federation." You can probably tell what we mean from context. .. toctree:: :maxdepth: 1 - node-cluster-federation + node-requirements installing-server running-unit-tests configuration diff --git a/docs/source/nodes/node-cluster-federation.md b/docs/source/nodes/node-cluster-federation.md deleted file mode 100644 index 57e336cf..00000000 --- a/docs/source/nodes/node-cluster-federation.md +++ /dev/null @@ -1,9 +0,0 @@ -# Terminology: Nodes, Clusters & Federations - -A **BigchainDB node** is a server or set of closely-linked servers running RethinkDB Server, BigchainDB Server, and other BigchainDB-related software. Each node is controlled by one person or organization. - -A set of BigchainDB nodes can connect to each other to form a **cluster**. Each node in the cluster runs the same software. A cluster contains one logical RethinkDB datastore. A cluster may have additional servers to do things such as cluster monitoring. - -The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization). A federation must have some sort of governance structure to make decisions. If a cluster is run by a single company, then the federation is just that company. - -Confusingly, sometimes we refer to a cluster as a federation. You can probably tell what we mean from context. From c34f383de8e7b70f3190703e2171b91d46d67213 Mon Sep 17 00:00:00 2001 From: troymc Date: Mon, 13 Jun 2016 10:55:31 +0200 Subject: [PATCH 12/27] Docs: First draft of BigchainDB Node Requirements section --- docs/source/nodes/node-requirements.md | 44 ++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/source/nodes/node-requirements.md diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md new file mode 100644 index 00000000..131b5379 --- /dev/null +++ b/docs/source/nodes/node-requirements.md @@ -0,0 +1,44 @@ +# Node Requirements (OS, Memory, Storage, etc.) + +For now, we will assume that a BigchainDB node is just one server. In the future, a node may consist of several closely-coupled servers run by one node operator (federation member). + + +## OS Requirements + +* RethinkDB Server [will run on any modern OS](https://www.rethinkdb.com/docs/install/). Note that the Fedora package isn't officially supported. Also, official support for Windows is fairly recent ([April 2016](https://rethinkdb.com/blog/2.3-release/)). +* Python 3.4+ [will run on any modern OS](https://docs.python.org/3.4/using/index.html). +* Some of the Python packages used by BigchainDB Server won't work on all OSes. In particular, [some functionality in the `multiprocessing` package doesn't work on OS X](https://docs.python.org/3.4/library/multiprocessing.html#multiprocessing.Queue.qsize). +* ZeroMQ [will run on any modern OS](http://zeromq.org/area:download). + +In summary, you can't use OS X, but any other modern OS should be okay. + +The BigchainDB core dev team uses Ubuntu 14.04 or Fedora 23. + + +## Memory Requirements + +Every OS has memory requirements; check the memory requirements of your OS. + +There is [documentation about RethinkDB's memory requirements](https://rethinkdb.com/docs/memory-usage/). In particular: "RethinkDB requires data structures in RAM on each server proportional to the size of the data on that server’s disk, usually around 1% of the size of the total data set." ([source](https://rethinkdb.com/limitations/)) + + +## Storage Requirements + +If you want a RethinkDB cluster to store an amount of data D, with a replication factor of R (on every table), and the cluster has N nodes, then each node will need to be able to store R×D/N data plus the storage required for the OS and various other software (RethinkDB, Python, etc.). The secondary indexes also require some storage. + +For failover to work, [every RethinkDB table must have at least three replicas](https://rethinkdb.com/docs/failover/), i.e. R ≥ 3. + +Also, RethinkDB tables can have [at most 64 shards](https://rethinkdb.com/limitations/). For example, if you have only one table and more than 64 nodes, some nodes won't have the primary of any shard, i.e. they will have replicas only. In other words, once you pass 64 nodes, adding more nodes won't provide storage space for new data; it will only add more space for shard replicas. If the biggest single-node storage available is d, then the most you can store in a RethinkDB cluster is < 64×d: accomplished by putting one primary shard in each of 64 nodes, with all replica shards on other nodes. (This is assuming one table. If there are T tables, then the most you can store is < 64×d×T.) + + +## Compatible File Systems + +RethinkDB has [issues with `btrfs`](https://github.com/rethinkdb/rethinkdb/issues/2781) (B-tree file system). + +If you want to use RethinkDB's `--direct-io` command-line option, then you need a file system which supports that. (Many compressed or encrypted file systems do not.) + + +## CPU Requirements + +Most servers will have enough CPUs (or vCPUs) to run a BigchainDB node. The more you have, the higher throughput will be. + From 53e14b3f9cc89eb059f369984b5353cc0e48cac4 Mon Sep 17 00:00:00 2001 From: troymc Date: Mon, 13 Jun 2016 11:26:59 +0200 Subject: [PATCH 13/27] Revised the intro section about installing BigchainDB Server --- docs/source/nodes/installing-server.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/source/nodes/installing-server.md b/docs/source/nodes/installing-server.md index 3a56fb7c..6a561d6e 100644 --- a/docs/source/nodes/installing-server.md +++ b/docs/source/nodes/installing-server.md @@ -1,19 +1,15 @@ -# Installing and Running BigchainDB Server +# Set Up and Run a Node -We're developing BigchainDB Server on Ubuntu 14.04, but it should work on any OS that runs RethinkDB Server and Python 3.4+. (BigchainDB Server is built on top of RethinkDB Server.) - -BigchainDB Server is intended to be run on each server in a large distributed cluster of servers; it's not very useful running by itself on a single computer. That's _why_ we're developing it on Ubuntu 14.04: it's one of the more common server operating systems. +First make sure your server(s) meet [the requirements for a BigchainDB node](node-requirements.html). Mac OS X users may get the best results [running BigchainDB Server with Docker](installing-server.html#run-bigchaindb-with-docker). -Windows users may get the best results [running BigchainDB Server in a VM with Vagrant](installing-server.html#how-to-install-bigchaindb-on-a-vm-with-vagrant). - -(BigchainDB clients should run on a much larger array of operating systems.) +We currently don't test BigchainDB on Windows. If you run into problems on Windows, then you may want to try [using Vagrant](installing-server.html#how-to-install-bigchaindb-on-a-vm-with-vagrant). ## Install and Run RethinkDB Server -If you don't already have RethinkDB Server installed on your server, you must install it. The RethinkDB documentation has instructions for [how to install RethinkDB Server on a variety of operating systems](http://rethinkdb.com/docs/install/). +If you don't already have RethinkDB Server installed, you must install it. The RethinkDB documentation has instructions for [how to install RethinkDB Server on a variety of operating systems](http://rethinkdb.com/docs/install/). RethinkDB Server doesn't require any special configuration. You can run it by opening a Terminal and entering: ```text From 06b261ebb425022003a48db46902f44d2131a19d Mon Sep 17 00:00:00 2001 From: troymc Date: Mon, 13 Jun 2016 11:27:53 +0200 Subject: [PATCH 14/27] Revised OS X remarks in Node Requirements --- docs/source/nodes/node-requirements.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md index 131b5379..7cc75639 100644 --- a/docs/source/nodes/node-requirements.md +++ b/docs/source/nodes/node-requirements.md @@ -7,11 +7,9 @@ For now, we will assume that a BigchainDB node is just one server. In the future * RethinkDB Server [will run on any modern OS](https://www.rethinkdb.com/docs/install/). Note that the Fedora package isn't officially supported. Also, official support for Windows is fairly recent ([April 2016](https://rethinkdb.com/blog/2.3-release/)). * Python 3.4+ [will run on any modern OS](https://docs.python.org/3.4/using/index.html). -* Some of the Python packages used by BigchainDB Server won't work on all OSes. In particular, [some functionality in the `multiprocessing` package doesn't work on OS X](https://docs.python.org/3.4/library/multiprocessing.html#multiprocessing.Queue.qsize). +* [Some functionality in the `multiprocessing` package doesn't work on OS X](https://docs.python.org/3.4/library/multiprocessing.html#multiprocessing.Queue.qsize). You can still use Mac OS X if you use Docker or a virtual machine. * ZeroMQ [will run on any modern OS](http://zeromq.org/area:download). -In summary, you can't use OS X, but any other modern OS should be okay. - The BigchainDB core dev team uses Ubuntu 14.04 or Fedora 23. From 0ce79f4b4753a11817a2cfdc849b1c89419c2569 Mon Sep 17 00:00:00 2001 From: troymc Date: Mon, 13 Jun 2016 11:39:51 +0200 Subject: [PATCH 15/27] Renamed installing-server.md to setup-run-node.md --- README.md | 4 ++-- docs/source/appendices/local-rethinkdb-cluster.md | 2 +- docs/source/nodes/index.rst | 2 +- docs/source/nodes/monitoring.md | 2 +- docs/source/nodes/python-server-api-examples.md | 2 +- docs/source/nodes/running-unit-tests.md | 2 +- docs/source/nodes/{installing-server.md => setup-run-node.md} | 4 ++-- 7 files changed, 9 insertions(+), 9 deletions(-) rename docs/source/nodes/{installing-server.md => setup-run-node.md} (97%) diff --git a/README.md b/README.md index 979239f7..56286ad9 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,8 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit ## Quick Start -### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.io/en/latest/nodes/installing-server.html) -### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.io/en/latest/installing-server.html#run-bigchaindb-with-docker) +### [Set Up and Run a BigchainDB Node](http://bigchaindb.readthedocs.io/en/latest/nodes/setup-run-node.html) +### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.io/en/latest/setup-run-node.html#run-bigchaindb-with-docker) ### [The Python Server API by Example](http://bigchaindb.readthedocs.io/en/latest/nodes/python-server-api-examples.html) ### [The Python Driver API by Example](http://bigchaindb.readthedocs.io/en/latest/drivers-clients/python-driver-api-examples.html) diff --git a/docs/source/appendices/local-rethinkdb-cluster.md b/docs/source/appendices/local-rethinkdb-cluster.md index 518fd261..948e3f40 100644 --- a/docs/source/appendices/local-rethinkdb-cluster.md +++ b/docs/source/appendices/local-rethinkdb-cluster.md @@ -4,7 +4,7 @@ This section explains one way to deploy a multi-node RethinkDB cluster on one ma ## Launching More RethinkDB Nodes -Assuming you've already [installed RethinkDB](../nodes/installing-server.html#install-and-run-rethinkdb-server) and have one RethinkDB node running, here's how you can launch two more nodes on the same machine. First, prepare two additional nodes. Note that the user who starts RethinkDB must have write access to the created directories: +Assuming you've already [installed RethinkDB](../nodes/setup-run-node.html#install-and-run-rethinkdb-server) and have one RethinkDB node running, here's how you can launch two more nodes on the same machine. First, prepare two additional nodes. Note that the user who starts RethinkDB must have write access to the created directories: mkdir -p /path/to/node2 mkdir -p /path/to/node3 diff --git a/docs/source/nodes/index.rst b/docs/source/nodes/index.rst index abfd966a..20fa4e33 100644 --- a/docs/source/nodes/index.rst +++ b/docs/source/nodes/index.rst @@ -20,7 +20,7 @@ Confusingly, we sometimes call a federation's cluster its "federation." You can :maxdepth: 1 node-requirements - installing-server + setup-run-node running-unit-tests configuration bigchaindb-cli diff --git a/docs/source/nodes/monitoring.md b/docs/source/nodes/monitoring.md index 1da58e4d..626230fe 100644 --- a/docs/source/nodes/monitoring.md +++ b/docs/source/nodes/monitoring.md @@ -28,7 +28,7 @@ You can view the Grafana dashboard in your web browser at: (You may want to replace `localhost` with another hostname in that URL, e.g. the hostname of a remote monitoring server.) -The login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](installing-server.html#run-bigchaindb) and load some test transactions: +The login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](setup-run-node.html#run-bigchaindb) and load some test transactions: ```text $ bigchaindb load ``` diff --git a/docs/source/nodes/python-server-api-examples.md b/docs/source/nodes/python-server-api-examples.md index 8aa97ee1..8b4ccd80 100644 --- a/docs/source/nodes/python-server-api-examples.md +++ b/docs/source/nodes/python-server-api-examples.md @@ -8,7 +8,7 @@ We create a digital asset, sign it, write it to a BigchainDB Server instance, re ## Getting Started -First, make sure you have RethinkDB and BigchainDB _installed and running_, i.e. you [installed them](installing-server.html) and you ran: +First, make sure you have RethinkDB and BigchainDB _installed and running_, i.e. you [installed them](setup-run-node.html) and you ran: ```text $ rethinkdb $ bigchaindb configure diff --git a/docs/source/nodes/running-unit-tests.md b/docs/source/nodes/running-unit-tests.md index f8fc4809..17c75256 100644 --- a/docs/source/nodes/running-unit-tests.md +++ b/docs/source/nodes/running-unit-tests.md @@ -2,7 +2,7 @@ Once you've installed BigchainDB Server, you may want to run all the unit tests. This section explains how. -First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. **Before you can run all the unit tests, you must [install BigchainDB from source](installing-server.html#how-to-install-bigchaindb-from-source).** +First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. **Before you can run all the unit tests, you must [install BigchainDB from source](setup-run-node.html#how-to-install-bigchaindb-from-source).** To run all the unit tests, first make sure you have RethinkDB running: diff --git a/docs/source/nodes/installing-server.md b/docs/source/nodes/setup-run-node.md similarity index 97% rename from docs/source/nodes/installing-server.md rename to docs/source/nodes/setup-run-node.md index 6a561d6e..bcc9f135 100644 --- a/docs/source/nodes/installing-server.md +++ b/docs/source/nodes/setup-run-node.md @@ -2,9 +2,9 @@ First make sure your server(s) meet [the requirements for a BigchainDB node](node-requirements.html). -Mac OS X users may get the best results [running BigchainDB Server with Docker](installing-server.html#run-bigchaindb-with-docker). +Mac OS X users may get the best results [running BigchainDB Server with Docker](setup-run-node.html#run-bigchaindb-with-docker). -We currently don't test BigchainDB on Windows. If you run into problems on Windows, then you may want to try [using Vagrant](installing-server.html#how-to-install-bigchaindb-on-a-vm-with-vagrant). +We currently don't test BigchainDB on Windows. If you run into problems on Windows, then you may want to try [using Vagrant](setup-run-node.html#how-to-install-bigchaindb-on-a-vm-with-vagrant). ## Install and Run RethinkDB Server From 1670362500d781b0a082630269b7fceeb34d6725 Mon Sep 17 00:00:00 2001 From: troymc Date: Mon, 13 Jun 2016 17:18:04 +0200 Subject: [PATCH 16/27] WIP: many revisions to node deployment (for clusters) --- docs/source/nodes/federation-set-up.md | 61 +++++++++++++++++---- docs/source/nodes/node-requirements.md | 5 ++ docs/source/nodes/setup-run-node.md | 76 +++++++++++++++++++++----- 3 files changed, 117 insertions(+), 25 deletions(-) diff --git a/docs/source/nodes/federation-set-up.md b/docs/source/nodes/federation-set-up.md index faff16f2..0df165a1 100644 --- a/docs/source/nodes/federation-set-up.md +++ b/docs/source/nodes/federation-set-up.md @@ -1,16 +1,57 @@ -# How to Set Up a Decentralized BigchainDB Federation +# Set Up and Run a Federation -This section outlines the steps to _set up_ a decentralized BigchainDB federation. - -If you want to deploy a centrally-controlled BigchainDB cluster on AWS (e.g. for testing), then see [the section about that](deploy-on-aws.html). +This section is about how to set up and run a BigchainDB _federation_, where each node is operated by a different operator. If you want to set up and run a BigchainDB cluster on AWS (where all nodes are operated by you), then see [the section about that](deploy-on-aws.html). -## INSERT SUBSECTION TITLE HERE +## Answer Basic Questions -What kinds of information will the federation store? BigchainDB is well-suited to storing asset registrations and ownership transfers, but it can handle many other kinds of data as well. - -How will assets be described? Do you have a data model (schema)? - -Who (or what organizations) will run the nodes? +* Do you have a governance process for making federation-level decisions, such as how to admit new members? +* What will you store in creation transactions (data payload)? Is there a data schema? +* Will you use transfer transactions? Will they include a non-empty data payload? +* Who will be allowed to submit transactions? Who will be allowed to read or query transactions? How will you enforce the access rules? +## Set Up the Initial Cluster + +When you first start a federation cluster, the initial nodes will all start at roughly the same time. Here are the steps to be taken by each node operator: + +**Create a RethinkDB Cluster** + +* Go through the steps of [Set Up and Run a Node](setup-run-node.html), up to "Configure RethinkDB Server" +* Determine the hostname of the server running RethinkDB (e.g. `rethinkdb2.farm3.organization5.com`) +* Share your RethinkDB hostname with all other members of the federation +* Once you have all the RethinkDB hostnames, add them to your local RethinkDB configuration file as explained in [Configure RethinkDB Server](setup-run-node.html#configure-rethinkdb-server) +* (TODO: Section with steps to make RethinkDB more secure) + +**Install, Configure and Run BigchainDB** + +* Continue with the steps of [Set Up and Run a Node](setup-run-node.html) +* You will generate a default `.bigchaindb` config file at some point. It contains your public/private keypair. +* Send your **public** key to all other federation members. If you accidentally send your private key, then delete your `.bigchaindb` config file, generate a new one, and send your new public key to all other federation members. +* Once you have the public keys of all other federation members, put them in the `keyring` (list) in your `.bigchaindb` file. (Your keyring should _not_ include your public key.) + +* TODO: Are there other things they should change in their `.bigchaindb` file? + +* Make sure RethinkDB is Running (TODO: say how) +* Start BigchainDB + +```text +$ bigchaindb start +``` + + + +## Documentation to Come + +* Backing Up & Restoring data +* Adding a node (including resharding etc.) +* Removing a node +* Node monitoring & crash recovery +* Node Security + * Securing your OS + * Firewalls and security groups + * (Private) key management + * RethinkDB security +* Cluster monitoring +* Internal watchdogs +* External watchdogs \ No newline at end of file diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md index 7cc75639..1942f167 100644 --- a/docs/source/nodes/node-requirements.md +++ b/docs/source/nodes/node-requirements.md @@ -12,6 +12,11 @@ For now, we will assume that a BigchainDB node is just one server. In the future The BigchainDB core dev team uses Ubuntu 14.04 or Fedora 23. +We don't test BigchainDB on Windows or Mac OS X, but you can try. + +* If you run into problems on Windows, then you may want to try using Vagrant. One of our community members ([@Mec-Is](https://github.com/Mec-iS)) wrote [a page about how to install BigchainDB on a VM with Vagrant](https://gist.github.com/Mec-iS/b84758397f1b21f21700). +* Mac OS X users may get the best results [running BigchainDB with Docker](setup-run-node.html#run-bigchaindb-with-docker). + ## Memory Requirements diff --git a/docs/source/nodes/setup-run-node.md b/docs/source/nodes/setup-run-node.md index bcc9f135..62827c75 100644 --- a/docs/source/nodes/setup-run-node.md +++ b/docs/source/nodes/setup-run-node.md @@ -1,25 +1,58 @@ # Set Up and Run a Node -First make sure your server(s) meet [the requirements for a BigchainDB node](node-requirements.html). +This section goes through the steps to set up a BigchainDB node (running RethinkDB Server, BigchainDB Server, etc.). There are instructions for two cases: -Mac OS X users may get the best results [running BigchainDB Server with Docker](setup-run-node.html#run-bigchaindb-with-docker). - -We currently don't test BigchainDB on Windows. If you run into problems on Windows, then you may want to try [using Vagrant](setup-run-node.html#how-to-install-bigchaindb-on-a-vm-with-vagrant). +1. Stand-Alone Node (useful for local development) +2. Cluster Node -## Install and Run RethinkDB Server +## Check the Node Requirements + +The first step is to make sure you have a server (or equivalent) which meets [the requirements for a BigchainDB node](node-requirements.html). + + +## Set Up the File System + +This step isn't always necessary. + +See `def install_rethinkdb()` in `deploy-cluster-aws/fabfile.py` for an example. + +TODO: This step needs more explanation. + + +## Install RethinkDB Server If you don't already have RethinkDB Server installed, you must install it. The RethinkDB documentation has instructions for [how to install RethinkDB Server on a variety of operating systems](http://rethinkdb.com/docs/install/). -RethinkDB Server doesn't require any special configuration. You can run it by opening a Terminal and entering: + +## Configure RethinkDB Server + +### Stand-Alone Node + +If you're setting up a stand-alone node (i.e. not intending for it to join a cluster), then the default RethinkDB configuration will work. + +### Cluster Node + +If you're setting up a node that will be part of a RethinkDB cluster, then you need to... + +TODO: (edit the default .conf file: add list of all hostnames in the federation) + +TODO: Steps to make the RethinkDB cluster more secure. + + +## Run RethinkDB Server + +You can run RethinkDB by opening a Terminal and entering: ```text $ rethinkdb ``` + ## Install Python 3.4+ If you don't already have it, then you should [install Python 3.4+](https://www.python.org/downloads/) (maybe in a virtual environment, so it doesn't conflict with other Python projects you're working on). + ## Install BigchainDB Server BigchainDB Server has some OS-level dependencies. @@ -40,6 +73,7 @@ $ sudo dnf install gcc-c++ redhat-rpm-config python3-devel With OS-level dependencies installed, you can install BigchainDB Server with `pip` or from source. + ### How to Install BigchainDB with pip BigchainDB (i.e. both the Server and the officially-supported drivers) is distributed as a Python package on PyPI so you can install it using `pip`. First, make sure you have a version of `pip` installed for Python 3.4+: @@ -78,9 +112,28 @@ and then install from source: $ python setup.py install ``` -### How to Install BigchainDB on a VM with Vagrant -One of our community members ([@Mec-Is](https://github.com/Mec-iS)) wrote [a page about how to install BigchainDB on a VM with Vagrant](https://gist.github.com/Mec-iS/b84758397f1b21f21700). +## Configure BigchainDB Server + +Start by creating a default BigchainDB configuration file (named `.bigchaindb`) in your `$HOME` directory using: +```text +$ bigchaindb -y configure +``` + +There's documentation for the `bigchaindb` command is in the section on [the BigchainDB Command Line Interface (CLI)](bigchaindb-cli.html). + +### Stand-Alone Node + +The default BigchainDB configuration file will work. + +### Cluster Node + +Open `$HOME/.bigchaindb` in your text editor and: + +* Change the keyring (list) to include the public keys of all the other nodes in the federation. The keyring should _not_ include your node's public key. +* TODO: Make other changes to the BigchainDB config file? + +For more information about the BigchainDB config file, see [Configuring a BigchainDB Node](configuration.html). ## Run BigchainDB Server @@ -90,13 +143,6 @@ Once you've installed BigchainDB Server, you can run it. First make sure you hav $ rethinkdb ``` -Then open a different terminal and run: -```text -$ bigchaindb -y configure -``` - -That creates a configuration file in `$HOME/.bigchaindb` (documented in [the section on configuration](configuration.html)). More documentation about the `bigchaindb` command is in the section on [the BigchainDB Command Line Interface (CLI)](bigchaindb-cli.html). - You can start BigchainDB Server using: ```text $ bigchaindb start From 9a3fd04e40adb9dc4fb950e698b89e8b0c9738bb Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 10:48:06 +0200 Subject: [PATCH 17/27] Docs on NTP daemon setup, RethinkDB config for cluster nodes --- docs/source/appendices/index.rst | 1 + docs/source/appendices/ntp-notes.md | 41 +++++++++++++ docs/source/nodes/setup-run-node.md | 89 +++++++++++++++++++---------- 3 files changed, 102 insertions(+), 29 deletions(-) create mode 100644 docs/source/appendices/ntp-notes.md diff --git a/docs/source/appendices/index.rst b/docs/source/appendices/index.rst index 353e7909..f621d4bd 100644 --- a/docs/source/appendices/index.rst +++ b/docs/source/appendices/index.rst @@ -11,5 +11,6 @@ Appendices cryptography the-Bigchain-class consensus + ntp-notes local-rethinkdb-cluster licenses \ No newline at end of file diff --git a/docs/source/appendices/ntp-notes.md b/docs/source/appendices/ntp-notes.md new file mode 100644 index 00000000..7d9c5ea5 --- /dev/null +++ b/docs/source/appendices/ntp-notes.md @@ -0,0 +1,41 @@ +# Notes on NTP Daemon Setup + +As mentioned [elsewhere in these docs](../nodes/setup-run-node.html), there are several NTP daemons available, including: + +* The reference NTP daemon from ntp.org; see [their support website](http://support.ntp.org/bin/view/Support/WebHome) +* [OpenNTPD](http://www.openntpd.org/) +* [chrony](https://chrony.tuxfamily.org/index.html) +* Maybe [NTPsec](https://www.ntpsec.org/), once it's production-ready +* Maybe [Ntimed](http://nwtime.org/projects/ntimed/), once it's production-ready +* [More](https://en.wikipedia.org/wiki/Ntpd#Implementations) + +It's tricky to make an NTP daemon setup secure. Always install the latest version and read the documentation about how to configure and run it securely. + + +## Ubuntu Packages + +The [Ubuntu 14.04 (Trusty Tahr) package `ntp`](https://launchpad.net/ubuntu/trusty/+source/ntp) is based on the reference implementation of an NTP daemon. + +The following commands will uninstall the `ntp` and `ntpdate` packages, install the latest `ntp` package (which _might not be based on the latest ntpd code_), and start the NTP daemon (a local NTP server). (`ntpdate` is not reinstalled because it's [deprecated](https://askubuntu.com/questions/297560/ntpd-vs-ntpdate-pros-and-cons) and you shouldn't use it.) +```text +sudo apt-get --purge remove ntp ntpdate +sudo apt-get autoremove +sudo apt-get update +sudo apt-get install ntp +# That should start the NTP daemon too, but just to be sure: +sudo service ntp restart +``` + +You can check if NTP is running using `sudo ntpq -p`. + +You may want to use different NTP time servers. You can change them by editing the NTP config file `/etc/ntp.conf`. + +Note: A server running the NTP daemon can be used by others for DRDoS amplification attacks. The above installation procedure should install a default NTP configuration file `/etc/ntp.conf` with the lines: +```text +restrict -4 default kod notrap nomodify nopeer noquery +restrict -6 default kod notrap nomodify nopeer noquery +``` + +Those lines should prevent the NTP daemon from being used in an attack. (The first line is for IPv4, the second for IPv6.) + +There are additional things you can do to make NTP more secure. See the [NTP Support Website](http://support.ntp.org/bin/view/Support/WebHome) for more details. diff --git a/docs/source/nodes/setup-run-node.md b/docs/source/nodes/setup-run-node.md index 62827c75..77b80f03 100644 --- a/docs/source/nodes/setup-run-node.md +++ b/docs/source/nodes/setup-run-node.md @@ -11,13 +11,33 @@ This section goes through the steps to set up a BigchainDB node (running Rethink The first step is to make sure you have a server (or equivalent) which meets [the requirements for a BigchainDB node](node-requirements.html). -## Set Up the File System +## System Clock Sync -This step isn't always necessary. +If you're just setting up a stand-alone node, then you can skip this step. -See `def install_rethinkdb()` in `deploy-cluster-aws/fabfile.py` for an example. +A BigchainDB node uses its system clock to generate timestamps for blocks and votes, so that clock should be kept in sync with some standard clock(s). The standard way to do that is to run an NTP daemon (Network Time Protocol daemon) on the node. (You could also use tlsdate, which uses TLS timestamps rather than NTP, but don't: it's not very accurate and it will break with TLS 1.3, which removes the timestamp.) -TODO: This step needs more explanation. +NTP is a standard protocol. There are many NTP daemons implementing it. We don't recommend a particular one. On the contrary, we recommend that different nodes in a federation run different NTP daemons, so that a problem with one daemon won't affect all nodes. Some options are: + +* The reference NTP daemon from ntp.org; see [their support website](http://support.ntp.org/bin/view/Support/WebHome) +* [OpenNTPD](http://www.openntpd.org/) +* [chrony](https://chrony.tuxfamily.org/index.html) +* Maybe [NTPsec](https://www.ntpsec.org/), once it's production-ready +* Maybe [Ntimed](http://nwtime.org/projects/ntimed/), once it's production-ready +* [More](https://en.wikipedia.org/wiki/Ntpd#Implementations) + +It's tricky to make an NTP daemon setup secure. Always install the latest version and read the documentation about how to configure and run it securely. + +The appendix has [some notes on NTP daemon setup](../appendices/ntp-notes.html). + + +## Set Up the File System for RethinkDB + +If you're just setting up a stand-alone node, then you can probably skip this step. + +See `def install_rethinkdb()` in `deploy-cluster-aws/fabfile.py` for an example of configuring a file system on an AWS instance running Ubuntu. + +TODO: This section needs more elaboration ## Install RethinkDB Server @@ -33,19 +53,28 @@ If you're setting up a stand-alone node (i.e. not intending for it to join a clu ### Cluster Node -If you're setting up a node that will be part of a RethinkDB cluster, then you need to... +If you're setting up a node that will be part of a RethinkDB cluster, then it needs to find out the hostnames of all the other nodes somehow. You _could_ designate one node to be the one that every other node asks, and put that node's hostname in the RethinkDB config file, but that wouldn't be very decentralized. Instead, we will list _every_ node in the list of nodes-to-ask: + +1. Download the [sample RethinkDB config file from RethinkDB](https://github.com/rethinkdb/rethinkdb/blob/next/packaging/assets/config/default.conf.sample). +2. Edit that file to add one line for each node in the federation, like so: +```text +join=node0_hostname:29015 +join=node1_hostname:29015 +join=node2_hostname:29015 +# continue until all federation node hostnames are included +``` + +where the hostnames must be replaced by the hostnames of the RethinkDB servers in the cluster, e.g. `jackfish.servers.organization45.net`. + +You may want to change other things in the RethinkDB config file as well. RethinkDB has [a documentation page with more info](https://www.rethinkdb.com/docs/config-file/). -TODO: (edit the default .conf file: add list of all hostnames in the federation) TODO: Steps to make the RethinkDB cluster more secure. ## Run RethinkDB Server -You can run RethinkDB by opening a Terminal and entering: -```text -$ rethinkdb -``` +You can run RethinkDB by opening a Terminal and entering `rethinkdb`. You could do that now or wait until just before you start BigchainDB. ## Install Python 3.4+ @@ -59,14 +88,14 @@ BigchainDB Server has some OS-level dependencies. On Ubuntu 14.04, we found that the following was enough: ```text -$ sudo apt-get update -$ sudo apt-get install g++ python3-dev +sudo apt-get update +sudo apt-get install g++ python3-dev ``` On Fedora 23, we found that the following was enough (tested in February 2015): ```text -$ sudo dnf update -$ sudo dnf install gcc-c++ redhat-rpm-config python3-devel +sudo dnf update +sudo dnf install gcc-c++ redhat-rpm-config python3-devel ``` (If you're using a version of Fedora before version 22, you may have to use `yum` instead of `dnf`.) @@ -78,14 +107,14 @@ With OS-level dependencies installed, you can install BigchainDB Server with `pi BigchainDB (i.e. both the Server and the officially-supported drivers) is distributed as a Python package on PyPI so you can install it using `pip`. First, make sure you have a version of `pip` installed for Python 3.4+: ```text -$ pip -V +pip -V ``` If it says that `pip` isn't installed, or it says `pip` is associated with a Python version less than 3.4, then you must install a `pip` version associated with Python 3.4+. See [the `pip` installation instructions](https://pip.pypa.io/en/stable/installing/). On Ubuntu 14.04, we found that this works: ```text -$ sudo apt-get install python3-setuptools -$ sudo easy_install3 pip -$ pip3 install --upgrade pip wheel setuptools +sudo apt-get install python3-setuptools +sudo easy_install3 pip +pip3 install --upgrade pip wheel setuptools ``` (Note: Using `sudo apt-get python3-pip` also installs a Python 3 version of `pip` (named `pip3`) but we found it installed a very old version and there were issues with updating it.) @@ -104,12 +133,12 @@ If you want to install BitchainDB from source because you want to contribute cod Otherwise, clone the public repository: ```text -$ git clone git@github.com:bigchaindb/bigchaindb.git +git clone git@github.com:bigchaindb/bigchaindb.git ``` and then install from source: ```text -$ python setup.py install +python setup.py install ``` @@ -117,7 +146,7 @@ $ python setup.py install Start by creating a default BigchainDB configuration file (named `.bigchaindb`) in your `$HOME` directory using: ```text -$ bigchaindb -y configure +bigchaindb -y configure ``` There's documentation for the `bigchaindb` command is in the section on [the BigchainDB Command Line Interface (CLI)](bigchaindb-cli.html). @@ -140,12 +169,14 @@ For more information about the BigchainDB config file, see [Configuring a Bigcha Once you've installed BigchainDB Server, you can run it. First make sure you have RethinkDB running: ```text -$ rethinkdb +rethinkdb ``` +You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at `http://rethinkdb-hostname:8080/`. If you're running RethinkDB on localhost, that would be [http://localhost:8080/](http://localhost:8080/). + You can start BigchainDB Server using: ```text -$ bigchaindb start +bigchaindb start ``` If it's the first time you've run `bigchaindb start`, then it creates the database (a RethinkDB database), the tables, the indexes, and the genesis block. It then starts BigchainDB. If you're run `bigchaindb start` or `bigchaindb init` before (and you haven't dropped the database), then `bigchaindb start` just starts BigchainDB. @@ -173,7 +204,7 @@ the `-y` option to accept all the default values. The configuration file will be stored in a file on your host machine at `~/bigchaindb_docker/.bigchaindb`: ```text -$ docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ +docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ bigchaindb/bigchaindb -y configure Generating keypair Configuration written to /data/.bigchaindb @@ -198,7 +229,7 @@ Let's analyze that command: After configuring the system, you can run BigchainDB with the following command: ```text -$ docker run -v "$HOME/bigchaindb_docker:/data" -d \ +docker run -v "$HOME/bigchaindb_docker:/data" -d \ --name bigchaindb \ -p "58080:8080" -p "59984:9984" \ bigchaindb/bigchaindb start @@ -242,7 +273,7 @@ You should see a container named `bigchaindb` in the list. You can load test the BigchainDB running in that container by running the `bigchaindb load` command in a second container: ```text -$ docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ +docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ --link bigchaindb \ bigchaindb/bigchaindb load ``` @@ -253,7 +284,7 @@ Aside: The `bigchaindb load` command has several options (e.g. `-m`). You can re If you look at the RethinkDB dashboard (in your web browser), you should see the effects of the load test. You can also see some effects in the Docker logs using: ```text -$ docker logs -f bigchaindb +docker logs -f bigchaindb ``` ### Building Your Own Image @@ -262,12 +293,12 @@ Assuming you have Docker installed, you would proceed as follows. In a terminal shell: ```text -$ git clone git@github.com:bigchaindb/bigchaindb.git +git clone git@github.com:bigchaindb/bigchaindb.git ``` Build the Docker image: ```text -$ docker build --tag local-bigchaindb . +docker build --tag local-bigchaindb . ``` Now you can use your own image to run BigchainDB containers. From 7741bae7e6cc6f25393d3cad9a3c642c0137e65f Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 10:49:43 +0200 Subject: [PATCH 18/27] Docs: minor edit to text re: Mac OS X --- docs/source/nodes/node-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md index 1942f167..e2c472fd 100644 --- a/docs/source/nodes/node-requirements.md +++ b/docs/source/nodes/node-requirements.md @@ -15,7 +15,7 @@ The BigchainDB core dev team uses Ubuntu 14.04 or Fedora 23. We don't test BigchainDB on Windows or Mac OS X, but you can try. * If you run into problems on Windows, then you may want to try using Vagrant. One of our community members ([@Mec-Is](https://github.com/Mec-iS)) wrote [a page about how to install BigchainDB on a VM with Vagrant](https://gist.github.com/Mec-iS/b84758397f1b21f21700). -* Mac OS X users may get the best results [running BigchainDB with Docker](setup-run-node.html#run-bigchaindb-with-docker). +* If you have Mac OS X and want to experiment with BigchainDB, then you could do that [using Docker](setup-run-node.html#run-bigchaindb-with-docker). ## Memory Requirements From 670c0bed0e7310b9d4605d589b2ba8ec249f7dbd Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 12:14:34 +0200 Subject: [PATCH 19/27] Docs: revised node requirements re: direct I/O support --- docs/source/nodes/node-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md index e2c472fd..5177edf5 100644 --- a/docs/source/nodes/node-requirements.md +++ b/docs/source/nodes/node-requirements.md @@ -38,7 +38,7 @@ Also, RethinkDB tables can have [at most 64 shards](https://rethinkdb.com/limita RethinkDB has [issues with `btrfs`](https://github.com/rethinkdb/rethinkdb/issues/2781) (B-tree file system). -If you want to use RethinkDB's `--direct-io` command-line option, then you need a file system which supports that. (Many compressed or encrypted file systems do not.) +It's best to have a file system that supports Direct I/O, because that will improve RethinkDB performance. Many compressed or encrypted file systems don't support Direct I/O. ## CPU Requirements From f80449643b3b871ca659712f8d5a4fd849605a75 Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 14:06:03 +0200 Subject: [PATCH 20/27] Docs: updates to file system setup, RethinkDB config, BigchainDB config --- docs/source/nodes/setup-run-node.md | 38 +++++++++++++++++++---------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/docs/source/nodes/setup-run-node.md b/docs/source/nodes/setup-run-node.md index 77b80f03..6e128e1e 100644 --- a/docs/source/nodes/setup-run-node.md +++ b/docs/source/nodes/setup-run-node.md @@ -35,8 +35,14 @@ The appendix has [some notes on NTP daemon setup](../appendices/ntp-notes.html). If you're just setting up a stand-alone node, then you can probably skip this step. +Ideally, use a file system that supports direct I/O (Input/Output), a feature whereby file reads and writes go directly from RethinkDB to the storage device, bypassing the operating system read and write caches. + +TODO: What file systems support direct I/O? How can you check? How do you enable it, if necessary? + See `def install_rethinkdb()` in `deploy-cluster-aws/fabfile.py` for an example of configuring a file system on an AWS instance running Ubuntu. +Mount the partition for RethinkDB on `/data`: we will tell RethinkDB to store its data there. + TODO: This section needs more elaboration @@ -53,23 +59,26 @@ If you're setting up a stand-alone node (i.e. not intending for it to join a clu ### Cluster Node -If you're setting up a node that will be part of a RethinkDB cluster, then it needs to find out the hostnames of all the other nodes somehow. You _could_ designate one node to be the one that every other node asks, and put that node's hostname in the RethinkDB config file, but that wouldn't be very decentralized. Instead, we will list _every_ node in the list of nodes-to-ask: - -1. Download the [sample RethinkDB config file from RethinkDB](https://github.com/rethinkdb/rethinkdb/blob/next/packaging/assets/config/default.conf.sample). -2. Edit that file to add one line for each node in the federation, like so: +Create a RethinkDB configuration file (text file) named `instance.conf` in the `/etc/rethinkdb/instances.d/` directory with the following contents (explained below): ```text +directory=/data +bind=all +direct-io +# Replace node?_hostname with actual node hostnames below, e.g. rdb.examples.com join=node0_hostname:29015 join=node1_hostname:29015 join=node2_hostname:29015 -# continue until all federation node hostnames are included +# continue until there's a join= line for each node in the federation ``` -where the hostnames must be replaced by the hostnames of the RethinkDB servers in the cluster, e.g. `jackfish.servers.organization45.net`. +* `directory=/data` tells the RethinkDB node to store its share of the database data in `/data`. +* `bind=all` binds RethinkDB to all local network interfaces (e.g. loopback, Ethernet, wireless, whatever is available), so it can communicate with the outside world. (The default is to bind only to local interfaces.) +* `direct-io` tells RethinkDB to use direct I/O (explained earlier). +* `join=hostname:29015` lines: A cluster node needs to find out the hostnames of all the other nodes somehow. You _could_ designate one node to be the one that every other node asks, and put that node's hostname in the config file, but that wouldn't be very decentralized. Instead, we include _every_ node in the list of nodes-to-ask. -You may want to change other things in the RethinkDB config file as well. RethinkDB has [a documentation page with more info](https://www.rethinkdb.com/docs/config-file/). +If you're curious about the RethinkDB config file, there's [a RethinkDB documentation page about it](https://www.rethinkdb.com/docs/config-file/). - -TODO: Steps to make the RethinkDB cluster more secure. +TODO: Explain how to configure the RethinkDB cluster to be more secure. ## Run RethinkDB Server @@ -79,12 +88,14 @@ You can run RethinkDB by opening a Terminal and entering `rethinkdb`. You could ## Install Python 3.4+ -If you don't already have it, then you should [install Python 3.4+](https://www.python.org/downloads/) (maybe in a virtual environment, so it doesn't conflict with other Python projects you're working on). +If you don't already have it, then you should [install Python 3.4+](https://www.python.org/downloads/). + +If you're testing or developing BigchainDB on a stand-alone node, then you should probably create a Python 3.4+ virtual environment and activate it (e.g. using virtualenv or conda). Later we will install several Python packages and you probably only want those installed in the virtual environment. ## Install BigchainDB Server -BigchainDB Server has some OS-level dependencies. +BigchainDB Server has some OS-level dependencies that must be installed. On Ubuntu 14.04, we found that the following was enough: ```text @@ -159,8 +170,9 @@ The default BigchainDB configuration file will work. Open `$HOME/.bigchaindb` in your text editor and: -* Change the keyring (list) to include the public keys of all the other nodes in the federation. The keyring should _not_ include your node's public key. -* TODO: Make other changes to the BigchainDB config file? +* Change `"server": {"bind": "localhost:9984", ... }` to `"server": {"bind": "0.0.0.0:9984", ... }`. This makes it so traffic can come from any IP address to port 9984 (the HTTP Client-Server API port). +* Change `"api_endpoint": "http://localhost:9984/api/v1"` to `"api_endpoint": "http://your_api_hostname:9984/api/v1"` +* Change `"keyring": []` to `"keyring": ["public_key_of_other_node_A", "public_key_of_other_node_B", "..."]` i.e. a list of the public keys of all the other nodes in the federation. The keyring should _not_ include your node's public key. For more information about the BigchainDB config file, see [Configuring a BigchainDB Node](configuration.html). From 13985b4bbc586f423a593c2e751f3741f4aca96b Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 14:07:03 +0200 Subject: [PATCH 21/27] Docs: minor additions to section on federation setup --- docs/source/nodes/federation-set-up.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/nodes/federation-set-up.md b/docs/source/nodes/federation-set-up.md index 0df165a1..68b47f3f 100644 --- a/docs/source/nodes/federation-set-up.md +++ b/docs/source/nodes/federation-set-up.md @@ -46,10 +46,11 @@ $ bigchaindb start * Backing Up & Restoring data * Adding a node (including resharding etc.) * Removing a node +* Logging * Node monitoring & crash recovery * Node Security * Securing your OS - * Firewalls and security groups + * Firewalls and security groups. Remember to open port 123 for NTP. * (Private) key management * RethinkDB security * Cluster monitoring From fd6828a0f67d4f3a07c754e2e0df61acc209941c Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 14:55:12 +0200 Subject: [PATCH 22/27] Move the Docker docs to a separate page --- docs/source/nodes/index.rst | 3 +- docs/source/nodes/run-with-docker.md | 120 +++++++++++++++++++++++++ docs/source/nodes/setup-run-node.md | 130 ++------------------------- 3 files changed, 130 insertions(+), 123 deletions(-) create mode 100644 docs/source/nodes/run-with-docker.md diff --git a/docs/source/nodes/index.rst b/docs/source/nodes/index.rst index 20fa4e33..f9a03180 100644 --- a/docs/source/nodes/index.rst +++ b/docs/source/nodes/index.rst @@ -20,11 +20,12 @@ Confusingly, we sometimes call a federation's cluster its "federation." You can :maxdepth: 1 node-requirements + federation-set-up setup-run-node + run-with-docker running-unit-tests configuration bigchaindb-cli python-server-api-examples - federation-set-up.md deploy-on-aws monitoring diff --git a/docs/source/nodes/run-with-docker.md b/docs/source/nodes/run-with-docker.md new file mode 100644 index 00000000..e39766d1 --- /dev/null +++ b/docs/source/nodes/run-with-docker.md @@ -0,0 +1,120 @@ +# Run BigchainDB with Docker + +**NOT for Production Use** + +For those who like using Docker and wish to experiment with BigchainDB in +non-production environments, we currently maintain a Docker image and a +`Dockerfile` that can be used to build an image for `bigchaindb`. + +## Pull and Run the Image from Docker Hub + +Assuming you have Docker installed, you would proceed as follows. + +In a terminal shell, pull the latest version of the BigchainDB Docker image using: +```text +docker pull bigchaindb/bigchaindb +``` + +then do a one-time configuration step to create the config file; we will use +the `-y` option to accept all the default values. The configuration file will +be stored in a file on your host machine at `~/bigchaindb_docker/.bigchaindb`: + +```text +docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ + bigchaindb/bigchaindb -y configure +Generating keypair +Configuration written to /data/.bigchaindb +Ready to go! +``` + +Let's analyze that command: + +* `docker run` tells Docker to run some image +* `--rm` remove the container once we are done +* `-v "$HOME/bigchaindb_docker:/data"` map the host directory + `$HOME/bigchaindb_docker` to the container directory `/data`; + this allows us to have the data persisted on the host machine, + you can read more in the [official Docker + documentation](https://docs.docker.com/engine/userguide/containers/dockervolumes/#mount-a-host-directory-as-a-data-volume) +* `-t` allocate a pseudo-TTY +* `-i` keep STDIN open even if not attached +* `bigchaindb/bigchaindb the image to use +* `-y configure` execute the `configure` sub-command (of the `bigchaindb` command) inside the container, with the `-y` option to automatically use all the default config values + + +After configuring the system, you can run BigchainDB with the following command: + +```text +docker run -v "$HOME/bigchaindb_docker:/data" -d \ + --name bigchaindb \ + -p "58080:8080" -p "59984:9984" \ + bigchaindb/bigchaindb start +``` + +The command is slightly different from the previous one, the differences are: + +* `-d` run the container in the background +* `--name bigchaindb` give a nice name to the container so it's easier to + refer to it later +* `-p "58080:8080"` map the host port `58080` to the container port `8080` + (the RethinkDB admin interface) +* `-p "59984:9984"` map the host port `59984` to the container port `9984` + (the BigchainDB API server) +* `start` start the BigchainDB service + +Another way to publish the ports exposed by the container is to use the `-P` (or +`--publish-all`) option. This will publish all exposed ports to random ports. You can +always run `docker ps` to check the random mapping. + +You can also access the RethinkDB dashboard at +[http://localhost:58080/](http://localhost:58080/) + +If that doesn't work, then replace `localhost` with the IP or hostname of the +machine running the Docker engine. If you are running docker-machine (e.g. on +Mac OS X) this will be the IP of the Docker machine (`docker-machine ip +machine_name`). + +### Load Testing with Docker + +Now that we have BigchainDB running in the Docker container named `bigchaindb`, we can +start another BigchainDB container to generate a load test for it. + +First, make sure the container named `bigchaindb` is still running. You can check that using: +```text +docker ps +``` + +You should see a container named `bigchaindb` in the list. + +You can load test the BigchainDB running in that container by running the `bigchaindb load` command in a second container: + +```text +docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ + --link bigchaindb \ + bigchaindb/bigchaindb load +``` + +Note the `--link` option to link to the first container (named `bigchaindb`). + +Aside: The `bigchaindb load` command has several options (e.g. `-m`). You can read more about it in [the documentation about the BigchainDB command line interface](bigchaindb-cli.html). + +If you look at the RethinkDB dashboard (in your web browser), you should see the effects of the load test. You can also see some effects in the Docker logs using: +```text +docker logs -f bigchaindb +``` + +## Building Your Own Image + +Assuming you have Docker installed, you would proceed as follows. + +In a terminal shell: +```text +git clone git@github.com:bigchaindb/bigchaindb.git +``` + +Build the Docker image: +```text +docker build --tag local-bigchaindb . +``` + +Now you can use your own image to run BigchainDB containers. diff --git a/docs/source/nodes/setup-run-node.md b/docs/source/nodes/setup-run-node.md index 6e128e1e..3011bf02 100644 --- a/docs/source/nodes/setup-run-node.md +++ b/docs/source/nodes/setup-run-node.md @@ -59,7 +59,7 @@ If you're setting up a stand-alone node (i.e. not intending for it to join a clu ### Cluster Node -Create a RethinkDB configuration file (text file) named `instance.conf` in the `/etc/rethinkdb/instances.d/` directory with the following contents (explained below): +Create a RethinkDB configuration file (text file) named `instance1.conf` in the `/etc/rethinkdb/instances.d/` directory with the following contents (explained below): ```text directory=/data bind=all @@ -184,133 +184,19 @@ Once you've installed BigchainDB Server, you can run it. First make sure you hav rethinkdb ``` +TODO: What about `/etc/init.d/rethinkdb restart` + You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at `http://rethinkdb-hostname:8080/`. If you're running RethinkDB on localhost, that would be [http://localhost:8080/](http://localhost:8080/). +TODO: Set shards using `bigchaindb set-shards numshards` + You can start BigchainDB Server using: ```text bigchaindb start ``` +OR: `screen -d -m bigchaindb -y start &` + + If it's the first time you've run `bigchaindb start`, then it creates the database (a RethinkDB database), the tables, the indexes, and the genesis block. It then starts BigchainDB. If you're run `bigchaindb start` or `bigchaindb init` before (and you haven't dropped the database), then `bigchaindb start` just starts BigchainDB. - -## Run BigchainDB with Docker - -**NOT for Production Use** - -For those who like using Docker and wish to experiment with BigchainDB in -non-production environments, we currently maintain a Docker image and a -`Dockerfile` that can be used to build an image for `bigchaindb`. - -### Pull and Run the Image from Docker Hub - -Assuming you have Docker installed, you would proceed as follows. - -In a terminal shell, pull the latest version of the BigchainDB Docker image using: -```text -docker pull bigchaindb/bigchaindb -``` - -then do a one-time configuration step to create the config file; we will use -the `-y` option to accept all the default values. The configuration file will -be stored in a file on your host machine at `~/bigchaindb_docker/.bigchaindb`: - -```text -docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ - bigchaindb/bigchaindb -y configure -Generating keypair -Configuration written to /data/.bigchaindb -Ready to go! -``` - -Let's analyze that command: - -* `docker run` tells Docker to run some image -* `--rm` remove the container once we are done -* `-v "$HOME/bigchaindb_docker:/data"` map the host directory - `$HOME/bigchaindb_docker` to the container directory `/data`; - this allows us to have the data persisted on the host machine, - you can read more in the [official Docker - documentation](https://docs.docker.com/engine/userguide/containers/dockervolumes/#mount-a-host-directory-as-a-data-volume) -* `-t` allocate a pseudo-TTY -* `-i` keep STDIN open even if not attached -* `bigchaindb/bigchaindb the image to use -* `-y configure` execute the `configure` sub-command (of the `bigchaindb` command) inside the container, with the `-y` option to automatically use all the default config values - - -After configuring the system, you can run BigchainDB with the following command: - -```text -docker run -v "$HOME/bigchaindb_docker:/data" -d \ - --name bigchaindb \ - -p "58080:8080" -p "59984:9984" \ - bigchaindb/bigchaindb start -``` - -The command is slightly different from the previous one, the differences are: - -* `-d` run the container in the background -* `--name bigchaindb` give a nice name to the container so it's easier to - refer to it later -* `-p "58080:8080"` map the host port `58080` to the container port `8080` - (the RethinkDB admin interface) -* `-p "59984:9984"` map the host port `59984` to the container port `9984` - (the BigchainDB API server) -* `start` start the BigchainDB service - -Another way to publish the ports exposed by the container is to use the `-P` (or -`--publish-all`) option. This will publish all exposed ports to random ports. You can -always run `docker ps` to check the random mapping. - -You can also access the RethinkDB dashboard at -[http://localhost:58080/](http://localhost:58080/) - -If that doesn't work, then replace `localhost` with the IP or hostname of the -machine running the Docker engine. If you are running docker-machine (e.g. on -Mac OS X) this will be the IP of the Docker machine (`docker-machine ip -machine_name`). - -#### Load Testing with Docker - -Now that we have BigchainDB running in the Docker container named `bigchaindb`, we can -start another BigchainDB container to generate a load test for it. - -First, make sure the container named `bigchaindb` is still running. You can check that using: -```text -docker ps -``` - -You should see a container named `bigchaindb` in the list. - -You can load test the BigchainDB running in that container by running the `bigchaindb load` command in a second container: - -```text -docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \ - --link bigchaindb \ - bigchaindb/bigchaindb load -``` - -Note the `--link` option to link to the first container (named `bigchaindb`). - -Aside: The `bigchaindb load` command has several options (e.g. `-m`). You can read more about it in [the documentation about the BigchainDB command line interface](bigchaindb-cli.html). - -If you look at the RethinkDB dashboard (in your web browser), you should see the effects of the load test. You can also see some effects in the Docker logs using: -```text -docker logs -f bigchaindb -``` - -### Building Your Own Image - -Assuming you have Docker installed, you would proceed as follows. - -In a terminal shell: -```text -git clone git@github.com:bigchaindb/bigchaindb.git -``` - -Build the Docker image: -```text -docker build --tag local-bigchaindb . -``` - -Now you can use your own image to run BigchainDB containers. From 6ac09e95c3cd03bfb7ae18e2501f9c8d9265b319 Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 15:32:41 +0200 Subject: [PATCH 23/27] New top-level section of docs: Clusters & Federations --- .../{nodes => clusters-feds}/deploy-on-aws.md | 2 +- .../source/clusters-feds/federation-set-up.md | 38 ++++++++++++ docs/source/clusters-feds/index.rst | 13 +++++ .../{nodes => clusters-feds}/monitoring.md | 2 +- docs/source/index.rst | 1 + docs/source/nodes/configuration.md | 2 +- docs/source/nodes/federation-set-up.md | 58 ------------------- docs/source/nodes/index.rst | 19 +----- docs/source/topic-guides/index.rst | 1 + docs/source/topic-guides/node-cluster-fed.md | 13 +++++ 10 files changed, 71 insertions(+), 78 deletions(-) rename docs/source/{nodes => clusters-feds}/deploy-on-aws.md (99%) create mode 100644 docs/source/clusters-feds/federation-set-up.md create mode 100644 docs/source/clusters-feds/index.rst rename docs/source/{nodes => clusters-feds}/monitoring.md (95%) delete mode 100644 docs/source/nodes/federation-set-up.md create mode 100644 docs/source/topic-guides/node-cluster-fed.md diff --git a/docs/source/nodes/deploy-on-aws.md b/docs/source/clusters-feds/deploy-on-aws.md similarity index 99% rename from docs/source/nodes/deploy-on-aws.md rename to docs/source/clusters-feds/deploy-on-aws.md index 4e3596cb..f725f59a 100644 --- a/docs/source/nodes/deploy-on-aws.md +++ b/docs/source/clusters-feds/deploy-on-aws.md @@ -124,7 +124,7 @@ fab --fabfile=fabfile-monitor.py --hosts= run_monitor For more information about monitoring (e.g. how to view the Grafana dashboard in your web browser), see the [Monitoring](monitoring.html) section of this documentation. -To configure a BigchainDB node to send monitoring data to the monitoring server, change the statsd host in the configuration of the BigchainDB node. The section on [Configuring a BigchainDB Node](configuration.html) explains how you can do that. (For example, you can change the statsd host in `$HOME/.bigchaindb`.) +To configure a BigchainDB node to send monitoring data to the monitoring server, change the statsd host in the configuration of the BigchainDB node. The section on [Configuring a BigchainDB Node](../nodes/configuration.html) explains how you can do that. (For example, you can change the statsd host in `$HOME/.bigchaindb`.) ## Deploy a BigchainDB Cluster diff --git a/docs/source/clusters-feds/federation-set-up.md b/docs/source/clusters-feds/federation-set-up.md new file mode 100644 index 00000000..018bb515 --- /dev/null +++ b/docs/source/clusters-feds/federation-set-up.md @@ -0,0 +1,38 @@ +# Set Up and Run a Federation + +This section is about how to set up and run a BigchainDB _federation_, where each node is operated by a different operator. If you want to set up and run a BigchainDB cluster on AWS (where all nodes are operated by you), then see [the section about that](deploy-on-aws.html). + + +## Initial Checklist + +* Do you have a governance process for making federation-level decisions, such as how to admit new members? +* What will you store in creation transactions (data payload)? Is there a data schema? +* Will you use transfer transactions? Will they include a non-empty data payload? +* Who will be allowed to submit transactions? Who will be allowed to read or query transactions? How will you enforce the access rules? + + +## Set Up the Initial Cluster + +When you first start a federation cluster, the initial nodes will all start at roughly the same time. + +The steps to set up a cluster node are outlined in the section titled [Set Up and Run a Node](../nodes/setup-run-node.html). You'll need two pieces of information from all other nodes in the federation: + +1. Their RethinkDB hostname, e.g. `rdb.farm2.organization.org` +2. Their BigchainDB public key, e.g. `Eky3nkbxDTMgkmiJC8i5hKyVFiAQNmPP4a2G4JdDxJCK` + + +## Documentation to Come + +* Backing Up & Restoring data +* Adding a node (including resharding etc.) +* Removing a node +* Logging +* Node monitoring & crash recovery +* Node Security + * Securing your OS + * Firewalls and security groups. Remember to open port 123 for NTP. + * (Private) key management + * RethinkDB security +* Cluster monitoring +* Internal watchdogs +* External watchdogs \ No newline at end of file diff --git a/docs/source/clusters-feds/index.rst b/docs/source/clusters-feds/index.rst new file mode 100644 index 00000000..dcda2e85 --- /dev/null +++ b/docs/source/clusters-feds/index.rst @@ -0,0 +1,13 @@ +.. You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +BigchainDB Clusters & Federations +================================= + +.. toctree:: + :maxdepth: 1 + + federation-set-up + deploy-on-aws + monitoring + \ No newline at end of file diff --git a/docs/source/nodes/monitoring.md b/docs/source/clusters-feds/monitoring.md similarity index 95% rename from docs/source/nodes/monitoring.md rename to docs/source/clusters-feds/monitoring.md index 626230fe..629ccbd5 100644 --- a/docs/source/nodes/monitoring.md +++ b/docs/source/clusters-feds/monitoring.md @@ -28,7 +28,7 @@ You can view the Grafana dashboard in your web browser at: (You may want to replace `localhost` with another hostname in that URL, e.g. the hostname of a remote monitoring server.) -The login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](setup-run-node.html#run-bigchaindb) and load some test transactions: +The login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](../nodes/setup-run-node.html#run-bigchaindb) and load some test transactions: ```text $ bigchaindb load ``` diff --git a/docs/source/index.rst b/docs/source/index.rst index 2ef1a8fc..9173d125 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -13,6 +13,7 @@ Table of Contents introduction topic-guides/index nodes/index + clusters-feds/index drivers-clients/index release-notes appendices/index diff --git a/docs/source/nodes/configuration.md b/docs/source/nodes/configuration.md index fad2b416..9fe13925 100644 --- a/docs/source/nodes/configuration.md +++ b/docs/source/nodes/configuration.md @@ -60,7 +60,7 @@ environment variables available are: - `BIGCHAINDB_KEYPAIR_PUBLIC` defines the public key of the BigchainDB node. - `BIGCHAINDB_KEYPAIR_PRIVATE` defines the private key of the BigchainDB node. - `BIGCHAINDB_KEYRING` is a colon-separated list of the public keys of all _other_ nodes in the cluster. -- `BIGCHAINDB_STATSD_HOST` defines the hostname of the statsd server for [monitoring](monitoring.html). +- `BIGCHAINDB_STATSD_HOST` defines the hostname of the statsd server for [monitoring](../clusters-feds/monitoring.html). - `BIGCHAINDB_STATSD_PORT` defines the port of the statsd server for monitoring. - `BIGCHAINDB_STATSD_RATE` is a float between `0` and `1` that defines the fraction of transaction operations sampled. - `BIGCHAINDB_API_ENDPOINT` defines the API endpoint to use (e.g. `http://localhost:9984/api/v1`). diff --git a/docs/source/nodes/federation-set-up.md b/docs/source/nodes/federation-set-up.md deleted file mode 100644 index 68b47f3f..00000000 --- a/docs/source/nodes/federation-set-up.md +++ /dev/null @@ -1,58 +0,0 @@ -# Set Up and Run a Federation - -This section is about how to set up and run a BigchainDB _federation_, where each node is operated by a different operator. If you want to set up and run a BigchainDB cluster on AWS (where all nodes are operated by you), then see [the section about that](deploy-on-aws.html). - - -## Answer Basic Questions - -* Do you have a governance process for making federation-level decisions, such as how to admit new members? -* What will you store in creation transactions (data payload)? Is there a data schema? -* Will you use transfer transactions? Will they include a non-empty data payload? -* Who will be allowed to submit transactions? Who will be allowed to read or query transactions? How will you enforce the access rules? - - -## Set Up the Initial Cluster - -When you first start a federation cluster, the initial nodes will all start at roughly the same time. Here are the steps to be taken by each node operator: - -**Create a RethinkDB Cluster** - -* Go through the steps of [Set Up and Run a Node](setup-run-node.html), up to "Configure RethinkDB Server" -* Determine the hostname of the server running RethinkDB (e.g. `rethinkdb2.farm3.organization5.com`) -* Share your RethinkDB hostname with all other members of the federation -* Once you have all the RethinkDB hostnames, add them to your local RethinkDB configuration file as explained in [Configure RethinkDB Server](setup-run-node.html#configure-rethinkdb-server) -* (TODO: Section with steps to make RethinkDB more secure) - -**Install, Configure and Run BigchainDB** - -* Continue with the steps of [Set Up and Run a Node](setup-run-node.html) -* You will generate a default `.bigchaindb` config file at some point. It contains your public/private keypair. -* Send your **public** key to all other federation members. If you accidentally send your private key, then delete your `.bigchaindb` config file, generate a new one, and send your new public key to all other federation members. -* Once you have the public keys of all other federation members, put them in the `keyring` (list) in your `.bigchaindb` file. (Your keyring should _not_ include your public key.) - -* TODO: Are there other things they should change in their `.bigchaindb` file? - -* Make sure RethinkDB is Running (TODO: say how) -* Start BigchainDB - -```text -$ bigchaindb start -``` - - - -## Documentation to Come - -* Backing Up & Restoring data -* Adding a node (including resharding etc.) -* Removing a node -* Logging -* Node monitoring & crash recovery -* Node Security - * Securing your OS - * Firewalls and security groups. Remember to open port 123 for NTP. - * (Private) key management - * RethinkDB security -* Cluster monitoring -* Internal watchdogs -* External watchdogs \ No newline at end of file diff --git a/docs/source/nodes/index.rst b/docs/source/nodes/index.rst index f9a03180..ecf92b8e 100644 --- a/docs/source/nodes/index.rst +++ b/docs/source/nodes/index.rst @@ -1,31 +1,16 @@ .. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -BigchainDB Nodes, Clusters & Federations -======================================== - -A **BigchainDB node** is a server or set of closely-linked servers running RethinkDB Server, BigchainDB Server, and other BigchainDB-related software. Each node is controlled by one person or organization. - -A set of BigchainDB nodes can connect to each other to form a **cluster**. Each node in the cluster runs the same software. A cluster contains one logical RethinkDB datastore. A cluster may have additional servers to do things such as cluster monitoring. - -The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization). A federation must have some sort of governance structure to make decisions. If a cluster is run by a single company, then the federation is just that company. - -**What's the Difference Between a Cluster and a Federation?** - -A cluster is just a bunch of connected nodes (computers). A cluster might be operated by just one person. A federation is an organization which has a cluster, and where each node in the cluster has a different operator. - -Confusingly, we sometimes call a federation's cluster its "federation." You can probably tell what we mean from context. +BigchainDB Nodes +================ .. toctree:: :maxdepth: 1 node-requirements - federation-set-up setup-run-node run-with-docker running-unit-tests configuration bigchaindb-cli python-server-api-examples - deploy-on-aws - monitoring diff --git a/docs/source/topic-guides/index.rst b/docs/source/topic-guides/index.rst index 309620e2..4785fcc9 100644 --- a/docs/source/topic-guides/index.rst +++ b/docs/source/topic-guides/index.rst @@ -14,3 +14,4 @@ Topic guides give background and explain concepts at a high level. immutable assets models + node-cluster-fed diff --git a/docs/source/topic-guides/node-cluster-fed.md b/docs/source/topic-guides/node-cluster-fed.md new file mode 100644 index 00000000..2987d477 --- /dev/null +++ b/docs/source/topic-guides/node-cluster-fed.md @@ -0,0 +1,13 @@ +# Nodes, Clusters & Federations + +A **BigchainDB node** is a server or set of closely-linked servers running RethinkDB Server, BigchainDB Server, and other BigchainDB-related software. Each node is controlled by one person or organization. + +A set of BigchainDB nodes can connect to each other to form a **cluster**. Each node in the cluster runs the same software. A cluster contains one logical RethinkDB datastore. A cluster may have additional servers to do things such as cluster monitoring. + +The people and organizations that run the nodes in a cluster belong to a **federation** (i.e. another organization). A federation must have some sort of governance structure to make decisions. If a cluster is run by a single company, then the federation is just that company. + +**What's the Difference Between a Cluster and a Federation?** + +A cluster is just a bunch of connected nodes (computers). A cluster might be operated by just one person. A federation is an organization which has a cluster, and where each node in the cluster has a different operator. + +Confusingly, we sometimes call a federation's cluster its "federation." You can probably tell what we mean from context. \ No newline at end of file From 1d108654a89dcf5a9170c6f7bbb99c353c967aaf Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 16:36:41 +0200 Subject: [PATCH 24/27] Docs: clarify steps to run RethinkDB & BigchainDB for clusters --- .../source/clusters-feds/federation-set-up.md | 2 + docs/source/nodes/setup-run-node.md | 44 +++++++++++++------ 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/docs/source/clusters-feds/federation-set-up.md b/docs/source/clusters-feds/federation-set-up.md index 018bb515..3fbe97be 100644 --- a/docs/source/clusters-feds/federation-set-up.md +++ b/docs/source/clusters-feds/federation-set-up.md @@ -20,6 +20,8 @@ The steps to set up a cluster node are outlined in the section titled [Set Up an 1. Their RethinkDB hostname, e.g. `rdb.farm2.organization.org` 2. Their BigchainDB public key, e.g. `Eky3nkbxDTMgkmiJC8i5hKyVFiAQNmPP4a2G4JdDxJCK` +One node must be chosen as the "designated fist node": they must run some RethinkDB configuration commands after all nodes have started RethinkDB, but before any node has started BigchainDB. + ## Documentation to Come diff --git a/docs/source/nodes/setup-run-node.md b/docs/source/nodes/setup-run-node.md index 3011bf02..2140601d 100644 --- a/docs/source/nodes/setup-run-node.md +++ b/docs/source/nodes/setup-run-node.md @@ -59,7 +59,7 @@ If you're setting up a stand-alone node (i.e. not intending for it to join a clu ### Cluster Node -Create a RethinkDB configuration file (text file) named `instance1.conf` in the `/etc/rethinkdb/instances.d/` directory with the following contents (explained below): +Create a RethinkDB configuration file (text file) named `instance1.conf` with the following contents (explained below): ```text directory=/data bind=all @@ -76,16 +76,11 @@ join=node2_hostname:29015 * `direct-io` tells RethinkDB to use direct I/O (explained earlier). * `join=hostname:29015` lines: A cluster node needs to find out the hostnames of all the other nodes somehow. You _could_ designate one node to be the one that every other node asks, and put that node's hostname in the config file, but that wouldn't be very decentralized. Instead, we include _every_ node in the list of nodes-to-ask. -If you're curious about the RethinkDB config file, there's [a RethinkDB documentation page about it](https://www.rethinkdb.com/docs/config-file/). +If you're curious about the RethinkDB config file, there's [a RethinkDB documentation page about it](https://www.rethinkdb.com/docs/config-file/). The [explanations of the RethinkDB command-line options](https://rethinkdb.com/docs/cli-options/) are another useful reference. TODO: Explain how to configure the RethinkDB cluster to be more secure. -## Run RethinkDB Server - -You can run RethinkDB by opening a Terminal and entering `rethinkdb`. You could do that now or wait until just before you start BigchainDB. - - ## Install Python 3.4+ If you don't already have it, then you should [install Python 3.4+](https://www.python.org/downloads/). @@ -177,26 +172,47 @@ Open `$HOME/.bigchaindb` in your text editor and: For more information about the BigchainDB config file, see [Configuring a BigchainDB Node](configuration.html). -## Run BigchainDB Server +## Run RethinkDB Server -Once you've installed BigchainDB Server, you can run it. First make sure you have RethinkDB running: +If you didn't create a RethinkDB config file (e.g. because you're running a stand-alone node), then you can start RethinkDB using: ```text rethinkdb ``` -TODO: What about `/etc/init.d/rethinkdb restart` +If you _did_ create a RethinkDB config file, then you should start RethinkDB using: +```text +rethinkdb --config-file path/to/instance1.conf +``` + +except replace the path with the actual path to `instance1.conf`. + +Note: It's possible to [make RethinkDB start at system startup](https://www.rethinkdb.com/docs/start-on-startup/). You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at `http://rethinkdb-hostname:8080/`. If you're running RethinkDB on localhost, that would be [http://localhost:8080/](http://localhost:8080/). -TODO: Set shards using `bigchaindb set-shards numshards` -You can start BigchainDB Server using: +## Run BigchainDB Server + +### Stand-Alone Node + +It should be enough to do: ```text bigchaindb start ``` -OR: `screen -d -m bigchaindb -y start &` +### Cluster Node +After all the cluster nodes have started RethinkDB, but before they start BigchainDB, a designated cluster node has to do some RethinkDB cluster configuration by running the following two commands: +```text +bigchaindb init +bigchaindb set-shards numshards +``` -If it's the first time you've run `bigchaindb start`, then it creates the database (a RethinkDB database), the tables, the indexes, and the genesis block. It then starts BigchainDB. If you're run `bigchaindb start` or `bigchaindb init` before (and you haven't dropped the database), then `bigchaindb start` just starts BigchainDB. +where `numshards` should be set equal to the number of nodes expected to be in the cluster (i.e. once all currently-expected nodes have joined). +(The `bigchain init` command creates the database within RethinkDB, the tables, the indexes, and the genesis block.) + +Once the designated node has run the above two commands, every node can start BigchainDB using: +```text +bigchaindb start +``` From ae88b8e0e3e1ba6e795b862fa8d6f0fa84a0047a Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 16:45:48 +0200 Subject: [PATCH 25/27] Fixed some links to the Docker docs --- README.md | 2 +- docs/source/nodes/node-requirements.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 56286ad9..6fc9f37b 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit ## Quick Start ### [Set Up and Run a BigchainDB Node](http://bigchaindb.readthedocs.io/en/latest/nodes/setup-run-node.html) -### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.io/en/latest/setup-run-node.html#run-bigchaindb-with-docker) +### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.io/en/latest/nodes/run-with-docker.html) ### [The Python Server API by Example](http://bigchaindb.readthedocs.io/en/latest/nodes/python-server-api-examples.html) ### [The Python Driver API by Example](http://bigchaindb.readthedocs.io/en/latest/drivers-clients/python-driver-api-examples.html) diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md index 5177edf5..0b8286ff 100644 --- a/docs/source/nodes/node-requirements.md +++ b/docs/source/nodes/node-requirements.md @@ -15,7 +15,7 @@ The BigchainDB core dev team uses Ubuntu 14.04 or Fedora 23. We don't test BigchainDB on Windows or Mac OS X, but you can try. * If you run into problems on Windows, then you may want to try using Vagrant. One of our community members ([@Mec-Is](https://github.com/Mec-iS)) wrote [a page about how to install BigchainDB on a VM with Vagrant](https://gist.github.com/Mec-iS/b84758397f1b21f21700). -* If you have Mac OS X and want to experiment with BigchainDB, then you could do that [using Docker](setup-run-node.html#run-bigchaindb-with-docker). +* If you have Mac OS X and want to experiment with BigchainDB, then you could do that [using Docker](run-with-docker.html). ## Memory Requirements From c7e1e015e2a421275dedf065b9e29bef80b2e102 Mon Sep 17 00:00:00 2001 From: troymc Date: Wed, 15 Jun 2016 16:49:36 +0200 Subject: [PATCH 26/27] Docs: 'Direct I/O' should be just 'direct I/O' --- docs/source/nodes/node-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/nodes/node-requirements.md b/docs/source/nodes/node-requirements.md index 0b8286ff..d43885bf 100644 --- a/docs/source/nodes/node-requirements.md +++ b/docs/source/nodes/node-requirements.md @@ -38,7 +38,7 @@ Also, RethinkDB tables can have [at most 64 shards](https://rethinkdb.com/limita RethinkDB has [issues with `btrfs`](https://github.com/rethinkdb/rethinkdb/issues/2781) (B-tree file system). -It's best to have a file system that supports Direct I/O, because that will improve RethinkDB performance. Many compressed or encrypted file systems don't support Direct I/O. +It's best to have a file system that supports direct I/O, because that will improve RethinkDB performance. Many compressed or encrypted file systems don't support direct I/O. ## CPU Requirements From 6ffb1f524392403374062b26b175d7c06d8b4ee1 Mon Sep 17 00:00:00 2001 From: troymc Date: Thu, 16 Jun 2016 10:22:02 +0200 Subject: [PATCH 27/27] Docs: Added subsection 'Secure Your Server' --- docs/source/nodes/setup-run-node.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/docs/source/nodes/setup-run-node.md b/docs/source/nodes/setup-run-node.md index 2140601d..a906b913 100644 --- a/docs/source/nodes/setup-run-node.md +++ b/docs/source/nodes/setup-run-node.md @@ -2,16 +2,27 @@ This section goes through the steps to set up a BigchainDB node (running RethinkDB Server, BigchainDB Server, etc.). There are instructions for two cases: -1. Stand-Alone Node (useful for local development) +1. Stand-Alone Node (useful for local testing and development) 2. Cluster Node -## Check the Node Requirements +## Get a Server -The first step is to make sure you have a server (or equivalent) which meets [the requirements for a BigchainDB node](node-requirements.html). +The first step is to get a server (or equivalent) which meets [the requirements for a BigchainDB node](node-requirements.html). -## System Clock Sync +## Secure Your Server + +The steps that you must take to secure your server depend on your server OS and where your server is physically located. There are many articles and books about how to secure a server. Ask a search engine. + +Here we just cover special considerations when securing a BigchainDB node. + +TODO: Special security considerations for BigchainDB nodes. + +TODO: Notes about firewall setup. What ports should be open, for what kinds of traffic, accepting connections from what IP addresses, etc. For example, NTP uses port 123. Can firewall setup be done now or does it have to happen later? + + +## Sync Your System Clock If you're just setting up a stand-alone node, then you can skip this step.