Merge pull request #366 from bigchaindb/reorganize-the-docs

Documentation reorganization + new sections on decentralization & immutability

README.md

@@ -13,10 +13,10 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit
 
 ## Quick Start
 
-### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.io/en/latest/installing-server.html)
+### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.io/en/latest/servers/installing-server.html)
 ### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.io/en/latest/installing-server.html#run-bigchaindb-with-docker)
-### [The Python Server API by Example](http://bigchaindb.readthedocs.io/en/latest/python-server-api-examples.html)
-### [The Python Driver API by Example](http://bigchaindb.readthedocs.io/en/latest/python-driver-api-examples.html)
+### [The Python Server API by Example](http://bigchaindb.readthedocs.io/en/latest/servers/python-server-api-examples.html)
+### [The Python Driver API by Example](http://bigchaindb.readthedocs.io/en/latest/drivers-clients/python-driver-api-examples.html)
 
 ## Links for Everyone
 * [BigchainDB.com](https://www.bigchaindb.com/) - the main BigchainDB website, including newsletter signup
@@ -38,4 +38,3 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit
 * [Licenses](LICENSES.md) - open source & open content
 * [Imprint](https://www.bigchaindb.com/imprint/)
 * [Contact Us](https://www.bigchaindb.com/contact/)
-

docs/source/appendices/consensus.md

@@ -23,7 +23,7 @@ To activate your plugin, you can either set the `consensus_plugin` field in your
 
 ## Plugin API
 
-BigchainDB's [current plugin API](../../bigchaindb/consensus.py) exposes five functions in an `AbstractConsensusRules` class:
+BigchainDB's [current plugin API](https://github.com/bigchaindb/bigchaindb/blob/master/bigchaindb/consensus.py) exposes five functions in an `AbstractConsensusRules` class:
 
 ```python
 validate_transaction(bigchain, transaction)

docs/source/appendices/glossary.md

@@ -4,7 +4,7 @@
 
 **BigchainDB cluster.** A collection of servers running BigchainDB Server software and communicating with each other via RethinkDB protocols. a BigchainDB cluster contains one logical RethinkDB datastore.
 
-**Block.** A collection of up to 1000 transactions, plus other things. They get chained together by votes. See [the section on the block model](models.html#the-block-model).
+**Block.** A collection of up to 1000 transactions, plus other things. They get chained together by votes. See [the section on the block model](../topic-guides/models.html#the-block-model).
 
 **Federation.** 1. An organization with members and some kind of governance structure. 2. The BigchainDB cluster owned and operated by a Federation's members.
 
@@ -12,6 +12,6 @@
 
 **Node.** See _Federation node_.
 
-**Transaction.** The basic informational unit. A transaction can represent the creation or transfer of a digital asset. See [the section on the transaction model](models.html#the-transaction-model).
+**Transaction.** The basic informational unit. A transaction can represent the creation or transfer of a digital asset. See [the section on the transaction model](../topic-guides/models.html#the-transaction-model).
 
-**Vote.** Each federation node in a federation is required to vote on the validity of every block (i.e. whether the block valid or not). A node's vote on a block also includes the id of the block it considers to be the previous block. See [the section on the vote model](models.html#the-vote-model).
+**Vote.** Each federation node in a federation is required to vote on the validity of every block (i.e. whether the block valid or not). A node's vote on a block also includes the id of the block it considers to be the previous block. See [the section on the vote model](../topic-guides/models.html#the-vote-model).

docs/source/appendices/index.rst

@@ -0,0 +1,16 @@
+.. You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Appendices
+==========
+
+.. toctree::
+   :maxdepth: 1
+
+   glossary
+   json-serialization
+   cryptography
+   the-Bigchain-class
+   consensus
+   local-rethinkdb-cluster
+   licenses

docs/source/appendices/local-rethinkdb-cluster.md

@@ -4,7 +4,7 @@ This section explains one way to deploy a multi-node RethinkDB cluster on one ma
 
 ## Launching More RethinkDB Nodes
 
-Assuming you've already [installed RethinkDB](installing-server.html#install-and-run-rethinkdb-server) and have one RethinkDB node running, here's how you can launch two more nodes on the same machine. First, prepare two additional nodes. Note that the user who starts RethinkDB must have write access to the created directories:
+Assuming you've already [installed RethinkDB](../servers/installing-server.html#install-and-run-rethinkdb-server) and have one RethinkDB node running, here's how you can launch two more nodes on the same machine. First, prepare two additional nodes. Note that the user who starts RethinkDB must have write access to the created directories:
 
     mkdir -p /path/to/node2
     mkdir -p /path/to/node3

docs/source/appendices/the-Bigchain-class.rst

@@ -0,0 +1,10 @@
+##################
+The Bigchain class
+##################
+
+The Bigchain class is the top-level Python API for BigchainDB. If you want to create and initialize a BigchainDB database, you create a Bigchain instance (object). Then you can use its various methods to create transactions, write transactions (to the object/database), read transactions, etc.
+
+.. autoclass:: bigchaindb.Bigchain
+   :members:
+
+   .. automethod:: bigchaindb.core.Bigchain.__init__

docs/source/contributing.md

@@ -1,3 +0,0 @@
-# Contributing to the BigchainDB Project
-
-If you're interested in contributing to the BigchainDB project, you can find out how in the [CONTRIBUTING.md file](https://github.com/BigchainDB/bigchaindb/blob/master/CONTRIBUTING.md) (in the root directory of the BigchainDB repository).

docs/source/developer-interface.rst

@@ -1,10 +0,0 @@
-###################
-Developer Interface
-###################
-
-The Bigchain class is the top-level API for BigchainDB. If you want to create and initialize a BigchainDB database, you create a Bigchain instance (object). Then you can use its various methods to create transactions, write transactions (to the object/database), read transactions, etc.
-
-.. autoclass:: bigchaindb.Bigchain
-   :members:
-
-   .. automethod:: bigchaindb.core.Bigchain.__init__

docs/source/drivers-clients/index.rst

@@ -0,0 +1,12 @@
+.. You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+BigchainDB Drivers & Clients
+============================
+
+.. toctree::
+   :maxdepth: 1
+
+   http-client-server-api
+   python-driver-api-examples
+   

docs/source/index.rst

@@ -7,37 +7,12 @@ BigchainDB Documentation
 Table of Contents
 -----------------
 
-
 .. toctree::
-   :maxdepth: 5
-   :numbered:
+   :maxdepth: 1
 
    introduction
-   installing-server
-   running-unit-tests
-   configuration
-   python-server-api-examples
-   bigchaindb-cli
-   http-client-server-api
-   python-driver-api-examples
-   local-rethinkdb-cluster
-   deploy-on-aws
-   json-serialization
-   cryptography
-   models
-   developer-interface
-   consensus
-   monitoring
-   licenses
-   contributing
+   topic-guides/index
+   servers/index
+   drivers-clients/index
    release-notes
-   glossary
-
-
-Indices and Tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
-
+   appendices/index

docs/source/introduction.md

@@ -1,6 +1,8 @@
 # Introduction
 
-BigchainDB is a scalable blockchain database. You can read about its motivations, goals and high-level architecture in the [BigchainDB whitepaper](https://www.bigchaindb.com/whitepaper/).
+BigchainDB is a scalable blockchain database. That is, it's a "big data" database with some blockchain characteristics, including decentralization, immutability and native support for assets.
+
+You can read about the motivations, goals and high-level architecture in the [BigchainDB whitepaper](https://www.bigchaindb.com/whitepaper/).
 
 
 ## Who this Documentation for?

docs/source/servers/configuration.md

@@ -64,7 +64,7 @@ environment variables available are:
 - `BIGCHAINDB_STATSD_PORT` defines the port of the statsd server for monitoring.
 - `BIGCHAINDB_STATSD_RATE` is a float between `0` and `1` that defines the fraction of transaction operations sampled.
 - `BIGCHAINDB_API_ENDPOINT` defines the API endpoint to use (e.g. `http://localhost:9984/api/v1`).
-- `BIGCHAINDB_CONSENSUS_PLUGIN` defines the name of the [consensus plugin](consensus.html) to use.
+- `BIGCHAINDB_CONSENSUS_PLUGIN` defines the name of the [consensus plugin](../appendices/consensus.html) to use.
 - `BIGCHAINDB_SERVER_BIND` defines where to bind the server socket, the format is `addr:port` (e.g. `localhost:9984`).
 - `BIGCHAINDB_SERVER_WORKERS` defines the [number of workers](http://docs.gunicorn.org/en/stable/settings.html#workers)
   to start for the server API.

docs/source/servers/deploy-on-aws.md

@@ -8,7 +8,7 @@ This section explains a way to deploy a cluster of BigchainDB nodes on Amazon We
 
 You might ask why one would want to deploy a centrally-controlled BigchainDB cluster. Isn't BigchainDB supposed to be decentralized, where each node is controlled by a different person or organization?
 
-That's true, but there are some reasons why one might want a centrally-controlled cluster: 1) for testing, and 2) for initial deployment. Afterwards, the control of each node can be handed over to a different entity.
+That's true, but there are some reasons why one might want a centrally-controlled cluster: 1) for testing, and 2) for initial deployment, after which the control of each node can be handed over to a different entity.
 
 ## Python Setup
 

docs/source/servers/index.rst

@@ -0,0 +1,16 @@
+.. You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+BigchainDB Servers (Nodes) & Clusters
+=====================================
+
+.. toctree::
+   :maxdepth: 1
+
+   installing-server
+   running-unit-tests
+   configuration
+   bigchaindb-cli
+   python-server-api-examples
+   deploy-on-aws
+   monitoring

docs/source/servers/monitoring.md

@@ -8,7 +8,7 @@ BigchainDB uses [StatsD](https://github.com/etsy/statsd) for monitoring. We requ
 
 We put each of those inside its own Docker container. The whole system is illustrated below.
 
-![BigchainDB monitoring system diagram: Application metrics flow from servers running BigchainDB to Telegraf to InfluxDB to Grafana](./_static/monitoring_system_diagram.png)
+![BigchainDB monitoring system diagram: Application metrics flow from servers running BigchainDB to Telegraf to InfluxDB to Grafana](../_static/monitoring_system_diagram.png)
 
 For ease of use, we've created a Docker [_Compose file_](https://docs.docker.com/compose/compose-file/) (named `docker-compose-monitor.yml`) to define the monitoring system setup. To use it, just go to to the top `bigchaindb` directory and run:
 ```text

docs/source/servers/python-server-api-examples.md

@@ -30,9 +30,7 @@ This instantiates an object `b` of class `Bigchain`. When instantiating a `Bigch
 
 In a federation of BigchainDB nodes, each node has its own `Bigchain` instance.
 
-The `Bigchain` class is the main API for all BigchainDB interactions, right now. It does things that BigchainDB nodes do, but it also does things that BigchainDB clients do. In the future, it will be broken apart into a node/server class and a client class.
-
-The `Bigchain` class is documented in the [Developer Interface](developer-interface.html) section.
+The `Bigchain` class is the main API for all BigchainDB interactions, right now. It does things that BigchainDB nodes do, but it also does things that BigchainDB clients do. In the future, it will be refactored into different parts. The `Bigchain` class is documented [elsewhere (link)](../appendices/the-Bigchain-class.html).
 
 ## Create a Digital Asset
 

docs/source/topic-guides/decentralized.md

@@ -0,0 +1,19 @@
+# How BigchainDB is Decentralized
+
+Decentralization means that no one owns or controls everything, and there is no single point of failure.
+
+Ideally, each node in a BigchainDB cluster is owned and controlled by a different person or organization. Even if the cluster lives within one organization, it's still preferable to have each node controlled by a different person or subdivision.
+
+We use the phrase "BigchainDB federation" (or just "federation") to refer to the set of people and/or organizations who run the nodes of a BigchainDB cluster. A federation requires some form of governance to make decisions such as membership and policies. The exact details of the governance process are determined by each federation, but it can be very decentralized (e.g. purely vote-based, where each node gets a vote, and there are no special leadership roles).
+
+The actual data is decentralized in that it doesn’t all get stored in one place. Each federation node stores the primary of one shard and replicas of some other shards. (A shard is a subset of the total set of documents.) Sharding and replication are handled by RethinkDB.
+
+A federation can increase its decentralization (and its resilience) by increasing its jurisdictional diversity, geographic diversity, and other kinds of diversity. This idea is expanded upon in [the section on node diversity](diversity.html).
+
+There’s no node that has a long-term special position in the federation. All nodes run the same software and perform the same duties.
+
+RethinkDB has an “admin” user which can’t be deleted and which can make big changes to the database, such as dropping a table. Right now, that’s a big security vulnerability, but we have plans to mitigate it by:
+1. Locking down the admin user as much as possible.
+2. Having all nodes inspect RethinkDB admin-type requests before acting on them. Requests can be checked against an evolving whitelist of allowed actions (voted on by federation nodes).
+
+It’s worth noting that the RethinkDB admin user can’t transfer assets, even today. The only way to create a valid transfer transaction is to fulfill the current (crypto) conditions on the asset, and the admin user can’t do that because the admin user doesn’t have the necessary private keys (or preimages, in the case of hashlock conditions). They’re not stored in the database.

docs/source/topic-guides/diversity.md

@@ -0,0 +1,11 @@
+# Kinds of Node Diversity
+
+Steps should be taken to make it difficult for any one actor or event to control or damage “enough” of the nodes. (“Enough” is usually a quorum.) There are many kinds of diversity to consider, listed below. It may be quite difficult to have high diversity of all kinds.
+
+1. **Jurisdictional diversity.** The nodes should be controlled by entities within multiple legal jurisdictions, so that it becomes difficult to use legal means to compel enough of them to do something.
+2. **Geographic diversity.** The servers should be physically located at multiple geographic locations, so that it becomes difficult for a natural disaster (such as a flood or earthquake) to damage enough of them to cause problems.
+3. **Hosting diversity.** The servers should be hosted by multiple hosting providers (e.g. Amazon Web Services, Microsoft Azure, Digital Ocean, Rackspace), so that it becomes difficult for one hosting provider to influence enough of the nodes.
+4. **Operating system diversity.** The servers should use a variety of operating systems, so that a security bug in one OS can’t be used to exploit enough of the nodes.
+5. **Diversity in general.** In general, membership diversity (of all kinds) confers many advantages on a federation. For example, it provides the federation with a source of various ideas for addressing challenges.
+
+Note: If all the nodes are running the same code, i.e. the same implementation of BigchainDB, then a bug in that code could be used to compromise all of the nodes. Ideally, there would be several different, well-maintained implementations of BigchainDB Server (e.g. one in Python, one in Go, etc.), so that a federation could also have a diversity of server implementations.

docs/source/topic-guides/immutable.md

@@ -0,0 +1,17 @@
+# How BigchainDB is Immutable / Tamper-Resistant
+
+The blockchain community often describes blockchains as “immutable.” If we interpret that word literally, it means that blockchain data is unchangeable or permanent, which is absurd. The data _can_ be changed. For example, a plague might drive humanity extinct; the data would then get corrupted over time due to water damage, thermal noise, and the general increase of entropy. In the case of Bitcoin, nothing so drastic is required: a 51% attack will suffice.
+
+It’s true that blockchain data is more difficult to change than usual: it’s more tamper-resistant than a typical file system or database. Therefore, in the context of blockchains, we interpret the word “immutable” to mean tamper-resistant. (Linguists would say that the word “immutable” is a _term of art_ in the blockchain community.)
+
+BigchainDB achieves strong tamper-resistance in the following ways:
+
+1. **Replication.** All data is sharded and shards are replicated in several (different) places. The replication factor can be set by the federation. The higher the replication factor, the more difficult it becomes to change or delete all replicas.
+2. **Internal watchdogs.** All nodes monitor all changes and if some unallowed change happens, then appropriate action is taken. For example, if a valid block is deleted, then it is put back.
+3. **External watchdogs.** Federations may opt to have trusted third-parties to  monitor and audit their data, looking for irregularities. For federations with publicly-readable data, the public can act as an auditor.
+4. **Cryptographic signatures** are used throughout BigchainDB as a way to check if messages (transactions, blocks and votes) have been tampered with enroute, and as a way to verify who signed the messages. Each block is signed by the node that created it. Each vote is signed by the node that cast it. A creation transaction is signed by the node that created it, although there are plans to improve that by adding signatures from the sending client and multiple nodes; see [Issue #347](https://github.com/bigchaindb/bigchaindb/issues/347). Transfer transactions can contain multiple fulfillments (one per asset transferred). Each fulfillment will typically contain one or more signatures from the owners (i.e. the owners before the transfer). Hashlock fulfillments are an exception; there’s an open issue ([#339](https://github.com/bigchaindb/bigchaindb/issues/339)) to address that.
+5. **Full or partial backups** of the database may be recorded from time to time, possibly on magnetic tape storage, other blockchains, printouts, etc.
+6. **Strong security.** Node owners can adopt and enforce strong security policies.
+7. **Node diversity.** Diversity makes it so that no one thing (e.g. natural disaster or operating system bug) can compromise enough of the nodes. See [the section on the kinds of node diversity](diversity.html).
+
+Some of these things come "for free" as part of the BigchainDB software, and others require some extra effort from the federation and node owners.

docs/source/topic-guides/index.rst

@@ -0,0 +1,15 @@
+.. You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Topic Guides
+============
+
+Topic guides give background and explain concepts at a high level.
+
+.. toctree::
+   :maxdepth: 1
+
+   decentralized
+   diversity
+   immutable
+   models

docs/source/topic-guides/models.md

@@ -5,7 +5,7 @@ BigchainDB stores all its records in JSON documents.
 The three main kinds of records are transactions, blocks and votes. 
 _Transactions_ are used to register, issue, create or transfer things (e.g. assets). Multiple transactions are combined with some other metadata to form _blocks_. Nodes append _votes_ to blocks. This section is a reference on the details of transactions, blocks and votes.
 
-Below we often refer to cryptographic hashes, keys and signatures. The details of those are covered in [the section on cryptography](cryptography.html).
+Below we often refer to cryptographic hashes, keys and signatures. The details of those are covered in [the section on cryptography](../appendices/cryptography.html).
 
 ## Transaction Concepts
 
@@ -25,7 +25,7 @@ When someone creates a transfer transaction with the goal of changing an asset's
 
 Every create transaction contains exactly one fulfillment-condition pair. A transfer transaction can contain multiple fulfillment-condition pairs: one per asset transferred. Every fulfillment in a transfer transaction (input) must correspond to a condition (output) in a previous transaction. The diagram below illustrates some of these concepts: transactions are represented by light grey boxes, fulfillments have a label like `f:0`, and conditions have a label like `c:0`.
 
-![Tracking the stories of three assets](./_static/stories_3_assets.png)
+![Tracking the stories of three assets](../_static/stories_3_assets.png)
 
 When a node is asked to check the validity of a transaction, it must do several things; the main things are: