add top-level bigchaindb README (#678)

bigchaindb/README.md

@@ -0,0 +1,45 @@
+# Overview
+
+A high-level description of the files and subdirectories of BigchainDB. Heavily used external dependencies include [`multipipes`](https://github.com/bigchaindb/multipipes) and [`bigchaindb-common`](https://github.com/bigchaindb/bigchaindb-common).
+
+There are three database tables which underpin BigchainDB: `backlog`, where incoming transactions are held temporarily until they can be consumed by; `bigchain`, where blocks of transactions are written permanently; and `votes`, where votes are written permanently.  It is the votes in the `votes` table which must be queried to determine block validity and order. For more in-depth explanation, see [the whitepaper](https://www.bigchaindb.com/whitepaper/).
+
+## Files
+
+### [`core.py`](./core.py)
+
+The `Bigchain` class is defined here.  Most operations outlined in the [whitepaper](https://www.bigchaindb.com/whitepaper/) as well as database interactions are found in this file.  This is the place to start if you are interested in implementing a server API, since many of these class methods concern BigchainDB interacting with the outside world.
+
+### [`models.py`](./models.py)
+
+`Block`, `Transaction`, and `Asset` classes are defined here.  The classes mirror the block and transaction structure from the [documentation](https://docs.bigchaindb.com/projects/server/en/latest/topic-guides/models.html), but also include methods for validation and signing.
+
+### [`consensus.py`](./config_utils.py)
+
+Base class for consensus methods (verification of votes, blocks, and transactions).  The actual logic is mostly found in `transaction` and `block` models, defined in [`models.py`](https://github.com/bigchaindb/bigchaindb/blob/master/bigchaindb/models.py).
+
+### [`processes.py`](./processes.py)
+
+Entry point for the BigchainDB process, after initialization.  All subprocesses are started here: processes to handle new blocks, votes, etc.
+
+### [`config_utils.py`](./config_utils.py)
+
+Methods for managing the configuration, including loading configuration files, automatically generating the configuration, and keeping the configuration consistent across BigchainDB instances.
+
+### [`monitor.py`](./monitor.py)
+
+Code for monitoring speed of various processes in BigchainDB via `statsd` and Grafana.  [See documentation.](https://docs.bigchaindb.com/projects/server/en/latest/clusters-feds/monitoring.html)
+
+## Folders
+
+### [`pipelines`](./pipelines)
+
+Structure and implementation of various subprocesses started in [`processes.py`](https://github.com/bigchaindb/bigchaindb/blob/master/bigchaindb/processes.py).
+
+### [`commands`](./commands)
+
+Contains code for the [CLI](https://docs.bigchaindb.com/projects/server/en/latest/server-reference/bigchaindb-cli.html) for BigchainDB.
+
+### [`db`](./db)
+
+Code for building the database connection, creating indexes, and other database setup tasks.

bigchaindb/pipelines/README.md

@@ -0,0 +1,39 @@
+# Overview
+
+Code in this module concerns the long-running processes of BigchainDB.  Some are run as single processes while others may run as many processes in parallel.  When changes are detected in the `backlog`, `block`, or `votes` tables, they are handled here.  Everything is started in [`processes.py`](../processes.py).
+
+All the classes defined in these files depend heavily on the [`multipipes`](https://github.com/bigchaindb/multipipes/) library.  Each contains a static method `create_pipeline` which describes how the `Pipeline` is set up. Consider `votes.py`:
+
+```python
+vote_pipeline = Pipeline([
+    Node(voter.validate_block),
+    Node(voter.ungroup),
+    Node(voter.validate_tx, fraction_of_cores=1),
+    Node(voter.vote),
+    Node(voter.write_vote)
+])
+```
+
+The process flow is described here: an incoming block is validated, then the transactions are ungrouped, validated individually (using all available cores in parallel), a vote is created, and finally written to the votes table.
+
+## Files
+
+### [`block.py`](./block.py)
+
+Handles inserts and updates to the backlog.  When a node adds a transaction to the backlog, a `BlockPipeline` instance will verify it. If the transaction is valid, it will add it to a new block; otherwise, it's dropped. Finally, after a block accumulates 1000 transactions or a timeout is reached, the process will write the block.
+
+### [`election.py`](./election.py)
+
+Listens for inserts to the vote table. When a new vote comes in, checks if there are enough votes on that block to declare it valid or invalid.  If the block has been elected invalid, the transactions in that block are put back in the backlog.
+
+### [`stale.py`](./stale.py)
+
+Doesn't listen for any changes.  Periodically checks the backlog for transactions that have been there too long and assigns them to a new node if possible.
+
+### [`vote.py`](./vote.py)
+
+Listens for inserts to the bigchain table, then votes the blocks valid or invalid.  When a new block is written, the node running this process checks the validity of the block by checking the validity of each transaction. Then the vote is created based on the block validity, and cast (written) to the votes table.
+
+### [`utils.py`](./utils.py)
+
+Contains the `ChangeFeed` class, an abstraction which combines `multipipes` with the RethinkDB changefeed.