Merge remote-tracking branch 'remotes/origin/develop' into feat/127/crypto-conditions-ilp-sha256-fulfillments

Conflicts:
	bigchaindb/core.py

PYTHON_STYLE_GUIDE.md

@@ -57,46 +57,10 @@ we use the `format()` version. The [official Python documentation says](https://
 
 ## Writing (Python) Tests
 
-We write tests for our Python code using the [pytest](http://pytest.org/latest/) framework.
+We write unit tests for our Python code using the [pytest](http://pytest.org/latest/) framework.
 
 All tests go in the `bigchaindb/tests` directory or one of its subdirectories. You can use the tests already in there as templates or examples.
 
-### Standard Ways to Run All Tests
+The BigchainDB Documentation has a [section explaining how to run all unit tests](http://bigchaindb.readthedocs.org/en/develop/running-unit-tests.html).
 
-To run all the tests, first make sure you have RethinkDB running:
+**Automated testing of pull requests.** We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.)
-```text
-$ rethinkdb
-```
-
-then in another terminal, do:
-```text
-$ py.test -v
-```
-
-If that doesn't work (e.g. maybe you are running in a conda virtual environment), try:
-```text
-$ python -m pytest -v
-```
-
-You can also run all tests via `setup.py`, using:
-```text
-$  python setup.py test
-```
-
-### Using `docker-compose` to Run the Tests
-
-You can use `docker-compose` to run the tests. (You don't have to start RethinkDB first: `docker-compose` does that on its own, when it reads the `docker-compose.yml` file.)
-
-First, build the images (~once), using:
-```text
-$ docker-compose build
-```
-
-then run the tests using:
-```text
-$ docker-compose run --rm bigchaindb py.test -v
-```
-
-### Automated Testing of All Pull Requests
-
-We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.)

ROADMAP.md

@@ -5,30 +5,89 @@ See also:
 * [Milestones](https://github.com/bigchaindb/bigchaindb/milestones) (i.e. issues to be closed before various releases)
 * [Open issues](https://github.com/bigchaindb/bigchaindb/issues) and [open pull requests](https://github.com/bigchaindb/bigchaindb/pulls)
 
-## BigchainDB Protocols
+Note: Below, #345 refers to Issue #345 in the BigchainDB repository on GitHub. #N refers to Issue #N.
-* Validation of other nodes
-* Fault tolerance
-* Permissions framework
-* Protocol audits including security audits
 
-## HTTP Client-Server API
+
-* Validate the structure of the transaction
+## Deployment and Federation Configuration/Management
+* Define how a federation is managed - [#126](https://github.com/bigchaindb/bigchaindb/issues/126)
+* Review current configuration mechanism - [#49](https://github.com/bigchaindb/bigchaindb/issues/49)
+* Make the configuration easier for Docker-based setup - [#36](https://github.com/bigchaindb/bigchaindb/issues/36)
+
+
+## Testing
+* (Unit-test writing and unit testing are ongoing.)
+* More Integration Testing, Validation Testing, System Testing, Benchmarking
+* Define some standard test systems (e.g. local virtual cluster, data center, WAN)
+* Develop standardized test descriptions and documentation (test setup, inputs, outputs)
+* Build up a suite of tests to test each identified fault
+* More tools for cluster benchmarking
+* Identify bottlenecks using profiling and monitoring
+* Fault-testing framework
+* Clean exit for the bigchaindb-benchmark process - [#122](https://github.com/bigchaindb/bigchaindb/issues/122)
+* Tool to bulk-upload transactions into biacklog table - [#114](https://github.com/bigchaindb/bigchaindb/issues/114)
+* Tool to deploy multiple clients for testing - [#113](https://github.com/bigchaindb/bigchaindb/issues/113)
+* Tool to read transactions from files for testing - [#112](https://github.com/bigchaindb/bigchaindb/issues/112)
+
+
+## Specific Bugs/Faults and Related Tests
+* Validation of other nodes
+* Changefeed watchdog
+* Non-deterministic assignment of tx in S is a DoS vulnerability - [#20](https://github.com/bigchaindb/bigchaindb/issues/20)
+* Queues are unbounded - [#124](https://github.com/bigchaindb/bigchaindb/issues/124)
+* Better handling of timeouts in block creation - [#123](https://github.com/bigchaindb/bigchaindb/issues/123)
+* Secure node-node communication - [#77](https://github.com/bigchaindb/bigchaindb/issues/77)
+* Checking if transactions are in a decided_valid block (or otherwise) when necessary - [#134](https://github.com/bigchaindb/bigchaindb/issues/134)
+* When validating an incoming transaction, check to ensure it isn't a duplicate - [#131](https://github.com/bigchaindb/bigchaindb/issues/131)
+* Consider secondary indexes on some queries - [#105](https://github.com/bigchaindb/bigchaindb/issues/105)
+
+
+## Transactions / Assets
+* Current Top-Level Goal: Define and implement "v2 transactions", that is, support multisig (done) and:
+* Support for multiple inputs and outputs - [#128](https://github.com/bigchaindb/bigchaindb/issues/128)
+* Crypto-conditions specific to ILP - [#127](https://github.com/bigchaindb/bigchaindb/issues/127)
+* Support divisible assets - [#129](https://github.com/bigchaindb/bigchaindb/issues/129)
+* Define a JSON template for digital assets - [#125](https://github.com/bigchaindb/bigchaindb/issues/125)
+* Revisit timestamps - [#132](https://github.com/bigchaindb/bigchaindb/issues/132)
+* Refactor structure of a transaction - [#98](https://github.com/bigchaindb/bigchaindb/issues/98)
+* Plugin or hook architecture e.g. for validate_transaction - [#90](https://github.com/bigchaindb/bigchaindb/issues/90)
+
+
+## Web API (HTTP Client-Server API)
+* Current Top-Level Goal: Support v2 transactions (see above)
+* Validate the structure of incoming transactions
 * Return the correct error code if something goes wrong
+* Validate transaction before writing it to the backlog - [#109](https://github.com/bigchaindb/bigchaindb/issues/109)
+* Better organization of transaction-related code - [#108](https://github.com/bigchaindb/bigchaindb/issues/108)
 * Add an endpoint to query unspents for a given public key
 * More endpoints
-* See [the relevant open issues](https://github.com/bigchaindb/bigchaindb/issues?q=is%3Aissue+is%3Aopen+label%3Arest-api)
+* See [open issues with the "rest-api" label](https://github.com/bigchaindb/bigchaindb/issues?q=is%3Aissue+is%3Aopen+label%3Arest-api)
 
-## Implementation/Code
+
-* Node validation framework (inspect and agree or not with what the other nodes are doing)
+## Drivers
-* Open public testing cluster (for people to try out a BigchainDB cluster and to test client software)
+* Update the reference driver (Python) to support v2 transactions and web API (see above)
-* Federation management tools
-* More tools for benchmarking a cluster
-* Descriptions and results of more benchmarking tests
-* AWS image and other easy deployment options
 * Drivers/SDKs for more client-side languages (e.g. JavaScript, Ruby, Java)
-* ORM to better-decouple BigchainDB from its data store (will make it easy to try other databases)
-* Code audits including security audits
 
-## Other/Future
+
-* Byzantine fault tolerance
+## Public Sandbox Testnet and Public BigchainDB
+* Deploy a 3-node Public Sandbox Testnet in a data center, open to all external users, refreshing daily
+* Deploy Public BigchaindB Testnet with more than 3 nodes and with nodes more globally-distributed
+* Public BigchainDB governance/voting system
+* Transaction (or usage) accounting
+* Billing system
+
+
+## Other
+* Get BigchainDB production-ready for submission to AWS Marketplace (as an AMI)
+
+
+## Future
+* Permissions framework
+* More Byzantine fault tolerance (BFT)
 * Better support for smart contract frameworks
+* Algorithm audits
+* Protocol audits
+* Code (implementation) audits
+* Security audits
+* IPFS interoperability - [#100](https://github.com/bigchaindb/bigchaindb/issues/100)
+* ORM to better-decouple BigchainDB from its data store (will make it easy to try other databases)
+* Support more server operating systems

bigchaindb/__init__.py

@@ -40,7 +40,8 @@ config = {
         'port': e('BIGCHAIN_STATSD_PORT', default=8125),
         'rate': e('BIGCHAIN_STATSD_SAMPLERATE', default=0.01)
     },
-    'api_endpoint': 'http://localhost:8008/api/v1'
+    'api_endpoint': 'http://localhost:8008/api/v1',
+    'consensus_plugin': e('BIGCHAIN_CONSENSUS_PLUGIN', default='default')
 }
 
 # We need to maintain a backup copy of the original config dict in case
@@ -48,4 +49,3 @@ config = {
 # for more info.
 _config = copy.deepcopy(config)
 from bigchaindb.core import Bigchain  # noqa
-

bigchaindb/client.py

@@ -15,7 +15,8 @@ class Client:
     In the future, a Client might connect to >1 hosts.
     """
 
-    def __init__(self, public_key=None, private_key=None, api_endpoint=None):
+    def __init__(self, public_key=None, private_key=None, api_endpoint=None,
+                 consensus_plugin=None):
         """Initialize the Client instance
 
         There are three ways in which the Client instance can get its parameters.
@@ -28,8 +29,11 @@ class Client:
         Args:
             public_key (str): the base58 encoded public key for the ECDSA secp256k1 curve.
             private_key (str): the base58 encoded private key for the ECDSA secp256k1 curve.
-            host (str): hostname where the rethinkdb is running.
+            api_endpoint (str): a URL where rethinkdb is running.
-            port (int): port in which rethinkb is running (usually 28015).
+                format: scheme://hostname:port
+            consensus_plugin (str): the registered name of your installed
+                consensus plugin. The `core` plugin is built into BigchainDB;
+                others must be installed via pip.
         """
 
         config_utils.autoconfigure()
@@ -37,6 +41,7 @@ class Client:
         self.public_key = public_key or bigchaindb.config['keypair']['public']
         self.private_key = private_key or bigchaindb.config['keypair']['private']
         self.api_endpoint = api_endpoint or bigchaindb.config['api_endpoint']
+        self.consensus = config_utils.load_consensus_plugin(consensus_plugin)
 
         if not self.public_key or not self.private_key:
             raise exceptions.KeypairNotFoundException()
@@ -51,8 +56,15 @@ class Client:
             The transaction pushed to the Federation.
         """
 
-        tx = util.create_tx(self.public_key, self.public_key, None, operation='CREATE', payload=payload)
+        tx = self.consensus.create_transaction(
-        signed_tx = util.sign_tx(tx, self.private_key)
+            current_owner=self.public_key,
+            new_owner=self.public_key,
+            tx_input=None,
+            operation='CREATE',
+            payload=payload)
+
+        signed_tx = self.consensus.sign_transaction(
+            tx, private_key=self.private_key)
         return self._push(signed_tx)
 
     def transfer(self, new_owner, tx_input, payload=None):
@@ -67,8 +79,15 @@ class Client:
             The transaction pushed to the Federation.
         """
 
-        tx = util.create_tx(self.public_key, new_owner, tx_input, operation='TRANSFER', payload=payload)
+        tx = self.consensus.create_transaction(
-        signed_tx = util.sign_tx(tx, self.private_key)
+            current_owner=self.public_key,
+            new_owner=new_owner,
+            tx_input=tx_input,
+            operation='TRANSFER',
+            payload=payload)
+
+        signed_tx = self.consensus.sign_transaction(
+            tx, private_key=self.private_key)
         return self._push(signed_tx)
 
     def _push(self, tx):

bigchaindb/config_utils.py

@@ -17,7 +17,10 @@ import json
 import logging
 import collections
 
+from pkg_resources import iter_entry_points, ResolutionError
+
 import bigchaindb
+from bigchaindb.consensus import AbstractConsensusRules
 
 logger = logging.getLogger(__name__)
 CONFIG_DEFAULT_PATH = os.environ.setdefault(
@@ -100,3 +103,38 @@ def autoconfigure():
     except FileNotFoundError:
         logger.warning('Cannot find your config file. Run `bigchaindb configure` to create one')
 
+
+def load_consensus_plugin(name=None):
+    """Find and load the chosen consensus plugin.
+
+    Args:
+        name (string): the name of the entry_point, as advertised in the
+            setup.py of the providing package.
+
+    Returns:
+        an uninstantiated subclass of ``bigchaindb.consensus.AbstractConsensusRules``
+    """
+    if not name:
+        name = bigchaindb.config.get('consensus_plugin', 'default')
+
+    # TODO: This will return the first plugin with group `bigchaindb.consensus`
+    #       and name `name` in the active WorkingSet.
+    #       We should probably support Requirements specs in the config, e.g.
+    #       consensus_plugin: 'my-plugin-package==0.0.1;default'
+    plugin = None
+    for entry_point in iter_entry_points('bigchaindb.consensus', name):
+        plugin = entry_point.load()
+
+    # No matching entry_point found
+    if not plugin:
+        raise ResolutionError(
+            'No plugin found in group `bigchaindb.consensus` with name `{}`'.
+            format(name))
+
+    # Is this strictness desireable?
+    # It will probably reduce developer headaches in the wild.
+    if not issubclass(plugin, (AbstractConsensusRules)):
+        raise TypeError("object of type '{}' does not implement `bigchaindb."
+                        "consensus.AbstractConsensusRules`".format(type(plugin)))
+
+    return plugin

bigchaindb/consensus.py

@@ -0,0 +1,221 @@
+from abc import ABCMeta, abstractmethod
+
+import bigchaindb.exceptions as exceptions
+from bigchaindb import util
+from bigchaindb.crypto import hash_data, PublicKey
+
+
+class AbstractConsensusRules(metaclass=ABCMeta):
+    """Abstract base class for Bigchain plugins which implement consensus logic.
+
+    A consensus plugin must expose a class inheriting from this one via an
+    entry_point.
+
+    All methods listed below must be implemented.
+    """
+
+    @abstractmethod
+    def validate_transaction(bigchain, transaction):
+        """Validate a transaction.
+
+        Args:
+            bigchain (Bigchain): an instantiated ``bigchaindb.Bigchain`` object.
+            transaction (dict): transaction to validate.
+
+        Returns:
+            The transaction if the transaction is valid else it raises an
+            exception describing the reason why the transaction is invalid.
+
+        Raises:
+            Descriptive exceptions indicating the reason the transaction failed.
+            See the `exceptions` module for bigchain-native error classes.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def validate_block(bigchain, block):
+        """Validate a block.
+
+        Args:
+            bigchain (Bigchain): an instantiated ``bigchaindb.Bigchain`` object.
+            block (dict): block to validate.
+
+        Returns:
+            The block if the block is valid else it raises an exception
+            describing the reason why the block is invalid.
+
+        Raises:
+            Descriptive exceptions indicating the reason the block failed.
+            See the `exceptions` module for bigchain-native error classes.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def create_transaction(*args, **kwargs):
+        """Create a new transaction.
+
+        Args:
+            The signature of this method is left to plugin authors to decide.
+
+        Returns:
+            dict: newly constructed transaction.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def sign_transaction(transaction, *args, **kwargs):
+        """Sign a transaction.
+
+        Args:
+            transaction (dict): transaction to sign.
+            any other arguments are left to plugin authors to decide.
+
+        Returns:
+            dict: transaction with any signatures applied.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def verify_signature(signed_transaction):
+        """Verify the signature of a transaction.
+
+        Args:
+            signed_transaction (dict): signed transaction to verify
+
+        Returns:
+            bool: True if the transaction's required signature data is present
+                and correct, False otherwise.
+        """
+        raise NotImplementedError
+
+
+class BaseConsensusRules(AbstractConsensusRules):
+    """Base consensus rules for Bigchain.
+
+    This class can be copied or overridden to write your own consensus rules!
+    """
+
+    @staticmethod
+    def validate_transaction(bigchain, transaction):
+        """Validate a transaction.
+
+        Args:
+            bigchain (Bigchain): an instantiated bigchaindb.Bigchain object.
+            transaction (dict): transaction to validate.
+
+        Returns:
+            The transaction if the transaction is valid else it raises an
+            exception describing the reason why the transaction is invalid.
+
+        Raises:
+            OperationError: if the transaction operation is not supported
+            TransactionDoesNotExist: if the input of the transaction is not found
+            TransactionOwnerError: if the new transaction is using an input it doesn't own
+            DoubleSpend: if the transaction is a double spend
+            InvalidHash: if the hash of the transaction is wrong
+            InvalidSignature: if the signature of the transaction is wrong
+        """
+
+        # If the operation is CREATE the transaction should have no inputs and
+        # should be signed by a federation node
+        if transaction['transaction']['operation'] == 'CREATE':
+            if transaction['transaction']['input']:
+                raise ValueError('A CREATE operation has no inputs')
+            if transaction['transaction']['current_owner'] not in (
+                    bigchain.federation_nodes + [bigchain.me]):
+                raise exceptions.OperationError(
+                    'Only federation nodes can use the operation `CREATE`')
+
+        else:
+            # check if the input exists, is owned by the current_owner
+            if not transaction['transaction']['input']:
+                raise ValueError(
+                    'Only `CREATE` transactions can have null inputs')
+
+            tx_input = bigchain.get_transaction(
+                transaction['transaction']['input'])
+
+            if not tx_input:
+                raise exceptions.TransactionDoesNotExist(
+                    'input `{}` does not exist in the bigchain'.format(
+                        transaction['transaction']['input']))
+
+            if (tx_input['transaction']['new_owner'] !=
+                transaction['transaction']['current_owner']):
+                raise exceptions.TransactionOwnerError(
+                    'current_owner `{}` does not own the input `{}`'.format(
+                        transaction['transaction']['current_owner'],
+                        transaction['transaction']['input']))
+
+            # check if the input was already spent by a transaction other than
+            # this one.
+            spent = bigchain.get_spent(tx_input['id'])
+            if spent and spent['id'] != transaction['id']:
+                raise exceptions.DoubleSpend(
+                    'input `{}` was already spent'.format(
+                        transaction['transaction']['input']))
+
+        # Check hash of the transaction
+        calculated_hash = hash_data(util.serialize(
+            transaction['transaction']))
+        if calculated_hash != transaction['id']:
+            raise exceptions.InvalidHash()
+
+        # Check signature
+        if not util.verify_signature(transaction):
+            raise exceptions.InvalidSignature()
+
+        return transaction
+
+    # TODO: Unsure if a bigchain parameter is really necessary here?
+    @staticmethod
+    def validate_block(bigchain, block):
+        """Validate a block.
+
+        Args:
+            bigchain (Bigchain): an instantiated bigchaindb.Bigchain object.
+            block (dict): block to validate.
+
+        Returns:
+            The block if the block is valid else it raises an exception
+            describing the reason why the block is invalid.
+
+        Raises:
+            InvalidHash: if the hash of the block is wrong.
+        """
+
+        # Check if current hash is correct
+        calculated_hash = hash_data(util.serialize(block['block']))
+        if calculated_hash != block['id']:
+            raise exceptions.InvalidHash()
+
+        return block
+
+    @staticmethod
+    def create_transaction(current_owner, new_owner, tx_input, operation,
+                           payload=None):
+        """Create a new transaction
+
+        Refer to the documentation of ``bigchaindb.util.create_tx``
+        """
+
+        return util.create_tx(current_owner, new_owner, tx_input, operation,
+                              payload)
+
+    @staticmethod
+    def sign_transaction(transaction, private_key):
+        """Sign a transaction
+
+        Refer to the documentation of ``bigchaindb.util.sign_tx``
+        """
+
+        return util.sign_tx(transaction, private_key)
+
+    @staticmethod
+    def verify_signature(signed_transaction):
+        """Verify the signature of a transaction.
+
+        Refer to the documentation of ``bigchaindb.util.verify_signature``
+        """
+
+        return util.verify_signature(signed_transaction)

bigchaindb/core.py

@@ -1,7 +1,8 @@
+import rethinkdb as r
 import random
+import json
 import rapidjson
 
-import rethinkdb as r
 
 import bigchaindb
 from bigchaindb import config_utils
@@ -24,7 +25,8 @@ class Bigchain(object):
     """
 
     def __init__(self, host=None, port=None, dbname=None,
-                 public_key=None, private_key=None, keyring=[]):
+                 public_key=None, private_key=None, keyring=[],
+                 consensus_plugin=None):
         """Initialize the Bigchain instance
 
         There are three ways in which the Bigchain instance can get its parameters.
@@ -50,6 +52,7 @@ class Bigchain(object):
         self.me = public_key or bigchaindb.config['keypair']['public']
         self.me_private = private_key or bigchaindb.config['keypair']['private']
         self.federation_nodes = keyring or bigchaindb.config['keyring']
+        self.consensus = config_utils.load_consensus_plugin(consensus_plugin)
 
         if not self.me or not self.me_private:
             raise exceptions.KeypairNotFoundException()
@@ -66,38 +69,40 @@ class Bigchain(object):
         return r.connect(host=self.host, port=self.port, db=self.dbname)
 
     @monitor.timer('create_transaction', rate=bigchaindb.config['statsd']['rate'])
-    def create_transaction(self, current_owner, new_owner, tx_input, operation, payload=None):
+    def create_transaction(self, *args, **kwargs):
         """Create a new transaction
 
-        Refer to the documentation of ``bigchaindb.util.create_tx``
+        Refer to the documentation of your consensus plugin.
+
+        Returns:
+            dict: newly constructed transaction.
         """
 
-        return util.create_tx(current_owner, new_owner, tx_input, operation, payload)
+        return self.consensus.create_transaction(*args, **kwargs)
 
-    def sign_transaction(self, transaction, private_key):
+    def sign_transaction(self, transaction, *args, **kwargs):
         """Sign a transaction
 
-        Refer to the documentation of ``bigchaindb.util.sign_tx``
+        Refer to the documentation of your consensus plugin.
+
+        Returns:
+            dict: transaction with any signatures applied.
         """
 
-        return util.sign_tx(transaction, private_key)
+        return self.consensus.sign_transaction(transaction, *args, **kwargs)
 
-    def verify_signature(self, signed_transaction):
+    def verify_signature(self, signed_transaction, *args, **kwargs):
-        """Verify the signature of a transaction.
+        """Verify the signature(s) of a transaction.
 
-        Refer to the documentation of ``bigchaindb.crypto.verify_signature``
+        Refer to the documentation of your consensus plugin.
+
+        Returns:
+            bool: True if the transaction's required signature data is present
+                and correct, False otherwise.
         """
 
-        data = signed_transaction.copy()
+        return self.consensus.verify_signature(
-
+            signed_transaction, *args, **kwargs)
-        # if assignee field in the transaction, remove it
-        if 'assignee' in data:
-            data.pop('assignee')
-
-        signature = data.pop('signature')
-        public_key_base58 = signed_transaction['transaction']['current_owner']
-        public_key = asymmetric.VerifyingKey(public_key_base58)
-        return public_key.verify(util.serialize(data), signature)
 
     @monitor.timer('write_transaction', rate=bigchaindb.config['statsd']['rate'])
     def write_transaction(self, signed_transaction, durability='soft'):
@@ -107,7 +112,7 @@ class Bigchain(object):
         it has been validated by the nodes of the federation.
 
         Args:
-            singed_transaction (dict): transaction with the `signature` included.
+            signed_transaction (dict): transaction with the `signature` included.
 
         Returns:
             dict: database response
@@ -242,49 +247,11 @@ class Bigchain(object):
             transaction (dict): transaction to validate.
 
         Returns:
-            The transaction if the transaction is valid else it raises and exception
+            The transaction if the transaction is valid else it raises an
-            describing the reason why the transaction is invalid.
+            exception describing the reason why the transaction is invalid.
-
-        Raises:
-            OperationError: if the transaction operation is not supported
-            TransactionDoesNotExist: if the input of the transaction is not found
-            TransactionOwnerError: if the new transaction is using an input it doesn't own
-            DoubleSpend: if the transaction is a double spend
-            InvalidHash: if the hash of the transaction is wrong
-            InvalidSignature: if the signature of the transaction is wrong
         """
 
-        # If the operation is CREATE the transaction should have no inputs and should be signed by a
+        return self.consensus.validate_transaction(self, transaction)
-        # federation node
-        if transaction['transaction']['operation'] == 'CREATE':
-            if transaction['transaction']['input']:
-                raise ValueError('A CREATE operation has no inputs')
-            if transaction['transaction']['current_owner'] not in self.federation_nodes + [self.me]:
-                raise exceptions.OperationError('Only federation nodes can use the operation `CREATE`')
-
-        else:
-            # check if the input exists, is owned by the current_owner
-            if not transaction['transaction']['input']:
-                raise ValueError('Only `CREATE` transactions can have null inputs')
-
-            tx_input = self.get_transaction(transaction['transaction']['input'])
-            if not tx_input:
-                raise exceptions.TransactionDoesNotExist('input `{}` does not exist in the bigchain'.format(
-                    transaction['transaction']['input']))
-
-            if tx_input['transaction']['new_owner'] != transaction['transaction']['current_owner']:
-                raise exceptions.TransactionOwnerError('current_owner `{}` does not own the input `{}`'.format(
-                    transaction['transaction']['current_owner'], transaction['transaction']['input']))
-
-            # check if the input was already spent by a transaction other then this one.
-            spent = self.get_spent(tx_input['id'])
-            if spent:
-                if spent['id'] != transaction['id']:
-                    raise exceptions.DoubleSpend('input `{}` was already spent'.format(
-                        transaction['transaction']['input']))
-
-        util.check_hash_and_signature(transaction)
-        return transaction
 
     def is_valid_transaction(self, transaction):
         """Check whether a transacion is valid or invalid.
@@ -354,12 +321,11 @@ class Bigchain(object):
             describing the reason why the block is invalid.
         """
 
-        # 1. Check if current hash is correct
+        # First: Run the plugin block validation logic
-        calculated_hash = asymmetric.hash_data(util.serialize(block['block']))
+        self.consensus.validate_block(self, block)
-        if calculated_hash != block['id']:
-            raise exceptions.InvalidHash()
 
-        # 2. Validate all transactions in the block
+        # Finally: Tentative assumption that every blockchain will want to
+        # validate all transactions in each block
         for transaction in block['block']['transactions']:
             if not self.is_valid_transaction(transaction):
                 # this will raise the exception
@@ -401,6 +367,8 @@ class Bigchain(object):
         response = r.table('bigchain').get_all(transaction_id, index='transaction_id').run(self.conn)
         return True if len(response.items) > 0 else False
 
+    # TODO: Unless we prescribe the signature of create_transaction, this will
+    #       also need to be moved into the plugin API.
     def create_genesis_block(self):
         """Create the genesis block
 
@@ -507,4 +475,3 @@ class Bigchain(object):
             unvoted.pop(0)
 
         return unvoted
-

bigchaindb/web/views.py

@@ -57,9 +57,10 @@ def create_transaction():
 
     if tx['transaction']['operation'] == 'CREATE':
         tx = util.transform_create(tx)
-        tx = util.sign_tx(tx, bigchain.me_private)
+        tx = bigchain.consensus.sign_transaction(
+            tx, private_key=bigchain.me_private)
 
-    if not util.verify_signature(tx):
+    if not bigchain.consensus.verify_signature(tx):
         val['error'] = 'Invalid transaction signature'
 
     val = bigchain.write_transaction(tx)

docs/source/consensus.md

@@ -0,0 +1,82 @@
+# BigchainDB Consensus Plugins
+
+BigchainDB has a pluggable block/transaction validation architecture. The default consensus rules can be extended or replaced entirely.
+
+
+## Installing a plugin
+
+Plugins can be installed via pip!
+
+```bash
+$ pip install bigchaindb-plugin-demo
+```
+
+Or using setuptools:
+
+```bash
+$ cd bigchaindb-plugin-demo/
+$ python setup.py install # (or develop)
+```
+
+To activate your plugin, you can either set the `consensus_plugin` field in your config file (usually `~/.bigchaindb`) or by setting the `BIGCHAIN_CONSENSUS_PLUGIN` environement variable to the name of your plugin (see the section on [Packaging a plugin](#packaging-a-plugin) for more about plugin names).
+
+
+## Plugin API
+
+BigchainDB's [current plugin API](../../bigchaindb/consensus.py) exposes five functions in an `AbstractConsensusRules` class:
+
+```python
+validate_transaction(bigchain, transaction)
+validate_block(bigchain, block)
+create_transaction(*args, **kwargs)
+sign_transaction(transaction, *args, **kwargs)
+verify_signature(transaction)
+```
+
+Together, these functions are sufficient for most customizations. For example:
+- Replace the crypto-system with one your hardware can accelerate
+- Re-implement an existing protocol
+- Delegate validation to another application
+- etc...
+
+
+## Extending BigchainDB behavior
+
+A default installation of BigchainDB will use the rules in the `BaseConsensusRules` class. If you only need to modify this behavior slightly, you can inherit from that class and call `super()` in any methods you change, so long as the return values remain the same.
+
+Here's a quick example of a plugin that adds nonsense rules:
+
+```python
+from bigchaindb.consensus import BaseConsensusRules
+
+class SillyConsensusRules(BaseConsensusRules):
+
+    @staticmethod
+    def validate_transaction(bigchain, transaction):
+        transaction = super().validate_transaction(bigchain, transaction)
+        # I only like transactions whose timestamps are even.
+        if transaction['transaction']['timestamp'] % 2 != 0:
+            raise StandardError("Odd... very odd indeed.")
+        return transaction
+
+    @staticmethod
+    def validate_block(bigchain, block):
+        block = super().validate_block(bigchain, block)
+        # I don't trust Alice, I think she's shady.
+        if block['block']['node_pubkey'] == '<ALICE_PUBKEY>':
+            raise StandardError("Alice is shady, everybody ignore her blocks!")
+       return block
+```
+
+
+## Packaging a plugin
+
+BigchainDB uses [setuptool's entry_point](https://pythonhosted.org/setuptools/setuptools.html) system to provide the plugin functionality. Any custom plugin needs to add this section to the `setup()` call in their `setup.py`:
+
+```python
+entry_points={
+    'bigchaindb.consensus': [
+        'PLUGIN_NAME=package.module:ConsensusRulesClass'
+    ]
+},
+```

docs/source/index.rst

@@ -14,6 +14,7 @@ Table of Contents
 
    introduction
    installing-server
+   running-unit-tests
    python-server-api-examples
    bigchaindb-cli
    http-client-server-api

docs/source/running-unit-tests.md

@@ -0,0 +1,41 @@
+# Running Unit Tests
+
+Once you've installed BigchainDB Server, you may want to run all the unit tests. This section explains how.
+
+First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. Before you can run all the unit tests, you must [install BigchainDB from source](http://bigchaindb.readthedocs.org/en/develop/installing-server.html#how-to-install-bigchaindb-from-source).
+
+To run all the unit tests, first make sure you have RethinkDB running:
+```text
+$ rethinkdb
+```
+
+then in another terminal, do:
+```text
+$ py.test -v
+```
+
+If the above command doesn't work (e.g. maybe you are running in a conda virtual environment), try:
+```text
+$ python -m pytest -v
+```
+
+(We write our unit tests using the [pytest](http://pytest.org/latest/) framework.)
+
+You can also run all unit tests via `setup.py`, using:
+```text
+$  python setup.py test
+```
+
+### Using `docker-compose` to Run the Tests
+
+You can also use `docker-compose` to run the unit tests. (You don't have to start RethinkDB first: `docker-compose` does that on its own, when it reads the `docker-compose.yml` file.)
+
+First, build the images (~once), using:
+```text
+$ docker-compose build
+```
+
+then run the unit tests using:
+```text
+$ docker-compose run --rm bigchaindb py.test -v
+```

setup.py

@@ -56,13 +56,21 @@ setup(
         'Operating System :: POSIX :: Linux',
     ],
 
-    packages=['bigchaindb', 'bigchaindb.commands', 'bigchaindb.db', 'bigchaindb.web'],
+    packages=[
+        'bigchaindb',
+        'bigchaindb.commands',
+        'bigchaindb.db',
+        'bigchaindb.web'
+    ],
 
     entry_points={
         'console_scripts': [
             'bigchaindb=bigchaindb.commands.bigchain:main',
             'bigchaindb-benchmark=bigchaindb.commands.bigchain_benchmark:main'
         ],
+        'bigchaindb.consensus': [
+            'default=bigchaindb.consensus:BaseConsensusRules'
+        ]
     },
     install_requires=[
         'rethinkdb==2.2.0.post4',

tests/test_core.py

@@ -15,6 +15,7 @@ def config(request, monkeypatch):
         },
         'keyring': [],
         'CONFIGURED': True,
+        'consensus_plugin': 'default'
     }
 
     monkeypatch.setattr('bigchaindb.config', config)
@@ -24,6 +25,7 @@ def config(request, monkeypatch):
 
 def test_bigchain_class_default_initialization(config):
     from bigchaindb.core import Bigchain
+    from bigchaindb.consensus import BaseConsensusRules
     bigchain = Bigchain()
     assert bigchain.host == config['database']['host']
     assert bigchain.port == config['database']['port']
@@ -31,11 +33,13 @@ def test_bigchain_class_default_initialization(config):
     assert bigchain.me == config['keypair']['public']
     assert bigchain.me_private == config['keypair']['private']
     assert bigchain.federation_nodes == config['keyring']
+    assert bigchain.consensus == BaseConsensusRules
     assert bigchain._conn is None
 
 
 def test_bigchain_class_initialization_with_parameters(config):
     from bigchaindb.core import Bigchain
+    from bigchaindb.consensus import BaseConsensusRules
     init_kwargs = {
         'host': 'some_node',
         'port': '12345',
@@ -43,6 +47,7 @@ def test_bigchain_class_initialization_with_parameters(config):
         'public_key': 'white',
         'private_key': 'black',
         'keyring': ['key_one', 'key_two'],
+        'consensus_plugin': 'default'
     }
     bigchain = Bigchain(**init_kwargs)
     assert bigchain.host == init_kwargs['host']
@@ -51,4 +56,5 @@ def test_bigchain_class_initialization_with_parameters(config):
     assert bigchain.me == init_kwargs['public_key']
     assert bigchain.me_private == init_kwargs['private_key']
     assert bigchain.federation_nodes == init_kwargs['keyring']
+    assert bigchain.consensus == BaseConsensusRules
     assert bigchain._conn is None

tests/utils/test_config_utils.py

@@ -37,3 +37,32 @@ def test_bigchain_instance_raises_when_not_configured(monkeypatch):
 
     with pytest.raises(exceptions.KeypairNotFoundException):
         bigchaindb.Bigchain()
+
+
+def test_load_consensus_plugin_loads_default_rules_without_name():
+    from bigchaindb import config_utils
+    from bigchaindb.consensus import BaseConsensusRules
+
+    assert config_utils.load_consensus_plugin() == BaseConsensusRules
+
+
+def test_load_consensus_plugin_raises_with_unknown_name():
+    from pkg_resources import ResolutionError
+    from bigchaindb import config_utils
+
+    with pytest.raises(ResolutionError):
+        config_utils.load_consensus_plugin('bogus')
+
+
+def test_load_consensus_plugin_raises_with_invalid_subclass(monkeypatch):
+    # Monkeypatch entry_point.load to return something other than a
+    # ConsensusRules instance
+    from bigchaindb import config_utils
+    monkeypatch.setattr(config_utils,
+                        'iter_entry_points',
+                        lambda *args: [ type('entry_point',
+                                             (object),
+                                             {'load': lambda: object}) ])
+
+    with pytest.raises(TypeError):
+        config_utils.load_consensus_plugin()