diff --git a/PYTHON_STYLE_GUIDE.md b/PYTHON_STYLE_GUIDE.md index dc7e68d7..4d4d4888 100644 --- a/PYTHON_STYLE_GUIDE.md +++ b/PYTHON_STYLE_GUIDE.md @@ -57,46 +57,10 @@ we use the `format()` version. The [official Python documentation says](https:// ## Writing (Python) Tests -We write tests for our Python code using the [pytest](http://pytest.org/latest/) framework. +We write unit tests for our Python code using the [pytest](http://pytest.org/latest/) framework. All tests go in the `bigchaindb/tests` directory or one of its subdirectories. You can use the tests already in there as templates or examples. -### Standard Ways to Run All Tests +The BigchainDB Documentation has a [section explaining how to run all unit tests](http://bigchaindb.readthedocs.org/en/develop/running-unit-tests.html). -To run all the tests, first make sure you have RethinkDB running: -```text -$ rethinkdb -``` - -then in another terminal, do: -```text -$ py.test -v -``` - -If that doesn't work (e.g. maybe you are running in a conda virtual environment), try: -```text -$ python -m pytest -v -``` - -You can also run all tests via `setup.py`, using: -```text -$ python setup.py test -``` - -### Using `docker-compose` to Run the Tests - -You can use `docker-compose` to run the tests. (You don't have to start RethinkDB first: `docker-compose` does that on its own, when it reads the `docker-compose.yml` file.) - -First, build the images (~once), using: -```text -$ docker-compose build -``` - -then run the tests using: -```text -$ docker-compose run --rm bigchaindb py.test -v -``` - -### Automated Testing of All Pull Requests - -We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.) +**Automated testing of pull requests.** We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.) diff --git a/ROADMAP.md b/ROADMAP.md index 79fadf02..9d319994 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,30 +5,89 @@ See also: * [Milestones](https://github.com/bigchaindb/bigchaindb/milestones) (i.e. issues to be closed before various releases) * [Open issues](https://github.com/bigchaindb/bigchaindb/issues) and [open pull requests](https://github.com/bigchaindb/bigchaindb/pulls) -## BigchainDB Protocols -* Validation of other nodes -* Fault tolerance -* Permissions framework -* Protocol audits including security audits +Note: Below, #345 refers to Issue #345 in the BigchainDB repository on GitHub. #N refers to Issue #N. -## HTTP Client-Server API -* Validate the structure of the transaction + +## Deployment and Federation Configuration/Management +* Define how a federation is managed - [#126](https://github.com/bigchaindb/bigchaindb/issues/126) +* Review current configuration mechanism - [#49](https://github.com/bigchaindb/bigchaindb/issues/49) +* Make the configuration easier for Docker-based setup - [#36](https://github.com/bigchaindb/bigchaindb/issues/36) + + +## Testing +* (Unit-test writing and unit testing are ongoing.) +* More Integration Testing, Validation Testing, System Testing, Benchmarking +* Define some standard test systems (e.g. local virtual cluster, data center, WAN) +* Develop standardized test descriptions and documentation (test setup, inputs, outputs) +* Build up a suite of tests to test each identified fault +* More tools for cluster benchmarking +* Identify bottlenecks using profiling and monitoring +* Fault-testing framework +* Clean exit for the bigchaindb-benchmark process - [#122](https://github.com/bigchaindb/bigchaindb/issues/122) +* Tool to bulk-upload transactions into biacklog table - [#114](https://github.com/bigchaindb/bigchaindb/issues/114) +* Tool to deploy multiple clients for testing - [#113](https://github.com/bigchaindb/bigchaindb/issues/113) +* Tool to read transactions from files for testing - [#112](https://github.com/bigchaindb/bigchaindb/issues/112) + + +## Specific Bugs/Faults and Related Tests +* Validation of other nodes +* Changefeed watchdog +* Non-deterministic assignment of tx in S is a DoS vulnerability - [#20](https://github.com/bigchaindb/bigchaindb/issues/20) +* Queues are unbounded - [#124](https://github.com/bigchaindb/bigchaindb/issues/124) +* Better handling of timeouts in block creation - [#123](https://github.com/bigchaindb/bigchaindb/issues/123) +* Secure node-node communication - [#77](https://github.com/bigchaindb/bigchaindb/issues/77) +* Checking if transactions are in a decided_valid block (or otherwise) when necessary - [#134](https://github.com/bigchaindb/bigchaindb/issues/134) +* When validating an incoming transaction, check to ensure it isn't a duplicate - [#131](https://github.com/bigchaindb/bigchaindb/issues/131) +* Consider secondary indexes on some queries - [#105](https://github.com/bigchaindb/bigchaindb/issues/105) + + +## Transactions / Assets +* Current Top-Level Goal: Define and implement "v2 transactions", that is, support multisig (done) and: +* Support for multiple inputs and outputs - [#128](https://github.com/bigchaindb/bigchaindb/issues/128) +* Crypto-conditions specific to ILP - [#127](https://github.com/bigchaindb/bigchaindb/issues/127) +* Support divisible assets - [#129](https://github.com/bigchaindb/bigchaindb/issues/129) +* Define a JSON template for digital assets - [#125](https://github.com/bigchaindb/bigchaindb/issues/125) +* Revisit timestamps - [#132](https://github.com/bigchaindb/bigchaindb/issues/132) +* Refactor structure of a transaction - [#98](https://github.com/bigchaindb/bigchaindb/issues/98) +* Plugin or hook architecture e.g. for validate_transaction - [#90](https://github.com/bigchaindb/bigchaindb/issues/90) + + +## Web API (HTTP Client-Server API) +* Current Top-Level Goal: Support v2 transactions (see above) +* Validate the structure of incoming transactions * Return the correct error code if something goes wrong +* Validate transaction before writing it to the backlog - [#109](https://github.com/bigchaindb/bigchaindb/issues/109) +* Better organization of transaction-related code - [#108](https://github.com/bigchaindb/bigchaindb/issues/108) * Add an endpoint to query unspents for a given public key * More endpoints -* See [the relevant open issues](https://github.com/bigchaindb/bigchaindb/issues?q=is%3Aissue+is%3Aopen+label%3Arest-api) +* See [open issues with the "rest-api" label](https://github.com/bigchaindb/bigchaindb/issues?q=is%3Aissue+is%3Aopen+label%3Arest-api) -## Implementation/Code -* Node validation framework (inspect and agree or not with what the other nodes are doing) -* Open public testing cluster (for people to try out a BigchainDB cluster and to test client software) -* Federation management tools -* More tools for benchmarking a cluster -* Descriptions and results of more benchmarking tests -* AWS image and other easy deployment options + +## Drivers +* Update the reference driver (Python) to support v2 transactions and web API (see above) * Drivers/SDKs for more client-side languages (e.g. JavaScript, Ruby, Java) -* ORM to better-decouple BigchainDB from its data store (will make it easy to try other databases) -* Code audits including security audits -## Other/Future -* Byzantine fault tolerance + +## Public Sandbox Testnet and Public BigchainDB +* Deploy a 3-node Public Sandbox Testnet in a data center, open to all external users, refreshing daily +* Deploy Public BigchaindB Testnet with more than 3 nodes and with nodes more globally-distributed +* Public BigchainDB governance/voting system +* Transaction (or usage) accounting +* Billing system + + +## Other +* Get BigchainDB production-ready for submission to AWS Marketplace (as an AMI) + + +## Future +* Permissions framework +* More Byzantine fault tolerance (BFT) * Better support for smart contract frameworks +* Algorithm audits +* Protocol audits +* Code (implementation) audits +* Security audits +* IPFS interoperability - [#100](https://github.com/bigchaindb/bigchaindb/issues/100) +* ORM to better-decouple BigchainDB from its data store (will make it easy to try other databases) +* Support more server operating systems diff --git a/bigchaindb/__init__.py b/bigchaindb/__init__.py index 4982052f..9283913f 100644 --- a/bigchaindb/__init__.py +++ b/bigchaindb/__init__.py @@ -40,7 +40,8 @@ config = { 'port': e('BIGCHAIN_STATSD_PORT', default=8125), 'rate': e('BIGCHAIN_STATSD_SAMPLERATE', default=0.01) }, - 'api_endpoint': 'http://localhost:8008/api/v1' + 'api_endpoint': 'http://localhost:8008/api/v1', + 'consensus_plugin': e('BIGCHAIN_CONSENSUS_PLUGIN', default='default') } # We need to maintain a backup copy of the original config dict in case @@ -48,4 +49,3 @@ config = { # for more info. _config = copy.deepcopy(config) from bigchaindb.core import Bigchain # noqa - diff --git a/bigchaindb/client.py b/bigchaindb/client.py index 0a9eed01..f7b37ad4 100644 --- a/bigchaindb/client.py +++ b/bigchaindb/client.py @@ -15,7 +15,8 @@ class Client: In the future, a Client might connect to >1 hosts. """ - def __init__(self, public_key=None, private_key=None, api_endpoint=None): + def __init__(self, public_key=None, private_key=None, api_endpoint=None, + consensus_plugin=None): """Initialize the Client instance There are three ways in which the Client instance can get its parameters. @@ -28,8 +29,11 @@ class Client: Args: public_key (str): the base58 encoded public key for the ECDSA secp256k1 curve. private_key (str): the base58 encoded private key for the ECDSA secp256k1 curve. - host (str): hostname where the rethinkdb is running. - port (int): port in which rethinkb is running (usually 28015). + api_endpoint (str): a URL where rethinkdb is running. + format: scheme://hostname:port + consensus_plugin (str): the registered name of your installed + consensus plugin. The `core` plugin is built into BigchainDB; + others must be installed via pip. """ config_utils.autoconfigure() @@ -37,6 +41,7 @@ class Client: self.public_key = public_key or bigchaindb.config['keypair']['public'] self.private_key = private_key or bigchaindb.config['keypair']['private'] self.api_endpoint = api_endpoint or bigchaindb.config['api_endpoint'] + self.consensus = config_utils.load_consensus_plugin(consensus_plugin) if not self.public_key or not self.private_key: raise exceptions.KeypairNotFoundException() @@ -51,8 +56,15 @@ class Client: The transaction pushed to the Federation. """ - tx = util.create_tx(self.public_key, self.public_key, None, operation='CREATE', payload=payload) - signed_tx = util.sign_tx(tx, self.private_key) + tx = self.consensus.create_transaction( + current_owner=self.public_key, + new_owner=self.public_key, + tx_input=None, + operation='CREATE', + payload=payload) + + signed_tx = self.consensus.sign_transaction( + tx, private_key=self.private_key) return self._push(signed_tx) def transfer(self, new_owner, tx_input, payload=None): @@ -67,8 +79,15 @@ class Client: The transaction pushed to the Federation. """ - tx = util.create_tx(self.public_key, new_owner, tx_input, operation='TRANSFER', payload=payload) - signed_tx = util.sign_tx(tx, self.private_key) + tx = self.consensus.create_transaction( + current_owner=self.public_key, + new_owner=new_owner, + tx_input=tx_input, + operation='TRANSFER', + payload=payload) + + signed_tx = self.consensus.sign_transaction( + tx, private_key=self.private_key) return self._push(signed_tx) def _push(self, tx): diff --git a/bigchaindb/config_utils.py b/bigchaindb/config_utils.py index bf3ed529..645780eb 100644 --- a/bigchaindb/config_utils.py +++ b/bigchaindb/config_utils.py @@ -17,7 +17,10 @@ import json import logging import collections +from pkg_resources import iter_entry_points, ResolutionError + import bigchaindb +from bigchaindb.consensus import AbstractConsensusRules logger = logging.getLogger(__name__) CONFIG_DEFAULT_PATH = os.environ.setdefault( @@ -100,3 +103,38 @@ def autoconfigure(): except FileNotFoundError: logger.warning('Cannot find your config file. Run `bigchaindb configure` to create one') + +def load_consensus_plugin(name=None): + """Find and load the chosen consensus plugin. + + Args: + name (string): the name of the entry_point, as advertised in the + setup.py of the providing package. + + Returns: + an uninstantiated subclass of ``bigchaindb.consensus.AbstractConsensusRules`` + """ + if not name: + name = bigchaindb.config.get('consensus_plugin', 'default') + + # TODO: This will return the first plugin with group `bigchaindb.consensus` + # and name `name` in the active WorkingSet. + # We should probably support Requirements specs in the config, e.g. + # consensus_plugin: 'my-plugin-package==0.0.1;default' + plugin = None + for entry_point in iter_entry_points('bigchaindb.consensus', name): + plugin = entry_point.load() + + # No matching entry_point found + if not plugin: + raise ResolutionError( + 'No plugin found in group `bigchaindb.consensus` with name `{}`'. + format(name)) + + # Is this strictness desireable? + # It will probably reduce developer headaches in the wild. + if not issubclass(plugin, (AbstractConsensusRules)): + raise TypeError("object of type '{}' does not implement `bigchaindb." + "consensus.AbstractConsensusRules`".format(type(plugin))) + + return plugin diff --git a/bigchaindb/consensus.py b/bigchaindb/consensus.py new file mode 100644 index 00000000..b65ab9ad --- /dev/null +++ b/bigchaindb/consensus.py @@ -0,0 +1,221 @@ +from abc import ABCMeta, abstractmethod + +import bigchaindb.exceptions as exceptions +from bigchaindb import util +from bigchaindb.crypto import hash_data, PublicKey + + +class AbstractConsensusRules(metaclass=ABCMeta): + """Abstract base class for Bigchain plugins which implement consensus logic. + + A consensus plugin must expose a class inheriting from this one via an + entry_point. + + All methods listed below must be implemented. + """ + + @abstractmethod + def validate_transaction(bigchain, transaction): + """Validate a transaction. + + Args: + bigchain (Bigchain): an instantiated ``bigchaindb.Bigchain`` object. + transaction (dict): transaction to validate. + + Returns: + The transaction if the transaction is valid else it raises an + exception describing the reason why the transaction is invalid. + + Raises: + Descriptive exceptions indicating the reason the transaction failed. + See the `exceptions` module for bigchain-native error classes. + """ + raise NotImplementedError + + @abstractmethod + def validate_block(bigchain, block): + """Validate a block. + + Args: + bigchain (Bigchain): an instantiated ``bigchaindb.Bigchain`` object. + block (dict): block to validate. + + Returns: + The block if the block is valid else it raises an exception + describing the reason why the block is invalid. + + Raises: + Descriptive exceptions indicating the reason the block failed. + See the `exceptions` module for bigchain-native error classes. + """ + raise NotImplementedError + + @abstractmethod + def create_transaction(*args, **kwargs): + """Create a new transaction. + + Args: + The signature of this method is left to plugin authors to decide. + + Returns: + dict: newly constructed transaction. + """ + raise NotImplementedError + + @abstractmethod + def sign_transaction(transaction, *args, **kwargs): + """Sign a transaction. + + Args: + transaction (dict): transaction to sign. + any other arguments are left to plugin authors to decide. + + Returns: + dict: transaction with any signatures applied. + """ + raise NotImplementedError + + @abstractmethod + def verify_signature(signed_transaction): + """Verify the signature of a transaction. + + Args: + signed_transaction (dict): signed transaction to verify + + Returns: + bool: True if the transaction's required signature data is present + and correct, False otherwise. + """ + raise NotImplementedError + + +class BaseConsensusRules(AbstractConsensusRules): + """Base consensus rules for Bigchain. + + This class can be copied or overridden to write your own consensus rules! + """ + + @staticmethod + def validate_transaction(bigchain, transaction): + """Validate a transaction. + + Args: + bigchain (Bigchain): an instantiated bigchaindb.Bigchain object. + transaction (dict): transaction to validate. + + Returns: + The transaction if the transaction is valid else it raises an + exception describing the reason why the transaction is invalid. + + Raises: + OperationError: if the transaction operation is not supported + TransactionDoesNotExist: if the input of the transaction is not found + TransactionOwnerError: if the new transaction is using an input it doesn't own + DoubleSpend: if the transaction is a double spend + InvalidHash: if the hash of the transaction is wrong + InvalidSignature: if the signature of the transaction is wrong + """ + + # If the operation is CREATE the transaction should have no inputs and + # should be signed by a federation node + if transaction['transaction']['operation'] == 'CREATE': + if transaction['transaction']['input']: + raise ValueError('A CREATE operation has no inputs') + if transaction['transaction']['current_owner'] not in ( + bigchain.federation_nodes + [bigchain.me]): + raise exceptions.OperationError( + 'Only federation nodes can use the operation `CREATE`') + + else: + # check if the input exists, is owned by the current_owner + if not transaction['transaction']['input']: + raise ValueError( + 'Only `CREATE` transactions can have null inputs') + + tx_input = bigchain.get_transaction( + transaction['transaction']['input']) + + if not tx_input: + raise exceptions.TransactionDoesNotExist( + 'input `{}` does not exist in the bigchain'.format( + transaction['transaction']['input'])) + + if (tx_input['transaction']['new_owner'] != + transaction['transaction']['current_owner']): + raise exceptions.TransactionOwnerError( + 'current_owner `{}` does not own the input `{}`'.format( + transaction['transaction']['current_owner'], + transaction['transaction']['input'])) + + # check if the input was already spent by a transaction other than + # this one. + spent = bigchain.get_spent(tx_input['id']) + if spent and spent['id'] != transaction['id']: + raise exceptions.DoubleSpend( + 'input `{}` was already spent'.format( + transaction['transaction']['input'])) + + # Check hash of the transaction + calculated_hash = hash_data(util.serialize( + transaction['transaction'])) + if calculated_hash != transaction['id']: + raise exceptions.InvalidHash() + + # Check signature + if not util.verify_signature(transaction): + raise exceptions.InvalidSignature() + + return transaction + + # TODO: Unsure if a bigchain parameter is really necessary here? + @staticmethod + def validate_block(bigchain, block): + """Validate a block. + + Args: + bigchain (Bigchain): an instantiated bigchaindb.Bigchain object. + block (dict): block to validate. + + Returns: + The block if the block is valid else it raises an exception + describing the reason why the block is invalid. + + Raises: + InvalidHash: if the hash of the block is wrong. + """ + + # Check if current hash is correct + calculated_hash = hash_data(util.serialize(block['block'])) + if calculated_hash != block['id']: + raise exceptions.InvalidHash() + + return block + + @staticmethod + def create_transaction(current_owner, new_owner, tx_input, operation, + payload=None): + """Create a new transaction + + Refer to the documentation of ``bigchaindb.util.create_tx`` + """ + + return util.create_tx(current_owner, new_owner, tx_input, operation, + payload) + + @staticmethod + def sign_transaction(transaction, private_key): + """Sign a transaction + + Refer to the documentation of ``bigchaindb.util.sign_tx`` + """ + + return util.sign_tx(transaction, private_key) + + @staticmethod + def verify_signature(signed_transaction): + """Verify the signature of a transaction. + + Refer to the documentation of ``bigchaindb.util.verify_signature`` + """ + + return util.verify_signature(signed_transaction) diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 3b2a29d1..f76dd2d6 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -26,7 +26,8 @@ class Bigchain(object): """ def __init__(self, host=None, port=None, dbname=None, - public_key=None, private_key=None, keyring=[]): + public_key=None, private_key=None, keyring=[], + consensus_plugin=None): """Initialize the Bigchain instance There are three ways in which the Bigchain instance can get its parameters. @@ -52,6 +53,7 @@ class Bigchain(object): self.me = public_key or bigchaindb.config['keypair']['public'] self.me_private = private_key or bigchaindb.config['keypair']['private'] self.federation_nodes = keyring or bigchaindb.config['keyring'] + self.consensus = config_utils.load_consensus_plugin(consensus_plugin) if not self.me or not self.me_private: raise exceptions.KeypairNotFoundException() @@ -68,38 +70,40 @@ class Bigchain(object): return r.connect(host=self.host, port=self.port, db=self.dbname) @monitor.timer('create_transaction', rate=bigchaindb.config['statsd']['rate']) - def create_transaction(self, current_owner, new_owner, tx_input, operation, payload=None): + def create_transaction(self, *args, **kwargs): """Create a new transaction - Refer to the documentation of ``bigchaindb.util.create_tx`` + Refer to the documentation of your consensus plugin. + + Returns: + dict: newly constructed transaction. """ - return util.create_tx(current_owner, new_owner, tx_input, operation, payload) + return self.consensus.create_transaction(*args, **kwargs) - def sign_transaction(self, transaction, private_key): + def sign_transaction(self, transaction, *args, **kwargs): """Sign a transaction - Refer to the documentation of ``bigchaindb.util.sign_tx`` + Refer to the documentation of your consensus plugin. + + Returns: + dict: transaction with any signatures applied. """ - return util.sign_tx(transaction, private_key) + return self.consensus.sign_transaction(transaction, *args, **kwargs) - def verify_signature(self, signed_transaction): - """Verify the signature of a transaction. + def verify_signature(self, signed_transaction, *args, **kwargs): + """Verify the signature(s) of a transaction. - Refer to the documentation of ``bigchaindb.crypto.verify_signature`` + Refer to the documentation of your consensus plugin. + + Returns: + bool: True if the transaction's required signature data is present + and correct, False otherwise. """ - data = signed_transaction.copy() - - # if assignee field in the transaction, remove it - if 'assignee' in data: - data.pop('assignee') - - signature = data.pop('signature') - public_key_base58 = signed_transaction['transaction']['current_owner'] - public_key = crypto.PublicKey(public_key_base58) - return public_key.verify(util.serialize(data), signature) + return self.consensus.verify_signature( + signed_transaction, *args, **kwargs) @monitor.timer('write_transaction', rate=bigchaindb.config['statsd']['rate']) def write_transaction(self, signed_transaction, durability='soft'): @@ -109,7 +113,7 @@ class Bigchain(object): it has been validated by the nodes of the federation. Args: - singed_transaction (dict): transaction with the `signature` included. + signed_transaction (dict): transaction with the `signature` included. Returns: dict: database response @@ -244,49 +248,11 @@ class Bigchain(object): transaction (dict): transaction to validate. Returns: - The transaction if the transaction is valid else it raises and exception - describing the reason why the transaction is invalid. - - Raises: - OperationError: if the transaction operation is not supported - TransactionDoesNotExist: if the input of the transaction is not found - TransactionOwnerError: if the new transaction is using an input it doesn't own - DoubleSpend: if the transaction is a double spend - InvalidHash: if the hash of the transaction is wrong - InvalidSignature: if the signature of the transaction is wrong + The transaction if the transaction is valid else it raises an + exception describing the reason why the transaction is invalid. """ - # If the operation is CREATE the transaction should have no inputs and should be signed by a - # federation node - if transaction['transaction']['operation'] == 'CREATE': - if transaction['transaction']['input']: - raise ValueError('A CREATE operation has no inputs') - if transaction['transaction']['current_owner'] not in self.federation_nodes + [self.me]: - raise exceptions.OperationError('Only federation nodes can use the operation `CREATE`') - - else: - # check if the input exists, is owned by the current_owner - if not transaction['transaction']['input']: - raise ValueError('Only `CREATE` transactions can have null inputs') - - tx_input = self.get_transaction(transaction['transaction']['input']) - if not tx_input: - raise exceptions.TransactionDoesNotExist('input `{}` does not exist in the bigchain'.format( - transaction['transaction']['input'])) - - if tx_input['transaction']['new_owner'] != transaction['transaction']['current_owner']: - raise exceptions.TransactionOwnerError('current_owner `{}` does not own the input `{}`'.format( - transaction['transaction']['current_owner'], transaction['transaction']['input'])) - - # check if the input was already spent by a transaction other then this one. - spent = self.get_spent(tx_input['id']) - if spent: - if spent['id'] != transaction['id']: - raise exceptions.DoubleSpend('input `{}` was already spent'.format( - transaction['transaction']['input'])) - - util.check_hash_and_signature(transaction) - return transaction + return self.consensus.validate_transaction(self, transaction) def is_valid_transaction(self, transaction): """Check whether a transacion is valid or invalid. @@ -356,12 +322,11 @@ class Bigchain(object): describing the reason why the block is invalid. """ - # 1. Check if current hash is correct - calculated_hash = crypto.hash_data(util.serialize(block['block'])) - if calculated_hash != block['id']: - raise exceptions.InvalidHash() + # First: Run the plugin block validation logic + self.consensus.validate_block(self, block) - # 2. Validate all transactions in the block + # Finally: Tentative assumption that every blockchain will want to + # validate all transactions in each block for transaction in block['block']['transactions']: if not self.is_valid_transaction(transaction): # this will raise the exception @@ -403,6 +368,8 @@ class Bigchain(object): response = r.table('bigchain').get_all(transaction_id, index='transaction_id').run(self.conn) return True if len(response.items) > 0 else False + # TODO: Unless we prescribe the signature of create_transaction, this will + # also need to be moved into the plugin API. def create_genesis_block(self): """Create the genesis block @@ -509,4 +476,3 @@ class Bigchain(object): unvoted.pop(0) return unvoted - diff --git a/bigchaindb/web/views.py b/bigchaindb/web/views.py index 04d3992d..53db6f66 100644 --- a/bigchaindb/web/views.py +++ b/bigchaindb/web/views.py @@ -57,9 +57,10 @@ def create_transaction(): if tx['transaction']['operation'] == 'CREATE': tx = util.transform_create(tx) - tx = util.sign_tx(tx, bigchain.me_private) + tx = bigchain.consensus.sign_transaction( + tx, private_key=bigchain.me_private) - if not util.verify_signature(tx): + if not bigchain.consensus.verify_signature(tx): val['error'] = 'Invalid transaction signature' val = bigchain.write_transaction(tx) diff --git a/docs/source/consensus.md b/docs/source/consensus.md new file mode 100644 index 00000000..552ce0d6 --- /dev/null +++ b/docs/source/consensus.md @@ -0,0 +1,82 @@ +# BigchainDB Consensus Plugins + +BigchainDB has a pluggable block/transaction validation architecture. The default consensus rules can be extended or replaced entirely. + + +## Installing a plugin + +Plugins can be installed via pip! + +```bash +$ pip install bigchaindb-plugin-demo +``` + +Or using setuptools: + +```bash +$ cd bigchaindb-plugin-demo/ +$ python setup.py install # (or develop) +``` + +To activate your plugin, you can either set the `consensus_plugin` field in your config file (usually `~/.bigchaindb`) or by setting the `BIGCHAIN_CONSENSUS_PLUGIN` environement variable to the name of your plugin (see the section on [Packaging a plugin](#packaging-a-plugin) for more about plugin names). + + +## Plugin API + +BigchainDB's [current plugin API](../../bigchaindb/consensus.py) exposes five functions in an `AbstractConsensusRules` class: + +```python +validate_transaction(bigchain, transaction) +validate_block(bigchain, block) +create_transaction(*args, **kwargs) +sign_transaction(transaction, *args, **kwargs) +verify_signature(transaction) +``` + +Together, these functions are sufficient for most customizations. For example: +- Replace the crypto-system with one your hardware can accelerate +- Re-implement an existing protocol +- Delegate validation to another application +- etc... + + +## Extending BigchainDB behavior + +A default installation of BigchainDB will use the rules in the `BaseConsensusRules` class. If you only need to modify this behavior slightly, you can inherit from that class and call `super()` in any methods you change, so long as the return values remain the same. + +Here's a quick example of a plugin that adds nonsense rules: + +```python +from bigchaindb.consensus import BaseConsensusRules + +class SillyConsensusRules(BaseConsensusRules): + + @staticmethod + def validate_transaction(bigchain, transaction): + transaction = super().validate_transaction(bigchain, transaction) + # I only like transactions whose timestamps are even. + if transaction['transaction']['timestamp'] % 2 != 0: + raise StandardError("Odd... very odd indeed.") + return transaction + + @staticmethod + def validate_block(bigchain, block): + block = super().validate_block(bigchain, block) + # I don't trust Alice, I think she's shady. + if block['block']['node_pubkey'] == '': + raise StandardError("Alice is shady, everybody ignore her blocks!") + return block +``` + + +## Packaging a plugin + +BigchainDB uses [setuptool's entry_point](https://pythonhosted.org/setuptools/setuptools.html) system to provide the plugin functionality. Any custom plugin needs to add this section to the `setup()` call in their `setup.py`: + +```python +entry_points={ + 'bigchaindb.consensus': [ + 'PLUGIN_NAME=package.module:ConsensusRulesClass' + ] +}, +``` diff --git a/docs/source/index.rst b/docs/source/index.rst index 44523c7d..fd2c991e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -14,6 +14,7 @@ Table of Contents introduction installing-server + running-unit-tests python-server-api-examples bigchaindb-cli http-client-server-api diff --git a/docs/source/running-unit-tests.md b/docs/source/running-unit-tests.md new file mode 100644 index 00000000..9e53dc77 --- /dev/null +++ b/docs/source/running-unit-tests.md @@ -0,0 +1,41 @@ +# Running Unit Tests + +Once you've installed BigchainDB Server, you may want to run all the unit tests. This section explains how. + +First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. Before you can run all the unit tests, you must [install BigchainDB from source](http://bigchaindb.readthedocs.org/en/develop/installing-server.html#how-to-install-bigchaindb-from-source). + +To run all the unit tests, first make sure you have RethinkDB running: +```text +$ rethinkdb +``` + +then in another terminal, do: +```text +$ py.test -v +``` + +If the above command doesn't work (e.g. maybe you are running in a conda virtual environment), try: +```text +$ python -m pytest -v +``` + +(We write our unit tests using the [pytest](http://pytest.org/latest/) framework.) + +You can also run all unit tests via `setup.py`, using: +```text +$ python setup.py test +``` + +### Using `docker-compose` to Run the Tests + +You can also use `docker-compose` to run the unit tests. (You don't have to start RethinkDB first: `docker-compose` does that on its own, when it reads the `docker-compose.yml` file.) + +First, build the images (~once), using: +```text +$ docker-compose build +``` + +then run the unit tests using: +```text +$ docker-compose run --rm bigchaindb py.test -v +``` diff --git a/setup.py b/setup.py index 63ce9428..6156b417 100644 --- a/setup.py +++ b/setup.py @@ -63,6 +63,9 @@ setup( 'bigchaindb=bigchaindb.commands.bigchain:main', 'bigchaindb-benchmark=bigchaindb.commands.bigchain_benchmark:main' ], + 'bigchaindb.consensus': [ + 'default=bigchaindb.consensus:BaseConsensusRules' + ] }, install_requires=[ 'rethinkdb==2.2.0.post4', diff --git a/tests/test_core.py b/tests/test_core.py index 9857b37f..f136f4e3 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -15,6 +15,7 @@ def config(request, monkeypatch): }, 'keyring': [], 'CONFIGURED': True, + 'consensus_plugin': 'default' } monkeypatch.setattr('bigchaindb.config', config) @@ -24,6 +25,7 @@ def config(request, monkeypatch): def test_bigchain_class_default_initialization(config): from bigchaindb.core import Bigchain + from bigchaindb.consensus import BaseConsensusRules bigchain = Bigchain() assert bigchain.host == config['database']['host'] assert bigchain.port == config['database']['port'] @@ -31,11 +33,13 @@ def test_bigchain_class_default_initialization(config): assert bigchain.me == config['keypair']['public'] assert bigchain.me_private == config['keypair']['private'] assert bigchain.federation_nodes == config['keyring'] + assert bigchain.consensus == BaseConsensusRules assert bigchain._conn is None def test_bigchain_class_initialization_with_parameters(config): from bigchaindb.core import Bigchain + from bigchaindb.consensus import BaseConsensusRules init_kwargs = { 'host': 'some_node', 'port': '12345', @@ -43,6 +47,7 @@ def test_bigchain_class_initialization_with_parameters(config): 'public_key': 'white', 'private_key': 'black', 'keyring': ['key_one', 'key_two'], + 'consensus_plugin': 'default' } bigchain = Bigchain(**init_kwargs) assert bigchain.host == init_kwargs['host'] @@ -51,4 +56,5 @@ def test_bigchain_class_initialization_with_parameters(config): assert bigchain.me == init_kwargs['public_key'] assert bigchain.me_private == init_kwargs['private_key'] assert bigchain.federation_nodes == init_kwargs['keyring'] + assert bigchain.consensus == BaseConsensusRules assert bigchain._conn is None diff --git a/tests/utils/test_config_utils.py b/tests/utils/test_config_utils.py index 2cf57242..fa63158d 100644 --- a/tests/utils/test_config_utils.py +++ b/tests/utils/test_config_utils.py @@ -37,3 +37,32 @@ def test_bigchain_instance_raises_when_not_configured(monkeypatch): with pytest.raises(exceptions.KeypairNotFoundException): bigchaindb.Bigchain() + + +def test_load_consensus_plugin_loads_default_rules_without_name(): + from bigchaindb import config_utils + from bigchaindb.consensus import BaseConsensusRules + + assert config_utils.load_consensus_plugin() == BaseConsensusRules + + +def test_load_consensus_plugin_raises_with_unknown_name(): + from pkg_resources import ResolutionError + from bigchaindb import config_utils + + with pytest.raises(ResolutionError): + config_utils.load_consensus_plugin('bogus') + + +def test_load_consensus_plugin_raises_with_invalid_subclass(monkeypatch): + # Monkeypatch entry_point.load to return something other than a + # ConsensusRules instance + from bigchaindb import config_utils + monkeypatch.setattr(config_utils, + 'iter_entry_points', + lambda *args: [ type('entry_point', + (object), + {'load': lambda: object}) ]) + + with pytest.raises(TypeError): + config_utils.load_consensus_plugin()