From cf1f253019dd11452e1897e7a221ceace20c7b88 Mon Sep 17 00:00:00 2001 From: Muawia Khan Date: Fri, 31 Aug 2018 16:56:34 +0200 Subject: [PATCH 01/77] Update bigchaindb-driver version -> 0.5.2 in acceptance test (#2503) --- acceptance/python/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/python/Dockerfile b/acceptance/python/Dockerfile index a473a56a..af29629a 100644 --- a/acceptance/python/Dockerfile +++ b/acceptance/python/Dockerfile @@ -5,5 +5,5 @@ RUN pip install --upgrade \ pycco \ websocket-client~=0.47.0 \ pytest~=3.0 \ - bigchaindb-driver==0.5.1 \ + bigchaindb-driver==0.5.2 \ blns From 905b1a514161343a642dae1c10af7867b90d1939 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 31 Aug 2018 20:43:08 +0200 Subject: [PATCH 02/77] Problem: Root docs page about decentralization has two incorrect sentences (#2505) Solution: Delete them --- docs/root/source/decentralized.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/root/source/decentralized.md b/docs/root/source/decentralized.md index fc88cf99..a5258143 100644 --- a/docs/root/source/decentralized.md +++ b/docs/root/source/decentralized.md @@ -12,10 +12,6 @@ Ideally, each node in a BigchainDB network is owned and controlled by a differen We use the phrase "BigchainDB consortium" (or just "consortium") to refer to the set of people and/or organizations who run the nodes of a BigchainDB network. A consortium requires some form of governance to make decisions such as membership and policies. The exact details of the governance process are determined by each consortium, but it can be very decentralized. -If sharding is turned on (i.e. if the number of shards is larger than one), then the actual data is decentralized in that no one node stores all the data. - -Every node has its own locally-stored list of the public keys of other consortium members: the so-called keyring. There's no centrally-stored or centrally-shared keyring. - A consortium can increase its decentralization (and its resilience) by increasing its jurisdictional diversity, geographic diversity, and other kinds of diversity. This idea is expanded upon in [the section on node diversity](diversity.html). There’s no node that has a long-term special position in the BigchainDB network. All nodes run the same software and perform the same duties. From 80b688179734d02f1b846cb3d669f53397ec8eed Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Sun, 2 Sep 2018 00:23:17 +0200 Subject: [PATCH 03/77] Problem: Docs page 'BigchainDB and Smart Contracts' is stale (#2504) Solution: Update that page to reflect our current thinking on how BigchainDB relates to smart contracts --- docs/root/source/permissions.rst | 2 ++ docs/root/source/smart-contracts.rst | 14 +++----------- 2 files changed, 5 insertions(+), 11 deletions(-) diff --git a/docs/root/source/permissions.rst b/docs/root/source/permissions.rst index 009929fb..33871016 100644 --- a/docs/root/source/permissions.rst +++ b/docs/root/source/permissions.rst @@ -3,6 +3,8 @@ SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) Code is Apache-2.0 and docs are CC-BY-4.0 +.. _permissions-in-bigchaindb: + Permissions in BigchainDB ------------------------- diff --git a/docs/root/source/smart-contracts.rst b/docs/root/source/smart-contracts.rst index a9cd59f7..86a6a033 100644 --- a/docs/root/source/smart-contracts.rst +++ b/docs/root/source/smart-contracts.rst @@ -8,16 +8,8 @@ BigchainDB and Smart Contracts One can store the source code of any smart contract (i.e. a computer program) in BigchainDB, but BigchainDB won't run arbitrary smart contracts. -BigchainDB will run the subset of smart contracts expressible using `Crypto-Conditions `_. +BigchainDB can be used to enforce who has permission to transfer assets, both fungible assets and non-fungible assets. It will prevent double-spending. In other words, a BigchainDB network could be used instead of an ERC-20 (fungible token) or ERC-721 (non-fungible token) smart contract. -The owners of an asset can impose conditions on it that must be met for the asset to be transferred to new owners. Examples of possible conditions (crypto-conditions) include: +Asset transfer permissions can also be interpreted as write permissions, so they can be used to control who can write to a log, journal or audit trail. There is more about that idea in :ref:`the page about permissions in BigchainDB `. -- The current owner must sign the transfer transaction (one which transfers ownership to new owners). -- Three out of five current owners must sign the transfer transaction. -- (Shannon and Kelly) or Morgan must sign the transfer transaction. - -Crypto-conditions can be quite complex. They can't include loops or recursion and therefore will always run/check in finite time. - -.. note:: - - We used the word "owners" somewhat loosely above. A more accurate word might be fulfillers, signers, controllers, or transfer-enablers. See the section titled **A Note about Owners** in the relevant `BigchainDB Transactions Spec `_. \ No newline at end of file +A BigchainDB network can be connected to other blockchain networks, via oracles or inter-chain communications protocols. That means BigchainDB can be used as part of a solution that uses *other* blockchains to run arbitrary smart contracts. From 8e55b11da2bec4b576e85bd9ee56c30b04812c28 Mon Sep 17 00:00:00 2001 From: Muawia Khan Date: Mon, 3 Sep 2018 13:25:58 +0200 Subject: [PATCH 04/77] Problem: cyrptoconditions dependency updated because of vulnerability (#2494) * Problem: cyrptoconditions dependency updated because of vulnerability CVE-2018-10903 * update cc to ~=0.7.2 * Fix test using b58encode * Fixing some more tests failing because of base58 update --- bigchaindb/common/transaction.py | 4 ++-- bigchaindb/upsert_validator/validator_election.py | 2 +- setup.py | 2 +- tests/common/test_transaction.py | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/bigchaindb/common/transaction.py b/bigchaindb/common/transaction.py index d5b3eef8..8baf8196 100644 --- a/bigchaindb/common/transaction.py +++ b/bigchaindb/common/transaction.py @@ -94,7 +94,7 @@ class Input(object): """ try: fulfillment = self.fulfillment.serialize_uri() - except (TypeError, AttributeError, ASN1EncodeError): + except (TypeError, AttributeError, ASN1EncodeError, ASN1DecodeError): fulfillment = _fulfillment_to_details(self.fulfillment) try: @@ -161,7 +161,7 @@ def _fulfillment_to_details(fulfillment): if fulfillment.type_name == 'ed25519-sha-256': return { 'type': 'ed25519-sha-256', - 'public_key': base58.b58encode(fulfillment.public_key), + 'public_key': base58.b58encode(fulfillment.public_key).decode(), } if fulfillment.type_name == 'threshold-sha-256': diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index 7d9a6fbc..95162442 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -172,7 +172,7 @@ class ValidatorElection(Transaction): @classmethod def to_public_key(cls, election_id): - return base58.b58encode(bytes.fromhex(election_id)) + return base58.b58encode(bytes.fromhex(election_id)).decode() @classmethod def count_votes(cls, election_pk, transactions, getter=getattr): diff --git a/setup.py b/setup.py index dcecca51..071ddd45 100644 --- a/setup.py +++ b/setup.py @@ -79,7 +79,7 @@ install_requires = [ # TODO Consider not installing the db drivers, or putting them in extras. 'pymongo~=3.6', 'pysha3~=1.0.2', - 'cryptoconditions~=0.6.0.dev', + 'cryptoconditions~=0.7.2', 'python-rapidjson~=0.6.0', 'logstats~=0.2.1', 'flask>=0.10.1', diff --git a/tests/common/test_transaction.py b/tests/common/test_transaction.py index 7b8dde99..4a1586ca 100644 --- a/tests/common/test_transaction.py +++ b/tests/common/test_transaction.py @@ -93,7 +93,7 @@ def test_output_serialization(user_Ed25519, user_pub): 'uri': user_Ed25519.condition_uri, 'details': { 'type': 'ed25519-sha-256', - 'public_key': b58encode(user_Ed25519.public_key), + 'public_key': b58encode(user_Ed25519.public_key).decode(), }, }, 'public_keys': [user_pub], @@ -114,7 +114,7 @@ def test_output_deserialization(user_Ed25519, user_pub): 'uri': user_Ed25519.condition_uri, 'details': { 'type': 'ed25519-sha-256', - 'public_key': b58encode(user_Ed25519.public_key), + 'public_key': b58encode(user_Ed25519.public_key).decode(), }, }, 'public_keys': [user_pub], From fe0a4c494bee3170ec54eb4deab45bbe07c08a9e Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Mon, 3 Sep 2018 15:09:16 +0200 Subject: [PATCH 05/77] Problem: No MongoDB support for election ID index. (#2507) Solution: Election ID has to be unique but not every validator set record has it. MongoDB does not support partial indexes, does not even allow for multiple Nones. This is a temporary fix since we are introducing an `election` collection to store election IDs in #2498. --- bigchaindb/backend/localmongodb/schema.py | 3 --- tests/backend/localmongodb/test_schema.py | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/bigchaindb/backend/localmongodb/schema.py b/bigchaindb/backend/localmongodb/schema.py index 1ad88928..0607c883 100644 --- a/bigchaindb/backend/localmongodb/schema.py +++ b/bigchaindb/backend/localmongodb/schema.py @@ -133,6 +133,3 @@ def create_validators_secondary_index(conn, dbname): conn.conn[dbname]['validators'].create_index('height', name='height', unique=True,) - conn.conn[dbname]['validators'].create_index('election_id', - name='election_id', - unique=True,) diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index 2481cd60..136715ab 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -44,7 +44,7 @@ def test_init_creates_db_tables_and_indexes(): assert set(indexes) == {'_id_', 'pre_commit_id'} indexes = conn.conn[dbname]['validators'].index_information().keys() - assert set(indexes) == {'_id_', 'height', 'election_id'} + assert set(indexes) == {'_id_', 'height'} def test_init_database_fails_if_db_exists(): From 230a5b2d692d62b5a3c64588a9e6517abad623a8 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Mon, 3 Sep 2018 15:41:49 +0200 Subject: [PATCH 06/77] ABCI chain migration conclusion (#2488) * Problem: No good way to check for val set absence. Solution: Make get_validator_set/get_validators return None/[] when there are no validators yet. * Problem: Incompatible ABCI chain upgrades. Solution: Record known chains and sync through InitChain. Triggering the migration and adjusting other ABCI endpoints will follow. --- bigchaindb/backend/localmongodb/query.py | 22 +- bigchaindb/backend/localmongodb/schema.py | 14 ++ bigchaindb/backend/query.py | 20 ++ bigchaindb/backend/schema.py | 2 +- bigchaindb/core.py | 83 +++++++- bigchaindb/lib.py | 34 ++- tests/backend/localmongodb/test_queries.py | 51 +++++ tests/backend/localmongodb/test_schema.py | 8 +- tests/tendermint/conftest.py | 3 +- tests/tendermint/test_core.py | 234 ++++++++++++++++++++- tests/tendermint/test_lib.py | 33 +++ tests/tendermint/test_utils.py | 1 + tests/utils.py | 10 +- 13 files changed, 492 insertions(+), 23 deletions(-) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index 29993b30..6587d494 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -296,7 +296,7 @@ def get_validator_set(conn, height=None): .limit(1) ) - return list(cursor)[0] + return next(cursor, None) @register_query(LocalMongoDBConnection) @@ -322,3 +322,23 @@ def get_asset_tokens_for_public_key(conn, asset_id, public_key): {'$project': {'_id': False}} ])) return cursor + + +@register_query(LocalMongoDBConnection) +def store_abci_chain(conn, height, chain_id, is_synced=True): + return conn.run( + conn.collection('abci_chains').replace_one( + {'height': height}, + {'height': height, 'chain_id': chain_id, + 'is_synced': is_synced}, + upsert=True, + ) + ) + + +@register_query(LocalMongoDBConnection) +def get_latest_abci_chain(conn): + return conn.run( + conn.collection('abci_chains') + .find_one(projection={'_id': False}, sort=[('height', DESCENDING)]) + ) diff --git a/bigchaindb/backend/localmongodb/schema.py b/bigchaindb/backend/localmongodb/schema.py index 0607c883..e58f1def 100644 --- a/bigchaindb/backend/localmongodb/schema.py +++ b/bigchaindb/backend/localmongodb/schema.py @@ -47,6 +47,7 @@ def create_indexes(conn, dbname): create_utxos_secondary_index(conn, dbname) create_pre_commit_secondary_index(conn, dbname) create_validators_secondary_index(conn, dbname) + create_abci_chains_indexes(conn, dbname) @register_schema(LocalMongoDBConnection) @@ -133,3 +134,16 @@ def create_validators_secondary_index(conn, dbname): conn.conn[dbname]['validators'].create_index('height', name='height', unique=True,) + + +def create_abci_chains_indexes(conn, dbname): + logger.info('Create `abci_chains.height` secondary index.') + + conn.conn[dbname]['abci_chains'].create_index('height', + name='height', + unique=True,) + + logger.info('Create `abci_chains.chain_id` secondary index.') + conn.conn[dbname]['abci_chains'].create_index('chain_id', + name='chain_id', + unique=True) diff --git a/bigchaindb/backend/query.py b/bigchaindb/backend/query.py index 4d62c633..7965d5ee 100644 --- a/bigchaindb/backend/query.py +++ b/bigchaindb/backend/query.py @@ -380,3 +380,23 @@ def get_asset_tokens_for_public_key(connection, asset_id, Iterator of transaction that list given owner in conditions. """ raise NotImplementedError + + +@singledispatch +def store_abci_chain(conn, height, chain_id, is_synced=True): + """Create or update an ABCI chain at the given height. + Usually invoked in the beginning of the ABCI communications (height=0) + or when ABCI client (like Tendermint) is migrated (any height). + + Args: + is_synced: True if the chain is known by both ABCI client and server + """ + raise NotImplementedError + + +@singledispatch +def get_latest_abci_chain(conn): + """Returns the ABCI chain stored at the biggest height, if any, + None otherwise. + """ + raise NotImplementedError diff --git a/bigchaindb/backend/schema.py b/bigchaindb/backend/schema.py index 04c1bdd0..108bd39b 100644 --- a/bigchaindb/backend/schema.py +++ b/bigchaindb/backend/schema.py @@ -16,7 +16,7 @@ logger = logging.getLogger(__name__) # Tables/collections that every backend database must create TABLES = ('transactions', 'blocks', 'assets', 'metadata', - 'validators', 'pre_commit', 'utxos') + 'validators', 'pre_commit', 'utxos', 'abci_chains') VALID_LANGUAGES = ('danish', 'dutch', 'english', 'finnish', 'french', 'german', 'hungarian', 'italian', 'norwegian', 'portuguese', 'romanian', diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 621960fa..a0983472 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -6,6 +6,7 @@ with Tendermint. """ import logging +import sys from abci.application import BaseApplication from abci.types_pb2 import ( @@ -47,22 +48,76 @@ class App(BaseApplication): self.block_transactions = [] self.validators = None self.new_height = None + self.chain = self.bigchaindb.get_latest_abci_chain() + + def log_abci_migration_error(self, chain_id, validators): + logger.error(f'An ABCI chain migration is in process. ' + + 'Download the new ABCI client and configure it with ' + + 'chain_id={chain_id} and validators={validators}.') + + def abort_if_abci_chain_is_not_synced(self): + if self.chain is None or self.chain['is_synced']: + return + + validators = self.bigchaindb.get_validators() + self.log_abci_migration_error(self.chain['chain_id'], validators) + sys.exit(1) def init_chain(self, genesis): - """Initialize chain with block of height 0""" + """Initialize chain upon genesis or a migration""" - validator_set = [vutils.decode_validator(v) for v in genesis.validators] - block = Block(app_hash='', height=0, transactions=[]) + app_hash = '' + height = 0 + + known_chain = self.bigchaindb.get_latest_abci_chain() + if known_chain is not None: + chain_id = known_chain['chain_id'] + + if known_chain['is_synced']: + msg = f'Got invalid InitChain ABCI request ({genesis}) - ' + \ + 'the chain {chain_id} is already synced.' + logger.error(msg) + sys.exit(1) + + if chain_id != genesis.chain_id: + validators = self.bigchaindb.get_validators() + self.log_abci_migration_error(chain_id, validators) + sys.exit(1) + + # set migration values for app hash and height + block = self.bigchaindb.get_latest_block() + app_hash = '' if block is None else block['app_hash'] + height = 0 if block is None else block['height'] + 1 + + known_validators = self.bigchaindb.get_validators() + validator_set = [vutils.decode_validator(v) + for v in genesis.validators] + + if known_validators and known_validators != validator_set: + self.log_abci_migration_error(known_chain['chain_id'], + known_validators) + sys.exit(1) + + block = Block(app_hash=app_hash, height=height, transactions=[]) self.bigchaindb.store_block(block._asdict()) - self.bigchaindb.store_validator_set(1, validator_set, None) + self.bigchaindb.store_validator_set(height + 1, validator_set, None) + abci_chain_height = 0 if known_chain is None else known_chain['height'] + self.bigchaindb.store_abci_chain(abci_chain_height, + genesis.chain_id, True) + self.chain = {'height': abci_chain_height, 'is_synced': True, + 'chain_id': genesis.chain_id} return ResponseInitChain() def info(self, request): """Return height of the latest committed block.""" + + self.abort_if_abci_chain_is_not_synced() + r = ResponseInfo() block = self.bigchaindb.get_latest_block() if block: - r.last_block_height = block['height'] + chain_shift = 0 if self.chain is None else self.chain['height'] + r.last_block_height = block['height'] - chain_shift r.last_block_app_hash = block['app_hash'].encode('utf-8') else: r.last_block_height = 0 @@ -77,6 +132,8 @@ class App(BaseApplication): raw_tx: a raw string (in bytes) transaction. """ + self.abort_if_abci_chain_is_not_synced() + logger.benchmark('CHECK_TX_INIT') logger.debug('check_tx: %s', raw_transaction) transaction = decode_transaction(raw_transaction) @@ -95,8 +152,11 @@ class App(BaseApplication): req_begin_block: block object which contains block header and block hash. """ + self.abort_if_abci_chain_is_not_synced() + + chain_shift = 0 if self.chain is None else self.chain['height'] logger.benchmark('BEGIN BLOCK, height:%s, num_txs:%s', - req_begin_block.header.height, + req_begin_block.header.height + chain_shift, req_begin_block.header.num_txs) self.block_txn_ids = [] @@ -109,6 +169,9 @@ class App(BaseApplication): Args: raw_tx: a raw string (in bytes) transaction. """ + + self.abort_if_abci_chain_is_not_synced() + logger.debug('deliver_tx: %s', raw_transaction) transaction = self.bigchaindb.is_valid_transaction( decode_transaction(raw_transaction), self.block_transactions) @@ -130,7 +193,11 @@ class App(BaseApplication): height (int): new height of the chain. """ - height = request_end_block.height + self.abort_if_abci_chain_is_not_synced() + + chain_shift = 0 if self.chain is None else self.chain['height'] + + height = request_end_block.height + chain_shift self.new_height = height block_txn_hash = calculate_hash(self.block_txn_ids) block = self.bigchaindb.get_latest_block() @@ -158,6 +225,8 @@ class App(BaseApplication): def commit(self): """Store the new height and along with block hash.""" + self.abort_if_abci_chain_is_not_synced() + data = self.block_txn_hash.encode('utf-8') # register a new block only when new transactions are received diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index 05ca3e69..fa72a506 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -426,8 +426,7 @@ class BigchainDB(object): def get_validators(self, height=None): result = self.get_validator_change(height) - validators = result['validators'] - return validators + return [] if result is None else result['validators'] def get_validators_by_election_id(self, election_id): result = backend.query.get_validator_set_by_election_id(self.connection, election_id) @@ -448,6 +447,37 @@ class BigchainDB(object): 'validators': validators, 'election_id': election_id}) + def store_abci_chain(self, height, chain_id, is_synced=True): + return backend.query.store_abci_chain(self.connection, height, + chain_id, is_synced) + + def get_latest_abci_chain(self): + return backend.query.get_latest_abci_chain(self.connection) + + def migrate_abci_chain(self): + """Generate and record a new ABCI chain ID. New blocks are not + accepted until we receive an InitChain ABCI request with + the matching chain ID and validator set. + + Chain ID is generated based on the current chain and height. + `chain-X` => `chain-X-migrated-at-height-5`. + `chain-X-migrated-at-height-5` => `chain-X-migrated-at-height-21`. + + If there is no known chain (we are at genesis), the function returns. + """ + latest_chain = self.get_latest_abci_chain() + if latest_chain is None: + return + + block = self.get_latest_block() + + suffix = '-migrated-at-height-' + chain_id = latest_chain['chain_id'] + block_height_str = str(block['height']) + new_chain_id = chain_id.split(suffix)[0] + suffix + block_height_str + + self.store_abci_chain(block['height'] + 1, new_chain_id, False) + Block = namedtuple('Block', ('app_hash', 'height', 'transactions')) diff --git a/tests/backend/localmongodb/test_queries.py b/tests/backend/localmongodb/test_queries.py index 77bfddd0..9c38d822 100644 --- a/tests/backend/localmongodb/test_queries.py +++ b/tests/backend/localmongodb/test_queries.py @@ -7,6 +7,9 @@ from copy import deepcopy import pytest import pymongo +from bigchaindb.backend import connect, query + + pytestmark = [pytest.mark.tendermint, pytest.mark.bdb] @@ -394,3 +397,51 @@ def test_validator_update(): v91 = query.get_validator_set(conn) assert v91['height'] == 91 + + +@pytest.mark.parametrize('description,stores,expected', [ + ( + 'Query empty database.', + [], + None, + ), + ( + 'Store one chain with the default value for `is_synced`.', + [ + {'height': 0, 'chain_id': 'some-id'}, + ], + {'height': 0, 'chain_id': 'some-id', 'is_synced': True}, + ), + ( + 'Store one chain with a custom value for `is_synced`.', + [ + {'height': 0, 'chain_id': 'some-id', 'is_synced': False}, + ], + {'height': 0, 'chain_id': 'some-id', 'is_synced': False}, + ), + ( + 'Store one chain, then update it.', + [ + {'height': 0, 'chain_id': 'some-id', 'is_synced': True}, + {'height': 0, 'chain_id': 'new-id', 'is_synced': False}, + ], + {'height': 0, 'chain_id': 'new-id', 'is_synced': False}, + ), + ( + 'Store a chain, update it, store another chain.', + [ + {'height': 0, 'chain_id': 'some-id', 'is_synced': True}, + {'height': 0, 'chain_id': 'some-id', 'is_synced': False}, + {'height': 10, 'chain_id': 'another-id', 'is_synced': True}, + ], + {'height': 10, 'chain_id': 'another-id', 'is_synced': True}, + ), +]) +def test_store_abci_chain(description, stores, expected): + conn = connect() + + for store in stores: + query.store_abci_chain(conn, **store) + + actual = query.get_latest_abci_chain(conn) + assert expected == actual, description diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index 136715ab..aeadbe05 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -24,7 +24,7 @@ def test_init_creates_db_tables_and_indexes(): collection_names = conn.conn[dbname].collection_names() assert set(collection_names) == { 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'pre_commit', - 'validators' + 'validators', 'abci_chains', } indexes = conn.conn[dbname]['assets'].index_information().keys() @@ -46,6 +46,9 @@ def test_init_creates_db_tables_and_indexes(): indexes = conn.conn[dbname]['validators'].index_information().keys() assert set(indexes) == {'_id_', 'height'} + indexes = conn.conn[dbname]['abci_chains'].index_information().keys() + assert set(indexes) == {'_id_', 'height', 'chain_id'} + def test_init_database_fails_if_db_exists(): import bigchaindb @@ -79,7 +82,8 @@ def test_create_tables(): collection_names = conn.conn[dbname].collection_names() assert set(collection_names) == { 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'validators', - 'pre_commit'} + 'pre_commit', 'abci_chains', + } def test_create_secondary_indexes(): diff --git a/tests/tendermint/conftest.py b/tests/tendermint/conftest.py index 8efa8e76..fccd2ade 100644 --- a/tests/tendermint/conftest.py +++ b/tests/tendermint/conftest.py @@ -16,7 +16,8 @@ def validator_pub_key(): @pytest.fixture def init_chain_request(): addr = codecs.decode(b'9FD479C869C7D7E7605BF99293457AA5D80C3033', 'hex') - pk = codecs.decode(b'VAgFZtYw8bNR5TMZHFOBDWk9cAmEu3/c6JgRBmddbbI=', 'base64') + pk = codecs.decode(b'VAgFZtYw8bNR5TMZHFOBDWk9cAmEu3/c6JgRBmddbbI=', + 'base64') val_a = types.Validator(address=addr, power=10, pub_key=types.PubKey(type='ed25519', data=pk)) diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index 40958aa2..64ee2887 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -2,17 +2,28 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 +import codecs import json import pytest +import random from abci.types_pb2 import ( + PubKey, + ResponseInitChain, + RequestInitChain, + RequestInfo, RequestBeginBlock, - RequestEndBlock + RequestEndBlock, + Validator, ) +from bigchaindb import App +from bigchaindb.backend.localmongodb import query +from bigchaindb.common.crypto import generate_key_pair from bigchaindb.core import (CodeTypeOk, CodeTypeError, ) +from bigchaindb.lib import Block from bigchaindb.upsert_validator.validator_utils import new_validator_set from bigchaindb.tendermint_utils import public_key_to_base64 @@ -24,6 +35,173 @@ def encode_tx_to_bytes(transaction): return json.dumps(transaction.to_dict()).encode('utf8') +def generate_address(): + return ''.join(random.choices('1,2,3,4,5,6,7,8,9,A,B,C,D,E,F'.split(','), + k=40)).encode() + + +def generate_validator(): + addr = codecs.decode(generate_address(), 'hex') + pk, _ = generate_key_pair() + pub_key = PubKey(type='ed25519', data=pk.encode()) + val = Validator(address=addr, power=10, pub_key=pub_key) + return val + + +def generate_init_chain_request(chain_id, vals=None): + vals = vals if vals is not None else [generate_validator()] + return RequestInitChain(validators=vals, chain_id=chain_id) + + +def test_init_chain_successfully_registers_chain(b): + request = generate_init_chain_request('chain-XYZ') + res = App(b).init_chain(request) + assert res == ResponseInitChain() + chain = query.get_latest_abci_chain(b.connection) + assert chain == {'height': 0, 'chain_id': 'chain-XYZ', 'is_synced': True} + assert query.get_latest_block(b.connection) == { + 'height': 0, + 'app_hash': '', + 'transactions': [], + } + + +def test_init_chain_ignores_invalid_init_chain_requests(b): + validators = [generate_validator()] + request = generate_init_chain_request('chain-XYZ', validators) + res = App(b).init_chain(request) + assert res == ResponseInitChain() + + validator_set = query.get_validator_set(b.connection) + + invalid_requests = [ + request, # the same request again + # different validator set + generate_init_chain_request('chain-XYZ'), + # different chain ID + generate_init_chain_request('chain-ABC', validators), + ] + for r in invalid_requests: + with pytest.raises(SystemExit): + App(b).init_chain(r) + # assert nothing changed - neither validator set, nor chain ID + new_validator_set = query.get_validator_set(b.connection) + assert new_validator_set == validator_set + new_chain_id = query.get_latest_abci_chain(b.connection)['chain_id'] + assert new_chain_id == 'chain-XYZ' + assert query.get_latest_block(b.connection) == { + 'height': 0, + 'app_hash': '', + 'transactions': [], + } + + +def test_init_chain_recognizes_new_chain_after_migration(b): + validators = [generate_validator()] + request = generate_init_chain_request('chain-XYZ', validators) + res = App(b).init_chain(request) + assert res == ResponseInitChain() + + validator_set = query.get_validator_set(b.connection)['validators'] + + # simulate a migration + query.store_block(b.connection, Block(app_hash='', height=1, + transactions=[])._asdict()) + b.migrate_abci_chain() + + # the same or other mismatching requests are ignored + invalid_requests = [ + request, + generate_init_chain_request('unknown', validators), + generate_init_chain_request('chain-XYZ'), + generate_init_chain_request('chain-XYZ-migrated-at-height-1'), + ] + for r in invalid_requests: + with pytest.raises(SystemExit): + App(b).init_chain(r) + assert query.get_latest_abci_chain(b.connection) == { + 'chain_id': 'chain-XYZ-migrated-at-height-1', + 'is_synced': False, + 'height': 2, + } + new_validator_set = query.get_validator_set(b.connection)['validators'] + assert new_validator_set == validator_set + + # a request with the matching chain ID and matching validator set + # completes the migration + request = generate_init_chain_request('chain-XYZ-migrated-at-height-1', + validators) + res = App(b).init_chain(request) + assert res == ResponseInitChain() + assert query.get_latest_abci_chain(b.connection) == { + 'chain_id': 'chain-XYZ-migrated-at-height-1', + 'is_synced': True, + 'height': 2, + } + assert query.get_latest_block(b.connection) == { + 'height': 2, + 'app_hash': '', + 'transactions': [], + } + + # requests with old chain ID and other requests are ignored + invalid_requests = [ + request, + generate_init_chain_request('chain-XYZ', validators), + generate_init_chain_request('chain-XYZ-migrated-at-height-1'), + ] + for r in invalid_requests: + with pytest.raises(SystemExit): + App(b).init_chain(r) + assert query.get_latest_abci_chain(b.connection) == { + 'chain_id': 'chain-XYZ-migrated-at-height-1', + 'is_synced': True, + 'height': 2, + } + new_validator_set = query.get_validator_set(b.connection)['validators'] + assert new_validator_set == validator_set + assert query.get_latest_block(b.connection) == { + 'height': 2, + 'app_hash': '', + 'transactions': [], + } + + +def test_info(b): + r = RequestInfo() + app = App(b) + + res = app.info(r) + assert res.last_block_height == 0 + assert res.last_block_app_hash == b'' + + b.store_block(Block(app_hash='1', height=1, transactions=[])._asdict()) + res = app.info(r) + assert res.last_block_height == 1 + assert res.last_block_app_hash == b'1' + + # simulate a migration and assert the height is shifted + b.store_abci_chain(2, 'chain-XYZ') + app = App(b) + b.store_block(Block(app_hash='2', height=2, transactions=[])._asdict()) + res = app.info(r) + assert res.last_block_height == 0 + assert res.last_block_app_hash == b'2' + + b.store_block(Block(app_hash='3', height=3, transactions=[])._asdict()) + res = app.info(r) + assert res.last_block_height == 1 + assert res.last_block_app_hash == b'3' + + # it's always the latest migration that is taken into account + b.store_abci_chain(4, 'chain-XYZ-new') + app = App(b) + b.store_block(Block(app_hash='4', height=4, transactions=[])._asdict()) + res = app.info(r) + assert res.last_block_height == 0 + assert res.last_block_app_hash == b'4' + + def test_check_tx__signed_create_is_ok(b): from bigchaindb import App from bigchaindb.models import Transaction @@ -57,7 +235,6 @@ def test_check_tx__unsigned_create_is_error(b): assert result.code == CodeTypeError -@pytest.mark.bdb def test_deliver_tx__valid_create_updates_db(b, init_chain_request): from bigchaindb import App from bigchaindb.models import Transaction @@ -225,6 +402,17 @@ def test_store_pre_commit_state_in_end_block(b, alice, init_chain_request): assert resp['height'] == 100 assert resp['transactions'] == [tx.id] + # simulate a chain migration and assert the height is shifted + b.store_abci_chain(100, 'new-chain') + app = App(b) + app.begin_block(begin_block) + app.deliver_tx(encode_tx_to_bytes(tx)) + app.end_block(RequestEndBlock(height=1)) + resp = query.get_pre_commit_state(b.connection, PRE_COMMIT_ID) + assert resp['commit_id'] == PRE_COMMIT_ID + assert resp['height'] == 101 + assert resp['transactions'] == [tx.id] + def test_new_validator_set(b): node1 = {'pub_key': {'type': 'ed25519', @@ -247,3 +435,45 @@ def test_new_validator_set(b): 'voting_power': u['power']}) assert updated_validator_set == updated_validators + + +def test_info_aborts_if_chain_is_not_synced(b): + b.store_abci_chain(0, 'chain-XYZ', False) + + with pytest.raises(SystemExit): + App(b).info(RequestInfo()) + + +def test_check_tx_aborts_if_chain_is_not_synced(b): + b.store_abci_chain(0, 'chain-XYZ', False) + + with pytest.raises(SystemExit): + App(b).check_tx('some bytes') + + +def test_begin_aborts_if_chain_is_not_synced(b): + b.store_abci_chain(0, 'chain-XYZ', False) + + with pytest.raises(SystemExit): + App(b).info(RequestBeginBlock()) + + +def test_deliver_tx_aborts_if_chain_is_not_synced(b): + b.store_abci_chain(0, 'chain-XYZ', False) + + with pytest.raises(SystemExit): + App(b).deliver_tx('some bytes') + + +def test_end_block_aborts_if_chain_is_not_synced(b): + b.store_abci_chain(0, 'chain-XYZ', False) + + with pytest.raises(SystemExit): + App(b).info(RequestEndBlock()) + + +def test_commit_aborts_if_chain_is_not_synced(b): + b.store_abci_chain(0, 'chain-XYZ', False) + + with pytest.raises(SystemExit): + App(b).commit() diff --git a/tests/tendermint/test_lib.py b/tests/tendermint/test_lib.py index 4e8ff6b3..3ea91329 100644 --- a/tests/tendermint/test_lib.py +++ b/tests/tendermint/test_lib.py @@ -15,6 +15,7 @@ import pytest from pymongo import MongoClient from bigchaindb import backend +from bigchaindb.lib import Block pytestmark = pytest.mark.tendermint @@ -441,3 +442,35 @@ def test_validation_with_transaction_buffer(b): assert not b.is_valid_transaction(create_tx, [create_tx]) assert not b.is_valid_transaction(transfer_tx, [create_tx, transfer_tx]) assert not b.is_valid_transaction(double_spend, [create_tx, transfer_tx]) + + +@pytest.mark.bdb +def test_migrate_abci_chain_yields_on_genesis(b): + b.migrate_abci_chain() + latest_chain = b.get_latest_abci_chain() + assert latest_chain is None + + +@pytest.mark.bdb +@pytest.mark.parametrize('chain,block_height,expected', [ + ( + (1, 'chain-XYZ', True), + 4, + {'height': 5, 'chain_id': 'chain-XYZ-migrated-at-height-4', + 'is_synced': False}, + ), + ( + (5, 'chain-XYZ-migrated-at-height-4', True), + 13, + {'height': 14, 'chain_id': 'chain-XYZ-migrated-at-height-13', + 'is_synced': False}, + ), +]) +def test_migrate_abci_chain_generates_new_chains(b, chain, block_height, + expected): + b.store_abci_chain(*chain) + b.store_block(Block(app_hash='', height=block_height, + transactions=[])._asdict()) + b.migrate_abci_chain() + latest_chain = b.get_latest_abci_chain() + assert latest_chain == expected diff --git a/tests/tendermint/test_utils.py b/tests/tendermint/test_utils.py index ae34104f..46b36525 100644 --- a/tests/tendermint/test_utils.py +++ b/tests/tendermint/test_utils.py @@ -12,6 +12,7 @@ except ImportError: import pytest + pytestmark = pytest.mark.tendermint diff --git a/tests/utils.py b/tests/utils.py index 349f7e5a..87c07a79 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -5,6 +5,7 @@ from functools import singledispatch from bigchaindb.backend.localmongodb.connection import LocalMongoDBConnection +from bigchaindb.backend.schema import TABLES @singledispatch @@ -14,13 +15,8 @@ def flush_db(connection, dbname): @flush_db.register(LocalMongoDBConnection) def flush_localmongo_db(connection, dbname): - connection.conn[dbname].bigchain.delete_many({}) - connection.conn[dbname].blocks.delete_many({}) - connection.conn[dbname].transactions.delete_many({}) - connection.conn[dbname].assets.delete_many({}) - connection.conn[dbname].metadata.delete_many({}) - connection.conn[dbname].utxos.delete_many({}) - connection.conn[dbname].validators.delete_many({}) + for t in TABLES: + getattr(connection.conn[dbname], t).delete_many({}) def generate_block(bigchain): From f8bb29535ae4e1b875360e44ebaae27488e3c433 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Tue, 4 Sep 2018 12:17:27 +0200 Subject: [PATCH 07/77] Problem: Code Reference docs hidden in appendices (#2473) Solution: Move the Code Reference docs into their own section --- docs/server/source/appendices/index.rst | 3 --- .../backend.rst | 0 .../commands.rst | 0 docs/server/source/code-reference/index.rst | 24 +++++++++++++++++++ .../the-bigchaindb-class.rst | 0 docs/server/source/index.rst | 1 + 6 files changed, 25 insertions(+), 3 deletions(-) rename docs/server/source/{appendices => code-reference}/backend.rst (100%) rename docs/server/source/{appendices => code-reference}/commands.rst (100%) create mode 100644 docs/server/source/code-reference/index.rst rename docs/server/source/{appendices => code-reference}/the-bigchaindb-class.rst (100%) diff --git a/docs/server/source/appendices/index.rst b/docs/server/source/appendices/index.rst index ea02aed6..24276f2d 100755 --- a/docs/server/source/appendices/index.rst +++ b/docs/server/source/appendices/index.rst @@ -11,9 +11,6 @@ Appendices json-serialization cryptography - the-bigchaindb-class - backend - commands aws-setup generate-key-pair-for-ssh firewall-notes diff --git a/docs/server/source/appendices/backend.rst b/docs/server/source/code-reference/backend.rst similarity index 100% rename from docs/server/source/appendices/backend.rst rename to docs/server/source/code-reference/backend.rst diff --git a/docs/server/source/appendices/commands.rst b/docs/server/source/code-reference/commands.rst similarity index 100% rename from docs/server/source/appendices/commands.rst rename to docs/server/source/code-reference/commands.rst diff --git a/docs/server/source/code-reference/index.rst b/docs/server/source/code-reference/index.rst new file mode 100644 index 00000000..11d42b66 --- /dev/null +++ b/docs/server/source/code-reference/index.rst @@ -0,0 +1,24 @@ + +.. Copyright BigchainDB GmbH and BigchainDB contributors + SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) + Code is Apache-2.0 and docs are CC-BY-4.0 + +Code Reference +============== + +This section contains auto-generated documentation of various functions, classes and methods +in the BigchainDB Server code, based on Python docstrings in the code itself. + +.. warning:: + + While we try to keep docstrings accurate, + if you want to know *for sure* what the code does, + then you have to read the code itself. + +.. toctree:: + :maxdepth: 1 + + the-bigchaindb-class + backend + commands + \ No newline at end of file diff --git a/docs/server/source/appendices/the-bigchaindb-class.rst b/docs/server/source/code-reference/the-bigchaindb-class.rst similarity index 100% rename from docs/server/source/appendices/the-bigchaindb-class.rst rename to docs/server/source/code-reference/the-bigchaindb-class.rst diff --git a/docs/server/source/index.rst b/docs/server/source/index.rst index ced2a433..534cc6fd 100644 --- a/docs/server/source/index.rst +++ b/docs/server/source/index.rst @@ -23,4 +23,5 @@ BigchainDB Server Documentation data-models/index k8s-deployment-template/index release-notes + code-reference/index appendices/index From c72c7a46263607b8e15912be889e2b48f7fd8599 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Tue, 4 Sep 2018 13:31:00 +0200 Subject: [PATCH 08/77] Problem: tests/README.md either wrong or redundant (#2479) The information on that page was either wrong or redundant, so I either deleted it or linked to where to find the same information in the docs about contributing to BigchainDB. --- tests/README.md | 104 ++++-------------------------------------------- 1 file changed, 7 insertions(+), 97 deletions(-) diff --git a/tests/README.md b/tests/README.md index 6ae9be24..fe63b32d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -4,103 +4,13 @@ SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) Code is Apache-2.0 and docs are CC-BY-4.0 ---> -# BigchainDB Server Tests +# BigchainDB Server Unit Tests -## The tests/ Folder +Most of the tests in the `tests/` folder are unit tests. For info about how to write and run tests, see [the docs about contributing to BigchainDB](http://docs.bigchaindb.com/projects/contributing/en/latest/index.html), especially: -The `tests/` folder is where all the tests for BigchainDB Server live. Most of them are unit tests. Integration tests are in the [`tests/integration/` folder](./integration/). +- [Write Code - Remember to Write Tests](http://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/write-code.html#remember-to-write-tests) +- [Notes on Running a Local Dev Node with Docker Compose](http://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/run-node-with-docker-compose.html), especially `make test` +- [ +Notes on Running a Local Dev Node as Processes (and Running All Tests)](http://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/run-node-as-processes.html) -A few notes: - -- [`tests/common/`](./common/) contains self-contained tests only testing - [`bigchaindb/common/`](../bigchaindb/common/) -- [`tests/backend/`](./backend/) contains tests requiring - the database backend (MongoDB) - - -## Writing Tests - -We write unit and integration tests for our Python code using the [pytest](http://pytest.org/latest/) framework. You can use the tests in the `tests/` folder as templates or examples. - - -## Running Tests - -### Running Tests Directly - -If you installed BigchainDB Server using `pip install bigchaindb`, then you -didn't install the tests. Before you can run all the tests, you must install -BigchainDB from source. The [`CONTRIBUTING.md` file](../CONTRIBUTING.md) has -instructions for how to do that. - -Next, make sure you have Local MongoDB running in the background. You -can run MongoDB using `mongod`. - -The `pytest` command has many options. If you want to learn about all the -things you can do with pytest, see [the pytest -documentation](http://pytest.org/latest/). We've also added a customization to -pytest: - -`--database-backend`: Defines the backend to use for the tests. It defaults to -`localmongodb`. - -Now you can run all tests using: -```text -pytest -v -``` - -or, if that doesn't work, try: -```text -python -m pytest -v -``` - -or: -```text -python setup.py test -``` - -How does `python setup.py test` work? The documentation for [pytest-runner](https://pypi.python.org/pypi/pytest-runner) explains. - -The `pytest` command has many options. If you want to learn about all the things you can do with pytest, see [the pytest documentation](http://pytest.org/latest/). We've also added a customization to pytest: - - -### Running Tests with Docker Compose - -You can also use [Docker Compose](https://docs.docker.com/compose/) to run all the tests. - -First, bring up all the services BigchainDB, MongoDB, Tendermint in the background: - -```text -$ docker-compose up -d bdb -``` - -then run the tests using: - -```text -$ docker-compose run --rm --no-deps bigchaindb pytest -v -``` - -## Automated Testing of All Pull Requests - -We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. We use the same `docker-compose.yml` for tests. It tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.) - - -### Tox - -We use [tox](https://tox.readthedocs.io/en/latest/) to run multiple suites of tests against multiple environments during automated testing. Generally you don't need to run this yourself, but it might be useful when troubleshooting a failing Travis CI build. - -To run all the tox tests, use: -```text -tox -``` - -or: -```text -python -m tox -``` - -To run only a few environments, use the `-e` flag: -```text -tox -e {ENVLIST} -``` - -where `{ENVLIST}` is one or more of the environments specified in the [tox.ini file](../tox.ini). +Note: There are acceptance tests in the `acceptance/` folder (at the same level in the hierarchy as the `tests/` folder). From cb22557771c2b75fa24607f58a1526e802fbe96f Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Tue, 4 Sep 2018 15:30:52 +0200 Subject: [PATCH 09/77] Problem: Validation code not optimized (#2490) Solution: memoize data --- bigchaindb/backend/localmongodb/query.py | 6 +- bigchaindb/common/memoize.py | 58 ++++++++++++ bigchaindb/common/transaction.py | 27 +++++- bigchaindb/lib.py | 15 ++- bigchaindb/models.py | 2 +- .../upsert_validator/validator_election.py | 6 -- tests/backend/localmongodb/test_queries.py | 4 +- tests/common/test_memoize.py | 92 +++++++++++++++++++ tests/common/test_transaction.py | 4 +- tests/conftest.py | 6 ++ 10 files changed, 195 insertions(+), 25 deletions(-) create mode 100644 bigchaindb/common/memoize.py create mode 100644 tests/common/test_memoize.py diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index 6587d494..6309c86d 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -91,10 +91,8 @@ def get_assets(conn, asset_ids): @register_query(LocalMongoDBConnection) def get_spent(conn, transaction_id, output): - query = {'inputs.fulfills': { - 'transaction_id': transaction_id, - 'output_index': output}} - + query = {'inputs.fulfills': {'transaction_id': transaction_id, + 'output_index': output}} return conn.run( conn.collection('transactions') .find(query, {'_id': 0})) diff --git a/bigchaindb/common/memoize.py b/bigchaindb/common/memoize.py new file mode 100644 index 00000000..b814e512 --- /dev/null +++ b/bigchaindb/common/memoize.py @@ -0,0 +1,58 @@ +import functools +import codecs +from functools import lru_cache + + +class HDict(dict): + def __hash__(self): + return hash(codecs.decode(self['id'], 'hex')) + + +@lru_cache(maxsize=16384) +def from_dict(func, *args, **kwargs): + return func(*args, **kwargs) + + +def memoize_from_dict(func): + + @functools.wraps(func) + def memoized_func(*args, **kwargs): + + if args[1].get('id', None): + args = list(args) + args[1] = HDict(args[1]) + new_args = tuple(args) + return from_dict(func, *new_args, **kwargs) + else: + return func(*args, **kwargs) + + return memoized_func + + +class ToDictWrapper(): + def __init__(self, tx): + self.tx = tx + + def __eq__(self, other): + return self.tx.id == other.tx.id + + def __hash__(self): + return hash(self.tx.id) + + +@lru_cache(maxsize=16384) +def to_dict(func, tx_wrapped): + return func(tx_wrapped.tx) + + +def memoize_to_dict(func): + + @functools.wraps(func) + def memoized_func(*args, **kwargs): + + if args[0].id: + return to_dict(func, ToDictWrapper(args[0])) + else: + return func(*args, **kwargs) + + return memoized_func diff --git a/bigchaindb/common/transaction.py b/bigchaindb/common/transaction.py index 8baf8196..40cc68e7 100644 --- a/bigchaindb/common/transaction.py +++ b/bigchaindb/common/transaction.py @@ -12,7 +12,8 @@ Attributes: """ from collections import namedtuple from copy import deepcopy -from functools import reduce +from functools import reduce, lru_cache +import rapidjson import base58 from cryptoconditions import Fulfillment, ThresholdSha256, Ed25519Sha256 @@ -27,6 +28,7 @@ from bigchaindb.common.exceptions import (KeypairMismatchException, AmountError, AssetIdMismatch, ThresholdTooDeep) from bigchaindb.common.utils import serialize +from .memoize import memoize_from_dict, memoize_to_dict UnspentOutput = namedtuple( @@ -82,6 +84,11 @@ class Input(object): # TODO: If `other !== Fulfillment` return `False` return self.to_dict() == other.to_dict() + # NOTE: This function is used to provide a unique key for a given + # Input to suppliment memoization + def __hash__(self): + return hash((self.fulfillment, self.fulfills)) + def to_dict(self): """Transforms the object to a Python dictionary. @@ -500,7 +507,7 @@ class Transaction(object): VERSION = '2.0' def __init__(self, operation, asset, inputs=None, outputs=None, - metadata=None, version=None, hash_id=None): + metadata=None, version=None, hash_id=None, tx_dict=None): """The constructor allows to create a customizable Transaction. Note: @@ -553,6 +560,7 @@ class Transaction(object): self.outputs = outputs or [] self.metadata = metadata self._id = hash_id + self.tx_dict = tx_dict @property def unspent_outputs(self): @@ -990,7 +998,7 @@ class Transaction(object): raise ValueError('Inputs and ' 'output_condition_uris must have the same count') - tx_dict = self.to_dict() + tx_dict = self.tx_dict if self.tx_dict else self.to_dict() tx_dict = Transaction._remove_signatures(tx_dict) tx_dict['id'] = None tx_serialized = Transaction._to_str(tx_dict) @@ -1003,6 +1011,7 @@ class Transaction(object): return all(validate(i, cond) for i, cond in enumerate(output_condition_uris)) + @lru_cache(maxsize=16384) def _input_valid(self, input_, operation, message, output_condition_uri=None): """Validates a single Input against a single Output. @@ -1048,6 +1057,11 @@ class Transaction(object): ffill_valid = parsed_ffill.validate(message=message.digest()) return output_valid and ffill_valid + # This function is required by `lru_cache` to create a key for memoization + def __hash__(self): + return hash(self.id) + + @memoize_to_dict def to_dict(self): """Transforms the object to a Python dictionary. @@ -1150,7 +1164,9 @@ class Transaction(object): tx_body (dict): The Transaction to be transformed. """ # NOTE: Remove reference to avoid side effects - tx_body = deepcopy(tx_body) + # tx_body = deepcopy(tx_body) + tx_body = rapidjson.loads(rapidjson.dumps(tx_body)) + try: proposed_tx_id = tx_body['id'] except KeyError: @@ -1167,6 +1183,7 @@ class Transaction(object): raise InvalidHash(err_msg.format(proposed_tx_id)) @classmethod + @memoize_from_dict def from_dict(cls, tx, skip_schema_validation=True): """Transforms a Python dictionary to a Transaction object. @@ -1184,7 +1201,7 @@ class Transaction(object): inputs = [Input.from_dict(input_) for input_ in tx['inputs']] outputs = [Output.from_dict(output) for output in tx['outputs']] return cls(tx['operation'], tx['asset'], inputs, outputs, - tx['metadata'], tx['version'], hash_id=tx['id']) + tx['metadata'], tx['version'], hash_id=tx['id'], tx_dict=tx) @classmethod def from_db(cls, bigchain, tx_dict_list): diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index fa72a506..017151db 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -9,6 +9,7 @@ MongoDB. import logging from collections import namedtuple from uuid import uuid4 +import rapidjson try: from hashlib import sha3_256 @@ -77,10 +78,11 @@ class BigchainDB(object): raise ValidationError('Mode must be one of the following {}.' .format(', '.join(self.mode_list))) + tx_dict = transaction.tx_dict if transaction.tx_dict else transaction.to_dict() payload = { 'method': mode, 'jsonrpc': '2.0', - 'params': [encode_transaction(transaction.to_dict())], + 'params': [encode_transaction(tx_dict)], 'id': str(uuid4()) } # TODO: handle connection errors! @@ -122,10 +124,9 @@ class BigchainDB(object): txns = [] assets = [] txn_metadatas = [] - for transaction_obj in transactions: - # self.update_utxoset(transaction) - transaction = transaction_obj.to_dict() - if transaction['operation'] == transaction_obj.CREATE: + for t in transactions: + transaction = t.tx_dict if t.tx_dict else rapidjson.loads(rapidjson.dumps(t.to_dict())) + if transaction['operation'] == t.CREATE: asset = transaction.pop('asset') asset['id'] = transaction['id'] assets.append(asset) @@ -224,6 +225,10 @@ class BigchainDB(object): return backend.query.delete_unspent_outputs( self.connection, *unspent_outputs) + def is_committed(self, transaction_id): + transaction = backend.query.get_transaction(self.connection, transaction_id) + return bool(transaction) + def get_transaction(self, transaction_id): transaction = backend.query.get_transaction(self.connection, transaction_id) diff --git a/bigchaindb/models.py b/bigchaindb/models.py index 894f510d..9df4dbd4 100644 --- a/bigchaindb/models.py +++ b/bigchaindb/models.py @@ -27,7 +27,7 @@ class Transaction(Transaction): if self.operation == Transaction.CREATE: duplicates = any(txn for txn in current_transactions if txn.id == self.id) - if bigchain.get_transaction(self.to_dict()['id']) or duplicates: + if bigchain.is_committed(self.id) or duplicates: raise DuplicateTransaction('transaction `{}` already exists' .format(self.id)) diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index 95162442..d7e63cf2 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -35,12 +35,6 @@ class ValidatorElection(Transaction): INCONCLUSIVE = 'inconclusive' ELECTION_THRESHOLD = 2 / 3 - def __init__(self, operation, asset, inputs, outputs, - metadata=None, version=None, hash_id=None): - # operation `CREATE` is being passed as argument as `VALIDATOR_ELECTION` is an extension - # of `CREATE` and any validation on `CREATE` in the parent class should apply to it - super().__init__(operation, asset, inputs, outputs, metadata, version, hash_id) - @classmethod def get_validator_change(cls, bigchain, height=None): """Return the latest change to the validator set diff --git a/tests/backend/localmongodb/test_queries.py b/tests/backend/localmongodb/test_queries.py index 9c38d822..2262d723 100644 --- a/tests/backend/localmongodb/test_queries.py +++ b/tests/backend/localmongodb/test_queries.py @@ -205,7 +205,7 @@ def test_get_owned_ids(signed_create_tx, user_pk): conn = connect() # insert a transaction - conn.db.transactions.insert_one(signed_create_tx.to_dict()) + conn.db.transactions.insert_one(deepcopy(signed_create_tx.to_dict())) txns = list(query.get_owned_ids(conn, user_pk)) @@ -224,7 +224,7 @@ def test_get_spending_transactions(user_pk, user_sk): tx2 = Transaction.transfer([inputs[0]], out, tx1.id).sign([user_sk]) tx3 = Transaction.transfer([inputs[1]], out, tx1.id).sign([user_sk]) tx4 = Transaction.transfer([inputs[2]], out, tx1.id).sign([user_sk]) - txns = [tx.to_dict() for tx in [tx1, tx2, tx3, tx4]] + txns = [deepcopy(tx.to_dict()) for tx in [tx1, tx2, tx3, tx4]] conn.db.transactions.insert_many(txns) links = [inputs[0].fulfills.to_dict(), inputs[2].fulfills.to_dict()] diff --git a/tests/common/test_memoize.py b/tests/common/test_memoize.py new file mode 100644 index 00000000..3cc7f62b --- /dev/null +++ b/tests/common/test_memoize.py @@ -0,0 +1,92 @@ +# Copyright BigchainDB GmbH and BigchainDB contributors +# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) +# Code is Apache-2.0 and docs are CC-BY-4.0 + +import pytest +from copy import deepcopy + +from bigchaindb.models import Transaction +from bigchaindb.common.crypto import generate_key_pair +from bigchaindb.common.memoize import to_dict, from_dict + + +pytestmark = [pytest.mark.tendermint, pytest.mark.bdb] + + +def test_memoize_to_dict(b): + alice = generate_key_pair() + asset = { + 'data': {'id': 'test_id'}, + } + + assert to_dict.cache_info().hits == 0 + assert to_dict.cache_info().misses == 0 + + tx = Transaction.create([alice.public_key], + [([alice.public_key], 1)], + asset=asset,)\ + .sign([alice.private_key]) + + tx.to_dict() + + assert to_dict.cache_info().hits == 0 + assert to_dict.cache_info().misses == 1 + + tx.to_dict() + tx.to_dict() + + assert to_dict.cache_info().hits == 2 + assert to_dict.cache_info().misses == 1 + + +def test_memoize_from_dict(b): + alice = generate_key_pair() + asset = { + 'data': {'id': 'test_id'}, + } + + assert from_dict.cache_info().hits == 0 + assert from_dict.cache_info().misses == 0 + + tx = Transaction.create([alice.public_key], + [([alice.public_key], 1)], + asset=asset,)\ + .sign([alice.private_key]) + tx_dict = deepcopy(tx.to_dict()) + + Transaction.from_dict(tx_dict) + + assert from_dict.cache_info().hits == 0 + assert from_dict.cache_info().misses == 1 + + Transaction.from_dict(tx_dict) + Transaction.from_dict(tx_dict) + + assert from_dict.cache_info().hits == 2 + assert from_dict.cache_info().misses == 1 + + +def test_memoize_input_valid(b): + alice = generate_key_pair() + asset = { + 'data': {'id': 'test_id'}, + } + + assert Transaction._input_valid.cache_info().hits == 0 + assert Transaction._input_valid.cache_info().misses == 0 + + tx = Transaction.create([alice.public_key], + [([alice.public_key], 1)], + asset=asset,)\ + .sign([alice.private_key]) + + tx.inputs_valid() + + assert Transaction._input_valid.cache_info().hits == 0 + assert Transaction._input_valid.cache_info().misses == 1 + + tx.inputs_valid() + tx.inputs_valid() + + assert Transaction._input_valid.cache_info().hits == 2 + assert Transaction._input_valid.cache_info().misses == 1 diff --git a/tests/common/test_transaction.py b/tests/common/test_transaction.py index 4a1586ca..fc6444dd 100644 --- a/tests/common/test_transaction.py +++ b/tests/common/test_transaction.py @@ -13,7 +13,7 @@ from cryptoconditions import Ed25519Sha256 from pytest import mark, raises from sha3 import sha3_256 -pytestmark = mark.tendermint +pytestmark = [mark.tendermint, mark.bdb] def test_input_serialization(ffill_uri, user_pub): @@ -533,7 +533,7 @@ def test_validate_input_with_invalid_parameters(utx): input_conditions = [out.fulfillment.condition_uri for out in utx.outputs] tx_dict = utx.to_dict() tx_serialized = Transaction._to_str(tx_dict) - valid = utx._input_valid(utx.inputs[0], tx_serialized, input_conditions) + valid = utx._input_valid(utx.inputs[0], tx_serialized, input_conditions[0]) assert not valid diff --git a/tests/conftest.py b/tests/conftest.py index 7faa66c6..12cd65e3 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -144,11 +144,17 @@ def _bdb(_setup_database, _configure_bigchaindb): from bigchaindb import config from bigchaindb.backend import connect from .utils import flush_db + from bigchaindb.common.memoize import to_dict, from_dict + from bigchaindb.models import Transaction conn = connect() yield dbname = config['database']['name'] flush_db(conn, dbname) + to_dict.cache_clear() + from_dict.cache_clear() + Transaction._input_valid.cache_clear() + # We need this function to avoid loading an existing # conf file located in the home of the user running From af2b5424c05bf781ac356338568a6471e5e2e818 Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Tue, 4 Sep 2018 15:45:48 +0200 Subject: [PATCH 10/77] Problem: Cryptoconditions not pinned which is casuing build failure (#2512) Solution: Pin cryptoconditions so that docs build properly --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 071ddd45..db7f1bd0 100644 --- a/setup.py +++ b/setup.py @@ -79,7 +79,7 @@ install_requires = [ # TODO Consider not installing the db drivers, or putting them in extras. 'pymongo~=3.6', 'pysha3~=1.0.2', - 'cryptoconditions~=0.7.2', + 'cryptoconditions==0.7.2', 'python-rapidjson~=0.6.0', 'logstats~=0.2.1', 'flask>=0.10.1', From bee9468bd0b698f85bbb105fc7bb4f36508bf897 Mon Sep 17 00:00:00 2001 From: Zachary Bowen Date: Wed, 5 Sep 2018 10:51:32 +0200 Subject: [PATCH 11/77] Problem: `test_make_sure_we_dont_remove_any_command` does not check for `upsert-validator show` (#2499). Solution: Extend the test. --- tests/commands/test_commands.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 7b2f8e28..8c1325a9 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -30,6 +30,7 @@ def test_make_sure_we_dont_remove_any_command(): '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command assert parser.parse_args(['upsert-validator', 'approve', 'ELECTION_ID', '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command + assert parser.parse_args(['upsert-validator', 'show', 'ELECTION_ID']).command @pytest.mark.tendermint From f6bee3b63e910272e096ff4ded69cf0efdac763b Mon Sep 17 00:00:00 2001 From: vrde Date: Wed, 5 Sep 2018 16:04:37 +0200 Subject: [PATCH 12/77] Problem: default config is slow (#2516) Solution: update the suggested `config.toml` file to increase the allowed network speed, and avoid rechecking transactions. --- .../network-setup.md | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index f42ca9ef..6b8057ef 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -8,7 +8,7 @@ Code is Apache-2.0 and docs are CC-BY-4.0 Note 1: These instructions will also work for a "network" with only one node. -Note 2: You might not need to set up your own network yet. You should start by creating a proof-of-concept app that writes to [the BigchainDB Testnet](https://testnet.bigchaindb.com/), and if that goes well, then you can look into setting up your own network. +Note 2: You might not need to set up your own network yet. You should start by creating a proof-of-concept app that writes to [the BigchainDB Testnet](https://testnet.bigchaindb.com/), and if that goes well, then you can look into setting up your own network. Note 3: If you want to set up a node or network so that you can contribute to developing and testing the BigchainDB code, then see [the docs about contributing to BigchainDB](https://docs.bigchaindb.com/projects/contributing/en/latest/index.html). @@ -169,51 +169,51 @@ Share the `node_id`, `pub_key.value` and hostname of your Node with all other Me At this point the Coordinator should have received the data from all the Members, and should combine them in the `.tendermint/config/genesis.json` file: ```json -{ +{ "genesis_time":"0001-01-01T00:00:00Z", "chain_id":"test-chain-la6HSr", - "consensus_params":{ - "block_size_params":{ + "consensus_params":{ + "block_size_params":{ "max_bytes":"22020096", "max_txs":"10000", "max_gas":"-1" }, - "tx_size_params":{ + "tx_size_params":{ "max_bytes":"10240", "max_gas":"-1" }, - "block_gossip_params":{ + "block_gossip_params":{ "block_part_size_bytes":"65536" }, - "evidence_params":{ + "evidence_params":{ "max_age":"100000" } }, - "validators":[ - { - "pub_key":{ + "validators":[ + { + "pub_key":{ "type":"AC26791624DE60", "value":"" }, "power":10, "name":"" }, - { - "pub_key":{ + { + "pub_key":{ "type":"AC26791624DE60", "value":"" }, "power":10, "name":"" }, - { - "...":{ + { + "...":{ }, }, - { - "pub_key":{ + { + "pub_key":{ "type":"AC26791624DE60", "value":"" }, @@ -242,14 +242,18 @@ The Member must copy the `genesis.json` file in the local `.tendermint/config` d The Member must edit the `.tendermint/config/config.toml` file and make the following changes: ``` -... - +moniker = "Name of our node" create_empty_blocks = false -... +log_level = "main:info,state:info,*:error" persistent_peers = "@:26656,\ @:26656,\ @:26656," + +send_rate = 102400000 +recv_rate = 102400000 + +recheck = false ``` ## Member: Start MongoDB From 9bf09324df1c63b7fbef0b24830deb5bd80b4dae Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Wed, 5 Sep 2018 17:18:17 +0200 Subject: [PATCH 13/77] Problem: Txs posted in async/sync can be lost. (#2514) Solution: Document when the transactions posted in the async and sync modes can be lost. Describe how they are stored. Mention that BigchainDB does not store or expose rejected transactions. --- docs/server/source/http-client-server-api.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/server/source/http-client-server-api.rst b/docs/server/source/http-client-server-api.rst index 0f47acb5..eadfe983 100644 --- a/docs/server/source/http-client-server-api.rst +++ b/docs/server/source/http-client-server-api.rst @@ -154,7 +154,15 @@ Transactions is in a committed block. .. note:: - + In the async and sync modes, after a successful HTTP response is returned, the transaction may still be rejected later on. All the transactions are recorded internally by Tendermint in WAL (Write-Ahead Log) before the HTTP response is returned. Nevertheless, the following should be noted: + + - Transactions in WAL including the failed ones are not exposed in any of the BigchainDB or Tendermint APIs. + - Transactions are never fetched from WAL. WAL is never replayed. + - A critical failure (e.g. the system is out of disk space) may occur preventing transactions from being stored in WAL, even when the HTTP response indicates a success. + - If a transaction fails the validation because it conflicts with the other transactions of the same block, Tendermint includes it into its block, but BigchainDB does not store these transactions and does not offer any information about them in the APIs. + + .. note:: + The posted transaction should be valid. The relevant `BigchainDB Transactions Spec `_ From 2656302c600f664f94b23b3e6896db0ff6f5d09d Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Thu, 6 Sep 2018 11:27:52 +0200 Subject: [PATCH 14/77] Problem: Bigchaindb crashes when restarted after init chain (#2519) Solution: Return appropirate height so that Tendermint doesn't execute init_chain more than once --- bigchaindb/core.py | 13 +++++++------ tests/tendermint/test_integration.py | 2 +- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/bigchaindb/core.py b/bigchaindb/core.py index a0983472..075525bc 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -232,12 +232,13 @@ class App(BaseApplication): # register a new block only when new transactions are received if self.block_txn_ids: self.bigchaindb.store_bulk_transactions(self.block_transactions) - block = Block(app_hash=self.block_txn_hash, - height=self.new_height, - transactions=self.block_txn_ids) - # NOTE: storing the block should be the last operation during commit - # this effects crash recovery. Refer BEP#8 for details - self.bigchaindb.store_block(block._asdict()) + + block = Block(app_hash=self.block_txn_hash, + height=self.new_height, + transactions=self.block_txn_ids) + # NOTE: storing the block should be the last operation during commit + # this effects crash recovery. Refer BEP#8 for details + self.bigchaindb.store_block(block._asdict()) logger.debug('Commit-ing new block with hash: apphash=%s ,' 'height=%s, txn ids=%s', data, self.new_height, diff --git a/tests/tendermint/test_integration.py b/tests/tendermint/test_integration.py index 683404bc..b860abdf 100644 --- a/tests/tendermint/test_integration.py +++ b/tests/tendermint/test_integration.py @@ -101,7 +101,7 @@ def test_app(b, init_chain_request): block0 = b.get_latest_block() assert block0 - assert block0['height'] == 1 + assert block0['height'] == 2 # when empty block is generated hash of previous block should be returned assert block0['app_hash'] == new_block_hash From 699494613ffeeaa343e9c59d37f48995718c2621 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Thu, 6 Sep 2018 15:49:00 +0200 Subject: [PATCH 15/77] Problem: The commit mode behaves incorrectly. (#2510) * Problem: The commit mode behaves incorrectly. Solution: Parse the Tendermint response properly. The functionality was disabled in https://github.com/bigchaindb/bigchaindb/pull/2235/files#diff-c6511560546a7dc577e7e647b5bfdaceL68 and was not fixed since then. * Add a test case for the sync mode. * Do not strictly expect deliver_tx in the response. * Fix post_mock in web/test_transactions.py. * Check for the error field first. --- bigchaindb/lib.py | 39 ++++++++++++---------------- tests/tendermint/test_integration.py | 19 +++++++------- tests/web/test_transactions.py | 6 +++++ 3 files changed, 33 insertions(+), 31 deletions(-) diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index 017151db..fae4c76a 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -56,9 +56,10 @@ class BigchainDB(object): A connection to the database. """ config_utils.autoconfigure() + self.mode_commit = 'broadcast_tx_commit' self.mode_list = ('broadcast_tx_async', 'broadcast_tx_sync', - 'broadcast_tx_commit') + self.mode_commit) self.tendermint_host = bigchaindb.config['tendermint']['host'] self.tendermint_port = bigchaindb.config['tendermint']['port'] self.endpoint = 'http://{}:{}/'.format(self.tendermint_host, self.tendermint_port) @@ -96,29 +97,23 @@ class BigchainDB(object): def _process_post_response(self, response, mode): logger.debug(response) - if response.get('error') is not None: - return (500, 'Internal error') + + error = response.get('error') + if error: + return (500, error) + + result = response['result'] + if mode == self.mode_commit: + check_tx_code = result.get('check_tx', {}).get('code', 0) + deliver_tx_code = result.get('deliver_tx', {}).get('code', 0) + error_code = check_tx_code or deliver_tx_code + else: + error_code = result.get('code', 0) + + if error_code: + return (500, 'Transaction validation failed') return (202, '') - # result = response['result'] - # if mode == self.mode_list[2]: - # return self._process_commit_mode_response(result) - # else: - # status_code = result['code'] - # return self._process_status_code(status_code, - # 'Error while processing transaction') - - # def _process_commit_mode_response(self, result): - # check_tx_status_code = result['check_tx']['code'] - # if check_tx_status_code == 0: - # deliver_tx_status_code = result['deliver_tx']['code'] - # return self._process_status_code(deliver_tx_status_code, - # 'Error while commiting the transaction') - # else: - # return (500, 'Error while validating the transaction') - - def process_status_code(self, status_code, failure_msg): - return (202, '') if status_code == 0 else (500, failure_msg) def store_bulk_transactions(self, transactions): txns = [] diff --git a/tests/tendermint/test_integration.py b/tests/tendermint/test_integration.py index b860abdf..d3f01612 100644 --- a/tests/tendermint/test_integration.py +++ b/tests/tendermint/test_integration.py @@ -130,12 +130,13 @@ def test_post_transaction_responses(tendermint_ws_url, b): code, message = b.write_transaction(tx_transfer, 'broadcast_tx_commit') assert code == 202 - # NOTE: DOESN'T WORK (double spend) - # Tendermint crashes with error: Unexpected result type - # carly = generate_key_pair() - # double_spend = Transaction.transfer(tx.to_inputs(), - # [([carly.public_key], 1)], - # asset_id=tx.id)\ - # .sign([alice.private_key]) - # code, message = b.write_transaction(double_spend, 'broadcast_tx_commit') - # assert code == 500 + carly = generate_key_pair() + double_spend = Transaction.transfer( + tx.to_inputs(), + [([carly.public_key], 1)], + asset_id=tx.id, + ).sign([alice.private_key]) + for mode in ('broadcast_tx_sync', 'broadcast_tx_commit'): + code, message = b.write_transaction(double_spend, mode) + assert code == 500 + assert message == 'Transaction validation failed' diff --git a/tests/web/test_transactions.py b/tests/web/test_transactions.py index 7c7f2b7a..97386da0 100644 --- a/tests/web/test_transactions.py +++ b/tests/web/test_transactions.py @@ -416,6 +416,12 @@ def test_transactions_get_list_bad(client): def test_post_transaction_valid_modes(mock_post, client, mode): from bigchaindb.models import Transaction from bigchaindb.common.crypto import generate_key_pair + + def _mock_post(*args, **kwargs): + return Mock(json=Mock(return_value={'result': {'code': 0}})) + + mock_post.side_effect = _mock_post + alice = generate_key_pair() tx = Transaction.create([alice.public_key], [([alice.public_key], 1)], From cbfbfa8fc4886f5d13872c508d8c18e596fb5d7b Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Thu, 6 Sep 2018 16:06:36 +0200 Subject: [PATCH 16/77] Problem: Public key format not standardized (#2508) Solution: Add 'type' field to public key so that it can be decoded properly --- bigchaindb/commands/bigchaindb.py | 5 +- bigchaindb/common/exceptions.py | 4 ++ .../transaction_validator_election_v2.0.yaml | 15 ++++- .../upsert_validator/validator_election.py | 13 ++++- .../upsert_validator/validator_utils.py | 56 ++++++++++++++++--- tests/commands/test_commands.py | 7 ++- tests/conftest.py | 8 +-- tests/tendermint/test_core.py | 14 +++-- tests/tendermint/test_integration.py | 2 +- tests/upsert_validator/conftest.py | 2 +- .../test_validator_election.py | 15 ++++- .../test_validator_election_vote.py | 17 +++--- 12 files changed, 119 insertions(+), 39 deletions(-) diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index 19cb0e2d..5f23953b 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -124,7 +124,8 @@ def run_upsert_validator_new(args, bigchain): """ new_validator = { - 'public_key': public_key_from_base64(args.public_key), + 'public_key': {'value': public_key_from_base64(args.public_key), + 'type': 'ed25519-base16'}, 'power': args.power, 'node_id': args.node_id } @@ -207,7 +208,7 @@ def run_upsert_validator_show(args, bigchain): new_validator = election.asset['data'] - public_key = public_key_to_base64(new_validator['public_key']) + public_key = public_key_to_base64(new_validator['public_key']['value']) power = new_validator['power'] node_id = new_validator['node_id'] status = election.get_status(bigchain) diff --git a/bigchaindb/common/exceptions.py b/bigchaindb/common/exceptions.py index adaff5c6..099ef92e 100644 --- a/bigchaindb/common/exceptions.py +++ b/bigchaindb/common/exceptions.py @@ -112,3 +112,7 @@ class UnequalValidatorSet(ValidationError): class InvalidPowerChange(ValidationError): """Raised if proposed power change in validator set is >=1/3 total power""" + + +class InvalidPublicKey(ValidationError): + """Raised if public key doesn't match the encoding type""" diff --git a/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml b/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml index 2c93886c..d849d516 100644 --- a/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml +++ b/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml @@ -23,7 +23,20 @@ properties: node_id: type: string public_key: - type: string + type: object + additionalProperties: false + required: + - value + - type + properties: + value: + type: string + type: + type: string + enum: + - ed25519-base16 + - ed25519-base32 + - ed25519-base64 power: "$ref": "#/definitions/positiveInteger" required: diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index d7e63cf2..0b36c268 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -19,7 +19,10 @@ from bigchaindb.common.schema import (_validate_schema, TX_SCHEMA_COMMON, TX_SCHEMA_CREATE) from . import ValidatorElectionVote -from .validator_utils import (new_validator_set, encode_validator) +from .validator_utils import (new_validator_set, + encode_validator, + encode_pk_to_base16, + validate_asset_public_key) class ValidatorElection(Transaction): @@ -58,7 +61,7 @@ class ValidatorElection(Transaction): validators = {} for validator in bigchain.get_validators(height): # NOTE: we assume that Tendermint encodes public key in base64 - public_key = public_key_from_ed25519_key(key_from_base64(validator['pub_key']['data'])) + public_key = public_key_from_ed25519_key(key_from_base64(validator['public_key']['value'])) validators[public_key] = validator['voting_power'] return validators @@ -155,6 +158,7 @@ class ValidatorElection(Transaction): _validate_schema(TX_SCHEMA_COMMON, tx) _validate_schema(TX_SCHEMA_CREATE, tx) _validate_schema(TX_SCHEMA_VALIDATOR_ELECTION, tx) + validate_asset_public_key(tx['asset']['data']['public_key']) @classmethod def create(cls, tx_signers, recipients, metadata=None, asset=None): @@ -236,7 +240,10 @@ class ValidatorElection(Transaction): updated_validator_set = [v for v in updated_validator_set if v['voting_power'] > 0] bigchain.store_validator_set(new_height+1, updated_validator_set, election.id) - return [encode_validator(election.asset['data'])] + + validator16 = encode_pk_to_base16(election.asset['data']) + return [encode_validator(validator16)] + return [] def get_validator_update_by_election_id(self, election_id, bigchain): diff --git a/bigchaindb/upsert_validator/validator_utils.py b/bigchaindb/upsert_validator/validator_utils.py index 75c7baf5..b0f8a81e 100644 --- a/bigchaindb/upsert_validator/validator_utils.py +++ b/bigchaindb/upsert_validator/validator_utils.py @@ -1,12 +1,14 @@ import codecs +import base64 +import binascii from abci.types_pb2 import (Validator, PubKey) -from bigchaindb.tendermint_utils import public_key_to_base64 +from bigchaindb.common.exceptions import InvalidPublicKey def encode_validator(v): - ed25519_public_key = v['public_key'] + ed25519_public_key = v['public_key']['value'] # NOTE: tendermint expects public to be encoded in go-amino format pub_key = PubKey(type='ed25519', data=bytes.fromhex(ed25519_public_key)) @@ -16,22 +18,60 @@ def encode_validator(v): def decode_validator(v): - return {'pub_key': {'type': v.pub_key.type, - 'data': codecs.encode(v.pub_key.data, 'base64').decode().rstrip('\n')}, + return {'public_key': {'type': 'ed25519-base64', + 'value': codecs.encode(v.pub_key.data, 'base64').decode().rstrip('\n')}, 'voting_power': v.power} def new_validator_set(validators, updates): validators_dict = {} for v in validators: - validators_dict[v['pub_key']['data']] = v + validators_dict[v['public_key']['value']] = v updates_dict = {} for u in updates: - public_key64 = public_key_to_base64(u['public_key']) - updates_dict[public_key64] = {'pub_key': {'type': 'ed25519', - 'data': public_key64}, + decoder = get_public_key_decoder(u['public_key']) + public_key64 = base64.b64encode(decoder(u['public_key']['value'])).decode('utf-8') + updates_dict[public_key64] = {'public_key': {'type': 'ed25519-base64', + 'value': public_key64}, 'voting_power': u['power']} new_validators_dict = {**validators_dict, **updates_dict} return list(new_validators_dict.values()) + + +def encode_pk_to_base16(validator): + pk = validator['public_key'] + decoder = get_public_key_decoder(pk) + public_key16 = base64.b16encode(decoder(pk['value'])).decode('utf-8') + + validator['public_key']['value'] = public_key16 + return validator + + +def validate_asset_public_key(pk): + pk_binary = pk['value'].encode('utf-8') + decoder = get_public_key_decoder(pk) + try: + pk_decoded = decoder(pk_binary) + if len(pk_decoded) != 32: + raise InvalidPublicKey('Public key should be of size 32 bytes') + + except binascii.Error as e: + raise InvalidPublicKey('Invalid `type` specified for public key `value`') + + +def get_public_key_decoder(pk): + encoding = pk['type'] + decoder = base64.b64decode + + if encoding == 'ed25519-base16': + decoder = base64.b16decode + elif encoding == 'ed25519-base32': + decoder = base64.b32decode + elif encoding == 'ed25519-base64': + decoder = base64.b64decode + else: + raise InvalidPublicKey('Invalid `type` specified for public key `value`') + + return decoder diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 8c1325a9..5a5d2deb 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -353,7 +353,7 @@ def test_upsert_validator_new_with_tendermint(b, priv_validator_path, user_sk, v from bigchaindb.commands.bigchaindb import run_upsert_validator_new new_args = Namespace(action='new', - public_key='8eJ8q9ZQpReWyQT5aFCiwtZ5wDZC4eDnCen88p3tQ6ie', + public_key='HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY=', power=1, node_id='unique_node_id_for_test_upsert_validator_new_with_tendermint', sk=priv_validator_path, @@ -444,6 +444,7 @@ def test_upsert_validator_approve_with_tendermint(b, priv_validator_path, user_s config={}) election_id = run_upsert_validator_new(new_args, b) + assert election_id args = Namespace(action='approve', election_id=election_id, @@ -524,8 +525,8 @@ def mock_get_validators(height): keys = node_keys() pub_key = list(keys.keys())[0] return [ - {'pub_key': {'data': pub_key, - 'type': 'tendermint/PubKeyEd25519'}, + {'public_key': {'value': pub_key, + 'type': 'ed25519-base64'}, 'voting_power': 10} ] diff --git a/tests/conftest.py b/tests/conftest.py index 12cd65e3..9ef51e21 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -649,9 +649,8 @@ def validators(b, node_keys): (public_key, private_key) = list(node_keys.items())[0] validator_set = [{'address': 'F5426F0980E36E03044F74DD414248D29ABCBDB2', - 'pub_key': { - 'data': public_key, - 'type': 'ed25519'}, + 'public_key': {'value': public_key, + 'type': 'ed25519-base64'}, 'voting_power': 10}] validator_update = {'validators': validator_set, @@ -687,6 +686,7 @@ def new_validator(): power = 1 node_id = 'fake_node_id' - return {'public_key': public_key, + return {'public_key': {'value': public_key, + 'type': 'ed25519-base16'}, 'power': power, 'node_id': node_id} diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index 64ee2887..c98ee1ea 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -415,12 +415,14 @@ def test_store_pre_commit_state_in_end_block(b, alice, init_chain_request): def test_new_validator_set(b): - node1 = {'pub_key': {'type': 'ed25519', - 'data': 'FxjS2/8AFYoIUqF6AcePTc87qOT7e4WGgH+sGCpTUDQ='}, + node1 = {'public_key': {'type': 'ed25519-base64', + 'value': 'FxjS2/8AFYoIUqF6AcePTc87qOT7e4WGgH+sGCpTUDQ='}, 'voting_power': 10} - node1_new_power = {'public_key': '1718D2DBFF00158A0852A17A01C78F4DCF3BA8E4FB7B8586807FAC182A535034', + node1_new_power = {'public_key': {'value': '1718D2DBFF00158A0852A17A01C78F4DCF3BA8E4FB7B8586807FAC182A535034', + 'type': 'ed25519-base16'}, 'power': 20} - node2 = {'public_key': '1888A353B181715CA2554701D06C1665BC42C5D936C55EA9C5DBCBDB8B3F02A3', + node2 = {'public_key': {'value': '1888A353B181715CA2554701D06C1665BC42C5D936C55EA9C5DBCBDB8B3F02A3', + 'type': 'ed25519-base16'}, 'power': 10} validators = [node1] @@ -430,8 +432,8 @@ def test_new_validator_set(b): updated_validators = [] for u in updates: - updated_validators.append({'pub_key': {'type': 'ed25519', - 'data': public_key_to_base64(u['public_key'])}, + updated_validators.append({'public_key': {'type': 'ed25519-base64', + 'value': public_key_to_base64(u['public_key']['value'])}, 'voting_power': u['power']}) assert updated_validator_set == updated_validators diff --git a/tests/tendermint/test_integration.py b/tests/tendermint/test_integration.py index d3f01612..e91c2be4 100644 --- a/tests/tendermint/test_integration.py +++ b/tests/tendermint/test_integration.py @@ -40,7 +40,7 @@ def test_app(b, init_chain_request): pk = codecs.encode(init_chain_request.validators[0].pub_key.data, 'base64').decode().strip('\n') [validator] = b.get_validators(height=1) - assert validator['pub_key']['data'] == pk + assert validator['public_key']['value'] == pk assert validator['voting_power'] == 10 alice = generate_key_pair() diff --git a/tests/upsert_validator/conftest.py b/tests/upsert_validator/conftest.py index 20906ada..c9bb4870 100644 --- a/tests/upsert_validator/conftest.py +++ b/tests/upsert_validator/conftest.py @@ -22,7 +22,7 @@ def mock_get_validators(network_validators): validators = [] for public_key, power in network_validators.items(): validators.append({ - 'pub_key': {'type': 'AC26791624DE60', 'data': public_key}, + 'public_key': {'type': 'ed25519-base64', 'value': public_key}, 'voting_power': power }) return validators diff --git a/tests/upsert_validator/test_validator_election.py b/tests/upsert_validator/test_validator_election.py index d09d6403..4c3849e4 100644 --- a/tests/upsert_validator/test_validator_election.py +++ b/tests/upsert_validator/test_validator_election.py @@ -24,6 +24,19 @@ def test_upsert_validator_valid_election(b_mock, new_validator, node_key): assert election.validate(b_mock) +def test_upsert_validator_invalid_election_public_key(b_mock, new_validator, node_key): + from bigchaindb.common.exceptions import InvalidPublicKey + + for iv in ['ed25519-base32', 'ed25519-base64']: + new_validator['public_key']['type'] = iv + voters = ValidatorElection.recipients(b_mock) + + with pytest.raises(InvalidPublicKey): + ValidatorElection.generate([node_key.public_key], + voters, + new_validator, None).sign([node_key.private_key]) + + def test_upsert_validator_invalid_power_election(b_mock, new_validator, node_key): voters = ValidatorElection.recipients(b_mock) new_validator['power'] = 30 @@ -147,7 +160,7 @@ def test_upsert_validator_show(caplog, ongoing_election, b): from bigchaindb.commands.bigchaindb import run_upsert_validator_show election_id = ongoing_election.id - public_key = public_key_to_base64(ongoing_election.asset['data']['public_key']) + public_key = public_key_to_base64(ongoing_election.asset['data']['public_key']['value']) power = ongoing_election.asset['data']['power'] node_id = ongoing_election.asset['data']['node_id'] status = ValidatorElection.ONGOING diff --git a/tests/upsert_validator/test_validator_election_vote.py b/tests/upsert_validator/test_validator_election_vote.py index 6c92af99..4504977a 100644 --- a/tests/upsert_validator/test_validator_election_vote.py +++ b/tests/upsert_validator/test_validator_election_vote.py @@ -228,8 +228,7 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): (node_pub, _) = list(node_keys.items())[0] - validators = [{'pub_key': {'type': 'ed25519', - 'data': node_pub}, + validators = [{'public_key': {'type': 'ed25519-base64', 'value': node_pub}, 'voting_power': 10}] latest_block = b.get_latest_block() @@ -239,7 +238,7 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): power = 1 public_key = '9B3119650DF82B9A5D8A12E38953EA47475C09F0C48A4E6A0ECE182944B24403' public_key64 = public_key_to_base64(public_key) - new_validator = {'public_key': public_key, + new_validator = {'public_key': {'value': public_key, 'type': 'ed25519-base16'}, 'node_id': 'some_node_id', 'power': power} @@ -268,7 +267,7 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): new_validator_set = b.get_validators() validator_pub_keys = [] for v in new_validator_set: - validator_pub_keys.append(v['pub_key']['data']) + validator_pub_keys.append(v['public_key']['value']) assert (public_key64 in validator_pub_keys) @@ -281,7 +280,7 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): power = 1 public_key = '9B3119650DF82B9A5D8A12E38953EA47475C09F0C48A4E6A0ECE182944B24403' public_key64 = public_key_to_base64(public_key) - new_validator = {'public_key': public_key, + new_validator = {'public_key': {'value': public_key, 'type': 'ed25519-base16'}, 'node_id': 'some_node_id', 'power': power} voters = ValidatorElection.recipients(b) @@ -316,7 +315,7 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): # remove validator power = 0 - new_validator = {'public_key': public_key, + new_validator = {'public_key': {'value': public_key, 'type': 'ed25519-base16'}, 'node_id': 'some_node_id', 'power': power} voters = ValidatorElection.recipients(b) @@ -339,7 +338,7 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): # assert that the public key is not a part of the current validator set for v in b.get_validators(10): - assert not v['pub_key']['data'] == public_key64 + assert not v['public_key']['value'] == public_key64 # ============================================================================ @@ -365,7 +364,7 @@ def gen_vote(election, i, ed25519_node_keys): def reset_validator_set(b, node_keys, height): validators = [] for (node_pub, _) in node_keys.items(): - validators.append({'pub_key': {'type': 'ed25519', - 'data': node_pub}, + validators.append({'public_key': {'type': 'ed25519-base64', + 'value': node_pub}, 'voting_power': 10}) b.store_validator_set(height, validators, 'election_id') From dd84d4eb6f14a7b45b74e653661bc5ff5b61b4cf Mon Sep 17 00:00:00 2001 From: codegeschrei Date: Mon, 10 Sep 2018 09:53:18 +0200 Subject: [PATCH 17/77] Problem: There are unnecessary markers (#2522) * Problem: we have unused and outdated fixtures Solution: clean up fixtures and tests accordingly * Problem: there are still unused fixtures Solution: remove Merlin keys * Problem: There are unnecessary markers Solution: remove the tendermint marker for tests --- pytest.ini | 2 +- tests/assets/test_digital_assets.py | 3 -- tests/assets/test_divisible_assets.py | 3 -- tests/backend/localmongodb/test_connection.py | 2 +- tests/backend/localmongodb/test_queries.py | 2 +- tests/backend/localmongodb/test_schema.py | 3 -- tests/backend/test_connection.py | 3 -- tests/backend/test_generics.py | 5 --- tests/backend/test_utils.py | 3 -- tests/commands/conftest.py | 6 ++-- tests/commands/test_commands.py | 36 ++++--------------- tests/commands/test_utils.py | 2 -- tests/common/test_memoize.py | 2 +- tests/common/test_schema.py | 5 --- tests/common/test_transaction.py | 2 +- tests/db/test_bigchain_api.py | 23 ------------ tests/tendermint/test_core.py | 2 +- tests/tendermint/test_event_stream.py | 3 -- tests/tendermint/test_fastquery.py | 2 +- tests/tendermint/test_integration.py | 1 - tests/tendermint/test_lib.py | 3 -- tests/tendermint/test_utils.py | 5 --- tests/test_config_utils.py | 3 -- tests/test_core.py | 2 -- tests/test_events.py | 1 - tests/test_txlist.py | 2 -- tests/test_utils.py | 2 -- .../test_validator_election.py | 2 +- .../test_validator_election_vote.py | 7 ---- .../validation/test_transaction_structure.py | 2 -- tests/web/test_assets.py | 4 --- tests/web/test_block_tendermint.py | 28 --------------- tests/web/test_blocks.py | 4 +-- tests/web/test_info.py | 1 + tests/web/test_metadata.py | 4 --- tests/web/test_outputs.py | 9 +---- tests/web/test_parameters.py | 2 -- tests/web/test_server.py | 4 --- tests/web/test_transactions.py | 4 --- tests/web/test_validators.py | 4 --- tests/web/test_websocket_server.py | 2 -- 41 files changed, 21 insertions(+), 184 deletions(-) diff --git a/pytest.ini b/pytest.ini index 006bc2bc..625c3e09 100644 --- a/pytest.ini +++ b/pytest.ini @@ -1,5 +1,5 @@ [pytest] testpaths = tests/ norecursedirs = .* *.egg *.egg-info env* devenv* docs -addopts = -m tendermint +addopts = -m "not abci" looponfailroots = bigchaindb tests diff --git a/tests/assets/test_digital_assets.py b/tests/assets/test_digital_assets.py index 029db261..9af59245 100644 --- a/tests/assets/test_digital_assets.py +++ b/tests/assets/test_digital_assets.py @@ -6,9 +6,6 @@ import pytest import random -pytestmark = pytest.mark.tendermint - - def test_asset_transfer(b, signed_create_tx, user_pk, user_sk): from bigchaindb.models import Transaction diff --git a/tests/assets/test_divisible_assets.py b/tests/assets/test_divisible_assets.py index a4d1241a..276495cd 100644 --- a/tests/assets/test_divisible_assets.py +++ b/tests/assets/test_divisible_assets.py @@ -8,9 +8,6 @@ import random from bigchaindb.common.exceptions import DoubleSpend -pytestmark = pytest.mark.tendermint - - # CREATE divisible asset # Single input # Single owners_before diff --git a/tests/backend/localmongodb/test_connection.py b/tests/backend/localmongodb/test_connection.py index 9bb3d9aa..eeb0adb8 100644 --- a/tests/backend/localmongodb/test_connection.py +++ b/tests/backend/localmongodb/test_connection.py @@ -9,7 +9,7 @@ import pymongo from pymongo import MongoClient -pytestmark = [pytest.mark.bdb, pytest.mark.tendermint] +pytestmark = pytest.mark.bdb @pytest.fixture diff --git a/tests/backend/localmongodb/test_queries.py b/tests/backend/localmongodb/test_queries.py index 2262d723..dd8621d4 100644 --- a/tests/backend/localmongodb/test_queries.py +++ b/tests/backend/localmongodb/test_queries.py @@ -10,7 +10,7 @@ import pymongo from bigchaindb.backend import connect, query -pytestmark = [pytest.mark.tendermint, pytest.mark.bdb] +pytestmark = pytest.mark.bdb def test_get_txids_filtered(signed_create_tx, signed_transfer_tx): diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index aeadbe05..0d49c4ec 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -5,9 +5,6 @@ import pytest -pytestmark = [pytest.mark.bdb, pytest.mark.tendermint] - - def test_init_creates_db_tables_and_indexes(): import bigchaindb from bigchaindb import backend diff --git a/tests/backend/test_connection.py b/tests/backend/test_connection.py index 71d42458..52b0e4b0 100644 --- a/tests/backend/test_connection.py +++ b/tests/backend/test_connection.py @@ -5,9 +5,6 @@ import pytest -pytestmark = pytest.mark.tendermint - - def test_get_connection_raises_a_configuration_error(monkeypatch): from bigchaindb.common.exceptions import ConfigurationError from bigchaindb.backend import connect diff --git a/tests/backend/test_generics.py b/tests/backend/test_generics.py index 98b56bbf..6ddd7331 100644 --- a/tests/backend/test_generics.py +++ b/tests/backend/test_generics.py @@ -2,14 +2,9 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -import pytest - from pytest import mark, raises -pytestmark = pytest.mark.tendermint - - @mark.parametrize('schema_func_name,args_qty', ( ('create_database', 1), ('create_tables', 1), diff --git a/tests/backend/test_utils.py b/tests/backend/test_utils.py index 61c123ac..16e390af 100644 --- a/tests/backend/test_utils.py +++ b/tests/backend/test_utils.py @@ -8,9 +8,6 @@ from types import ModuleType import pytest -pytestmark = pytest.mark.tendermint - - @pytest.fixture def mock_module(): return ModuleType('mock_module') diff --git a/tests/commands/conftest.py b/tests/commands/conftest.py index ce09ff24..d4591720 100644 --- a/tests/commands/conftest.py +++ b/tests/commands/conftest.py @@ -27,8 +27,8 @@ def mock_db_init_with_existing_db(monkeypatch): @pytest.fixture def mock_processes_start(monkeypatch): - from bigchaindb.utils import Process - monkeypatch.setattr(Process, 'run', lambda *args: None) + from bigchaindb import start + monkeypatch.setattr(start, 'start', lambda *args: None) @pytest.fixture @@ -56,7 +56,7 @@ def run_start_args(request): @pytest.fixture def mocked_setup_logging(mocker): return mocker.patch( - 'bigchaindb.commands.utils.setup_logging', + 'bigchaindb.log.setup_logging', autospec=True, spec_set=True, ) diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 5a5d2deb..6ed54425 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -14,7 +14,6 @@ from bigchaindb import ValidatorElection from tests.conftest import node_keys -@pytest.mark.tendermint def test_make_sure_we_dont_remove_any_command(): # thanks to: http://stackoverflow.com/a/18161115/597097 from bigchaindb.commands.bigchaindb import create_parser @@ -33,7 +32,6 @@ def test_make_sure_we_dont_remove_any_command(): assert parser.parse_args(['upsert-validator', 'show', 'ELECTION_ID']).command -@pytest.mark.tendermint @patch('bigchaindb.commands.utils.start') def test_main_entrypoint(mock_start): from bigchaindb.commands.bigchaindb import main @@ -42,22 +40,21 @@ def test_main_entrypoint(mock_start): assert mock_start.called -def test_bigchain_run_start(mock_run_configure, - mock_processes_start, - mock_db_init_with_existing_db, - mocked_setup_logging): - from bigchaindb import config +@patch('bigchaindb.log.setup_logging') +@patch('bigchaindb.commands.bigchaindb._run_init') +@patch('bigchaindb.config_utils.autoconfigure') +def test_bigchain_run_start(mock_setup_logging, mock_run_init, + mock_autoconfigure, mock_processes_start): from bigchaindb.commands.bigchaindb import run_start args = Namespace(config=None, yes=True, skip_initialize_database=False) run_start(args) - mocked_setup_logging.assert_called_once_with(user_log_config=config['log']) + assert mock_setup_logging.called # TODO Please beware, that if debugging, the "-s" switch for pytest will # interfere with capsys. # See related issue: https://github.com/pytest-dev/pytest/issues/128 -@pytest.mark.tendermint @pytest.mark.usefixtures('ignore_local_config_file') def test_bigchain_show_config(capsys): from bigchaindb.commands.bigchaindb import run_show_config @@ -78,7 +75,6 @@ def test_bigchain_show_config(capsys): assert output_config == config -@pytest.mark.tendermint def test_bigchain_run_init_when_db_exists(mocker, capsys): from bigchaindb.commands.bigchaindb import run_init from bigchaindb.common.exceptions import DatabaseAlreadyExists @@ -98,7 +94,6 @@ def test_bigchain_run_init_when_db_exists(mocker, capsys): ) -@pytest.mark.tendermint def test__run_init(mocker): from bigchaindb.commands.bigchaindb import _run_init bigchain_mock = mocker.patch( @@ -114,7 +109,6 @@ def test__run_init(mocker): connection=bigchain_mock.return_value.connection) -@pytest.mark.tendermint @patch('bigchaindb.backend.schema.drop_database') def test_drop_db_when_assumed_yes(mock_db_drop): from bigchaindb.commands.bigchaindb import run_drop @@ -124,7 +118,6 @@ def test_drop_db_when_assumed_yes(mock_db_drop): assert mock_db_drop.called -@pytest.mark.tendermint @patch('bigchaindb.backend.schema.drop_database') def test_drop_db_when_interactive_yes(mock_db_drop, monkeypatch): from bigchaindb.commands.bigchaindb import run_drop @@ -136,7 +129,6 @@ def test_drop_db_when_interactive_yes(mock_db_drop, monkeypatch): assert mock_db_drop.called -@pytest.mark.tendermint @patch('bigchaindb.backend.schema.drop_database') def test_drop_db_when_db_does_not_exist(mock_db_drop, capsys): from bigchaindb import config @@ -151,7 +143,6 @@ def test_drop_db_when_db_does_not_exist(mock_db_drop, capsys): name=config['database']['name']) -@pytest.mark.tendermint @patch('bigchaindb.backend.schema.drop_database') def test_drop_db_does_not_drop_when_interactive_no(mock_db_drop, monkeypatch): from bigchaindb.commands.bigchaindb import run_drop @@ -166,7 +157,6 @@ def test_drop_db_does_not_drop_when_interactive_no(mock_db_drop, monkeypatch): # TODO Beware if you are putting breakpoints in there, and using the '-s' # switch with pytest. It will just hang. Seems related to the monkeypatching of # input_on_stderr. -@pytest.mark.tendermint def test_run_configure_when_config_does_not_exist(monkeypatch, mock_write_config, mock_generate_key_pair, @@ -179,7 +169,6 @@ def test_run_configure_when_config_does_not_exist(monkeypatch, assert return_value is None -@pytest.mark.tendermint def test_run_configure_when_config_does_exist(monkeypatch, mock_write_config, mock_generate_key_pair, @@ -201,7 +190,6 @@ def test_run_configure_when_config_does_exist(monkeypatch, @pytest.mark.skip -@pytest.mark.tendermint @pytest.mark.parametrize('backend', ( 'localmongodb', )) @@ -235,10 +223,9 @@ def test_run_start_when_db_already_exists(mocker, monkeypatch, run_start_args, mocked_setup_logging): - from bigchaindb import config from bigchaindb.commands.bigchaindb import run_start from bigchaindb.common.exceptions import DatabaseAlreadyExists - mocked_start = mocker.patch('bigchaindb.processes.start') + mocked_start = mocker.patch('bigchaindb.start.start') def mock_run_init(): raise DatabaseAlreadyExists() @@ -246,11 +233,9 @@ def test_run_start_when_db_already_exists(mocker, monkeypatch.setattr( 'bigchaindb.commands.bigchaindb._run_init', mock_run_init) run_start(run_start_args) - mocked_setup_logging.assert_called_once_with(user_log_config=config['log']) assert mocked_start.called -@pytest.mark.tendermint @patch('bigchaindb.commands.utils.start') def test_calling_main(start_mock, monkeypatch): from bigchaindb.commands.bigchaindb import main @@ -295,7 +280,6 @@ def test_recover_db_on_start(mock_run_recover, assert mock_start.called -@pytest.mark.tendermint @pytest.mark.bdb def test_run_recover(b, alice, bob): from bigchaindb.commands.bigchaindb import run_recover @@ -364,7 +348,6 @@ def test_upsert_validator_new_with_tendermint(b, priv_validator_path, user_sk, v assert b.get_transaction(election_id) -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_new_without_tendermint(caplog, b, priv_validator_path, user_sk): from bigchaindb.commands.bigchaindb import run_upsert_validator_new @@ -389,7 +372,6 @@ def test_upsert_validator_new_without_tendermint(caplog, b, priv_validator_path, assert b.get_transaction(election_id) -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_new_invalid_election(caplog, b, priv_validator_path, user_sk): from bigchaindb.commands.bigchaindb import run_upsert_validator_new @@ -406,7 +388,6 @@ def test_upsert_validator_new_invalid_election(caplog, b, priv_validator_path, u assert caplog.records[0].msg.__class__ == FileNotFoundError -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_new_election_invalid_power(caplog, b, priv_validator_path, user_sk): from bigchaindb.commands.bigchaindb import run_upsert_validator_new @@ -456,7 +437,6 @@ def test_upsert_validator_approve_with_tendermint(b, priv_validator_path, user_s @pytest.mark.bdb -@pytest.mark.tendermint def test_upsert_validator_approve_without_tendermint(caplog, b, priv_validator_path, new_validator, node_key): from bigchaindb.commands.bigchaindb import run_upsert_validator_approve from argparse import Namespace @@ -476,7 +456,6 @@ def test_upsert_validator_approve_without_tendermint(caplog, b, priv_validator_p assert b.get_transaction(approval_id) -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_approve_failure(caplog, b, priv_validator_path, new_validator, node_key): from bigchaindb.commands.bigchaindb import run_upsert_validator_approve @@ -501,7 +480,6 @@ def test_upsert_validator_approve_failure(caplog, b, priv_validator_path, new_va assert caplog.records[0].msg == 'Failed to commit vote' -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_approve_called_with_bad_key(caplog, b, bad_validator_path, new_validator, node_key): from bigchaindb.commands.bigchaindb import run_upsert_validator_approve diff --git a/tests/commands/test_utils.py b/tests/commands/test_utils.py index 11ff41a8..eeac67fc 100644 --- a/tests/commands/test_utils.py +++ b/tests/commands/test_utils.py @@ -10,8 +10,6 @@ import pytest from unittest.mock import patch -pytestmark = pytest.mark.tendermint - @pytest.fixture def reset_bigchaindb_config(monkeypatch): diff --git a/tests/common/test_memoize.py b/tests/common/test_memoize.py index 3cc7f62b..20c84f31 100644 --- a/tests/common/test_memoize.py +++ b/tests/common/test_memoize.py @@ -10,7 +10,7 @@ from bigchaindb.common.crypto import generate_key_pair from bigchaindb.common.memoize import to_dict, from_dict -pytestmark = [pytest.mark.tendermint, pytest.mark.bdb] +pytestmark = pytest.mark.bdb def test_memoize_to_dict(b): diff --git a/tests/common/test_schema.py b/tests/common/test_schema.py index 93fe2e3c..0ca1be68 100644 --- a/tests/common/test_schema.py +++ b/tests/common/test_schema.py @@ -8,8 +8,6 @@ properties related to validation. from unittest.mock import patch -import pytest - from hypothesis import given from hypothesis_regex import regex from pytest import raises @@ -25,9 +23,6 @@ UNSUPPORTED_CRYPTOCONDITION_TYPES = ( 'preimage-sha-256', 'prefix-sha-256', 'rsa-sha-256') -pytestmark = pytest.mark.tendermint - - ################################################################################ # Test of schema utils diff --git a/tests/common/test_transaction.py b/tests/common/test_transaction.py index fc6444dd..8e5a2b6c 100644 --- a/tests/common/test_transaction.py +++ b/tests/common/test_transaction.py @@ -13,7 +13,7 @@ from cryptoconditions import Ed25519Sha256 from pytest import mark, raises from sha3 import sha3_256 -pytestmark = [mark.tendermint, mark.bdb] +pytestmark = mark.bdb def test_input_serialization(ffill_uri, user_pub): diff --git a/tests/db/test_bigchain_api.py b/tests/db/test_bigchain_api.py index 1ec1904c..1af41e37 100644 --- a/tests/db/test_bigchain_api.py +++ b/tests/db/test_bigchain_api.py @@ -12,7 +12,6 @@ pytestmark = pytest.mark.bdb class TestBigchainApi(object): - @pytest.mark.tendermint def test_get_spent_with_double_spend_detected(self, b, alice): from bigchaindb.models import Transaction from bigchaindb.common.exceptions import DoubleSpend @@ -43,7 +42,6 @@ class TestBigchainApi(object): with pytest.raises(CriticalDoubleSpend): b.get_spent(tx.id, 0) - @pytest.mark.tendermint def test_double_inclusion(self, b, alice): from bigchaindb.models import Transaction from bigchaindb.backend.exceptions import OperationError @@ -56,7 +54,6 @@ class TestBigchainApi(object): with pytest.raises(OperationError): b.store_bulk_transactions([tx]) - @pytest.mark.tendermint def test_text_search(self, b, alice): from bigchaindb.models import Transaction @@ -81,7 +78,6 @@ class TestBigchainApi(object): assert len(assets) == 3 @pytest.mark.usefixtures('inputs') - @pytest.mark.tendermint def test_non_create_input_not_found(self, b, user_pk): from cryptoconditions import Ed25519Sha256 from bigchaindb.common.exceptions import InputDoesNotExist @@ -97,7 +93,6 @@ class TestBigchainApi(object): with pytest.raises(InputDoesNotExist): tx.validate(b) - @pytest.mark.tendermint def test_write_transaction(self, b, user_sk, user_pk, alice, create_tx): from bigchaindb.models import Transaction @@ -120,7 +115,6 @@ class TestBigchainApi(object): class TestTransactionValidation(object): - @pytest.mark.tendermint def test_non_create_input_not_found(self, b, signed_transfer_tx): from bigchaindb.common.exceptions import InputDoesNotExist from bigchaindb.common.transaction import TransactionLink @@ -129,7 +123,6 @@ class TestTransactionValidation(object): with pytest.raises(InputDoesNotExist): b.validate_transaction(signed_transfer_tx) - @pytest.mark.tendermint @pytest.mark.usefixtures('inputs') def test_non_create_valid_input_wrong_owner(self, b, user_pk): from bigchaindb.common.crypto import generate_key_pair @@ -147,7 +140,6 @@ class TestTransactionValidation(object): with pytest.raises(InvalidSignature): b.validate_transaction(tx) - @pytest.mark.tendermint @pytest.mark.usefixtures('inputs') def test_non_create_double_spend(self, b, signed_create_tx, signed_transfer_tx, double_spend_tx): @@ -161,7 +153,6 @@ class TestTransactionValidation(object): class TestMultipleInputs(object): - @pytest.mark.tendermint def test_transfer_single_owner_single_input(self, b, inputs, user_pk, user_sk): from bigchaindb.common import crypto @@ -181,7 +172,6 @@ class TestMultipleInputs(object): assert len(tx.inputs) == 1 assert len(tx.outputs) == 1 - @pytest.mark.tendermint def test_single_owner_before_multiple_owners_after_single_input(self, b, user_sk, user_pk, @@ -203,7 +193,6 @@ class TestMultipleInputs(object): assert len(tx.inputs) == 1 assert len(tx.outputs) == 1 - @pytest.mark.tendermint @pytest.mark.usefixtures('inputs') def test_multiple_owners_before_single_owner_after_single_input(self, b, user_sk, @@ -232,7 +221,6 @@ class TestMultipleInputs(object): assert len(transfer_tx.inputs) == 1 assert len(transfer_tx.outputs) == 1 - @pytest.mark.tendermint @pytest.mark.usefixtures('inputs') def test_multiple_owners_before_multiple_owners_after_single_input(self, b, user_sk, @@ -262,7 +250,6 @@ class TestMultipleInputs(object): assert len(tx.inputs) == 1 assert len(tx.outputs) == 1 - @pytest.mark.tendermint def test_get_owned_ids_single_tx_single_output(self, b, user_sk, user_pk, alice): from bigchaindb.common import crypto from bigchaindb.common.transaction import TransactionLink @@ -290,7 +277,6 @@ class TestMultipleInputs(object): assert owned_inputs_user1 == [TransactionLink(tx.id, 0)] assert owned_inputs_user2 == [TransactionLink(tx_transfer.id, 0)] - @pytest.mark.tendermint def test_get_owned_ids_single_tx_multiple_outputs(self, b, user_sk, user_pk, alice): from bigchaindb.common import crypto @@ -326,7 +312,6 @@ class TestMultipleInputs(object): assert owned_inputs_user2 == [TransactionLink(tx_transfer.id, 0), TransactionLink(tx_transfer.id, 1)] - @pytest.mark.tendermint def test_get_owned_ids_multiple_owners(self, b, user_sk, user_pk, alice): from bigchaindb.common import crypto from bigchaindb.common.transaction import TransactionLink @@ -359,7 +344,6 @@ class TestMultipleInputs(object): assert owned_inputs_user1 == owned_inputs_user2 assert not spent_user1 - @pytest.mark.tendermint def test_get_spent_single_tx_single_output(self, b, user_sk, user_pk, alice): from bigchaindb.common import crypto from bigchaindb.models import Transaction @@ -386,7 +370,6 @@ class TestMultipleInputs(object): spent_inputs_user1 = b.get_spent(input_txid, 0) assert spent_inputs_user1 == tx - @pytest.mark.tendermint def test_get_spent_single_tx_multiple_outputs(self, b, user_sk, user_pk, alice): from bigchaindb.common import crypto from bigchaindb.models import Transaction @@ -424,7 +407,6 @@ class TestMultipleInputs(object): # spendable by BigchainDB assert b.get_spent(tx_create.to_inputs()[2].fulfills.txid, 2) is None - @pytest.mark.tendermint def test_get_spent_multiple_owners(self, b, user_sk, user_pk, alice): from bigchaindb.common import crypto from bigchaindb.models import Transaction @@ -461,7 +443,6 @@ class TestMultipleInputs(object): assert b.get_spent(unspent.id, 0) is None -@pytest.mark.tendermint def test_get_outputs_filtered_only_unspent(): from bigchaindb.common.transaction import TransactionLink from bigchaindb.lib import BigchainDB @@ -478,7 +459,6 @@ def test_get_outputs_filtered_only_unspent(): assert out == [TransactionLink('b', 2)] -@pytest.mark.tendermint def test_get_outputs_filtered_only_spent(): from bigchaindb.common.transaction import TransactionLink from bigchaindb.lib import BigchainDB @@ -494,7 +474,6 @@ def test_get_outputs_filtered_only_spent(): assert out == [TransactionLink('b', 2)] -@pytest.mark.tendermint @patch('bigchaindb.fastquery.FastQuery.filter_unspent_outputs') @patch('bigchaindb.fastquery.FastQuery.filter_spent_outputs') def test_get_outputs_filtered(filter_spent, filter_unspent): @@ -512,7 +491,6 @@ def test_get_outputs_filtered(filter_spent, filter_unspent): assert out == get_outputs.return_value -@pytest.mark.tendermint def test_cant_spend_same_input_twice_in_tx(b, alice): """Recreate duplicated fulfillments bug https://github.com/bigchaindb/bigchaindb/issues/1099 @@ -536,7 +514,6 @@ def test_cant_spend_same_input_twice_in_tx(b, alice): tx_transfer_signed.validate(b) -@pytest.mark.tendermint def test_transaction_unicode(b, alice): import copy from bigchaindb.common.utils import serialize diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index c98ee1ea..b59cc0c6 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -28,7 +28,7 @@ from bigchaindb.upsert_validator.validator_utils import new_validator_set from bigchaindb.tendermint_utils import public_key_to_base64 -pytestmark = [pytest.mark.tendermint, pytest.mark.bdb] +pytestmark = pytest.mark.bdb def encode_tx_to_bytes(transaction): diff --git a/tests/tendermint/test_event_stream.py b/tests/tendermint/test_event_stream.py index 365ea1e0..f23eeaba 100644 --- a/tests/tendermint/test_event_stream.py +++ b/tests/tendermint/test_event_stream.py @@ -10,7 +10,6 @@ from aiohttp import ClientSession import pytest -@pytest.mark.tendermint def test_process_event_new_block(): from bigchaindb.event_stream import process_event @@ -48,7 +47,6 @@ def test_process_event_new_block(): assert isinstance(block.data['height'], int) -@pytest.mark.tendermint def test_process_event_empty_block(): from bigchaindb.event_stream import process_event @@ -69,7 +67,6 @@ def test_process_event_empty_block(): assert event_queue.empty() -@pytest.mark.tendermint def test_process_unknown_event(): from bigchaindb.event_stream import process_event diff --git a/tests/tendermint/test_fastquery.py b/tests/tendermint/test_fastquery.py index 20eb1889..bf93850f 100644 --- a/tests/tendermint/test_fastquery.py +++ b/tests/tendermint/test_fastquery.py @@ -8,7 +8,7 @@ from bigchaindb.common.transaction import TransactionLink from bigchaindb.models import Transaction -pytestmark = [pytest.mark.bdb, pytest.mark.tendermint] +pytestmark = pytest.mark.bdb @pytest.fixture diff --git a/tests/tendermint/test_integration.py b/tests/tendermint/test_integration.py index e91c2be4..490da16b 100644 --- a/tests/tendermint/test_integration.py +++ b/tests/tendermint/test_integration.py @@ -14,7 +14,6 @@ from abci.encoding import read_messages from io import BytesIO -@pytest.mark.tendermint @pytest.mark.bdb def test_app(b, init_chain_request): from bigchaindb import App diff --git a/tests/tendermint/test_lib.py b/tests/tendermint/test_lib.py index 3ea91329..6b384b33 100644 --- a/tests/tendermint/test_lib.py +++ b/tests/tendermint/test_lib.py @@ -18,9 +18,6 @@ from bigchaindb import backend from bigchaindb.lib import Block -pytestmark = pytest.mark.tendermint - - @pytest.mark.bdb def test_asset_is_separated_from_transaciton(b): import copy diff --git a/tests/tendermint/test_utils.py b/tests/tendermint/test_utils.py index 46b36525..39d0b44e 100644 --- a/tests/tendermint/test_utils.py +++ b/tests/tendermint/test_utils.py @@ -10,11 +10,6 @@ try: except ImportError: from sha3 import sha3_256 -import pytest - - -pytestmark = pytest.mark.tendermint - def test_encode_decode_transaction(b): from bigchaindb.tendermint_utils import (encode_transaction, diff --git a/tests/test_config_utils.py b/tests/test_config_utils.py index 35f0e486..72dc4e29 100644 --- a/tests/test_config_utils.py +++ b/tests/test_config_utils.py @@ -13,9 +13,6 @@ import bigchaindb ORIGINAL_CONFIG = copy.deepcopy(bigchaindb._config) -pytestmark = pytest.mark.tendermint - - @pytest.fixture(scope='function', autouse=True) def clean_config(monkeypatch, request): original_config = copy.deepcopy(ORIGINAL_CONFIG) diff --git a/tests/test_core.py b/tests/test_core.py index 03701267..72e5bc1d 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -4,8 +4,6 @@ import pytest -pytestmark = pytest.mark.tendermint - @pytest.fixture def config(request, monkeypatch): diff --git a/tests/test_events.py b/tests/test_events.py index 77e9a0f5..ab088b99 100644 --- a/tests/test_events.py +++ b/tests/test_events.py @@ -3,7 +3,6 @@ # Code is Apache-2.0 and docs are CC-BY-4.0 import pytest -pytestmark = pytest.mark.tendermint def test_event_handler(): diff --git a/tests/test_txlist.py b/tests/test_txlist.py index d2192632..8566f73d 100644 --- a/tests/test_txlist.py +++ b/tests/test_txlist.py @@ -8,8 +8,6 @@ This test module defines it's own fixture which is used by all the tests. """ import pytest -pytestmark = pytest.mark.tendermint - @pytest.fixture def txlist(b, user_pk, user2_pk, user_sk, user2_sk): diff --git a/tests/test_utils.py b/tests/test_utils.py index 31549be4..46b01bf7 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -7,8 +7,6 @@ from unittest.mock import patch, call import pytest -pytestmark = pytest.mark.tendermint - @pytest.fixture def mock_queue(monkeypatch): diff --git a/tests/upsert_validator/test_validator_election.py b/tests/upsert_validator/test_validator_election.py index 4c3849e4..bad475a3 100644 --- a/tests/upsert_validator/test_validator_election.py +++ b/tests/upsert_validator/test_validator_election.py @@ -13,7 +13,7 @@ from bigchaindb.common.exceptions import (DuplicateTransaction, MultipleInputsError, InvalidPowerChange) -pytestmark = [pytest.mark.tendermint, pytest.mark.bdb] +pytestmark = pytest.mark.bdb def test_upsert_validator_valid_election(b_mock, new_validator, node_key): diff --git a/tests/upsert_validator/test_validator_election_vote.py b/tests/upsert_validator/test_validator_election_vote.py index 4504977a..c395bbc1 100644 --- a/tests/upsert_validator/test_validator_election_vote.py +++ b/tests/upsert_validator/test_validator_election_vote.py @@ -15,7 +15,6 @@ from tests.utils import generate_block pytestmark = [pytest.mark.execute] -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_valid_election_vote(b_mock, valid_election, ed25519_node_keys): b_mock.store_bulk_transactions([valid_election]) @@ -34,7 +33,6 @@ def test_upsert_validator_valid_election_vote(b_mock, valid_election, ed25519_no assert vote.validate(b_mock) -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_valid_non_election_vote(b_mock, valid_election, ed25519_node_keys): b_mock.store_bulk_transactions([valid_election]) @@ -54,7 +52,6 @@ def test_upsert_validator_valid_non_election_vote(b_mock, valid_election, ed2551 .sign([key0.private_key]) -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_delegate_election_vote(b_mock, valid_election, ed25519_node_keys): alice = generate_key_pair() @@ -91,7 +88,6 @@ def test_upsert_validator_delegate_election_vote(b_mock, valid_election, ed25519 assert key0_casted_vote.validate(b_mock) -@pytest.mark.tendermint @pytest.mark.bdb def test_upsert_validator_invalid_election_vote(b_mock, valid_election, ed25519_node_keys): b_mock.store_bulk_transactions([valid_election]) @@ -112,7 +108,6 @@ def test_upsert_validator_invalid_election_vote(b_mock, valid_election, ed25519_ assert vote.validate(b_mock) -@pytest.mark.tendermint @pytest.mark.bdb def test_valid_election_votes_received(b_mock, valid_election, ed25519_node_keys): alice = generate_key_pair() @@ -158,7 +153,6 @@ def test_valid_election_votes_received(b_mock, valid_election, ed25519_node_keys assert valid_election.get_commited_votes(b_mock) == votes-2 -@pytest.mark.tendermint @pytest.mark.bdb def test_valid_election_conclude(b_mock, valid_election, ed25519_node_keys): @@ -272,7 +266,6 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): assert (public_key64 in validator_pub_keys) -@pytest.mark.tendermint @pytest.mark.bdb def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): reset_validator_set(b, node_keys, 1) diff --git a/tests/validation/test_transaction_structure.py b/tests/validation/test_transaction_structure.py index cecb51d7..32622c87 100644 --- a/tests/validation/test_transaction_structure.py +++ b/tests/validation/test_transaction_structure.py @@ -17,8 +17,6 @@ from bigchaindb.common.exceptions import (AmountError, ThresholdTooDeep) from bigchaindb.models import Transaction -pytestmark = pytest.mark.tendermint - ################################################################################ # Helper functions diff --git a/tests/web/test_assets.py b/tests/web/test_assets.py index 2c658cdc..a5d3260e 100644 --- a/tests/web/test_assets.py +++ b/tests/web/test_assets.py @@ -7,7 +7,6 @@ import pytest ASSETS_ENDPOINT = '/api/v1/assets/' -@pytest.mark.tendermint def test_get_assets_with_empty_text_search(client): res = client.get(ASSETS_ENDPOINT + '?search=') assert res.json == {'status': 400, @@ -15,14 +14,12 @@ def test_get_assets_with_empty_text_search(client): assert res.status_code == 400 -@pytest.mark.tendermint def test_get_assets_with_missing_text_search(client): res = client.get(ASSETS_ENDPOINT) assert res.status_code == 400 @pytest.mark.bdb -@pytest.mark.tendermint def test_get_assets_tendermint(client, b, alice): from bigchaindb.models import Transaction @@ -49,7 +46,6 @@ def test_get_assets_tendermint(client, b, alice): @pytest.mark.bdb -@pytest.mark.tendermint def test_get_assets_limit_tendermint(client, b, alice): from bigchaindb.models import Transaction diff --git a/tests/web/test_block_tendermint.py b/tests/web/test_block_tendermint.py index a92822c4..2f789c55 100644 --- a/tests/web/test_block_tendermint.py +++ b/tests/web/test_block_tendermint.py @@ -9,8 +9,6 @@ from bigchaindb.lib import Block BLOCKS_ENDPOINT = '/api/v1/blocks/' -pytestmark = pytest.mark.tendermint - @pytest.mark.bdb @pytest.mark.usefixtures('inputs') @@ -73,29 +71,3 @@ def test_get_blocks_by_txid_endpoint_returns_empty_list_not_found(client): res = client.get(BLOCKS_ENDPOINT + '?transaction_id=123') assert res.status_code == 200 assert len(res.json) == 0 - - -@pytest.mark.bdb -def test_get_blocks_by_txid_endpoint_returns_400_bad_query_params(client): - res = client.get(BLOCKS_ENDPOINT) - assert res.status_code == 400 - - res = client.get(BLOCKS_ENDPOINT + '?ts_id=123') - assert res.status_code == 400 - assert res.json == { - 'message': { - 'transaction_id': 'Missing required parameter in the JSON body or the post body or the query string' - } - } - - res = client.get(BLOCKS_ENDPOINT + '?transaction_id=123&foo=123') - assert res.status_code == 400 - assert res.json == { - 'message': 'Unknown arguments: foo' - } - - res = client.get(BLOCKS_ENDPOINT + '?transaction_id=123&status=123') - assert res.status_code == 400 - assert res.json == { - 'message': 'Unknown arguments: status' - } diff --git a/tests/web/test_blocks.py b/tests/web/test_blocks.py index aa792b7f..e0130613 100644 --- a/tests/web/test_blocks.py +++ b/tests/web/test_blocks.py @@ -50,7 +50,5 @@ def test_get_blocks_by_txid_endpoint_returns_400_bad_query_params(client): res = client.get(BLOCKS_ENDPOINT + '?transaction_id=123&status=123') assert res.status_code == 400 assert res.json == { - 'message': { - 'status': '123 is not a valid choice' - } + 'message': 'Unknown arguments: status' } diff --git a/tests/web/test_info.py b/tests/web/test_info.py index c2ba9bc3..b91b6f60 100644 --- a/tests/web/test_info.py +++ b/tests/web/test_info.py @@ -43,6 +43,7 @@ def test_api_v1_endpoint(client, wsserver_base_url): 'streams': '{}/api/v1/streams/valid_transactions'.format( wsserver_base_url), 'metadata': '/metadata/', + 'validators': '/validators' } res = client.get('/api/v1') assert res.json == api_v1_info diff --git a/tests/web/test_metadata.py b/tests/web/test_metadata.py index 248ef946..c1e6c5ed 100644 --- a/tests/web/test_metadata.py +++ b/tests/web/test_metadata.py @@ -7,7 +7,6 @@ import pytest METADATA_ENDPOINT = '/api/v1/metadata/' -@pytest.mark.tendermint def test_get_metadata_with_empty_text_search(client): res = client.get(METADATA_ENDPOINT + '?search=') assert res.json == {'status': 400, @@ -15,14 +14,12 @@ def test_get_metadata_with_empty_text_search(client): assert res.status_code == 400 -@pytest.mark.tendermint def test_get_metadata_with_missing_text_search(client): res = client.get(METADATA_ENDPOINT) assert res.status_code == 400 @pytest.mark.bdb -@pytest.mark.tendermint def test_get_metadata_tendermint(client, b, alice): from bigchaindb.models import Transaction @@ -50,7 +47,6 @@ def test_get_metadata_tendermint(client, b, alice): @pytest.mark.bdb -@pytest.mark.tendermint def test_get_metadata_limit_tendermint(client, b, alice): from bigchaindb.models import Transaction diff --git a/tests/web/test_outputs.py b/tests/web/test_outputs.py index 7cf1ab5e..db7b718d 100644 --- a/tests/web/test_outputs.py +++ b/tests/web/test_outputs.py @@ -10,7 +10,6 @@ pytestmark = [pytest.mark.bdb, pytest.mark.usefixtures('inputs')] OUTPUTS_ENDPOINT = '/api/v1/outputs/' -@pytest.mark.tendermint def test_get_outputs_endpoint(client, user_pk): m = MagicMock() m.txid = 'a' @@ -26,7 +25,6 @@ def test_get_outputs_endpoint(client, user_pk): gof.assert_called_once_with(user_pk, None) -@pytest.mark.tendermint def test_get_outputs_endpoint_unspent(client, user_pk): m = MagicMock() m.txid = 'a' @@ -40,7 +38,6 @@ def test_get_outputs_endpoint_unspent(client, user_pk): gof.assert_called_once_with(user_pk, False) -@pytest.mark.tendermint def test_get_outputs_endpoint_spent(client, user_pk): m = MagicMock() m.txid = 'a' @@ -54,13 +51,11 @@ def test_get_outputs_endpoint_spent(client, user_pk): gof.assert_called_once_with(user_pk, True) -@pytest.mark.tendermint def test_get_outputs_endpoint_without_public_key(client): res = client.get(OUTPUTS_ENDPOINT) assert res.status_code == 400 -@pytest.mark.tendermint def test_get_outputs_endpoint_with_invalid_public_key(client): expected = {'message': {'public_key': 'Invalid base58 ed25519 key'}} res = client.get(OUTPUTS_ENDPOINT + '?public_key=abc') @@ -68,7 +63,6 @@ def test_get_outputs_endpoint_with_invalid_public_key(client): assert res.status_code == 400 -@pytest.mark.tendermint def test_get_outputs_endpoint_with_invalid_spent(client, user_pk): expected = {'message': {'spent': 'Boolean value must be "true" or "false" (lowercase)'}} params = '?spent=tru&public_key={}'.format(user_pk) @@ -87,8 +81,7 @@ def test_get_divisble_transactions_returns_500(b, client): TX_ENDPOINT = '/api/v1/transactions' def mine(tx_list): - block = b.create_block(tx_list) - b.write_block(block) + b.store_bulk_transactions(tx_list) alice_priv, alice_pub = crypto.generate_key_pair() bob_priv, bob_pub = crypto.generate_key_pair() diff --git a/tests/web/test_parameters.py b/tests/web/test_parameters.py index 3eb8ab2a..ccf3119b 100644 --- a/tests/web/test_parameters.py +++ b/tests/web/test_parameters.py @@ -4,8 +4,6 @@ import pytest -pytestmark = pytest.mark.tendermint - def test_valid_txid(): from bigchaindb.web.views.parameters import valid_txid diff --git a/tests/web/test_server.py b/tests/web/test_server.py index 5fce3629..215aa980 100644 --- a/tests/web/test_server.py +++ b/tests/web/test_server.py @@ -2,10 +2,6 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -import pytest - -pytestmark = pytest.mark.tendermint - def test_settings(): import bigchaindb diff --git a/tests/web/test_transactions.py b/tests/web/test_transactions.py index 97386da0..68df7f8b 100644 --- a/tests/web/test_transactions.py +++ b/tests/web/test_transactions.py @@ -23,7 +23,6 @@ def test_get_transaction_endpoint(client, posted_create_tx): assert res.status_code == 200 -@pytest.mark.tendermint def test_get_transaction_returns_404_if_not_found(client): res = client.get(TX_ENDPOINT + '123') assert res.status_code == 404 @@ -361,7 +360,6 @@ def test_post_wrong_asset_division_transfer_returns_400(b, client, user_pk): assert res.json['message'] == expected_error_message -@pytest.mark.tendermint def test_transactions_get_list_good(client): from functools import partial @@ -388,7 +386,6 @@ def test_transactions_get_list_good(client): ] -@pytest.mark.tendermint def test_transactions_get_list_bad(client): def should_not_be_called(): assert False @@ -405,7 +402,6 @@ def test_transactions_get_list_bad(client): assert client.get(url).status_code == 400 -@pytest.mark.tendermint @patch('requests.post') @pytest.mark.parametrize('mode', [ ('', 'broadcast_tx_async'), diff --git a/tests/web/test_validators.py b/tests/web/test_validators.py index 52420828..0574204f 100644 --- a/tests/web/test_validators.py +++ b/tests/web/test_validators.py @@ -2,10 +2,6 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -import pytest - -pytestmark = pytest.mark.tendermint - VALIDATORS_ENDPOINT = '/api/v1/validators/' diff --git a/tests/web/test_websocket_server.py b/tests/web/test_websocket_server.py index a3de508a..dff74f21 100644 --- a/tests/web/test_websocket_server.py +++ b/tests/web/test_websocket_server.py @@ -10,8 +10,6 @@ from unittest.mock import patch import pytest -pytestmark = pytest.mark.tendermint - class MockWebSocket: def __init__(self): From c79848d66a684420b717e36bed2ec00281f57855 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Tue, 11 Sep 2018 14:28:55 +0200 Subject: [PATCH 18/77] Problem: Docs say install highest Tendermint version (#2524) Solution: Change the docs to recommend installing a specific Tendermint version. Also update our release process so we change that version number before a release, if necessary. --- RELEASE_PROCESS.md | 1 + .../run-node-as-processes.md | 2 +- docs/server/source/simple-deployment-template/network-setup.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/RELEASE_PROCESS.md b/RELEASE_PROCESS.md index f5c7af39..3c24dedf 100644 --- a/RELEASE_PROCESS.md +++ b/RELEASE_PROCESS.md @@ -45,6 +45,7 @@ The following steps are what we do to release a new version of _BigchainDB Serve - In `bigchaindb/version.py`: - update `__version__` to e.g. `0.9.0` (with no `.dev` on the end) - update `__short_version__` to e.g. `0.9` (with no `.dev` on the end) + - In the docs about installing BigchainDB (and Tendermint), and in the associated scripts, recommend/install a version of Tendermint that _actually works_ with the soon-to-be-released version of BigchainDB. You can find all such references by doing a search for the previously-recommended version number, such as `0.22.8`. - In `setup.py`, _maybe_ update the development status item in the `classifiers` list. For example, one allowed value is `"Development Status :: 5 - Production/Stable"`. The [allowed values are listed at pypi.python.org](https://pypi.python.org/pypi?%3Aaction=list_classifiers). 1. **Wait for all the tests to pass!** diff --git a/docs/contributing/source/dev-setup-coding-and-contribution-process/run-node-as-processes.md b/docs/contributing/source/dev-setup-coding-and-contribution-process/run-node-as-processes.md index 7c1556a3..98a5aeb8 100644 --- a/docs/contributing/source/dev-setup-coding-and-contribution-process/run-node-as-processes.md +++ b/docs/contributing/source/dev-setup-coding-and-contribution-process/run-node-as-processes.md @@ -25,7 +25,7 @@ After the installation of MongoDB is complete, run MongoDB using `sudo mongod` ### Installing a Tendermint Executable -Find [the version number of the latest Tendermint release](https://github.com/tendermint/tendermint/releases) and install it using the following, where 0.22.8 should be replaced by the latest released version number: +The version of BigchainDB Server described in these docs only works well with Tendermint 0.22.8 (not a higher version number). Install that: ```bash $ sudo apt install -y unzip diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index 6b8057ef..ed6498fc 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -85,7 +85,7 @@ Note: The `mongodb` package is _not_ the official MongoDB package from MongoDB t #### Install Tendermint -Install a [recent version of Tendermint][tendermint:releases]. BigchainDB Server requires version 0.22.8 or newer. +The version of BigchainDB Server described in these docs only works well with Tendermint 0.22.8 (not a higher version number). Install that: ``` sudo apt install -y unzip From 0fe749d830a88ceaaf285e9bc89ac4b9e7d90962 Mon Sep 17 00:00:00 2001 From: Zachary Bowen Date: Tue, 11 Sep 2018 15:39:46 +0200 Subject: [PATCH 19/77] Create abstract election class (#2498). * Problem: `ValidatorElection` and `MigrationElection` need to inherit from a common `Election` class. Solution: Factored the common logic out of `ValidatorElection` and moved it to `Election` parent class. * Problem: No need to store different types of elections in their own tables Solution: Remove `DB_TABLE` property from `Election` class. Solution: Created the `elections` table with secondary_index `election_id`. * Problem: `UpsertValidatorVote` can be generalized to just be `Vote` Solution: Renamed, refactored and moved the `Vote` class to tie in with the more general `Election` base class. * Problem: `election_id` is not unique if two elections have the same properties. Solution: Added a random `uuid4` seed to enforce uniqueness. --- bigchaindb/__init__.py | 6 +- bigchaindb/backend/localmongodb/query.py | 16 +- bigchaindb/backend/localmongodb/schema.py | 10 + bigchaindb/backend/query.py | 13 +- bigchaindb/backend/schema.py | 2 +- bigchaindb/commands/bigchaindb.py | 9 +- bigchaindb/common/schema/__init__.py | 3 +- .../common/schema/transaction_v2.0.yaml | 2 +- .../transaction_validator_election_v2.0.yaml | 2 + ...e_v2.0.yaml => transaction_vote_v2.0.yaml} | 4 +- bigchaindb/core.py | 11 +- bigchaindb/elections/__init__.py | 0 bigchaindb/elections/election.py | 254 +++++++++++++++++ .../vote.py} | 20 +- bigchaindb/lib.py | 20 +- bigchaindb/upsert_validator/__init__.py | 1 - .../upsert_validator/validator_election.py | 260 ++---------------- tests/backend/localmongodb/test_schema.py | 11 +- tests/tendermint/test_core.py | 6 +- tests/tendermint/test_lib.py | 6 +- tests/upsert_validator/conftest.py | 28 +- ..._vote.py => test_upsert_validator_vote.py} | 110 ++++---- .../test_validator_election.py | 8 +- tests/web/test_validators.py | 2 +- 24 files changed, 448 insertions(+), 356 deletions(-) rename bigchaindb/common/schema/{transaction_validator_election_vote_v2.0.yaml => transaction_vote_v2.0.yaml} (87%) create mode 100644 bigchaindb/elections/__init__.py create mode 100644 bigchaindb/elections/election.py rename bigchaindb/{upsert_validator/validator_election_vote.py => elections/vote.py} (79%) rename tests/upsert_validator/{test_validator_election_vote.py => test_upsert_validator_vote.py} (73%) diff --git a/bigchaindb/__init__.py b/bigchaindb/__init__.py index 5e69b684..e1762f5d 100644 --- a/bigchaindb/__init__.py +++ b/bigchaindb/__init__.py @@ -94,9 +94,9 @@ _config = copy.deepcopy(config) from bigchaindb.common.transaction import Transaction # noqa from bigchaindb import models # noqa from bigchaindb.upsert_validator import ValidatorElection # noqa -from bigchaindb.upsert_validator import ValidatorElectionVote # noqa +from bigchaindb.elections.vote import Vote # noqa Transaction.register_type(Transaction.CREATE, models.Transaction) Transaction.register_type(Transaction.TRANSFER, models.Transaction) -Transaction.register_type(ValidatorElection.VALIDATOR_ELECTION, ValidatorElection) -Transaction.register_type(ValidatorElectionVote.VALIDATOR_ELECTION_VOTE, ValidatorElectionVote) +Transaction.register_type(ValidatorElection.OPERATION, ValidatorElection) +Transaction.register_type(Vote.OPERATION, Vote) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index 6309c86d..4c016f18 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -281,6 +281,18 @@ def store_validator_set(conn, validators_update): ) +@register_query(LocalMongoDBConnection) +def store_election_results(conn, election): + height = election['height'] + return conn.run( + conn.collection('elections').replace_one( + {'height': height}, + election, + upsert=True + ) + ) + + @register_query(LocalMongoDBConnection) def get_validator_set(conn, height=None): query = {} @@ -298,11 +310,11 @@ def get_validator_set(conn, height=None): @register_query(LocalMongoDBConnection) -def get_validator_set_by_election_id(conn, election_id): +def get_election(conn, election_id): query = {'election_id': election_id} cursor = conn.run( - conn.collection('validators') + conn.collection('elections') .find(query, projection={'_id': False}) ) diff --git a/bigchaindb/backend/localmongodb/schema.py b/bigchaindb/backend/localmongodb/schema.py index e58f1def..342a22b7 100644 --- a/bigchaindb/backend/localmongodb/schema.py +++ b/bigchaindb/backend/localmongodb/schema.py @@ -48,6 +48,7 @@ def create_indexes(conn, dbname): create_pre_commit_secondary_index(conn, dbname) create_validators_secondary_index(conn, dbname) create_abci_chains_indexes(conn, dbname) + create_elections_secondary_index(conn, dbname) @register_schema(LocalMongoDBConnection) @@ -144,6 +145,15 @@ def create_abci_chains_indexes(conn, dbname): unique=True,) logger.info('Create `abci_chains.chain_id` secondary index.') + conn.conn[dbname]['abci_chains'].create_index('chain_id', name='chain_id', unique=True) + + +def create_elections_secondary_index(conn, dbname): + logger.info('Create `elections` secondary index.') + + conn.conn[dbname]['elections'].create_index('election_id', + name='election_id', + unique=True,) diff --git a/bigchaindb/backend/query.py b/bigchaindb/backend/query.py index 7965d5ee..d8f60320 100644 --- a/bigchaindb/backend/query.py +++ b/bigchaindb/backend/query.py @@ -351,6 +351,13 @@ def store_validator_set(conn, validator_update): raise NotImplementedError +@singledispatch +def store_election_results(conn, election): + """Store election results""" + + raise NotImplementedError + + @singledispatch def get_validator_set(conn, height): """Get validator set for a given `height`, if `height` is not specified @@ -361,7 +368,7 @@ def get_validator_set(conn, height): @singledispatch -def get_validator_set_by_election_id(conn, election_id): +def get_election(conn, election_id): """Return a validator set change with the specified election_id """ @@ -369,13 +376,11 @@ def get_validator_set_by_election_id(conn, election_id): @singledispatch -def get_asset_tokens_for_public_key(connection, asset_id, - public_key, operation): +def get_asset_tokens_for_public_key(connection, asset_id, public_key): """Retrieve a list of tokens of type `asset_id` that are owned by the `public_key`. Args: asset_id (str): Id of the token. public_key (str): base58 encoded public key - operation: filter transaction based on `operation` Returns: Iterator of transaction that list given owner in conditions. """ diff --git a/bigchaindb/backend/schema.py b/bigchaindb/backend/schema.py index 108bd39b..c1d0343d 100644 --- a/bigchaindb/backend/schema.py +++ b/bigchaindb/backend/schema.py @@ -16,7 +16,7 @@ logger = logging.getLogger(__name__) # Tables/collections that every backend database must create TABLES = ('transactions', 'blocks', 'assets', 'metadata', - 'validators', 'pre_commit', 'utxos', 'abci_chains') + 'validators', 'elections', 'pre_commit', 'utxos', 'abci_chains') VALID_LANGUAGES = ('danish', 'dutch', 'english', 'finnish', 'french', 'german', 'hungarian', 'italian', 'norwegian', 'portuguese', 'romanian', diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index 5f23953b..d548d7b0 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -17,9 +17,10 @@ from bigchaindb.utils import load_node_key from bigchaindb.common.exceptions import (DatabaseAlreadyExists, DatabaseDoesNotExist, ValidationError) +from bigchaindb.elections.vote import Vote import bigchaindb from bigchaindb import (backend, ValidatorElection, - BigchainDB, ValidatorElectionVote) + BigchainDB) from bigchaindb.backend import schema from bigchaindb.backend import query from bigchaindb.backend.query import PRE_COMMIT_ID @@ -176,9 +177,9 @@ def run_upsert_validator_approve(args, bigchain): inputs = [i for i in tx.to_inputs() if key.public_key in i.owners_before] election_pub_key = ValidatorElection.to_public_key(tx.id) - approval = ValidatorElectionVote.generate(inputs, - [([election_pub_key], voting_power)], - tx.id).sign([key.private_key]) + approval = Vote.generate(inputs, + [([election_pub_key], voting_power)], + tx.id).sign([key.private_key]) approval.validate(bigchain) resp = bigchain.write_transaction(approval, 'broadcast_tx_commit') diff --git a/bigchaindb/common/schema/__init__.py b/bigchaindb/common/schema/__init__.py index 914e5196..25943675 100644 --- a/bigchaindb/common/schema/__init__.py +++ b/bigchaindb/common/schema/__init__.py @@ -37,8 +37,7 @@ _, TX_SCHEMA_TRANSFER = _load_schema('transaction_transfer_' + _, TX_SCHEMA_VALIDATOR_ELECTION = _load_schema('transaction_validator_election_' + TX_SCHEMA_VERSION) -_, TX_SCHEMA_VALIDATOR_ELECTION_VOTE = _load_schema('transaction_validator_election_vote_' + - TX_SCHEMA_VERSION) +_, TX_SCHEMA_VOTE = _load_schema('transaction_vote_' + TX_SCHEMA_VERSION) def _validate_schema(schema, body): diff --git a/bigchaindb/common/schema/transaction_v2.0.yaml b/bigchaindb/common/schema/transaction_v2.0.yaml index acc8c6b5..562c0d86 100644 --- a/bigchaindb/common/schema/transaction_v2.0.yaml +++ b/bigchaindb/common/schema/transaction_v2.0.yaml @@ -63,7 +63,7 @@ definitions: - CREATE - TRANSFER - VALIDATOR_ELECTION - - VALIDATOR_ELECTION_VOTE + - VOTE asset: type: object additionalProperties: false diff --git a/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml b/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml index d849d516..602ef9e2 100644 --- a/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml +++ b/bigchaindb/common/schema/transaction_validator_election_v2.0.yaml @@ -22,6 +22,8 @@ properties: properties: node_id: type: string + seed: + type: string public_key: type: object additionalProperties: false diff --git a/bigchaindb/common/schema/transaction_validator_election_vote_v2.0.yaml b/bigchaindb/common/schema/transaction_vote_v2.0.yaml similarity index 87% rename from bigchaindb/common/schema/transaction_validator_election_vote_v2.0.yaml rename to bigchaindb/common/schema/transaction_vote_v2.0.yaml index c17fb229..5e7c4763 100644 --- a/bigchaindb/common/schema/transaction_validator_election_vote_v2.0.yaml +++ b/bigchaindb/common/schema/transaction_vote_v2.0.yaml @@ -5,14 +5,14 @@ --- "$schema": "http://json-schema.org/draft-04/schema#" type: object -title: Validator Election Vote Schema - Vote on a validator set change +title: Vote Schema - Vote on an election required: - operation - outputs properties: operation: type: string - value: "VALIDATOR_ELECTION_VOTE" + value: "VOTE" outputs: type: array items: diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 075525bc..62bc6ec3 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -100,7 +100,7 @@ class App(BaseApplication): block = Block(app_hash=app_hash, height=height, transactions=[]) self.bigchaindb.store_block(block._asdict()) - self.bigchaindb.store_validator_set(height + 1, validator_set, None) + self.bigchaindb.store_validator_set(height + 1, validator_set) abci_chain_height = 0 if known_chain is None else known_chain['height'] self.bigchaindb.store_abci_chain(abci_chain_height, genesis.chain_id, True) @@ -209,9 +209,10 @@ class App(BaseApplication): # Check if the current block concluded any validator elections and # update the locally tracked validator set - validator_updates = ValidatorElection.get_validator_update(self.bigchaindb, - self.new_height, - self.block_transactions) + validator_update = ValidatorElection.approved_update(self.bigchaindb, + self.new_height, + self.block_transactions) + update = [validator_update] if validator_update else [] # Store pre-commit state to recover in case there is a crash # during `commit` @@ -220,7 +221,7 @@ class App(BaseApplication): transactions=self.block_txn_ids) logger.debug('Updating PreCommitState: %s', self.new_height) self.bigchaindb.store_pre_commit_state(pre_commit_state._asdict()) - return ResponseEndBlock(validator_updates=validator_updates) + return ResponseEndBlock(validator_updates=update) def commit(self): """Store the new height and along with block hash.""" diff --git a/bigchaindb/elections/__init__.py b/bigchaindb/elections/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/bigchaindb/elections/election.py b/bigchaindb/elections/election.py new file mode 100644 index 00000000..0c0219bd --- /dev/null +++ b/bigchaindb/elections/election.py @@ -0,0 +1,254 @@ +# Copyright BigchainDB GmbH and BigchainDB contributors +# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) +# Code is Apache-2.0 and docs are CC-BY-4.0 + +import base58 +from uuid import uuid4 + +from bigchaindb import backend +from bigchaindb.elections.vote import Vote +from bigchaindb.common.exceptions import (InvalidSignature, + MultipleInputsError, + InvalidProposer, + UnequalValidatorSet, + DuplicateTransaction) +from bigchaindb.tendermint_utils import key_from_base64 +from bigchaindb.common.crypto import (public_key_from_ed25519_key) +from bigchaindb.common.transaction import Transaction +from bigchaindb.common.schema import (_validate_schema, + TX_SCHEMA_COMMON, + TX_SCHEMA_CREATE) + + +class Election(Transaction): + + # NOTE: this transaction class extends create so the operation inheritance is achieved + # by setting an ELECTION_TYPE and renaming CREATE = ELECTION_TYPE and ALLOWED_OPERATIONS = (ELECTION_TYPE,) + OPERATION = None + # Custom validation schema + TX_SCHEMA_CUSTOM = None + # Election Statuses: + ONGOING = 'ongoing' + CONCLUDED = 'concluded' + INCONCLUSIVE = 'inconclusive' + # Vote ratio to approve an election + ELECTION_THRESHOLD = 2 / 3 + + @classmethod + def get_validator_change(cls, bigchain, height=None): + """Return the latest change to the validator set + + :return: { + 'height': , + 'validators': , + 'election_id': + } + """ + return bigchain.get_validator_change(height) + + @classmethod + def get_validators(cls, bigchain, height=None): + """Return a dictionary of validators with key as `public_key` and + value as the `voting_power` + """ + validators = {} + for validator in bigchain.get_validators(height): + # NOTE: we assume that Tendermint encodes public key in base64 + public_key = public_key_from_ed25519_key(key_from_base64(validator['public_key']['value'])) + validators[public_key] = validator['voting_power'] + + return validators + + @classmethod + def recipients(cls, bigchain): + """Convert validator dictionary to a recipient list for `Transaction`""" + + recipients = [] + for public_key, voting_power in cls.get_validators(bigchain).items(): + recipients.append(([public_key], voting_power)) + + return recipients + + @classmethod + def is_same_topology(cls, current_topology, election_topology): + voters = {} + for voter in election_topology: + if len(voter.public_keys) > 1: + return False + + [public_key] = voter.public_keys + voting_power = voter.amount + voters[public_key] = voting_power + + # Check whether the voters and their votes is same to that of the + # validators and their voting power in the network + return current_topology == voters + + def validate(self, bigchain, current_transactions=[]): + """Validate election transaction + + NOTE: + * A valid election is initiated by an existing validator. + + * A valid election is one where voters are validators and votes are + allocated according to the voting power of each validator node. + + Args: + :param bigchain: (BigchainDB) an instantiated bigchaindb.lib.BigchainDB object. + :param current_transactions: (list) A list of transactions to be validated along with the election + + Returns: + Election: a Election object or an object of the derived Election subclass. + + Raises: + ValidationError: If the election is invalid + """ + input_conditions = [] + + duplicates = any(txn for txn in current_transactions if txn.id == self.id) + if bigchain.is_committed(self.id) or duplicates: + raise DuplicateTransaction('transaction `{}` already exists' + .format(self.id)) + + if not self.inputs_valid(input_conditions): + raise InvalidSignature('Transaction signature is invalid.') + + current_validators = self.get_validators(bigchain) + + # NOTE: Proposer should be a single node + if len(self.inputs) != 1 or len(self.inputs[0].owners_before) != 1: + raise MultipleInputsError('`tx_signers` must be a list instance of length one') + + # NOTE: Check if the proposer is a validator. + [election_initiator_node_pub_key] = self.inputs[0].owners_before + if election_initiator_node_pub_key not in current_validators.keys(): + raise InvalidProposer('Public key is not a part of the validator set') + + # NOTE: Check if all validators have been assigned votes equal to their voting power + if not self.is_same_topology(current_validators, self.outputs): + raise UnequalValidatorSet('Validator set much be exactly same to the outputs of election') + + return self + + @classmethod + def generate(cls, initiator, voters, election_data, metadata=None): + # Break symmetry in case we need to call an election with the same properties twice + uuid = uuid4() + election_data['seed'] = str(uuid) + + (inputs, outputs) = cls.validate_create(initiator, voters, election_data, metadata) + election = cls(cls.OPERATION, {'data': election_data}, inputs, outputs, metadata) + cls.validate_schema(election.to_dict()) + return election + + @classmethod + def validate_schema(cls, tx): + """Validate the election transaction. Since `ELECTION` extends `CREATE` transaction, all the validations for + `CREATE` transaction should be inherited + """ + _validate_schema(TX_SCHEMA_COMMON, tx) + _validate_schema(TX_SCHEMA_CREATE, tx) + if cls.TX_SCHEMA_CUSTOM: + _validate_schema(cls.TX_SCHEMA_CUSTOM, tx) + + @classmethod + def create(cls, tx_signers, recipients, metadata=None, asset=None): + raise NotImplementedError + + @classmethod + def transfer(cls, tx_signers, recipients, metadata=None, asset=None): + raise NotImplementedError + + @classmethod + def to_public_key(cls, election_id): + return base58.b58encode(bytes.fromhex(election_id)).decode() + + @classmethod + def count_votes(cls, election_pk, transactions, getter=getattr): + votes = 0 + for txn in transactions: + if getter(txn, 'operation') == Vote.OPERATION: + for output in getter(txn, 'outputs'): + # NOTE: We enforce that a valid vote to election id will have only + # election_pk in the output public keys, including any other public key + # along with election_pk will lead to vote being not considered valid. + if len(getter(output, 'public_keys')) == 1 and [election_pk] == getter(output, 'public_keys'): + votes = votes + int(getter(output, 'amount')) + return votes + + def get_commited_votes(self, bigchain, election_pk=None): + if election_pk is None: + election_pk = self.to_public_key(self.id) + txns = list(backend.query.get_asset_tokens_for_public_key(bigchain.connection, + self.id, + election_pk)) + return self.count_votes(election_pk, txns, dict.get) + + @classmethod + def has_concluded(cls, bigchain, election_id, current_votes=[], height=None): + """Check if the given `election_id` can be concluded or not + NOTE: + * Election is concluded iff the current validator set is exactly equal + to the validator set encoded in election outputs + * Election can concluded only if the current votes achieves a supermajority + """ + election = bigchain.get_transaction(election_id) + + if election: + election_pk = election.to_public_key(election.id) + votes_committed = election.get_commited_votes(bigchain, election_pk) + votes_current = election.count_votes(election_pk, current_votes) + current_validators = election.get_validators(bigchain, height) + + if election.is_same_topology(current_validators, election.outputs): + total_votes = sum(current_validators.values()) + if (votes_committed < (2/3)*total_votes) and \ + (votes_committed + votes_current >= (2/3)*total_votes): + return election + return False + + def get_status(self, bigchain): + concluded = self.get_election(self.id, bigchain) + if concluded: + return self.CONCLUDED + + latest_change = self.get_validator_change(bigchain) + latest_change_height = latest_change['height'] + election_height = bigchain.get_block_containing_tx(self.id)[0] + + if latest_change_height >= election_height: + return self.INCONCLUSIVE + else: + return self.ONGOING + + def get_election(self, election_id, bigchain): + result = bigchain.get_election(election_id) + return result + + @classmethod + def store_election_results(cls, bigchain, election, height): + bigchain.store_election_results(height, election) + + @classmethod + def approved_update(cls, bigchain, new_height, txns): + votes = {} + for txn in txns: + if not isinstance(txn, Vote): + continue + + election_id = txn.asset['id'] + election_votes = votes.get(election_id, []) + election_votes.append(txn) + votes[election_id] = election_votes + + election = cls.has_concluded(bigchain, election_id, election_votes, new_height) + # Once an election concludes any other conclusion for the same + # or any other election is invalidated + if election: + cls.store_election_results(bigchain, election, new_height) + return cls.on_approval(bigchain, election, new_height) + return None + + @classmethod + def on_approval(cls, bigchain, election, new_height): + raise NotImplementedError diff --git a/bigchaindb/upsert_validator/validator_election_vote.py b/bigchaindb/elections/vote.py similarity index 79% rename from bigchaindb/upsert_validator/validator_election_vote.py rename to bigchaindb/elections/vote.py index 7620b289..01175290 100644 --- a/bigchaindb/upsert_validator/validator_election_vote.py +++ b/bigchaindb/elections/vote.py @@ -6,16 +6,18 @@ from bigchaindb.common.transaction import Transaction from bigchaindb.common.schema import (_validate_schema, TX_SCHEMA_COMMON, TX_SCHEMA_TRANSFER, - TX_SCHEMA_VALIDATOR_ELECTION_VOTE) + TX_SCHEMA_VOTE) -class ValidatorElectionVote(Transaction): +class Vote(Transaction): - VALIDATOR_ELECTION_VOTE = 'VALIDATOR_ELECTION_VOTE' + OPERATION = 'VOTE' # NOTE: This class inherits TRANSFER txn type. The `TRANSFER` property is # overriden to re-use methods from parent class - TRANSFER = VALIDATOR_ELECTION_VOTE - ALLOWED_OPERATIONS = (VALIDATOR_ELECTION_VOTE,) + TRANSFER = OPERATION + ALLOWED_OPERATIONS = (OPERATION,) + # Custom validation schema + TX_SCHEMA_CUSTOM = TX_SCHEMA_VOTE def validate(self, bigchain, current_transactions=[]): """Validate election vote transaction @@ -28,7 +30,7 @@ class ValidatorElectionVote(Transaction): bigchain (BigchainDB): an instantiated bigchaindb.lib.BigchainDB object. Returns: - ValidatorElectionVote object + Vote: a Vote object Raises: ValidationError: If the election vote is invalid @@ -39,20 +41,20 @@ class ValidatorElectionVote(Transaction): @classmethod def generate(cls, inputs, recipients, election_id, metadata=None): (inputs, outputs) = cls.validate_transfer(inputs, recipients, election_id, metadata) - election_vote = cls(cls.VALIDATOR_ELECTION_VOTE, {'id': election_id}, inputs, outputs, metadata) + election_vote = cls(cls.OPERATION, {'id': election_id}, inputs, outputs, metadata) cls.validate_schema(election_vote.to_dict(), skip_id=True) return election_vote @classmethod def validate_schema(cls, tx, skip_id=False): - """Validate the validator election vote transaction. Since `VALIDATOR_ELECTION_VOTE` extends `TRANFER` + """Validate the validator election vote transaction. Since `VOTE` extends `TRANSFER` transaction, all the validations for `CREATE` transaction should be inherited """ if not skip_id: cls.validate_id(tx) _validate_schema(TX_SCHEMA_COMMON, tx) _validate_schema(TX_SCHEMA_TRANSFER, tx) - _validate_schema(TX_SCHEMA_VALIDATOR_ELECTION_VOTE, tx) + _validate_schema(cls.TX_SCHEMA_CUSTOM, tx) @classmethod def create(cls, tx_signers, recipients, metadata=None, asset=None): diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index fae4c76a..f6761aa6 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -428,24 +428,20 @@ class BigchainDB(object): result = self.get_validator_change(height) return [] if result is None else result['validators'] - def get_validators_by_election_id(self, election_id): - result = backend.query.get_validator_set_by_election_id(self.connection, election_id) + def get_election(self, election_id): + result = backend.query.get_election(self.connection, election_id) return result - def delete_validator_update(self): - return backend.query.delete_validator_update(self.connection) - def store_pre_commit_state(self, state): return backend.query.store_pre_commit_state(self.connection, state) - def store_validator_set(self, height, validators, election_id): + def store_validator_set(self, height, validators): """Store validator set at a given `height`. NOTE: If the validator set already exists at that `height` then an exception will be raised. """ return backend.query.store_validator_set(self.connection, {'height': height, - 'validators': validators, - 'election_id': election_id}) + 'validators': validators}) def store_abci_chain(self, height, chain_id, is_synced=True): return backend.query.store_abci_chain(self.connection, height, @@ -478,6 +474,14 @@ class BigchainDB(object): self.store_abci_chain(block['height'] + 1, new_chain_id, False) + def store_election_results(self, height, election): + """Store election results + :param height: the block height at which the election concluded + :param election: a concluded election + """ + return backend.query.store_election_results(self.connection, {'height': height, + 'election_id': election.id}) + Block = namedtuple('Block', ('app_hash', 'height', 'transactions')) diff --git a/bigchaindb/upsert_validator/__init__.py b/bigchaindb/upsert_validator/__init__.py index 90a02a0b..0aa49f35 100644 --- a/bigchaindb/upsert_validator/__init__.py +++ b/bigchaindb/upsert_validator/__init__.py @@ -3,5 +3,4 @@ # Code is Apache-2.0 and docs are CC-BY-4.0 -from bigchaindb.upsert_validator.validator_election_vote import ValidatorElectionVote # noqa from bigchaindb.upsert_validator.validator_election import ValidatorElection # noqa diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index 0b36c268..3daf22eb 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -2,264 +2,48 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -import base58 - -from bigchaindb import backend -from bigchaindb.common.exceptions import (InvalidSignature, - MultipleInputsError, - InvalidProposer, - UnequalValidatorSet, - InvalidPowerChange, - DuplicateTransaction) -from bigchaindb.tendermint_utils import key_from_base64 -from bigchaindb.common.crypto import (public_key_from_ed25519_key) -from bigchaindb.common.transaction import Transaction -from bigchaindb.common.schema import (_validate_schema, - TX_SCHEMA_VALIDATOR_ELECTION, - TX_SCHEMA_COMMON, - TX_SCHEMA_CREATE) -from . import ValidatorElectionVote -from .validator_utils import (new_validator_set, - encode_validator, - encode_pk_to_base16, - validate_asset_public_key) +from bigchaindb.common.exceptions import InvalidPowerChange +from bigchaindb.elections.election import Election +from bigchaindb.common.schema import (TX_SCHEMA_VALIDATOR_ELECTION) +from .validator_utils import (new_validator_set, encode_validator, validate_asset_public_key) -class ValidatorElection(Transaction): +class ValidatorElection(Election): - VALIDATOR_ELECTION = 'VALIDATOR_ELECTION' + OPERATION = 'VALIDATOR_ELECTION' # NOTE: this transaction class extends create so the operation inheritence is achieved # by renaming CREATE to VALIDATOR_ELECTION - CREATE = VALIDATOR_ELECTION - ALLOWED_OPERATIONS = (VALIDATOR_ELECTION,) - # Election Statuses: - ONGOING = 'ongoing' - CONCLUDED = 'concluded' - INCONCLUSIVE = 'inconclusive' - ELECTION_THRESHOLD = 2 / 3 - - @classmethod - def get_validator_change(cls, bigchain, height=None): - """Return the latest change to the validator set - - :return: { - 'height': , - 'asset': { - 'height': , - 'validators': , - 'election_id': - } - } - """ - return bigchain.get_validator_change(height) - - @classmethod - def get_validators(cls, bigchain, height=None): - """Return a dictionary of validators with key as `public_key` and - value as the `voting_power` - """ - validators = {} - for validator in bigchain.get_validators(height): - # NOTE: we assume that Tendermint encodes public key in base64 - public_key = public_key_from_ed25519_key(key_from_base64(validator['public_key']['value'])) - validators[public_key] = validator['voting_power'] - - return validators - - @classmethod - def recipients(cls, bigchain): - """Convert validator dictionary to a recipient list for `Transaction`""" - - recipients = [] - for public_key, voting_power in cls.get_validators(bigchain).items(): - recipients.append(([public_key], voting_power)) - - return recipients - - @classmethod - def is_same_topology(cls, current_topology, election_topology): - voters = {} - for voter in election_topology: - if len(voter.public_keys) > 1: - return False - - [public_key] = voter.public_keys - voting_power = voter.amount - voters[public_key] = voting_power - - # Check whether the voters and their votes is same to that of the - # validators and their voting power in the network - return (current_topology == voters) + CREATE = OPERATION + ALLOWED_OPERATIONS = (OPERATION,) + TX_SCHEMA_CUSTOM = TX_SCHEMA_VALIDATOR_ELECTION def validate(self, bigchain, current_transactions=[]): - """Validate election transaction - For more details refer BEP-21: https://github.com/bigchaindb/BEPs/tree/master/21 - - NOTE: - * A valid election is initiated by an existing validator. - - * A valid election is one where voters are validators and votes are - alloacted according to the voting power of each validator node. - - Args: - bigchain (BigchainDB): an instantiated bigchaindb.lib.BigchainDB object. - - Returns: - ValidatorElection object - - Raises: - ValidationError: If the election is invalid + """For more details refer BEP-21: https://github.com/bigchaindb/BEPs/tree/master/21 """ - input_conditions = [] - - duplicates = any(txn for txn in current_transactions if txn.id == self.id) - if bigchain.get_transaction(self.id) or duplicates: - raise DuplicateTransaction('transaction `{}` already exists' - .format(self.id)) - - if not self.inputs_valid(input_conditions): - raise InvalidSignature('Transaction signature is invalid.') current_validators = self.get_validators(bigchain) - # NOTE: Proposer should be a single node - if len(self.inputs) != 1 or len(self.inputs[0].owners_before) != 1: - raise MultipleInputsError('`tx_signers` must be a list instance of length one') + super(ValidatorElection, self).validate(bigchain, current_transactions=current_transactions) # NOTE: change more than 1/3 of the current power is not allowed if self.asset['data']['power'] >= (1/3)*sum(current_validators.values()): raise InvalidPowerChange('`power` change must be less than 1/3 of total power') - # NOTE: Check if the proposer is a validator. - [election_initiator_node_pub_key] = self.inputs[0].owners_before - if election_initiator_node_pub_key not in current_validators.keys(): - raise InvalidProposer('Public key is not a part of the validator set') - - # NOTE: Check if all validators have been assigned votes equal to their voting power - if not self.is_same_topology(current_validators, self.outputs): - raise UnequalValidatorSet('Validator set much be exactly same to the outputs of election') - return self @classmethod - def generate(cls, initiator, voters, election_data, metadata=None): - (inputs, outputs) = cls.validate_create(initiator, voters, election_data, metadata) - election = cls(cls.VALIDATOR_ELECTION, {'data': election_data}, inputs, outputs, metadata) - cls.validate_schema(election.to_dict(), skip_id=True) - return election - - @classmethod - def validate_schema(cls, tx, skip_id=False): - """Validate the validator election transaction. Since `VALIDATOR_ELECTION` extends `CREATE` - transaction, all the validations for `CREATE` transaction should be inherited - """ - if not skip_id: - cls.validate_id(tx) - _validate_schema(TX_SCHEMA_COMMON, tx) - _validate_schema(TX_SCHEMA_CREATE, tx) - _validate_schema(TX_SCHEMA_VALIDATOR_ELECTION, tx) + def validate_schema(cls, tx): + super(ValidatorElection, cls).validate_schema(tx) validate_asset_public_key(tx['asset']['data']['public_key']) @classmethod - def create(cls, tx_signers, recipients, metadata=None, asset=None): - raise NotImplementedError + def on_approval(cls, bigchain, election, new_height): + # The new validator set comes into effect from height = new_height+1 + validator_updates = [election.asset['data']] + curr_validator_set = bigchain.get_validators(new_height) + updated_validator_set = new_validator_set(curr_validator_set, + validator_updates) - @classmethod - def transfer(cls, tx_signers, recipients, metadata=None, asset=None): - raise NotImplementedError - - @classmethod - def to_public_key(cls, election_id): - return base58.b58encode(bytes.fromhex(election_id)).decode() - - @classmethod - def count_votes(cls, election_pk, transactions, getter=getattr): - votes = 0 - for txn in transactions: - if getter(txn, 'operation') == 'VALIDATOR_ELECTION_VOTE': - for output in getter(txn, 'outputs'): - # NOTE: We enforce that a valid vote to election id will have only - # election_pk in the output public keys, including any other public key - # along with election_pk will lead to vote being not considered valid. - if len(getter(output, 'public_keys')) == 1 and [election_pk] == getter(output, 'public_keys'): - votes = votes + int(getter(output, 'amount')) - return votes - - def get_commited_votes(self, bigchain, election_pk=None): - if election_pk is None: - election_pk = self.to_public_key(self.id) - txns = list(backend.query.get_asset_tokens_for_public_key(bigchain.connection, - self.id, - election_pk)) - return self.count_votes(election_pk, txns, dict.get) - - @classmethod - def has_concluded(cls, bigchain, election_id, current_votes=[], height=None): - """Check if the given `election_id` can be concluded or not - NOTE: - * Election is concluded iff the current validator set is exactly equal - to the validator set encoded in election outputs - * Election can concluded only if the current votes achieves a supermajority - """ - election = bigchain.get_transaction(election_id) - - if election: - election_pk = election.to_public_key(election.id) - votes_commited = election.get_commited_votes(bigchain, election_pk) - votes_current = election.count_votes(election_pk, current_votes) - current_validators = election.get_validators(bigchain, height) - - if election.is_same_topology(current_validators, election.outputs): - total_votes = sum(current_validators.values()) - if (votes_commited < (2/3)*total_votes) and \ - (votes_commited + votes_current >= (2/3)*total_votes): - return election - return False - - @classmethod - def get_validator_update(cls, bigchain, new_height, txns): - votes = {} - for txn in txns: - if not isinstance(txn, ValidatorElectionVote): - continue - - election_id = txn.asset['id'] - election_votes = votes.get(election_id, []) - election_votes.append(txn) - votes[election_id] = election_votes - - election = cls.has_concluded(bigchain, election_id, election_votes, new_height) - # Once an election concludes any other conclusion for the same - # or any other election is invalidated - if election: - # The new validator set comes into effect from height = new_height+1 - validator_updates = [election.asset['data']] - curr_validator_set = bigchain.get_validators(new_height) - updated_validator_set = new_validator_set(curr_validator_set, - validator_updates) - - updated_validator_set = [v for v in updated_validator_set if v['voting_power'] > 0] - bigchain.store_validator_set(new_height+1, updated_validator_set, election.id) - - validator16 = encode_pk_to_base16(election.asset['data']) - return [encode_validator(validator16)] - - return [] - - def get_validator_update_by_election_id(self, election_id, bigchain): - result = bigchain.get_validators_by_election_id(election_id) - return result - - def get_status(self, bigchain): - concluded = self.get_validator_update_by_election_id(self.id, bigchain) - if concluded: - return self.CONCLUDED - - latest_change = self.get_validator_change(bigchain) - latest_change_height = latest_change['height'] - election_height = bigchain.get_block_containing_tx(self.id)[0] - - if latest_change_height >= election_height: - return self.INCONCLUSIVE - else: - return self.ONGOING + updated_validator_set = [v for v in updated_validator_set if v['voting_power'] > 0] + bigchain.store_validator_set(new_height+1, updated_validator_set) + return encode_validator(election.asset['data']) diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index 0d49c4ec..21a9ac4f 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -21,7 +21,7 @@ def test_init_creates_db_tables_and_indexes(): collection_names = conn.conn[dbname].collection_names() assert set(collection_names) == { 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'pre_commit', - 'validators', 'abci_chains', + 'validators', 'elections', 'abci_chains', } indexes = conn.conn[dbname]['assets'].index_information().keys() @@ -46,6 +46,9 @@ def test_init_creates_db_tables_and_indexes(): indexes = conn.conn[dbname]['abci_chains'].index_information().keys() assert set(indexes) == {'_id_', 'height', 'chain_id'} + indexes = conn.conn[dbname]['elections'].index_information().keys() + assert set(indexes) == {'_id_', 'election_id'} + def test_init_database_fails_if_db_exists(): import bigchaindb @@ -78,7 +81,7 @@ def test_create_tables(): collection_names = conn.conn[dbname].collection_names() assert set(collection_names) == { - 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'validators', + 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'validators', 'elections', 'pre_commit', 'abci_chains', } @@ -117,6 +120,10 @@ def test_create_secondary_indexes(): assert index_info['utxo']['key'] == [('transaction_id', 1), ('output_index', 1)] + indexes = conn.conn[dbname]['elections'].index_information() + assert set(indexes.keys()) == {'_id_', 'election_id'} + assert indexes['election_id']['unique'] + indexes = conn.conn[dbname]['pre_commit'].index_information() assert set(indexes.keys()) == {'_id_', 'pre_commit_id'} assert indexes['pre_commit_id']['unique'] diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index b59cc0c6..31b4e100 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -366,8 +366,8 @@ def test_end_block_return_validator_updates(b, init_chain_request): resp = app.end_block(RequestEndBlock(height=99)) assert resp.validator_updates[0] == encode_validator(validator) - updates = b.get_validator_update() - assert updates == [] + updates = b.approved_update() + assert not updates def test_store_pre_commit_state_in_end_block(b, alice, init_chain_request): @@ -427,7 +427,7 @@ def test_new_validator_set(b): validators = [node1] updates = [node1_new_power, node2] - b.store_validator_set(1, validators, 'election_id') + b.store_validator_set(1, validators) updated_validator_set = new_validator_set(b.get_validators(1), updates) updated_validators = [] diff --git a/tests/tendermint/test_lib.py b/tests/tendermint/test_lib.py index 6b384b33..3eec8903 100644 --- a/tests/tendermint/test_lib.py +++ b/tests/tendermint/test_lib.py @@ -161,11 +161,11 @@ def test_validator_updates(b, validator_pub_key): 'update_id': VALIDATOR_UPDATE_ID} query.store_validator_update(b.connection, validator_update) - updates = b.get_validator_update() - assert updates == [validator_update['validator']] + updates = b.approved_updates() + assert updates == validator_update['validator'] b.delete_validator_update() - assert b.get_validator_update() == [] + assert not b.approved_updates() @pytest.mark.bdb diff --git a/tests/upsert_validator/conftest.py b/tests/upsert_validator/conftest.py index c9bb4870..9ab2dad8 100644 --- a/tests/upsert_validator/conftest.py +++ b/tests/upsert_validator/conftest.py @@ -1,10 +1,11 @@ # Copyright BigchainDB GmbH and BigchainDB contributors # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 +from unittest.mock import patch import pytest -from bigchaindb import ValidatorElectionVote +from bigchaindb import Vote from bigchaindb.backend.localmongodb import query from bigchaindb.lib import Block from bigchaindb.upsert_validator import ValidatorElection @@ -46,6 +47,15 @@ def valid_election_b(b, node_key, new_validator): new_validator, None).sign([node_key.private_key]) +@pytest.fixture +@patch('bigchaindb.elections.election.uuid4', lambda: 'mock_uuid4') +def fixed_seed_election(b_mock, node_key, new_validator): + voters = ValidatorElection.recipients(b_mock) + return ValidatorElection.generate([node_key.public_key], + voters, + new_validator, None).sign([node_key.private_key]) + + @pytest.fixture def ongoing_election(b, valid_election, ed25519_node_keys): validators = b.get_validators(height=1) @@ -62,12 +72,10 @@ def ongoing_election(b, valid_election, ed25519_node_keys): @pytest.fixture def concluded_election(b, ongoing_election, ed25519_node_keys): - validators = b.get_validators(height=1) - validator_update = {'validators': validators, - 'height': 2, - 'election_id': ongoing_election.id} + election_result = {'height': 2, + 'election_id': ongoing_election.id} - query.store_validator_set(b.connection, validator_update) + query.store_election_results(b.connection, election_result) return ongoing_election @@ -91,9 +99,9 @@ def vote(election, voter, keys, b): election_pub_key = ValidatorElection.to_public_key(election.id) - v = ValidatorElectionVote.generate([election_input], - [([election_pub_key], votes)], - election_id=election.id)\ - .sign([key.private_key]) + v = Vote.generate([election_input], + [([election_pub_key], votes)], + election_id=election.id)\ + .sign([key.private_key]) b.store_bulk_transactions([v]) return v diff --git a/tests/upsert_validator/test_validator_election_vote.py b/tests/upsert_validator/test_upsert_validator_vote.py similarity index 73% rename from tests/upsert_validator/test_validator_election_vote.py rename to tests/upsert_validator/test_upsert_validator_vote.py index c395bbc1..8567397c 100644 --- a/tests/upsert_validator/test_validator_election_vote.py +++ b/tests/upsert_validator/test_upsert_validator_vote.py @@ -6,10 +6,11 @@ import pytest import codecs from bigchaindb.tendermint_utils import public_key_to_base64 -from bigchaindb.upsert_validator import ValidatorElection, ValidatorElectionVote +from bigchaindb.upsert_validator import ValidatorElection from bigchaindb.common.exceptions import AmountError from bigchaindb.common.crypto import generate_key_pair from bigchaindb.common.exceptions import ValidationError +from bigchaindb.elections.vote import Vote from tests.utils import generate_block pytestmark = [pytest.mark.execute] @@ -26,10 +27,10 @@ def test_upsert_validator_valid_election_vote(b_mock, valid_election, ed25519_no election_pub_key = ValidatorElection.to_public_key(valid_election.id) - vote = ValidatorElectionVote.generate([input0], - [([election_pub_key], votes)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + vote = Vote.generate([input0], + [([election_pub_key], votes)], + election_id=valid_election.id)\ + .sign([key0.private_key]) assert vote.validate(b_mock) @@ -46,10 +47,10 @@ def test_upsert_validator_valid_non_election_vote(b_mock, valid_election, ed2551 # Ensure that threshold conditions are now allowed with pytest.raises(ValidationError): - ValidatorElectionVote.generate([input0], - [([election_pub_key, key0.public_key], votes)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + Vote.generate([input0], + [([election_pub_key, key0.public_key], votes)], + election_id=valid_election.id)\ + .sign([key0.private_key]) @pytest.mark.bdb @@ -63,10 +64,10 @@ def test_upsert_validator_delegate_election_vote(b_mock, valid_election, ed25519 public_key0 = input0.owners_before[0] key0 = ed25519_node_keys[public_key0] - delegate_vote = ValidatorElectionVote.generate([input0], - [([alice.public_key], 3), ([key0.public_key], votes-3)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + delegate_vote = Vote.generate([input0], + [([alice.public_key], 3), ([key0.public_key], votes-3)], + election_id=valid_election.id)\ + .sign([key0.private_key]) assert delegate_vote.validate(b_mock) @@ -74,17 +75,17 @@ def test_upsert_validator_delegate_election_vote(b_mock, valid_election, ed25519 election_pub_key = ValidatorElection.to_public_key(valid_election.id) alice_votes = delegate_vote.to_inputs()[0] - alice_casted_vote = ValidatorElectionVote.generate([alice_votes], - [([election_pub_key], 3)], - election_id=valid_election.id)\ - .sign([alice.private_key]) + alice_casted_vote = Vote.generate([alice_votes], + [([election_pub_key], 3)], + election_id=valid_election.id)\ + .sign([alice.private_key]) assert alice_casted_vote.validate(b_mock) key0_votes = delegate_vote.to_inputs()[1] - key0_casted_vote = ValidatorElectionVote.generate([key0_votes], - [([election_pub_key], votes-3)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + key0_casted_vote = Vote.generate([key0_votes], + [([election_pub_key], votes-3)], + election_id=valid_election.id)\ + .sign([key0.private_key]) assert key0_casted_vote.validate(b_mock) @@ -99,10 +100,10 @@ def test_upsert_validator_invalid_election_vote(b_mock, valid_election, ed25519_ election_pub_key = ValidatorElection.to_public_key(valid_election.id) - vote = ValidatorElectionVote.generate([input0], - [([election_pub_key], votes+1)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + vote = Vote.generate([input0], + [([election_pub_key], votes+1)], + election_id=valid_election.id)\ + .sign([key0.private_key]) with pytest.raises(AmountError): assert vote.validate(b_mock) @@ -120,10 +121,10 @@ def test_valid_election_votes_received(b_mock, valid_election, ed25519_node_keys key0 = ed25519_node_keys[public_key0] # delegate some votes to alice - delegate_vote = ValidatorElectionVote.generate([input0], - [([alice.public_key], 4), ([key0.public_key], votes-4)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + delegate_vote = Vote.generate([input0], + [([alice.public_key], 4), ([key0.public_key], votes-4)], + election_id=valid_election.id)\ + .sign([key0.private_key]) b_mock.store_bulk_transactions([delegate_vote]) assert valid_election.get_commited_votes(b_mock) == 0 @@ -131,10 +132,10 @@ def test_valid_election_votes_received(b_mock, valid_election, ed25519_node_keys alice_votes = delegate_vote.to_inputs()[0] key0_votes = delegate_vote.to_inputs()[1] - alice_casted_vote = ValidatorElectionVote.generate([alice_votes], - [([election_public_key], 2), ([alice.public_key], 2)], - election_id=valid_election.id)\ - .sign([alice.private_key]) + alice_casted_vote = Vote.generate([alice_votes], + [([election_public_key], 2), ([alice.public_key], 2)], + election_id=valid_election.id)\ + .sign([alice.private_key]) assert alice_casted_vote.validate(b_mock) b_mock.store_bulk_transactions([alice_casted_vote]) @@ -142,10 +143,10 @@ def test_valid_election_votes_received(b_mock, valid_election, ed25519_node_keys # Check if the delegated vote is count as valid vote assert valid_election.get_commited_votes(b_mock) == 2 - key0_casted_vote = ValidatorElectionVote.generate([key0_votes], - [([election_public_key], votes-4)], - election_id=valid_election.id)\ - .sign([key0.private_key]) + key0_casted_vote = Vote.generate([key0_votes], + [([election_public_key], votes-4)], + election_id=valid_election.id)\ + .sign([key0.private_key]) assert key0_casted_vote.validate(b_mock) b_mock.store_bulk_transactions([key0_casted_vote]) @@ -227,7 +228,7 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): latest_block = b.get_latest_block() # reset the validator set - b.store_validator_set(latest_block['height'], validators, 'previous_election_id') + b.store_validator_set(latest_block['height'], validators) power = 1 public_key = '9B3119650DF82B9A5D8A12E38953EA47475C09F0C48A4E6A0ECE182944B24403' @@ -291,19 +292,19 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): assert not ValidatorElection.has_concluded(b, election.id, [tx_vote0, tx_vote1]) assert ValidatorElection.has_concluded(b, election.id, [tx_vote0, tx_vote1, tx_vote2]) - assert ValidatorElection.get_validator_update(b, 4, [tx_vote0]) == [] - assert ValidatorElection.get_validator_update(b, 4, [tx_vote0, tx_vote1]) == [] + assert not ValidatorElection.approved_update(b, 4, [tx_vote0]) + assert not ValidatorElection.approved_update(b, 4, [tx_vote0, tx_vote1]) - update = ValidatorElection.get_validator_update(b, 4, [tx_vote0, tx_vote1, tx_vote2]) - update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') - assert len(update) == 1 + update = ValidatorElection.approved_update(b, 4, [tx_vote0, tx_vote1, tx_vote2]) + assert update + update_public_key = codecs.encode(update.pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 b.store_bulk_transactions([tx_vote0, tx_vote1]) - update = ValidatorElection.get_validator_update(b, 4, [tx_vote2]) - update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') - assert len(update) == 1 + update = ValidatorElection.approved_update(b, 4, [tx_vote2]) + assert update + update_public_key = codecs.encode(update.pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 # remove validator @@ -324,9 +325,10 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): b.store_bulk_transactions([tx_vote0, tx_vote1]) - update = ValidatorElection.get_validator_update(b, 9, [tx_vote2]) - update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') - assert len(update) == 1 + update = ValidatorElection.approved_update(b, 9, [tx_vote2]) + if update: + update_public_key = codecs.encode(update.pub_key.data, 'base64').decode().rstrip('\n') + assert update assert update_public_key == public_key64 # assert that the public key is not a part of the current validator set @@ -348,10 +350,10 @@ def to_inputs(election, i, ed25519_node_keys): def gen_vote(election, i, ed25519_node_keys): (input_i, votes_i, key_i) = to_inputs(election, i, ed25519_node_keys) election_pub_key = ValidatorElection.to_public_key(election.id) - return ValidatorElectionVote.generate([input_i], - [([election_pub_key], votes_i)], - election_id=election.id)\ - .sign([key_i.private_key]) + return Vote.generate([input_i], + [([election_pub_key], votes_i)], + election_id=election.id)\ + .sign([key_i.private_key]) def reset_validator_set(b, node_keys, height): @@ -360,4 +362,4 @@ def reset_validator_set(b, node_keys, height): validators.append({'public_key': {'type': 'ed25519-base64', 'value': node_pub}, 'voting_power': 10}) - b.store_validator_set(height, validators, 'election_id') + b.store_validator_set(height, validators) diff --git a/tests/upsert_validator/test_validator_election.py b/tests/upsert_validator/test_validator_election.py index bad475a3..fe30331a 100644 --- a/tests/upsert_validator/test_validator_election.py +++ b/tests/upsert_validator/test_validator_election.py @@ -2,6 +2,7 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 from argparse import Namespace +from unittest.mock import patch import pytest @@ -72,16 +73,17 @@ def test_upsert_validator_invalid_inputs_election(b_mock, new_validator, node_ke election.validate(b_mock) -def test_upsert_validator_invalid_election(b_mock, new_validator, node_key, valid_election): +@patch('bigchaindb.elections.election.uuid4', lambda: 'mock_uuid4') +def test_upsert_validator_invalid_election(b_mock, new_validator, node_key, fixed_seed_election): voters = ValidatorElection.recipients(b_mock) duplicate_election = ValidatorElection.generate([node_key.public_key], voters, new_validator, None).sign([node_key.private_key]) with pytest.raises(DuplicateTransaction): - valid_election.validate(b_mock, [duplicate_election]) + fixed_seed_election.validate(b_mock, [duplicate_election]) - b_mock.store_bulk_transactions([valid_election]) + b_mock.store_bulk_transactions([fixed_seed_election]) with pytest.raises(DuplicateTransaction): duplicate_election.validate(b_mock) diff --git a/tests/web/test_validators.py b/tests/web/test_validators.py index 0574204f..1daeb18f 100644 --- a/tests/web/test_validators.py +++ b/tests/web/test_validators.py @@ -10,7 +10,7 @@ def test_get_validators_endpoint(b, client): 'pub_key': {'data': '4E2685D9016126864733225BE00F005515200727FBAB1312FC78C8B76831255A', 'type': 'ed25519'}, 'voting_power': 10}] - b.store_validator_set(23, validator_set, 'election_id') + b.store_validator_set(23, validator_set) res = client.get(VALIDATORS_ENDPOINT) assert is_validator(res.json[0]) From b33e3808a65bd4b9f3f861395378dab29383d08e Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Tue, 11 Sep 2018 16:41:27 +0200 Subject: [PATCH 20/77] Problem: Test fails if Tendermint is not up. (#2530) Solution: Mark the test with the abci mark since it requires running BigchainDB and Tendermint. Fails since the mark cleanup in #2522. --- tests/web/test_outputs.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/web/test_outputs.py b/tests/web/test_outputs.py index db7b718d..77a3f258 100644 --- a/tests/web/test_outputs.py +++ b/tests/web/test_outputs.py @@ -71,6 +71,7 @@ def test_get_outputs_endpoint_with_invalid_spent(client, user_pk): assert res.status_code == 400 +@pytest.mark.abci @pytest.mark.bdb @pytest.mark.usefixtures('inputs') def test_get_divisble_transactions_returns_500(b, client): From 5a440843b69a1519a81a437153b7416007df0056 Mon Sep 17 00:00:00 2001 From: Muawia Khan Date: Wed, 12 Sep 2018 11:14:51 +0200 Subject: [PATCH 21/77] Problem: No log rotation when BigchainDB and Tendermint started with Monit (#2528) * Log rotation for tendermint and BigchainDB using Monit * Change file size from 100KB used for testing to 200 MB for test envs * Add docs for log rotation * remove --consensus.create_empty_blocks from monit start Because already taken care of in the config file * do not run bigchaindb with debug flag instead use config file * Address comments I - Rephrase log rotation to imply monit is not default - Add statement to explain where bigchaindb log files are created - Fix HTML comment - Add log rotation to index * Address comments II --- docs/server/source/appendices/index.rst | 1 + docs/server/source/appendices/log-rotation.md | 48 +++++++ .../network-setup.md | 4 + pkg/scripts/bigchaindb-monit-config | 126 +++++++++++------- 4 files changed, 129 insertions(+), 50 deletions(-) create mode 100644 docs/server/source/appendices/log-rotation.md diff --git a/docs/server/source/appendices/index.rst b/docs/server/source/appendices/index.rst index 24276f2d..ac67be25 100755 --- a/docs/server/source/appendices/index.rst +++ b/docs/server/source/appendices/index.rst @@ -17,3 +17,4 @@ Appendices ntp-notes licenses all-in-one-bigchaindb + log-rotation diff --git a/docs/server/source/appendices/log-rotation.md b/docs/server/source/appendices/log-rotation.md new file mode 100644 index 00000000..25c0cd6e --- /dev/null +++ b/docs/server/source/appendices/log-rotation.md @@ -0,0 +1,48 @@ + + +# Log rotation for a BigchainDB node + +Each BigchainDB node comprises of 3 main services: + - BigchainDB server + - Tendermint + - MongoDB + +To run a BigchainDB test network/dev node, that is expected to run for relatively longer periods +of time, we need to consider the log rotation of these services i.e. we do not want the logs taking +up large amounts of storage and making the node unresponsive or get into bad state. + +## Log rotation for MongoDB + +Currently, we leave the log rotation of MongoDB to the BigchainDB administrator. For more notes on MongoDB log rotation +please refer to [MongoDB docs](https://docs.mongodb.com/v3.6/tutorial/rotate-log-files/). + +## Log rotation for BigchainDB + +Log rotation is baked into BigchainDB server using the `logging` module. BigchainDB server logs information into the following files: + - `bigchaindb.log` + - `bigchaindb-errors.log` + - `bigchaindb-benchmark.log` + +These log files are created by default in the directory from where you run `bigchaindb start`, if you are using `monit`, from +[How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide, the default directory is: `$HOME/.bigchaindb-monit/logs` + +The logs for BigchainDB server are rotated when any of the above mentioned file exceeds `209715200 bytes (i.e. approximately 209 MB).`. + + +## Log rotation for Tendermint + +In order to set up log rotation of Tendermint, you will need to use the [Monit]( https://www.mmonit.com/monit) scripts provided by us. Covered in the [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide. + +```bash +$ monit -d 1 +``` + +Monit monitors both Tendermint and BigchainDB processes as well as the Tendermint log files, `tendermint.out.log` and `tendermint.err.log`. Default location for these log files is: +`$HOME/.bigchaindb-monit/logs`. + +Tendermint logs are rotated if any of the above mentioned log files exceeds `200 MB` in size. + diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index ed6498fc..9536c65e 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -390,6 +390,10 @@ If you started BigchainDB in the foreground, a `Ctrl + C` or `Ctrl + Z` would sh TBD. +## Log rotation + +Please go over the [document describing log rotation of a BigchainDB node](../appendices/log-rotation.md). + [bdb:software]: https://github.com/bigchaindb/bigchaindb/ [bdb:pypi]: https://pypi.org/project/BigchainDB/#history diff --git a/pkg/scripts/bigchaindb-monit-config b/pkg/scripts/bigchaindb-monit-config index 14caaf95..90225328 100644 --- a/pkg/scripts/bigchaindb-monit-config +++ b/pkg/scripts/bigchaindb-monit-config @@ -4,7 +4,7 @@ set -o nounset # Check if directory for monit logs exists if [ ! -d "$HOME/.bigchaindb-monit" ]; then - mkdir -p "$HOME/.bigchaindb-monit" + mkdir -p "$HOME/.bigchaindb-monit" fi monit_pid_path=${MONIT_PID_PATH:=$HOME/.bigchaindb-monit/monit_processes} @@ -13,7 +13,7 @@ monit_log_path=${MONIT_LOG_PATH:=$HOME/.bigchaindb-monit/logs} monitrc_path=${MONITRC_PATH:=$HOME/.monitrc} function usage() { - cat <${monit_script_path} <> \$3/bigchaindb.out.log 2>> \$3/bigchaindb.err.log & + nohup bigchaindb start > /dev/null 2>&1 & echo \$! > \$2 popd @@ -110,7 +110,8 @@ case \$1 in start_tendermint) pushd \$4 - nohup tendermint node --consensus.create_empty_blocks=false >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log & + + nohup tendermint node >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log & echo \$! > \$2 popd @@ -129,21 +130,37 @@ exit 0 EOF chmod +x ${monit_script_path} +cat >${monit_script_path}_logrotate < 200 MB then + exec "${monit_script_path}_logrotate rotate_tendermint_logs ${monit_log_path}/tendermint.out.log $monit_pid_path/tendermint.pid" + +check file tendermint.err.log with path ${monit_log_path}/tendermint.err.log + if size > 200 MB then + exec "${monit_script_path}_logrotate rotate_tendermint_logs ${monit_log_path}/tendermint.err.log $monit_pid_path/tendermint.pid" + EOF # Setting permissions for control file From 8a7650c13ab83955f6c4f3f142333c7e371f1b89 Mon Sep 17 00:00:00 2001 From: Zachary Bowen Date: Wed, 12 Sep 2018 13:41:16 +0200 Subject: [PATCH 22/77] Generalize election management commands (#2515) * Problem: `ValidatorElection` and `MigrationElection` need to inherit from a common `Election` class Solution: Factored the common logic out of `ValidatorElection` and moved it to `Election` parent class * Problem: Adding base58 as a requirement seems to break the build... Solution: Reverting the changes * Problem: Responding to a suggestion for improved method naming Solution: Refactored `get_result_by_election_id` to `get_election_result_by_id` * Problem: No need to store different types of elections in their own tables Solution: Remove `DB_TABLE` property from `Election` class * Revert "Problem: No need to store different types of elections in their own tables" This reverts commit db45374d3c690429d18a25bcc319f8056c016500. * Problem: Missed a method in `Bigchain` class when updating the naming for an election method Solution: Finished the refactoring * Problem: Need a table to store data for all elections Solution: Created the `elections` table with secondary_index `election_id` * Problem: `Election` class needs to be updated to store concluded elections in the `elections` table Solution: Updated the class to use the new table * Problem: `UpsertValidatorVote` can be generalized to just be `Vote` Solution: Renamed, refactored and moved the `Vote` class to tie in with the more general `Election` base class * Problem: Error in docstring return signature Solution: Fixed the docstring * Problem: Hardcoded reference to the `VOTE_TYPE` in `Election` base class Solution: Pointed the reference to the class variable * Problem: Schema still refers to `ValidatorElectionVote` instead of `Vote` Solution: Renamed `TX_SCHEMA_VALIDATOR_ELECTION_VOTE` as `TX_SCHEMA_VOTE` * Problem: `Election` class variable `ELECTION_TYPE` is overly specific Solution: Renamed `ELECTION_TYPE` to `OPERATION` * Problem: Command line options for `upsert-validator` can be generalized to manage any type of election Solution: Refactored the CLI to manage generalized elections * Problem: Default for `show_election` not implemented for `Election` class Solution: Create a default method that work if all fields in the 'asset[data]' can be displayed without additional formatting * Problem: Multiple small issues with style etc. Solution: Addressed comments from PR * Problem: `Election` class variable to `VOTE_TYPE` unnecessary Solution: Removed the variable and hardcoded everything to use the `Vote` class * Problem: Minor style issues with PR Solution: Addressing comments * Problem: Changes to format for validator keys broke some tests Solution: Aligned the tests to reflect the changed key format * Problem: `election show` command displaying the base56 public key Solution: Cast any public key to base64 * Problem: `election_parser` help message still refers to upsert-validator Solution: Updated the help message --- bigchaindb/commands/bigchaindb.py | 76 +++++++++---------- bigchaindb/commands/election_types.py | 20 +++++ bigchaindb/elections/election.py | 14 +++- .../source/server-reference/bigchaindb-cli.md | 42 +++++----- tests/commands/test_commands.py | 63 ++++++++------- .../test_validator_election.py | 4 +- 6 files changed, 127 insertions(+), 92 deletions(-) create mode 100644 bigchaindb/commands/election_types.py diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index d548d7b0..d5b2e9ac 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -28,7 +28,8 @@ from bigchaindb.commands import utils from bigchaindb.commands.utils import (configure_bigchaindb, input_on_stderr) from bigchaindb.log import setup_logging -from bigchaindb.tendermint_utils import public_key_from_base64, public_key_to_base64 +from bigchaindb.tendermint_utils import public_key_from_base64 +from bigchaindb.commands.election_types import elections logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) @@ -101,16 +102,20 @@ def run_configure(args): @configure_bigchaindb -def run_upsert_validator(args): - """Initiate and manage elections to change the validator set""" +def run_election(args): + """Initiate and manage elections""" b = BigchainDB() # Call the function specified by args.action, as defined above - globals()[f'run_upsert_validator_{args.action}'](args, b) + globals()[f'run_election_{args.action}'](args, b) -def run_upsert_validator_new(args, bigchain): +def run_election_new(args, bigchain): + globals()[f'run_election_new_{args.election_type}'](args, bigchain) + + +def run_election_new_upsert_validator(args, bigchain): """Initiates an election to add/update/remove a validator to an existing BigchainDB network :param args: dict @@ -154,8 +159,8 @@ def run_upsert_validator_new(args, bigchain): return False -def run_upsert_validator_approve(args, bigchain): - """Approve an election to add/update/remove a validator to an existing BigchainDB network +def run_election_approve(args, bigchain): + """Approve an election :param args: dict args = { @@ -192,8 +197,8 @@ def run_upsert_validator_approve(args, bigchain): return False -def run_upsert_validator_show(args, bigchain): - """Retrieves information about an upsert-validator election +def run_election_show(args, bigchain): + """Retrieves information about an election :param args: dict args = { @@ -207,14 +212,7 @@ def run_upsert_validator_show(args, bigchain): logger.error(f'No election found with election_id {args.election_id}') return - new_validator = election.asset['data'] - - public_key = public_key_to_base64(new_validator['public_key']['value']) - power = new_validator['power'] - node_id = new_validator['node_id'] - status = election.get_status(bigchain) - - response = f'public_key={public_key}\npower={power}\nnode_id={node_id}\nstatus={status}' + response = election.show_election(bigchain) logger.info(response) @@ -317,41 +315,37 @@ def create_parser(): help='The backend to use. It can only be ' '"localmongodb", currently.') - # parser for managing validator elections - validator_parser = subparsers.add_parser('upsert-validator', - help='Add/update/delete a validator.') + # parser for managing elections + election_parser = subparsers.add_parser('election', + help='Manage elections.') - validator_subparser = validator_parser.add_subparsers(title='Action', - dest='action') + election_subparser = election_parser.add_subparsers(title='Action', + dest='action') - new_election_parser = validator_subparser.add_parser('new', - help='Calls a new election.') + new_election_parser = election_subparser.add_parser('new', + help='Calls a new election.') - new_election_parser.add_argument('public_key', - help='Public key of the validator to be added/updated/removed.') + new_election_subparser = new_election_parser.add_subparsers(title='Election_Type', + dest='election_type') - new_election_parser.add_argument('power', - type=int, - help='The proposed power for the validator. ' - 'Setting to 0 will remove the validator.') + # Parser factory for each type of new election, so we get a bunch of commands that look like this: + # election new ... + for name, data in elections.items(): + args = data['args'] + generic_parser = new_election_subparser.add_parser(name, help=data['help']) + for arg, kwargs in args.items(): + generic_parser.add_argument(arg, **kwargs) - new_election_parser.add_argument('node_id', - help='The node_id of the validator.') - - new_election_parser.add_argument('--private-key', - dest='sk', - help='Path to the private key of the election initiator.') - - approve_election_parser = validator_subparser.add_parser('approve', - help='Approve the election.') + approve_election_parser = election_subparser.add_parser('approve', + help='Approve the election.') approve_election_parser.add_argument('election_id', help='The election_id of the election.') approve_election_parser.add_argument('--private-key', dest='sk', help='Path to the private key of the election initiator.') - show_election_parser = validator_subparser.add_parser('show', - help='Provides information about an election.') + show_election_parser = election_subparser.add_parser('show', + help='Provides information about an election.') show_election_parser.add_argument('election_id', help='The transaction id of the election you wish to query.') diff --git a/bigchaindb/commands/election_types.py b/bigchaindb/commands/election_types.py new file mode 100644 index 00000000..b57522f6 --- /dev/null +++ b/bigchaindb/commands/election_types.py @@ -0,0 +1,20 @@ +elections = { + 'upsert-validator': { + 'help': 'Propose a change to the validator set', + 'args': { + 'public_key': { + 'help': 'Public key of the validator to be added/updated/removed.' + }, + 'power': { + 'type': int, + 'help': 'The proposed power for the validator. Setting to 0 will remove the validator.'}, + 'node_id': { + 'help': 'The node_id of the validator.' + }, + '--private-key': { + 'dest': 'sk', + 'help': 'Path to the private key of the election initiator.' + } + } + } +} diff --git a/bigchaindb/elections/election.py b/bigchaindb/elections/election.py index 0c0219bd..b78e1456 100644 --- a/bigchaindb/elections/election.py +++ b/bigchaindb/elections/election.py @@ -12,7 +12,7 @@ from bigchaindb.common.exceptions import (InvalidSignature, InvalidProposer, UnequalValidatorSet, DuplicateTransaction) -from bigchaindb.tendermint_utils import key_from_base64 +from bigchaindb.tendermint_utils import key_from_base64, public_key_to_base64 from bigchaindb.common.crypto import (public_key_from_ed25519_key) from bigchaindb.common.transaction import Transaction from bigchaindb.common.schema import (_validate_schema, @@ -229,6 +229,18 @@ class Election(Transaction): def store_election_results(cls, bigchain, election, height): bigchain.store_election_results(height, election) + def show_election(self, bigchain): + data = self.asset['data'] + if 'public_key' in data.keys(): + data['public_key'] = public_key_to_base64(data['public_key']['value']) + response = '' + for k, v in data.items(): + if k != 'seed': + response += f'{k}={v}\n' + response += f'status={self.get_status(bigchain)}' + + return response + @classmethod def approved_update(cls, bigchain, new_height, txns): votes = {} diff --git a/docs/server/source/server-reference/bigchaindb-cli.md b/docs/server/source/server-reference/bigchaindb-cli.md index 80dbccdb..b67ca296 100644 --- a/docs/server/source/server-reference/bigchaindb-cli.md +++ b/docs/server/source/server-reference/bigchaindb-cli.md @@ -81,20 +81,25 @@ configuration file as documented under [Configuration Settings](configuration.html). -## bigchaindb upsert-validator +## bigchaindb election -Manage elections to add, update, or remove a validator from the validators set. The upsert-validator subcommands implement [BEP-21](https://github.com/bigchaindb/BEPs/tree/master/21), please refer it for more details. +Manage elections to manage the BigChainDB network. The specifics of the election process are defined in [BEP-18](https://github.com/bigchaindb/BEPs/tree/master/18), please refer it for more details. Election management is broken into several subcommands. Below is the command line syntax for each, -#### upsert-validator new +#### election new -Create a new election which proposes a change to the validator set. An election can be used to add/update/remove a validator from the validator set. +Create a new election which proposes a change to your BigChainDB network. + +There are multiple types of election, which each take different parameters. Below is a short description of each type of election, as well as their command line syntax and the return value. + +###### election new upsert-validator + +Create an election to add/update/remove a validator from the validator set. -Below is the command line syntax and the return value, ```bash -$ bigchaindb upsert-validator new E_PUBKEY E_POWER E_NODE_ID --private-key PATH_TO_YOUR_PRIVATE_KEY +$ bigchaindb election new upsert-validator E_PUBKEY E_POWER E_NODE_ID --private-key PATH_TO_YOUR_PRIVATE_KEY [SUCCESS] Submitted proposal with id: ``` @@ -109,7 +114,7 @@ NOTE: A change to the validator set can only be proposed by one of the exisitng Example usage, ```bash -$ bigchaindb upsert-validator new HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY= 1 fb7140f03a4ffad899fabbbf655b97e0321add66 --private-key /home/user/.tendermint/config/priv_validator.json +$ bigchaindb election new upsert-validator HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY= 1 fb7140f03a4ffad899fabbbf655b97e0321add66 --private-key /home/user/.tendermint/config/priv_validator.json [SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa ``` @@ -119,13 +124,14 @@ If the command succeeds, it will create an election and return an `election_id`. **NOTE**: The election proposal consists of vote tokens allocated to each current validator as per their voting power. Validators then cast their votes to approve the change to the validator set by spending their vote tokens. -#### upsert-validator approve +#### election approve + +Approve an election by voting for it. The proposal generated by executing `bigchaindb election new ...` can be approved by the validators using this command. The validator who is approving the proposal will spend all their votes i.e. if the validator has a network power of `10` then they will cast `10` votes for the proposal. -Approve an election by voting for it. The propsal generated by executing `bigchaindb upsert-valdiator approve ...` can approved by the validators using this command. The validator who is approving the proposal will spend all their votes i.e. if the validator has a network power of `10` then they will cast `10` votes for the proposal.` Below is the command line syntax and the return value, ```bash -$ bigchaindb upsert-validator approve --private-key PATH_TO_YOUR_PRIVATE_KEY +$ bigchaindb election approve --private-key PATH_TO_YOUR_PRIVATE_KEY [SUCCESS] Your vote has been submitted ``` @@ -134,24 +140,22 @@ $ bigchaindb upsert-validator approve --private-key PATH_TO_YOUR_P Example usage, ```bash -$ bigchaindb upsert-validator approve 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa --private-key /home/user/.tendermint/config/priv_validator.json +$ bigchaindb election approve 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa --private-key /home/user/.tendermint/config/priv_validator.json [SUCCESS] Your vote has been submitted ``` -If the command succeeds a message will be returned stating that the vote was submitted successfully. Once a proposal has been approved by sufficent validators (more than `2/3` of the total voting power) then the proposed change is applied to the network. For example, consider a network wherein the total power is `90` then the proposed changed applied only after `60` (`2/3 * 90`) have been received. +If the command succeeds a message will be returned stating that the vote was submitted successfully. Once a proposal has been approved by sufficent validators (more than `2/3` of the total voting power) then the proposed change is applied to the network. For example, consider a network wherein the total power is `90` then the proposed changed is applied only after `60` (`2/3 * 90`) have been received. -#### upsert-validator show +#### election show -Retrieves information about an election initiated by `upsert-validator new`. +Retrieves information about an election initiated by `election new`. Below is the command line syntax and the return value, ```bash -$ bigchaindb upsert-validator show ELECTION_ID -public_key= -power= -node_id= +$ bigchaindb election show ELECTION_ID + status= ``` -The `public_key`, `power`, and `node_id` are the same values used in the `upsert-validator new` command that originally triggered the election. `status` takes three possible values, `ongoing`, if the election has not yet reached a 2/3 majority, `concluded`, if the election reached the 2/3 majority needed to pass, or `inconclusive`, if the validator set changed while the election was in process, rendering it undecidable. \ No newline at end of file +The election data is the same set of arguments used in the `election new` command that originally triggered the election. `status` takes three possible values, `ongoing`, if the election has not yet reached a 2/3 majority, `concluded`, if the election reached the 2/3 majority needed to pass, or `inconclusive`, if the validator set changed while the election was in process, rendering it undecidable. \ No newline at end of file diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 6ed54425..1511531e 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -25,11 +25,11 @@ def test_make_sure_we_dont_remove_any_command(): assert parser.parse_args(['init']).command assert parser.parse_args(['drop']).command assert parser.parse_args(['start']).command - assert parser.parse_args(['upsert-validator', 'new', 'TEMP_PUB_KEYPAIR', '10', 'TEMP_NODE_ID', + assert parser.parse_args(['election', 'new', 'upsert-validator', 'TEMP_PUB_KEYPAIR', '10', 'TEMP_NODE_ID', '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command - assert parser.parse_args(['upsert-validator', 'approve', 'ELECTION_ID', '--private-key', + assert parser.parse_args(['election', 'approve', 'ELECTION_ID', '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command - assert parser.parse_args(['upsert-validator', 'show', 'ELECTION_ID']).command + assert parser.parse_args(['election', 'show', 'ELECTION_ID']).command @patch('bigchaindb.commands.utils.start') @@ -333,24 +333,25 @@ class MockResponse(): @pytest.mark.abci -def test_upsert_validator_new_with_tendermint(b, priv_validator_path, user_sk, validators): - from bigchaindb.commands.bigchaindb import run_upsert_validator_new +def test_election_new_upsert_validator_with_tendermint(b, priv_validator_path, user_sk, validators): + from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator new_args = Namespace(action='new', + election_type='upsert_validator', public_key='HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY=', power=1, node_id='unique_node_id_for_test_upsert_validator_new_with_tendermint', sk=priv_validator_path, config={}) - election_id = run_upsert_validator_new(new_args, b) + election_id = run_election_new_upsert_validator(new_args, b) assert b.get_transaction(election_id) @pytest.mark.bdb -def test_upsert_validator_new_without_tendermint(caplog, b, priv_validator_path, user_sk): - from bigchaindb.commands.bigchaindb import run_upsert_validator_new +def test_election_new_upsert_validator_without_tendermint(caplog, b, priv_validator_path, user_sk): + from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator def mock_write(tx, mode): b.store_bulk_transactions([tx]) @@ -360,6 +361,7 @@ def test_upsert_validator_new_without_tendermint(caplog, b, priv_validator_path, b.write_transaction = mock_write args = Namespace(action='new', + election_type='upsert_validator', public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=', power=1, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', @@ -367,16 +369,17 @@ def test_upsert_validator_new_without_tendermint(caplog, b, priv_validator_path, config={}) with caplog.at_level(logging.INFO): - election_id = run_upsert_validator_new(args, b) + election_id = run_election_new_upsert_validator(args, b) assert caplog.records[0].msg == '[SUCCESS] Submitted proposal with id: ' + election_id assert b.get_transaction(election_id) @pytest.mark.bdb -def test_upsert_validator_new_invalid_election(caplog, b, priv_validator_path, user_sk): - from bigchaindb.commands.bigchaindb import run_upsert_validator_new +def test_election_new_upsert_validator_invalid_election(caplog, b, priv_validator_path, user_sk): + from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator args = Namespace(action='new', + election_type='upsert_validator', public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=', power=10, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', @@ -384,13 +387,13 @@ def test_upsert_validator_new_invalid_election(caplog, b, priv_validator_path, u config={}) with caplog.at_level(logging.ERROR): - assert not run_upsert_validator_new(args, b) + assert not run_election_new_upsert_validator(args, b) assert caplog.records[0].msg.__class__ == FileNotFoundError @pytest.mark.bdb -def test_upsert_validator_new_election_invalid_power(caplog, b, priv_validator_path, user_sk): - from bigchaindb.commands.bigchaindb import run_upsert_validator_new +def test_election_new_upsert_validator_invalid_power(caplog, b, priv_validator_path, user_sk): + from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator from bigchaindb.common.exceptions import InvalidPowerChange def mock_write(tx, mode): @@ -400,6 +403,7 @@ def test_upsert_validator_new_election_invalid_power(caplog, b, priv_validator_p b.write_transaction = mock_write b.get_validators = mock_get_validators args = Namespace(action='new', + election_type='upsert_validator', public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=', power=10, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', @@ -407,38 +411,39 @@ def test_upsert_validator_new_election_invalid_power(caplog, b, priv_validator_p config={}) with caplog.at_level(logging.ERROR): - assert not run_upsert_validator_new(args, b) + assert not run_election_new_upsert_validator(args, b) assert caplog.records[0].msg.__class__ == InvalidPowerChange @pytest.mark.abci -def test_upsert_validator_approve_with_tendermint(b, priv_validator_path, user_sk, validators): - from bigchaindb.commands.bigchaindb import (run_upsert_validator_new, - run_upsert_validator_approve) +def test_election_approve_with_tendermint(b, priv_validator_path, user_sk, validators): + from bigchaindb.commands.bigchaindb import (run_election_new_upsert_validator, + run_election_approve) public_key = 'CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=' new_args = Namespace(action='new', + election_type='upsert_validator', public_key=public_key, power=1, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', sk=priv_validator_path, config={}) - election_id = run_upsert_validator_new(new_args, b) + election_id = run_election_new_upsert_validator(new_args, b) assert election_id args = Namespace(action='approve', election_id=election_id, sk=priv_validator_path, config={}) - approve = run_upsert_validator_approve(args, b) + approve = run_election_approve(args, b) assert b.get_transaction(approve) @pytest.mark.bdb -def test_upsert_validator_approve_without_tendermint(caplog, b, priv_validator_path, new_validator, node_key): - from bigchaindb.commands.bigchaindb import run_upsert_validator_approve +def test_election_approve_without_tendermint(caplog, b, priv_validator_path, new_validator, node_key): + from bigchaindb.commands.bigchaindb import run_election_approve from argparse import Namespace b, election_id = call_election(b, new_validator, node_key) @@ -451,14 +456,14 @@ def test_upsert_validator_approve_without_tendermint(caplog, b, priv_validator_p # assert returned id is in the db with caplog.at_level(logging.INFO): - approval_id = run_upsert_validator_approve(args, b) + approval_id = run_election_approve(args, b) assert caplog.records[0].msg == '[SUCCESS] Your vote has been submitted' assert b.get_transaction(approval_id) @pytest.mark.bdb -def test_upsert_validator_approve_failure(caplog, b, priv_validator_path, new_validator, node_key): - from bigchaindb.commands.bigchaindb import run_upsert_validator_approve +def test_election_approve_failure(caplog, b, priv_validator_path, new_validator, node_key): + from bigchaindb.commands.bigchaindb import run_election_approve from argparse import Namespace b, election_id = call_election(b, new_validator, node_key) @@ -476,13 +481,13 @@ def test_upsert_validator_approve_failure(caplog, b, priv_validator_path, new_va config={}) with caplog.at_level(logging.ERROR): - assert not run_upsert_validator_approve(args, b) + assert not run_election_approve(args, b) assert caplog.records[0].msg == 'Failed to commit vote' @pytest.mark.bdb -def test_upsert_validator_approve_called_with_bad_key(caplog, b, bad_validator_path, new_validator, node_key): - from bigchaindb.commands.bigchaindb import run_upsert_validator_approve +def test_election_approve_called_with_bad_key(caplog, b, bad_validator_path, new_validator, node_key): + from bigchaindb.commands.bigchaindb import run_election_approve from argparse import Namespace b, election_id = call_election(b, new_validator, node_key) @@ -494,7 +499,7 @@ def test_upsert_validator_approve_called_with_bad_key(caplog, b, bad_validator_p config={}) with caplog.at_level(logging.ERROR): - assert not run_upsert_validator_approve(args, b) + assert not run_election_approve(args, b) assert caplog.records[0].msg == 'The key you provided does not match any of '\ 'the eligible voters in this election.' diff --git a/tests/upsert_validator/test_validator_election.py b/tests/upsert_validator/test_validator_election.py index fe30331a..c1f56fff 100644 --- a/tests/upsert_validator/test_validator_election.py +++ b/tests/upsert_validator/test_validator_election.py @@ -159,7 +159,7 @@ def test_get_status_inconclusive(b, inconclusive_election, new_validator): def test_upsert_validator_show(caplog, ongoing_election, b): - from bigchaindb.commands.bigchaindb import run_upsert_validator_show + from bigchaindb.commands.bigchaindb import run_election_show election_id = ongoing_election.id public_key = public_key_to_base64(ongoing_election.asset['data']['public_key']['value']) @@ -170,6 +170,6 @@ def test_upsert_validator_show(caplog, ongoing_election, b): show_args = Namespace(action='show', election_id=election_id) - msg = run_upsert_validator_show(show_args, b) + msg = run_election_show(show_args, b) assert msg == f'public_key={public_key}\npower={power}\nnode_id={node_id}\nstatus={status}' From 35e35ecd57c64eaca5f30c81e90223a4f5d45737 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Wed, 12 Sep 2018 14:39:07 +0200 Subject: [PATCH 23/77] Problem: New collections are not created in existing DB. (#2520) Solution: Do not abort the initialisation if a collection exists. Unify the index creation. --- bigchaindb/backend/localmongodb/schema.py | 171 +++++++--------------- bigchaindb/backend/schema.py | 20 --- bigchaindb/commands/bigchaindb.py | 21 +-- bigchaindb/common/exceptions.py | 4 - tests/backend/localmongodb/test_schema.py | 23 +-- tests/backend/test_generics.py | 1 - tests/commands/test_commands.py | 36 ----- tests/conftest.py | 36 ++--- tests/web/test_outputs.py | 13 +- 9 files changed, 82 insertions(+), 243 deletions(-) diff --git a/bigchaindb/backend/localmongodb/schema.py b/bigchaindb/backend/localmongodb/schema.py index 342a22b7..053cfc3d 100644 --- a/bigchaindb/backend/localmongodb/schema.py +++ b/bigchaindb/backend/localmongodb/schema.py @@ -7,9 +7,9 @@ import logging from pymongo import ASCENDING, DESCENDING, TEXT +from pymongo.errors import CollectionInvalid from bigchaindb import backend -from bigchaindb.common import exceptions from bigchaindb.backend.utils import module_dispatch_registrar from bigchaindb.backend.localmongodb.connection import LocalMongoDBConnection @@ -18,12 +18,47 @@ logger = logging.getLogger(__name__) register_schema = module_dispatch_registrar(backend.schema) +INDEXES = { + 'transactions': [ + ('id', dict(unique=True, name='transaction_id')), + ('asset.id', dict(name='asset_id')), + ('outputs.public_keys', dict(name='outputs')), + ([('inputs.fulfills.transaction_id', ASCENDING), + ('inputs.fulfills.output_index', ASCENDING)], dict(name='inputs')), + ], + 'assets': [ + ('id', dict(name='asset_id', unique=True)), + ([('$**', TEXT)], dict(name='text')), + ], + 'blocks': [ + ([('height', DESCENDING)], dict(name='height', unique=True)), + ], + 'metadata': [ + ('id', dict(name='transaction_id', unique=True)), + ([('$**', TEXT)], dict(name='text')), + ], + 'utxos': [ + ([('transaction_id', ASCENDING), + ('output_index', ASCENDING)], dict(name='utxo', unique=True)), + ], + 'pre_commit': [ + ('commit_id', dict(name='pre_commit_id', unique=True)), + ], + 'elections': [ + ('election_id', dict(name='election_id', unique=True)), + ], + 'validators': [ + ('height', dict(name='height', unique=True)), + ], + 'abci_chains': [ + ('height', dict(name='height', unique=True)), + ('chain_id', dict(name='chain_id', unique=True)), + ], +} + + @register_schema(LocalMongoDBConnection) def create_database(conn, dbname): - if dbname in conn.conn.database_names(): - raise exceptions.DatabaseAlreadyExists('Database `{}` already exists' - .format(dbname)) - logger.info('Create database `%s`.', dbname) # TODO: read and write concerns can be declared here conn.conn.get_database(dbname) @@ -32,128 +67,22 @@ def create_database(conn, dbname): @register_schema(LocalMongoDBConnection) def create_tables(conn, dbname): for table_name in backend.schema.TABLES: - logger.info('Create `%s` table.', table_name) # create the table # TODO: read and write concerns can be declared here - conn.conn[dbname].create_collection(table_name) + try: + logger.info(f'Create `{table_name}` table.') + conn.conn[dbname].create_collection(table_name) + except CollectionInvalid: + logger.info(f'Collection {table_name} already exists.') + create_indexes(conn, dbname, table_name, INDEXES[table_name]) -@register_schema(LocalMongoDBConnection) -def create_indexes(conn, dbname): - create_transactions_secondary_index(conn, dbname) - create_assets_secondary_index(conn, dbname) - create_blocks_secondary_index(conn, dbname) - create_metadata_secondary_index(conn, dbname) - create_utxos_secondary_index(conn, dbname) - create_pre_commit_secondary_index(conn, dbname) - create_validators_secondary_index(conn, dbname) - create_abci_chains_indexes(conn, dbname) - create_elections_secondary_index(conn, dbname) +def create_indexes(conn, dbname, collection, indexes): + logger.info(f'Ensure secondary indexes for `{collection}`.') + for fields, kwargs in indexes: + conn.conn[dbname][collection].create_index(fields, **kwargs) @register_schema(LocalMongoDBConnection) def drop_database(conn, dbname): conn.conn.drop_database(dbname) - - -def create_transactions_secondary_index(conn, dbname): - logger.info('Create `transactions` secondary index.') - - # to query the transactions for a transaction id, this field is unique - conn.conn[dbname]['transactions'].create_index('id', - unique=True, - name='transaction_id') - - # secondary index for asset uuid, this field is unique - conn.conn[dbname]['transactions']\ - .create_index('asset.id', name='asset_id') - - # secondary index on the public keys of outputs - conn.conn[dbname]['transactions']\ - .create_index('outputs.public_keys', - name='outputs') - - # secondary index on inputs/transaction links (transaction_id, output) - conn.conn[dbname]['transactions']\ - .create_index([ - ('inputs.fulfills.transaction_id', ASCENDING), - ('inputs.fulfills.output_index', ASCENDING), - ], name='inputs') - - -def create_assets_secondary_index(conn, dbname): - logger.info('Create `assets` secondary index.') - - # unique index on the id of the asset. - # the id is the txid of the transaction that created the asset - conn.conn[dbname]['assets'].create_index('id', - name='asset_id', - unique=True) - - # full text search index - conn.conn[dbname]['assets'].create_index([('$**', TEXT)], name='text') - - -def create_blocks_secondary_index(conn, dbname): - conn.conn[dbname]['blocks']\ - .create_index([('height', DESCENDING)], name='height', unique=True) - - -def create_metadata_secondary_index(conn, dbname): - logger.info('Create `assets` secondary index.') - - # the id is the txid of the transaction where metadata was defined - conn.conn[dbname]['metadata'].create_index('id', - name='transaction_id', - unique=True) - - # full text search index - conn.conn[dbname]['metadata'].create_index([('$**', TEXT)], name='text') - - -def create_utxos_secondary_index(conn, dbname): - logger.info('Create `utxos` secondary index.') - - conn.conn[dbname]['utxos'].create_index( - [('transaction_id', ASCENDING), ('output_index', ASCENDING)], - name='utxo', - unique=True, - ) - - -def create_pre_commit_secondary_index(conn, dbname): - logger.info('Create `pre_commit` secondary index.') - - conn.conn[dbname]['pre_commit'].create_index('commit_id', - name='pre_commit_id', - unique=True) - - -def create_validators_secondary_index(conn, dbname): - logger.info('Create `validators` secondary index.') - - conn.conn[dbname]['validators'].create_index('height', - name='height', - unique=True,) - - -def create_abci_chains_indexes(conn, dbname): - logger.info('Create `abci_chains.height` secondary index.') - - conn.conn[dbname]['abci_chains'].create_index('height', - name='height', - unique=True,) - - logger.info('Create `abci_chains.chain_id` secondary index.') - - conn.conn[dbname]['abci_chains'].create_index('chain_id', - name='chain_id', - unique=True) - - -def create_elections_secondary_index(conn, dbname): - logger.info('Create `elections` secondary index.') - - conn.conn[dbname]['elections'].create_index('election_id', - name='election_id', - unique=True,) diff --git a/bigchaindb/backend/schema.py b/bigchaindb/backend/schema.py index c1d0343d..08fdbc0e 100644 --- a/bigchaindb/backend/schema.py +++ b/bigchaindb/backend/schema.py @@ -31,10 +31,6 @@ def create_database(connection, dbname): Args: dbname (str): the name of the database to create. - - Raises: - :exc:`~DatabaseAlreadyExists`: If the given :attr:`dbname` already - exists as a database. """ raise NotImplementedError @@ -51,17 +47,6 @@ def create_tables(connection, dbname): raise NotImplementedError -@singledispatch -def create_indexes(connection, dbname): - """Create the indexes to be used by BigchainDB. - - Args: - dbname (str): the name of the database to create indexes for. - """ - - raise NotImplementedError - - @singledispatch def drop_database(connection, dbname): """Drop the database used by BigchainDB. @@ -90,10 +75,6 @@ def init_database(connection=None, dbname=None): dbname (str): the name of the database to create. Defaults to the database name given in the BigchainDB configuration. - - Raises: - :exc:`~DatabaseAlreadyExists`: If the given :attr:`dbname` already - exists as a database. """ connection = connection or connect() @@ -101,7 +82,6 @@ def init_database(connection=None, dbname=None): create_database(connection, dbname) create_tables(connection, dbname) - create_indexes(connection, dbname) def validate_language_key(obj, key): diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index d5b2e9ac..082b2753 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -14,8 +14,7 @@ import json import sys from bigchaindb.utils import load_node_key -from bigchaindb.common.exceptions import (DatabaseAlreadyExists, - DatabaseDoesNotExist, +from bigchaindb.common.exceptions import (DatabaseDoesNotExist, ValidationError) from bigchaindb.elections.vote import Vote import bigchaindb @@ -228,14 +227,7 @@ def _run_init(): @configure_bigchaindb def run_init(args): """Initialize the database""" - # TODO Provide mechanism to: - # 1. prompt the user to inquire whether they wish to drop the db - # 2. force the init, (e.g., via -f flag) - try: - _run_init() - except DatabaseAlreadyExists: - print('The database already exists.', file=sys.stderr) - print('If you wish to re-initialize it, first drop it.', file=sys.stderr) + _run_init() @configure_bigchaindb @@ -279,12 +271,9 @@ def run_start(args): logger.info('BigchainDB Version %s', bigchaindb.__version__) run_recover(bigchaindb.lib.BigchainDB()) - try: - if not args.skip_initialize_database: - logger.info('Initializing database') - _run_init() - except DatabaseAlreadyExists: - pass + if not args.skip_initialize_database: + logger.info('Initializing database') + _run_init() logger.info('Starting BigchainDB main process.') from bigchaindb.start import start diff --git a/bigchaindb/common/exceptions.py b/bigchaindb/common/exceptions.py index 099ef92e..ab2c1a6d 100644 --- a/bigchaindb/common/exceptions.py +++ b/bigchaindb/common/exceptions.py @@ -11,10 +11,6 @@ class ConfigurationError(BigchainDBError): """Raised when there is a problem with server configuration""" -class DatabaseAlreadyExists(BigchainDBError): - """Raised when trying to create the database but the db is already there""" - - class DatabaseDoesNotExist(BigchainDBError): """Raised when trying to delete the database but the db is not there""" diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index 21a9ac4f..d02a8fde 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -2,8 +2,6 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -import pytest - def test_init_creates_db_tables_and_indexes(): import bigchaindb @@ -50,11 +48,10 @@ def test_init_creates_db_tables_and_indexes(): assert set(indexes) == {'_id_', 'election_id'} -def test_init_database_fails_if_db_exists(): +def test_init_database_is_graceful_if_db_exists(): import bigchaindb from bigchaindb import backend from bigchaindb.backend.schema import init_database - from bigchaindb.common import exceptions conn = backend.connect() dbname = bigchaindb.config['database']['name'] @@ -62,8 +59,7 @@ def test_init_database_fails_if_db_exists(): # The db is set up by the fixtures assert dbname in conn.conn.database_names() - with pytest.raises(exceptions.DatabaseAlreadyExists): - init_database() + init_database() def test_create_tables(): @@ -85,21 +81,6 @@ def test_create_tables(): 'pre_commit', 'abci_chains', } - -def test_create_secondary_indexes(): - import bigchaindb - from bigchaindb import backend - from bigchaindb.backend import schema - - conn = backend.connect() - dbname = bigchaindb.config['database']['name'] - - # The db is set up by the fixtures so we need to remove it - conn.conn.drop_database(dbname) - schema.create_database(conn, dbname) - schema.create_tables(conn, dbname) - schema.create_indexes(conn, dbname) - indexes = conn.conn[dbname]['assets'].index_information().keys() assert set(indexes) == {'_id_', 'asset_id', 'text'} diff --git a/tests/backend/test_generics.py b/tests/backend/test_generics.py index 6ddd7331..68644181 100644 --- a/tests/backend/test_generics.py +++ b/tests/backend/test_generics.py @@ -8,7 +8,6 @@ from pytest import mark, raises @mark.parametrize('schema_func_name,args_qty', ( ('create_database', 1), ('create_tables', 1), - ('create_indexes', 1), ('drop_database', 1), )) def test_schema(schema_func_name, args_qty): diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 1511531e..a9cb9d6a 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -75,25 +75,6 @@ def test_bigchain_show_config(capsys): assert output_config == config -def test_bigchain_run_init_when_db_exists(mocker, capsys): - from bigchaindb.commands.bigchaindb import run_init - from bigchaindb.common.exceptions import DatabaseAlreadyExists - init_db_mock = mocker.patch( - 'bigchaindb.commands.bigchaindb.schema.init_database', - autospec=True, - spec_set=True, - ) - init_db_mock.side_effect = DatabaseAlreadyExists - args = Namespace(config=None) - run_init(args) - output_message = capsys.readouterr()[1] - print(output_message) - assert output_message == ( - 'The database already exists.\n' - 'If you wish to re-initialize it, first drop it.\n' - ) - - def test__run_init(mocker): from bigchaindb.commands.bigchaindb import _run_init bigchain_mock = mocker.patch( @@ -219,23 +200,6 @@ def test_run_configure_with_backend(backend, monkeypatch, mock_write_config): assert value['return'] == expected_config -def test_run_start_when_db_already_exists(mocker, - monkeypatch, - run_start_args, - mocked_setup_logging): - from bigchaindb.commands.bigchaindb import run_start - from bigchaindb.common.exceptions import DatabaseAlreadyExists - mocked_start = mocker.patch('bigchaindb.start.start') - - def mock_run_init(): - raise DatabaseAlreadyExists() - monkeypatch.setattr('builtins.input', lambda: '\x03') - monkeypatch.setattr( - 'bigchaindb.commands.bigchaindb._run_init', mock_run_init) - run_start(run_start_args) - assert mocked_start.called - - @patch('bigchaindb.commands.utils.start') def test_calling_main(start_mock, monkeypatch): from bigchaindb.commands.bigchaindb import main diff --git a/tests/conftest.py b/tests/conftest.py index 9ef51e21..fb9615b1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -23,8 +23,10 @@ from pymongo import MongoClient from bigchaindb.common import crypto from bigchaindb.log import setup_logging from bigchaindb.tendermint_utils import key_from_base64 +from bigchaindb.backend import schema from bigchaindb.common.crypto import (key_pair_from_ed25519_key, public_key_from_ed25519_key) +from bigchaindb.common.exceptions import DatabaseDoesNotExist from bigchaindb.lib import Block @@ -113,17 +115,12 @@ def _configure_bigchaindb(request): @pytest.fixture(scope='session') def _setup_database(_configure_bigchaindb): from bigchaindb import config - from bigchaindb.backend import connect, schema - from bigchaindb.common.exceptions import DatabaseDoesNotExist + from bigchaindb.backend import connect print('Initializing test db') dbname = config['database']['name'] conn = connect() - try: - schema.drop_database(conn, dbname) - except DatabaseDoesNotExist: - pass - + _drop_db(conn, dbname) schema.init_database(conn) print('Finishing init database') @@ -131,10 +128,7 @@ def _setup_database(_configure_bigchaindb): print('Deleting `{}` database'.format(dbname)) conn = connect() - try: - schema.drop_database(conn, dbname) - except DatabaseDoesNotExist: - pass + _drop_db(conn, dbname) print('Finished deleting `{}`'.format(dbname)) @@ -270,7 +264,6 @@ def signed_create_tx(alice, create_tx): return create_tx.sign([alice.private_key]) -@pytest.mark.abci @pytest.fixture def posted_create_tx(b, signed_create_tx): res = b.post_transaction(signed_create_tx, 'broadcast_tx_commit') @@ -321,20 +314,22 @@ def inputs(user_pk, b, alice): @pytest.fixture def dummy_db(request): - from bigchaindb.backend import connect, schema - from bigchaindb.common.exceptions import (DatabaseDoesNotExist, - DatabaseAlreadyExists) + from bigchaindb.backend import connect + conn = connect() dbname = request.fixturename xdist_suffix = getattr(request.config, 'slaveinput', {}).get('slaveid') if xdist_suffix: dbname = '{}_{}'.format(dbname, xdist_suffix) - try: - schema.init_database(conn, dbname) - except DatabaseAlreadyExists: - schema.drop_database(conn, dbname) - schema.init_database(conn, dbname) + + _drop_db(conn, dbname) # make sure we start with a clean DB + schema.init_database(conn, dbname) yield dbname + + _drop_db(conn, dbname) + + +def _drop_db(conn, dbname): try: schema.drop_database(conn, dbname) except DatabaseDoesNotExist: @@ -430,7 +425,6 @@ def event_loop(): loop.close() -@pytest.mark.bdb @pytest.fixture(scope='session') def abci_server(): from abci import ABCIServer diff --git a/tests/web/test_outputs.py b/tests/web/test_outputs.py index 77a3f258..0e0267fe 100644 --- a/tests/web/test_outputs.py +++ b/tests/web/test_outputs.py @@ -5,11 +5,12 @@ import pytest from unittest.mock import MagicMock, patch -pytestmark = [pytest.mark.bdb, pytest.mark.usefixtures('inputs')] OUTPUTS_ENDPOINT = '/api/v1/outputs/' +@pytest.mark.bdb +@pytest.mark.userfixtures('inputs') def test_get_outputs_endpoint(client, user_pk): m = MagicMock() m.txid = 'a' @@ -38,6 +39,8 @@ def test_get_outputs_endpoint_unspent(client, user_pk): gof.assert_called_once_with(user_pk, False) +@pytest.mark.bdb +@pytest.mark.userfixtures('inputs') def test_get_outputs_endpoint_spent(client, user_pk): m = MagicMock() m.txid = 'a' @@ -51,11 +54,15 @@ def test_get_outputs_endpoint_spent(client, user_pk): gof.assert_called_once_with(user_pk, True) +@pytest.mark.bdb +@pytest.mark.userfixtures('inputs') def test_get_outputs_endpoint_without_public_key(client): res = client.get(OUTPUTS_ENDPOINT) assert res.status_code == 400 +@pytest.mark.bdb +@pytest.mark.userfixtures('inputs') def test_get_outputs_endpoint_with_invalid_public_key(client): expected = {'message': {'public_key': 'Invalid base58 ed25519 key'}} res = client.get(OUTPUTS_ENDPOINT + '?public_key=abc') @@ -63,6 +70,8 @@ def test_get_outputs_endpoint_with_invalid_public_key(client): assert res.status_code == 400 +@pytest.mark.bdb +@pytest.mark.userfixtures('inputs') def test_get_outputs_endpoint_with_invalid_spent(client, user_pk): expected = {'message': {'spent': 'Boolean value must be "true" or "false" (lowercase)'}} params = '?spent=tru&public_key={}'.format(user_pk) @@ -72,8 +81,6 @@ def test_get_outputs_endpoint_with_invalid_spent(client, user_pk): @pytest.mark.abci -@pytest.mark.bdb -@pytest.mark.usefixtures('inputs') def test_get_divisble_transactions_returns_500(b, client): from bigchaindb.models import Transaction from bigchaindb.common import crypto From 0f41869bea2d5277da3c9e97be223ab95ac20622 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Wed, 12 Sep 2018 16:53:07 +0200 Subject: [PATCH 24/77] Updated the Python Driver version in acceptance/python/Dockerfile (#2532) * Updated the Python Driver version in acceptance/python/Dockerfile * Let package manager auto-update to latest patch version --- acceptance/python/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/python/Dockerfile b/acceptance/python/Dockerfile index af29629a..e5f96d62 100644 --- a/acceptance/python/Dockerfile +++ b/acceptance/python/Dockerfile @@ -5,5 +5,5 @@ RUN pip install --upgrade \ pycco \ websocket-client~=0.47.0 \ pytest~=3.0 \ - bigchaindb-driver==0.5.2 \ + bigchaindb-driver~=0.5.0 \ blns From 4b54e702f8c2477a8bd56a8dffeda25b241a88a0 Mon Sep 17 00:00:00 2001 From: vrde Date: Wed, 12 Sep 2018 16:55:23 +0200 Subject: [PATCH 25/77] Problem: Events API is unreliable (#2529) Solution: Stop getting events from the Tendermint events API. Get the event right before returning from the COMMIT phase of Tendermint, and publish it to the events queue. --- bigchaindb/core.py | 12 +- bigchaindb/event_stream.py | 89 -------------- bigchaindb/start.py | 10 +- bigchaindb/web/websocket_server.py | 10 +- .../events/websocket-event-stream-api.rst | 4 +- tests/tendermint/test_core.py | 8 +- tests/tendermint/test_event_stream.py | 111 ------------------ .../test_upsert_validator_vote.py | 1 + tests/web/test_websocket_server.py | 37 +++--- 9 files changed, 47 insertions(+), 235 deletions(-) delete mode 100644 bigchaindb/event_stream.py delete mode 100644 tests/tendermint/test_event_stream.py diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 62bc6ec3..2f9fa23f 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -26,6 +26,7 @@ from bigchaindb.lib import Block, PreCommitState from bigchaindb.backend.query import PRE_COMMIT_ID from bigchaindb.upsert_validator import ValidatorElection import bigchaindb.upsert_validator.validator_utils as vutils +from bigchaindb.events import EventTypes, Event CodeTypeOk = 0 @@ -41,7 +42,8 @@ class App(BaseApplication): State Machine. """ - def __init__(self, bigchaindb=None): + def __init__(self, bigchaindb=None, events_queue=None): + self.events_queue = events_queue self.bigchaindb = bigchaindb or BigchainDB() self.block_txn_ids = [] self.block_txn_hash = '' @@ -245,4 +247,12 @@ class App(BaseApplication): 'height=%s, txn ids=%s', data, self.new_height, self.block_txn_ids) logger.benchmark('COMMIT_BLOCK, height:%s', self.new_height) + + if self.events_queue: + event = Event(EventTypes.BLOCK_VALID, { + 'height': self.new_height, + 'transactions': self.block_transactions + }) + self.events_queue.put(event) + return ResponseCommit(data=data) diff --git a/bigchaindb/event_stream.py b/bigchaindb/event_stream.py deleted file mode 100644 index a7d39187..00000000 --- a/bigchaindb/event_stream.py +++ /dev/null @@ -1,89 +0,0 @@ -# Copyright BigchainDB GmbH and BigchainDB contributors -# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) -# Code is Apache-2.0 and docs are CC-BY-4.0 - -import asyncio -import json -import logging -import time - -import aiohttp - -from bigchaindb import config -from bigchaindb.common.utils import gen_timestamp -from bigchaindb.events import EventTypes, Event -from bigchaindb.tendermint_utils import decode_transaction_base64 - - -HOST = config['tendermint']['host'] -PORT = config['tendermint']['port'] -URL = 'ws://{}:{}/websocket'.format(HOST, PORT) - -logger = logging.getLogger(__name__) - - -@asyncio.coroutine -def connect_and_recv(event_queue): - session = aiohttp.ClientSession() - ws = yield from session.ws_connect(URL) - - logger.info('Connected to tendermint ws server') - - stream_id = 'bigchaindb_stream_{}'.format(gen_timestamp()) - yield from subscribe_events(ws, stream_id) - - while True: - msg = yield from ws.receive() - process_event(event_queue, msg.data, stream_id) - - if msg.type in (aiohttp.WSMsgType.CLOSED, - aiohttp.WSMsgType.ERROR): - session.close() - raise aiohttp.ClientConnectionError() - - -def process_event(event_queue, event, stream_id): - event_stream_id = stream_id + '#event' - event = json.loads(event) - - if (event['id'] == event_stream_id and event['result']['query'] == 'tm.event=\'NewBlock\''): - block = event['result']['data']['value']['block'] - block_id = block['header']['height'] - block_txs = block['data']['txs'] - - # Only push non empty blocks - if block_txs: - block_txs = [decode_transaction_base64(txn) for txn in block_txs] - new_block = {'height': block_id, 'transactions': block_txs} - event = Event(EventTypes.BLOCK_VALID, new_block) - event_queue.put(event) - - -@asyncio.coroutine -def subscribe_events(ws, stream_id): - payload = { - 'method': 'subscribe', - 'jsonrpc': '2.0', - 'params': ['tm.event=\'NewBlock\''], - 'id': stream_id - } - yield from ws.send_str(json.dumps(payload)) - - -@asyncio.coroutine -def try_connect_and_recv(event_queue): - try: - yield from connect_and_recv(event_queue) - - except Exception as e: - logger.warning('WebSocket connection failed with exception %s', e) - time.sleep(3) - yield from try_connect_and_recv(event_queue) - - -def start(event_queue): - loop = asyncio.get_event_loop() - try: - loop.run_until_complete(try_connect_and_recv(event_queue)) - except (KeyboardInterrupt, SystemExit): - logger.info('Shutting down Tendermint event stream connection') diff --git a/bigchaindb/start.py b/bigchaindb/start.py index 2b9667b5..41173be0 100644 --- a/bigchaindb/start.py +++ b/bigchaindb/start.py @@ -9,7 +9,6 @@ import bigchaindb from bigchaindb.lib import BigchainDB from bigchaindb.core import App from bigchaindb.web import server, websocket_server -from bigchaindb import event_stream from bigchaindb.events import Exchange, EventTypes from bigchaindb.utils import Process @@ -57,13 +56,6 @@ def start(): args=(exchange.get_subscriber_queue(EventTypes.BLOCK_VALID),)) p_websocket_server.start() - # connect to tendermint event stream - p_websocket_client = Process(name='bigchaindb_ws_to_tendermint', - target=event_stream.start, - daemon=True, - args=(exchange.get_publisher_queue(),)) - p_websocket_client.start() - p_exchange = Process(name='bigchaindb_exchange', target=exchange.run, daemon=True) p_exchange.start() @@ -75,7 +67,7 @@ def start(): setproctitle.setproctitle('bigchaindb') # Start the ABCIServer - app = ABCIServer(app=App()) + app = ABCIServer(app=App(events_queue=exchange.get_publisher_queue())) app.run() diff --git a/bigchaindb/web/websocket_server.py b/bigchaindb/web/websocket_server.py index f0a9f886..efe10151 100644 --- a/bigchaindb/web/websocket_server.py +++ b/bigchaindb/web/websocket_server.py @@ -50,13 +50,13 @@ def _multiprocessing_to_asyncio(in_queue, out_queue, loop): def eventify_block(block): for tx in block['transactions']: - try: - asset_id = tx['asset']['id'] - except KeyError: - asset_id = tx['id'] + if tx.asset: + asset_id = tx.asset.get('id', tx.id) + else: + asset_id = tx.id yield {'height': block['height'], 'asset_id': asset_id, - 'transaction_id': tx['id']} + 'transaction_id': tx.id} class Dispatcher: diff --git a/docs/server/source/events/websocket-event-stream-api.rst b/docs/server/source/events/websocket-event-stream-api.rst index 526df494..967b3464 100644 --- a/docs/server/source/events/websocket-event-stream-api.rst +++ b/docs/server/source/events/websocket-event-stream-api.rst @@ -90,7 +90,7 @@ Valid Transactions Streams an event for any newly valid transactions committed to a block. Message bodies contain the transaction's ID, associated asset ID, and containing -block's ID. +block's height. Example message: @@ -99,7 +99,7 @@ Example message: { "transaction_id": "", "asset_id": "", - "block_id": "" + "height": } diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index 31b4e100..5875ed9b 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -235,19 +235,21 @@ def test_check_tx__unsigned_create_is_error(b): assert result.code == CodeTypeError -def test_deliver_tx__valid_create_updates_db(b, init_chain_request): +def test_deliver_tx__valid_create_updates_db_and_emits_event(b, init_chain_request): + import multiprocessing as mp from bigchaindb import App from bigchaindb.models import Transaction from bigchaindb.common.crypto import generate_key_pair alice = generate_key_pair() bob = generate_key_pair() + events = mp.Queue() tx = Transaction.create([alice.public_key], [([bob.public_key], 1)])\ .sign([alice.private_key]) - app = App(b) + app = App(b, events) app.init_chain(init_chain_request) @@ -260,6 +262,8 @@ def test_deliver_tx__valid_create_updates_db(b, init_chain_request): app.end_block(RequestEndBlock(height=99)) app.commit() assert b.get_transaction(tx.id).id == tx.id + block_event = events.get() + assert block_event.data['transactions'] == [tx] # unspent_outputs = b.get_unspent_outputs() # unspent_output = next(unspent_outputs) diff --git a/tests/tendermint/test_event_stream.py b/tests/tendermint/test_event_stream.py deleted file mode 100644 index f23eeaba..00000000 --- a/tests/tendermint/test_event_stream.py +++ /dev/null @@ -1,111 +0,0 @@ -# Copyright BigchainDB GmbH and BigchainDB contributors -# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) -# Code is Apache-2.0 and docs are CC-BY-4.0 - -import json -import base64 -from queue import Queue - -from aiohttp import ClientSession -import pytest - - -def test_process_event_new_block(): - from bigchaindb.event_stream import process_event - - event = '{"jsonrpc": "2.0", "id": "test_stream_id#event", "result": {'\ - '"query": "tm.event=\'NewBlock\'", "data": { "type": "CF18EA939D3240",'\ - '"value": { "block": { "header": { "chain_id": "test-chain-ipQIAa",'\ - '"height": 1, "time": "2018-04-23T14:49:30.509920098Z", "num_txs": 1,'\ - '"last_block_id": { "hash": "", "parts": { "total": 0, "hash": "" }},'\ - '"total_txs": 1, "last_commit_hash": "", "data_hash": "38792142CE6D7F6F46F71777CB53F94CD9497B23",'\ - '"validators_hash": "BF0D0EC2E13C76E69FA572516B6D93E64F3C58EF",'\ - '"consensus_hash": "F66EF1DF8BA6DAC7A1ECCE40CC84E54A1CEBC6A5", "app_hash": "",'\ - '"last_results_hash": "", "evidence_hash": "" }, "data": {"txs": ['\ - '"eyJpbnB1dHMiOiBbeyJvd25lcnNfYmVmb3JlIjogWyJFb2Z0Z0FNd2hKQXM0cW81b'\ - '0dhOU1GWXF5dFp5WEdaNmVmZFVYc1dXTDdmZSJdLCAiZnVsZmlsbHMiOiBudWxsLCA'\ - 'iZnVsZmlsbG1lbnQiOiAicEdTQUlNMGNueFFGeTZrSE1PcGxBbzh1ZncwNDlsZ2VxN'\ - 'HBOeDFNdksya0pjRjBCZ1VETjN2RTlsWmhaT21jMWZHbFpLUFZmZDdCTi1RVTdBa0N'\ - 'TZ1NKWVRPYzB3YVlmQ1RXc1FQS1VmOE5fODFKd21YOUJxcnlLejYyTmVubHg0dGszN'\ - 'GtVRCJ9XSwgIm91dHB1dHMiOiBbeyJwdWJsaWNfa2V5cyI6IFsiRW9mdGdBTXdoSkF'\ - 'zNHFvNW9HYTlNRllxeXRaeVhHWjZlZmRVWHNXV0w3ZmUiXSwgImNvbmRpdGlvbiI6I'\ - 'HsiZGV0YWlscyI6IHsidHlwZSI6ICJlZDI1NTE5LXNoYS0yNTYiLCAicHVibGljX2t'\ - 'leSI6ICJFb2Z0Z0FNd2hKQXM0cW81b0dhOU1GWXF5dFp5WEdaNmVmZFVYc1dXTDdmZ'\ - 'SJ9LCAidXJpIjogIm5pOi8vL3NoYS0yNTY7cFJZWTJQQUE0S3dHd0dUNVQtUXRCQUY'\ - '0VWY1WG5JcVkxWmExVER0N0hMQT9mcHQ9ZWQyNTUxOS1zaGEtMjU2JmNvc3Q9MTMxM'\ - 'DcyIn0sICJhbW91bnQiOiAiMSJ9XSwgIm9wZXJhdGlvbiI6ICJDUkVBVEUiLCAibWV'\ - '0YWRhdGEiOiBudWxsLCAiYXNzZXQiOiB7ImRhdGEiOiBudWxsfSwgInZlcnNpb24iO'\ - 'iAiMi4wIiwgImlkIjogImUwMmM0ZWM3MmExYzUzMmJkNjUyNWZkNGMxODU3ZDhmN2E'\ - 'wYWVkYTgyNGVjY2NhZGY4NTlmNzc0Zjc3ZTgwZGUifQ=="]}, "evidence": {'\ - '"evidence": null}, "last_commit": { "blockID": { "hash": "", "parts":'\ - '{"total": 0, "hash": ""} }, "precommits": null } } } } } }' - - event_queue = Queue() - process_event(event_queue, event, 'test_stream_id') - assert not event_queue.empty() - block = event_queue.get() - assert isinstance(block.data['height'], int) - - -def test_process_event_empty_block(): - from bigchaindb.event_stream import process_event - - event = '{"jsonrpc": "2.0", "id": "bigchaindb_stream_1524555674#event",'\ - '"result": {"query": "tm.event=\'NewBlock\'", "data": {"type": '\ - '"CF18EA939D3240", "value": {"block": {"header": {"chain_id": '\ - '"test-chain-ipQIAa", "height": 1, "time": "2018-04-24T07:41:16.838038877Z",'\ - '"num_txs": 0, "last_block_id": {"hash": "", "parts": {"total": 0, "hash": ""}},'\ - '"total_txs": 0, "last_commit_hash": "", "data_hash": "", "validators_hash":'\ - '"BF0D0EC2E13C76E69FA572516B6D93E64F3C58EF", "consensus_hash": '\ - '"F66EF1DF8BA6DAC7A1ECCE40CC84E54A1CEBC6A5", "app_hash": "", '\ - '"last_results_hash": "", "evidence_hash": ""}, "data": {"txs": null},'\ - '"evidence": {"evidence": null}, "last_commit": {"blockID": {"hash": "", '\ - '"parts": {"total": 0, "hash": ""}}, "precommits": null}}}}}}' - - event_queue = Queue() - process_event(event_queue, event, 'test_stream_id') - assert event_queue.empty() - - -def test_process_unknown_event(): - from bigchaindb.event_stream import process_event - - event = '{"jsonrpc": "2.0", "id": "test_stream_id#event",'\ - ' "result": { "query": "tm.event=\'UnknownEvent\'" }}' - - event_queue = Queue() - process_event(event_queue, event, 'test_stream_id') - assert event_queue.empty() - - -@pytest.mark.asyncio -@pytest.mark.abci -async def test_subscribe_events(tendermint_ws_url, b): - from bigchaindb.event_stream import subscribe_events - from bigchaindb.common.crypto import generate_key_pair - from bigchaindb.models import Transaction - - session = ClientSession() - ws = await session.ws_connect(tendermint_ws_url) - stream_id = 'bigchaindb_stream_01' - await subscribe_events(ws, stream_id) - msg = await ws.receive() - assert msg.data - msg_data_dict = json.loads(msg.data) - assert msg_data_dict['id'] == stream_id - assert msg_data_dict['jsonrpc'] == '2.0' - assert msg_data_dict['result'] == {} - - alice = generate_key_pair() - tx = Transaction.create([alice.public_key], - [([alice.public_key], 1)], - asset=None)\ - .sign([alice.private_key]) - - b.post_transaction(tx, 'broadcast_tx_async') - msg = await ws.receive() - msg_data_dict = json.loads(msg.data) - raw_txn = msg_data_dict['result']['data']['value']['block']['data']['txs'][0] - transaction = json.loads(base64.b64decode(raw_txn).decode('utf8')) - - assert transaction == tx.to_dict() diff --git a/tests/upsert_validator/test_upsert_validator_vote.py b/tests/upsert_validator/test_upsert_validator_vote.py index 8567397c..b2e23af7 100644 --- a/tests/upsert_validator/test_upsert_validator_vote.py +++ b/tests/upsert_validator/test_upsert_validator_vote.py @@ -229,6 +229,7 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): latest_block = b.get_latest_block() # reset the validator set b.store_validator_set(latest_block['height'], validators) + generate_block(b) power = 1 public_key = '9B3119650DF82B9A5D8A12E38953EA47475C09F0C48A4E6A0ECE182944B24403' diff --git a/tests/web/test_websocket_server.py b/tests/web/test_websocket_server.py index dff74f21..7f72b795 100644 --- a/tests/web/test_websocket_server.py +++ b/tests/web/test_websocket_server.py @@ -21,25 +21,30 @@ class MockWebSocket: def test_eventify_block_works_with_any_transaction(): from bigchaindb.web.websocket_server import eventify_block + from bigchaindb.common.crypto import generate_key_pair + from bigchaindb.lib import Transaction - block = { - 'height': 1, - 'transactions': [{ - 'id': 1 - }, { - 'id': 2, - 'asset': {'id': 1} - }] - } + alice = generate_key_pair() + + tx = Transaction.create([alice.public_key], + [([alice.public_key], 1)])\ + .sign([alice.private_key]) + tx_transfer = Transaction.transfer(tx.to_inputs(), + [([alice.public_key], 1)], + asset_id=tx.id)\ + .sign([alice.private_key]) + + block = {'height': 1, + 'transactions': [tx, tx_transfer]} expected_events = [{ 'height': 1, - 'asset_id': 1, - 'transaction_id': 1 + 'asset_id': tx.id, + 'transaction_id': tx.id }, { 'height': 1, - 'asset_id': 1, - 'transaction_id': 2 + 'asset_id': tx_transfer.asset['id'], + 'transaction_id': tx_transfer.id }] for event, expected in zip(eventify_block(block), expected_events): @@ -144,7 +149,7 @@ def test_websocket_block_event(b, test_client, loop): app = init_app(event_source, loop=loop) client = yield from test_client(app) ws = yield from client.ws_connect(EVENTS_ENDPOINT) - block = {'height': 1, 'transactions': [tx.to_dict()]} + block = {'height': 1, 'transactions': [tx]} block_event = events.Event(events.EventTypes.BLOCK_VALID, block) yield from event_source.put(block_event) @@ -152,9 +157,9 @@ def test_websocket_block_event(b, test_client, loop): for tx in block['transactions']: result = yield from ws.receive() json_result = json.loads(result.data) - assert json_result['transaction_id'] == tx['id'] + assert json_result['transaction_id'] == tx.id # Since the transactions are all CREATEs, asset id == transaction id - assert json_result['asset_id'] == tx['id'] + assert json_result['asset_id'] == tx.id assert json_result['height'] == block['height'] yield from event_source.put(POISON_PILL) From 901b6b9d0289a79719ecf635b1629b373920269a Mon Sep 17 00:00:00 2001 From: vrde Date: Thu, 13 Sep 2018 12:05:01 +0200 Subject: [PATCH 26/77] Problem: Sphinx deprecated a config option (#2537) Solution: Move from `autodoc_default_flags` to `autodoc_default_options`. Related issue: - https://github.com/sphinx-doc/sphinx/issues/5421 --- docs/server/source/conf.py | 6 +++--- setup.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/server/source/conf.py b/docs/server/source/conf.py index 459877aa..dc8acd81 100644 --- a/docs/server/source/conf.py +++ b/docs/server/source/conf.py @@ -60,9 +60,9 @@ extensions = [ # autodoc settings autodoc_member_order = 'bysource' -autodoc_default_flags = [ - 'members', -] +autodoc_default_options = { + 'members': None, +} todo_include_todos = True diff --git a/setup.py b/setup.py index db7f1bd0..a004178d 100644 --- a/setup.py +++ b/setup.py @@ -44,7 +44,7 @@ dev_require = [ ] docs_require = [ - 'Sphinx>=1.4.8', + 'Sphinx~=1.0', 'recommonmark>=0.4.0', 'sphinx-rtd-theme>=0.1.9', 'sphinxcontrib-httpdomain>=1.5.0', From 241aae335d87d0cfbbf96a9256a0b1e5a5c99bd8 Mon Sep 17 00:00:00 2001 From: vrde Date: Thu, 13 Sep 2018 12:16:45 +0200 Subject: [PATCH 27/77] Problem: validation is slow (#2489) Solution: parallelize the validation of transactions. This patch adds a new flag for the `start` command, namely `--experimental-parallel-validation`, that can be used to enable this experimental feature. --- bigchaindb/commands/bigchaindb.py | 8 +- bigchaindb/parallel_validation.py | 127 ++++++++++++++++++++++++++++ bigchaindb/start.py | 9 +- tests/test_parallel_validation.py | 133 ++++++++++++++++++++++++++++++ 4 files changed, 273 insertions(+), 4 deletions(-) create mode 100644 bigchaindb/parallel_validation.py create mode 100644 tests/test_parallel_validation.py diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index 082b2753..f657b9e8 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -277,7 +277,7 @@ def run_start(args): logger.info('Starting BigchainDB main process.') from bigchaindb.start import start - start() + start(args) def create_parser(): @@ -360,6 +360,12 @@ def create_parser(): action='store_true', help='Skip database initialization') + start_parser.add_argument('--experimental-parallel-validation', + dest='experimental_parallel_validation', + default=False, + action='store_true', + help='💀 EXPERIMENTAL: parallelize validation for better throughput 💀') + return parser diff --git a/bigchaindb/parallel_validation.py b/bigchaindb/parallel_validation.py new file mode 100644 index 00000000..92a4d515 --- /dev/null +++ b/bigchaindb/parallel_validation.py @@ -0,0 +1,127 @@ +# Copyright BigchainDB GmbH and BigchainDB contributors +# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) +# Code is Apache-2.0 and docs are CC-BY-4.0 + +import multiprocessing as mp +from collections import defaultdict + +from abci.types_pb2 import ResponseCheckTx, ResponseDeliverTx + +from bigchaindb import BigchainDB, App +from bigchaindb.tendermint_utils import decode_transaction + + +CodeTypeOk = 0 + + +class ParallelValidationApp(App): + def __init__(self, bigchaindb=None, events_queue=None): + super().__init__(bigchaindb, events_queue) + self.parallel_validator = ParallelValidator() + self.parallel_validator.start() + + def check_tx(self, raw_transaction): + return ResponseCheckTx(code=CodeTypeOk) + + def deliver_tx(self, raw_transaction): + self.parallel_validator.validate(raw_transaction) + return ResponseDeliverTx(code=CodeTypeOk) + + def end_block(self, request_end_block): + result = self.parallel_validator.result(timeout=30) + for transaction in result: + if transaction: + self.block_txn_ids.append(transaction.id) + self.block_transactions.append(transaction) + + return super().end_block(request_end_block) + + +RESET = 'reset' +EXIT = 'exit' + + +class ParallelValidator: + def __init__(self, number_of_workers=mp.cpu_count()): + self.number_of_workers = number_of_workers + self.transaction_index = 0 + self.routing_queues = [mp.Queue() for _ in range(self.number_of_workers)] + self.workers = [] + self.results_queue = mp.Queue() + + def start(self): + for routing_queue in self.routing_queues: + worker = ValidationWorker(routing_queue, self.results_queue) + process = mp.Process(target=worker.run) + process.start() + self.workers.append(process) + + def stop(self): + for routing_queue in self.routing_queues: + routing_queue.put(EXIT) + + def validate(self, raw_transaction): + dict_transaction = decode_transaction(raw_transaction) + index = int(dict_transaction['id'], 16) % self.number_of_workers + self.routing_queues[index].put((self.transaction_index, dict_transaction)) + self.transaction_index += 1 + + def result(self, timeout=None): + result_buffer = [None] * self.transaction_index + for _ in range(self.transaction_index): + index, transaction = self.results_queue.get(timeout=timeout) + result_buffer[index] = transaction + self.transaction_index = 0 + for routing_queue in self.routing_queues: + routing_queue.put(RESET) + return result_buffer + + +class ValidationWorker: + """Run validation logic in a loop. This Worker is suitable for a Process + life: no thrills, just a queue to get some values, and a queue to return results. + + Note that a worker is expected to validate multiple transactions in + multiple rounds, and it needs to keep in memory all transactions already + validated, until a new round starts. To trigger a new round of validation, + a ValidationWorker expects a `RESET` message. To exit the infinite loop the + worker is in, it expects an `EXIT` message. + """ + + def __init__(self, in_queue, results_queue): + self.in_queue = in_queue + self.results_queue = results_queue + self.bigchaindb = BigchainDB() + self.reset() + + def reset(self): + # We need a place to store already validated transactions, + # in case of dependant transactions in the same block. + # `validated_transactions` maps an `asset_id` with the list + # of all other transactions sharing the same asset. + self.validated_transactions = defaultdict(list) + + def validate(self, dict_transaction): + try: + asset_id = dict_transaction['asset']['id'] + except KeyError: + asset_id = dict_transaction['id'] + + transaction = self.bigchaindb.is_valid_transaction( + dict_transaction, + self.validated_transactions[asset_id]) + + if transaction: + self.validated_transactions[asset_id].append(transaction) + return transaction + + def run(self): + while True: + message = self.in_queue.get() + if message == RESET: + self.reset() + elif message == EXIT: + return + else: + index, transaction = message + self.results_queue.put((index, self.validate(transaction))) diff --git a/bigchaindb/start.py b/bigchaindb/start.py index 41173be0..c23b5441 100644 --- a/bigchaindb/start.py +++ b/bigchaindb/start.py @@ -8,6 +8,7 @@ import setproctitle import bigchaindb from bigchaindb.lib import BigchainDB from bigchaindb.core import App +from bigchaindb.parallel_validation import ParallelValidationApp from bigchaindb.web import server, websocket_server from bigchaindb.events import Exchange, EventTypes from bigchaindb.utils import Process @@ -34,7 +35,7 @@ BANNER = """ """ -def start(): +def start(args): # Exchange object for event stream api logger.info('Starting BigchainDB') exchange = Exchange() @@ -46,7 +47,6 @@ def start(): p_webapi = Process(name='bigchaindb_webapi', target=app_server.run, daemon=True) p_webapi.start() - # start message logger.info(BANNER.format(bigchaindb.config['server']['bind'])) # start websocket server @@ -67,7 +67,10 @@ def start(): setproctitle.setproctitle('bigchaindb') # Start the ABCIServer - app = ABCIServer(app=App(events_queue=exchange.get_publisher_queue())) + if args.experimental_parallel_validation: + app = ABCIServer(app=ParallelValidationApp(events_queue=exchange.get_publisher_queue())) + else: + app = ABCIServer(app=App(events_queue=exchange.get_publisher_queue())) app.run() diff --git a/tests/test_parallel_validation.py b/tests/test_parallel_validation.py new file mode 100644 index 00000000..60fdf0b6 --- /dev/null +++ b/tests/test_parallel_validation.py @@ -0,0 +1,133 @@ +# Copyright BigchainDB GmbH and BigchainDB contributors +# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) +# Code is Apache-2.0 and docs are CC-BY-4.0 + +import pytest + +from bigchaindb.common.crypto import generate_key_pair +from bigchaindb.models import Transaction + + +pytestmark = pytest.mark.tendermint + + +def generate_create_and_transfer(keypair=None): + if not keypair: + keypair = generate_key_pair() + priv_key, pub_key = keypair + create_tx = Transaction.create([pub_key], [([pub_key], 10)]).sign([priv_key]) + transfer_tx = Transaction.transfer( + create_tx.to_inputs(), + [([pub_key], 10)], + asset_id=create_tx.id).sign([priv_key]) + return create_tx, transfer_tx + + +def test_validation_worker_process_multiple_transactions(b): + import multiprocessing as mp + from bigchaindb.parallel_validation import ValidationWorker, RESET, EXIT + + keypair = generate_key_pair() + create_tx, transfer_tx = generate_create_and_transfer(keypair) + double_spend = Transaction.transfer( + create_tx.to_inputs(), + [([keypair.public_key], 10)], + asset_id=create_tx.id).sign([keypair.private_key]) + + in_queue, results_queue = mp.Queue(), mp.Queue() + vw = ValidationWorker(in_queue, results_queue) + + # Note: in the following instructions, the worker will encounter two + # `RESET` messages, and an `EXIT` message. When a worker processes a + # `RESET` message, it forgets all transactions it has validated. This allow + # us to re-validate the same transactions. This won't happen in real life, + # but it's quite handy to check if the worker actually forgot about the + # past transactions (if not, it will return `False` because the + # transactions look like a double spend). + # `EXIT` makes the worker to stop the infinite loop. + in_queue.put((0, create_tx.to_dict())) + in_queue.put((10, transfer_tx.to_dict())) + in_queue.put((20, double_spend.to_dict())) + in_queue.put(RESET) + in_queue.put((0, create_tx.to_dict())) + in_queue.put((5, transfer_tx.to_dict())) + in_queue.put(RESET) + in_queue.put((20, create_tx.to_dict())) + in_queue.put((25, double_spend.to_dict())) + in_queue.put((30, transfer_tx.to_dict())) + in_queue.put(EXIT) + + vw.run() + + assert results_queue.get() == (0, create_tx) + assert results_queue.get() == (10, transfer_tx) + assert results_queue.get() == (20, False) + assert results_queue.get() == (0, create_tx) + assert results_queue.get() == (5, transfer_tx) + assert results_queue.get() == (20, create_tx) + assert results_queue.get() == (25, double_spend) + assert results_queue.get() == (30, False) + + +def test_parallel_validator_routes_transactions_correctly(b, monkeypatch): + import os + from collections import defaultdict + import multiprocessing as mp + from json import dumps + from bigchaindb.parallel_validation import ParallelValidator + + # We want to make sure that the load is distributed across all workers. + # Since introspection on an object running on a different process is + # difficult, we create an additional queue where every worker can emit its + # PID every time validation is called. + validation_called_by = mp.Queue() + + # Validate is now a passthrough, and every time it is called it will emit + # the PID of its worker to the designated queue. + def validate(self, dict_transaction): + validation_called_by.put((os.getpid(), dict_transaction['id'])) + return dict_transaction + + monkeypatch.setattr( + 'bigchaindb.parallel_validation.ValidationWorker.validate', + validate) + + # Transaction routing uses the `id` of the transaction. This test strips + # down a transaction to just its `id`. We have two workers, so even ids + # will be processed by one worker, odd ids by the other. + transactions = [{'id': '0'}, {'id': '1'}, {'id': '2'}, {'id': '3'}] + + pv = ParallelValidator(number_of_workers=2) + pv.start() + + # ParallelValidator is instantiated once, and then used several times. + # Here we simulate this scenario by running it an arbitrary number of + # times. + # Note that the `ParallelValidator.result` call resets the object, and + # makes it ready to validate a new set of transactions. + for _ in range(2): + # First, we push the transactions to the parallel validator instance + for transaction in transactions: + pv.validate(dumps(transaction).encode('utf8')) + + assert pv.result(timeout=1) == transactions + + # Now we analize the transaction processed by the workers + worker_to_transactions = defaultdict(list) + for _ in transactions: + worker_pid, transaction_id = validation_called_by.get() + worker_to_transactions[worker_pid].append(transaction_id) + + # The transactions are stored in two buckets. + for _, transaction_ids in worker_to_transactions.items(): + assert len(transaction_ids) == 2 + + # We have two workers, hence we have two different routes for + # transactions. We have the route for even transactions, and the + # route for odd transactions. Since we don't know which worker + # processed what, we test that the transactions processed by a + # worker are all even or all odd. + assert (all(filter(lambda x: int(x) % 2 == 0, transaction_ids)) or + all(filter(lambda x: int(x) % 2 == 1, transaction_ids))) + + pv.stop() From 6a9064196ad49baf3933051be6e116a3440fbad2 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 13 Sep 2018 14:11:15 +0200 Subject: [PATCH 28/77] Make requirements.txt files consistent with setup.py (#2539) --- docs/contributing/requirements.txt | 2 +- docs/root/requirements.txt | 2 +- docs/server/requirements.txt | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/contributing/requirements.txt b/docs/contributing/requirements.txt index 08bb8d75..d2406438 100644 --- a/docs/contributing/requirements.txt +++ b/docs/contributing/requirements.txt @@ -1,4 +1,4 @@ -Sphinx>=1.4.8 +Sphinx~=1.0 recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 wget diff --git a/docs/root/requirements.txt b/docs/root/requirements.txt index 4321f44b..b092a32c 100644 --- a/docs/root/requirements.txt +++ b/docs/root/requirements.txt @@ -1,4 +1,4 @@ -Sphinx>=1.4.8 +Sphinx~=1.0 recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 sphinxcontrib-napoleon>=0.4.4 diff --git a/docs/server/requirements.txt b/docs/server/requirements.txt index 54e2cd4d..c9fa59e6 100644 --- a/docs/server/requirements.txt +++ b/docs/server/requirements.txt @@ -1,7 +1,7 @@ -Sphinx>=1.4.8 +Sphinx~=1.0 recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 sphinxcontrib-napoleon>=0.4.4 sphinxcontrib-httpdomain>=1.5.0 -pyyaml>=3.12 +pyyaml~=3.12 aafigure>=0.6 From f55f22144fd9a32d08ffb75ee1ea563c37627ed5 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Thu, 13 Sep 2018 15:35:24 +0200 Subject: [PATCH 29/77] Problem: the skip_id param is redundant. (#2531) Solution: remove the param; only call validate_id where it is needed. --- bigchaindb/common/transaction.py | 2 ++ bigchaindb/elections/vote.py | 6 ++---- bigchaindb/models.py | 1 - 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/bigchaindb/common/transaction.py b/bigchaindb/common/transaction.py index 40cc68e7..a1dfc626 100644 --- a/bigchaindb/common/transaction.py +++ b/bigchaindb/common/transaction.py @@ -1195,7 +1195,9 @@ class Transaction(object): """ operation = tx.get('operation', Transaction.CREATE) if isinstance(tx, dict) else Transaction.CREATE cls = Transaction.resolve_class(operation) + if not skip_schema_validation: + cls.validate_id(tx) cls.validate_schema(tx) inputs = [Input.from_dict(input_) for input_ in tx['inputs']] diff --git a/bigchaindb/elections/vote.py b/bigchaindb/elections/vote.py index 01175290..adafb7f3 100644 --- a/bigchaindb/elections/vote.py +++ b/bigchaindb/elections/vote.py @@ -42,16 +42,14 @@ class Vote(Transaction): def generate(cls, inputs, recipients, election_id, metadata=None): (inputs, outputs) = cls.validate_transfer(inputs, recipients, election_id, metadata) election_vote = cls(cls.OPERATION, {'id': election_id}, inputs, outputs, metadata) - cls.validate_schema(election_vote.to_dict(), skip_id=True) + cls.validate_schema(election_vote.to_dict()) return election_vote @classmethod - def validate_schema(cls, tx, skip_id=False): + def validate_schema(cls, tx): """Validate the validator election vote transaction. Since `VOTE` extends `TRANSFER` transaction, all the validations for `CREATE` transaction should be inherited """ - if not skip_id: - cls.validate_id(tx) _validate_schema(TX_SCHEMA_COMMON, tx) _validate_schema(TX_SCHEMA_TRANSFER, tx) _validate_schema(cls.TX_SCHEMA_CUSTOM, tx) diff --git a/bigchaindb/models.py b/bigchaindb/models.py index 9df4dbd4..f53aed02 100644 --- a/bigchaindb/models.py +++ b/bigchaindb/models.py @@ -45,7 +45,6 @@ class Transaction(Transaction): @classmethod def validate_schema(cls, tx_body): - cls.validate_id(tx_body) validate_transaction_schema(tx_body) validate_txn_obj('asset', tx_body['asset'], 'data', validate_key) validate_txn_obj('metadata', tx_body, 'metadata', validate_key) From c68fab6c3137ddfd88fbad987f119e30e2550981 Mon Sep 17 00:00:00 2001 From: Shahbaz Nazir Date: Thu, 13 Sep 2018 16:03:22 +0200 Subject: [PATCH 30/77] Problem: A duplicate tx may result in HTTP 500. * Improve parsing of errors received from Tendermint RPC. --- bigchaindb/lib.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index f6761aa6..2dadcb90 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -100,7 +100,14 @@ class BigchainDB(object): error = response.get('error') if error: - return (500, error) + status_code = 500 + message = error.get('message', 'Internal Error') + data = error.get('data', '') + + if 'Tx already exists in cache' in data: + status_code = 400 + + return (status_code, message + ' - ' + data) result = response['result'] if mode == self.mode_commit: From 61ce427e1d4596edd87752c840c25248204ffad7 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 13 Sep 2018 16:43:15 +0200 Subject: [PATCH 31/77] Problem: Unclear that we want users to use Monit (#2533) Solution: Edit the "How to Set Up a BigchainDB Network" page so that the Monit way is the only fully-documented way. Give a tip for do-it-yourselfers. --- .../network-setup.md | 37 ++++--------------- 1 file changed, 8 insertions(+), 29 deletions(-) diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index 9536c65e..2d5bbe8b 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -264,28 +264,9 @@ If MongoDB isn't running, then you can start it using the command `mongod`, but If you installed MongoDB using `sudo apt install mongodb`, then a MongoDB startup script should already be installed (so MongoDB will start automatically when the machine is restarted). Otherwise, you should install a startup script for MongoDB. -## Member: Start BigchainDB and Tendermint +## Member: Start BigchainDB and Tendermint Using Monit -If you want to use a process manager, jump to the [next section](member-start-bigchaindb-and-tendermint-using-monit). - -To start BigchainDB, one uses the command `bigchaindb start` but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use `nohup`, `tmux`, or `screen`. For example, `nohup bigchaindb start 2>&1 > bigchaindb.log &` - -The _recommended_ approach is to create a startup script for BigchainDB, so it will start right after the boot of the operating system. (As mentioned earlier, MongoDB should already have a startup script.) - -To start Tendermint, one uses the command `tendermint node` but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use `nohup`, `tmux`, or `screen`. For example, `nohup tendermint node 2>&1 > tendermint.log &` - -The _recommended_ approach is to create a startup script for Tendermint, so it will start right after the boot of the operating system. - -Note: We'll share some example startup scripts in the future. This document is a work in progress. - -If you followed the recommended approach and created startup scripts for BigchainDB and Tendermint, then you can reboot the machine now. MongoDB, BigchainDB and Tendermint should all start. - - -### Member: Start BigchainDB and Tendermint using Monit - -This section describes how to manage the BigchainDB and Tendermint processes using [Monit][monit] - a small open-source utility for managing and monitoring Unix processes. - -This section assumes that you followed the guide down to the [start MongoDB section](#member-start-mongodb) inclusive. +This section describes how to manage the BigchainDB and Tendermint processes using [Monit][monit], a small open-source utility for managing and monitoring Unix processes. BigchainDB and Tendermint are managed together, because if BigchainDB is stopped (or crashes) and is restarted, *Tendermint won't try reconnecting to it*. (That's not a bug. It's just how Tendermint works.) Install Monit: @@ -293,32 +274,30 @@ Install Monit: sudo apt install monit ``` -If you installed the `bigchaindb` Python package, you should have the `bigchaindb-monit-config` script in your `PATH` now. - -Run the script: +If you installed the `bigchaindb` Python package as above, you should have the `bigchaindb-monit-config` script in your `PATH` now. Run the script to build a configuration file for Monit: ``` bigchaindb-monit-config ``` -The script builds a configuration file for Monit. - Run Monit as a daemon, instructing it to wake up every second to check on processes: ``` monit -d 1 ``` -It will run the processes and restart them when they crash. If the root `bigchaindb_` process crashes, Monit will also restart the Tendermint process. +Monit will run the BigchainDB and Tendermint processes and restart them when they crash. If the root `bigchaindb_` process crashes, Monit will also restart the Tendermint process. -Check the status by running `monit status` or `monit summary`. +You can check the status by running `monit status` or `monit summary`. By default, it will collect program logs into the `~/.bigchaindb-monit/logs` folder. -Consult `monit -h` or [the Monit documentation][monit-manual] to know more about the operational power you've just got the taste of. +To learn more about Monit, use `monit -h` (help) or read [the Monit documentation][monit-manual]. Check `bigchaindb-monit-config -h` if you want to arrange a different folder for logs or some of the Monit internal artifacts. +If you want to start and manage the BigchainDB and Tendermint processes yourself, then look inside the file [bigchaindb/pkg/scripts/bigchaindb-monit-config](https://github.com/bigchaindb/bigchaindb/blob/master/pkg/scripts/bigchaindb-monit-config) to see how *it* starts BigchainDB and Tendermint. + ## How Others Can Access Your Node If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL `http://hostname:9984` (where hostname is something like `bdb7.canada.vmsareus.net` or `17.122.200.76`). That is, anyone can interact with your node using the [BigchainDB HTTP API](http-client-server-api.html) exposed at that address. The most common way to do that is to use one of the [BigchainDB Drivers](./drivers-clients/index.html). From 53940541320bcc9eacca60907f16027dff1a3efd Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 13 Sep 2018 16:46:02 +0200 Subject: [PATCH 32/77] Add Troubleshooting page to Simple Deployment Template (#2517) --- .../simple-deployment-template/index.rst | 1 + .../network-setup.md | 13 --------- .../troubleshooting.md | 29 +++++++++++++++++++ 3 files changed, 30 insertions(+), 13 deletions(-) create mode 100644 docs/server/source/simple-deployment-template/troubleshooting.md diff --git a/docs/server/source/simple-deployment-template/index.rst b/docs/server/source/simple-deployment-template/index.rst index 90ca20ef..9ee713c2 100644 --- a/docs/server/source/simple-deployment-template/index.rst +++ b/docs/server/source/simple-deployment-template/index.rst @@ -18,3 +18,4 @@ We also have a :ref:`kubernetes-deployment-template` (not simple). :maxdepth: 1 network-setup + troubleshooting diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index 2d5bbe8b..a89c3ac4 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -302,19 +302,6 @@ If you want to start and manage the BigchainDB and Tendermint processes yourself If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL `http://hostname:9984` (where hostname is something like `bdb7.canada.vmsareus.net` or `17.122.200.76`). That is, anyone can interact with your node using the [BigchainDB HTTP API](http-client-server-api.html) exposed at that address. The most common way to do that is to use one of the [BigchainDB Drivers](./drivers-clients/index.html). -## Troubleshooting - -To check which nodes your node is connected to (via Tendermint protocols), do: - -```text -# if you don't jq installed, then install it -sudo apt install jq -# then do -curl -s localhost:26657/net_info | jq ".result.peers[].node_info | {id, listen_addr, moniker}" -``` - -Tendermint has other endpoints besides `/net_info`: see [the Tendermint RPC docs](https://tendermint.github.io/slate/?shell#introduction). - ## Refreshing Your Node If you want to refresh your node back to a fresh empty state, then your best bet is to terminate it and deploy a new virtual machine, but if that's not an option, then you can: diff --git a/docs/server/source/simple-deployment-template/troubleshooting.md b/docs/server/source/simple-deployment-template/troubleshooting.md new file mode 100644 index 00000000..2d0311fd --- /dev/null +++ b/docs/server/source/simple-deployment-template/troubleshooting.md @@ -0,0 +1,29 @@ +# Troubleshooting + +## General Tips + +- Check the BigchainDB, Tendermint and MongoDB logs. +- Try Googling the error message. + +## Resolving Tendermint Connectivity Problems + +To check which nodes your node is connected to (via Tendermint protocols), do: + +```text +# if you don't have jq installed, then install it +sudo apt install jq +# then do +curl -s localhost:26657/net_info | jq ".result.peers[].node_info | {id, listen_addr, moniker}" +``` + +Note: Tendermint has other endpoints besides `/net_info`: see [the Tendermint RPC docs](https://tendermint.github.io/slate/?shell#introduction). + +If you're running your network inside a [private network](https://en.wikipedia.org/wiki/Private_network), e.g. with IP addresses of the form 192.168.x.y, then you may have to change the following setting in `config.toml`: + +```text +addr_book_strict = false +``` + +## Other Problems + +If you're stuck, maybe [file a new issue on GitHub](https://github.com/bigchaindb/bigchaindb/issues/new). If your problem occurs often enough, we'll write about it here. From 754730a045ab502d7b1dbc0e80de4a345aa563f1 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 14 Sep 2018 11:47:39 +0200 Subject: [PATCH 33/77] Update the Quickstart page (#2538) * Update the Quickstart page * Add links to docs about how to deploy a local node --- docs/server/source/quickstart.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/docs/server/source/quickstart.md b/docs/server/source/quickstart.md index 92193e64..4359577f 100644 --- a/docs/server/source/quickstart.md +++ b/docs/server/source/quickstart.md @@ -51,16 +51,15 @@ Code is Apache-2.0 and docs are CC-BY-4.0 Create a BigchainDB transaction and post it to a BigchainDB network in 20 seconds or less:
- Try BigchainDB Now + Try BigchainDB Now
## Develop an App -To develop an app that talks to a BigchainDB network, you'll want a test network to test it against. The BigchainDB Testnet is a free-to-use, publicly-available BigchainDB network that you can test against. +To develop an app that talks to a BigchainDB network, you'll want a test network to test it against. You have a few options: -
- Get started with the BigchainDB Testnet -
+1. The BigchainDB Test Network (or "Testnet") is a free-to-use, publicly-available BigchainDB network that you can test against. You can find out more about the BigchainDB Testnet on [the BigchainDB "Get Started" page](https://www.bigchaindb.com/developers/getstarted/#server). +1. You could also run a BigchainDB node on you local machine. One way is to use the [Simple Deployment Template](./simple-deployment-template/index.html), with a one-node "network." Another way is to use one of the deployment methods listed in the [the docs about contributing to BigchainDB](https://docs.bigchaindb.com/projects/contributing/en/latest/index.html). Regardless of which BigchainDB network you use, you'll probably use one of the [BigchainDB drivers or tools](https://www.bigchaindb.com/getstarted/#drivers). @@ -68,10 +67,6 @@ Regardless of which BigchainDB network you use, you'll probably use one of the [ To help develop BigchainDB Server (the core software in each BigchainDB node), see [the docs about contributing to BigchainDB](https://docs.bigchaindb.com/projects/contributing/en/latest/index.html). -## Old Quickstart - -If you want something like the old Quickstart page, i.e. some command-line instructions to set up and run a BigchainDB node, then see [the notes on running a local dev node in the docs about contributing to BigchainDB](https://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/index.html). -

From bd390765224c9b8755fc7dcd9ac468a690236f25 Mon Sep 17 00:00:00 2001 From: Muawia Khan Date: Fri, 14 Sep 2018 15:37:18 +0200 Subject: [PATCH 34/77] Problem: Users trying to use an incompatible version of Tendermint (#2541) - Hard-wire the supported Tendermint version(s) right in the code of BigchainDB Server. Check the version of Tendermint and disconnect if Tendermint version is an unsupported one. - Expose a CLI command bigchaindb tendermint-version to display the supported Tendermint versions. - PR also takes care the long list of warnings we get when we run tests. - Updated deprecated pymongo methods - Do not call pytest fixtures directly. - Also added the doc for the new cli command --- bigchaindb/backend/localmongodb/query.py | 4 +- bigchaindb/commands/bigchaindb.py | 13 +++++ bigchaindb/core.py | 10 ++++ bigchaindb/utils.py | 23 ++++++++- bigchaindb/version.py | 2 + .../source/server-reference/bigchaindb-cli.md | 15 +++++- tests/backend/localmongodb/test_queries.py | 48 +++++++++---------- tests/backend/localmongodb/test_schema.py | 10 ++-- tests/commands/test_commands.py | 21 ++++++-- tests/conftest.py | 14 +----- tests/tendermint/test_core.py | 3 +- tests/tendermint/test_integration.py | 15 +++++- tests/tendermint/test_lib.py | 48 +++++++++---------- 13 files changed, 151 insertions(+), 75 deletions(-) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index 4c016f18..75fa7aa5 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -234,7 +234,7 @@ def store_unspent_outputs(conn, *unspent_outputs): def delete_unspent_outputs(conn, *unspent_outputs): if unspent_outputs: return conn.run( - conn.collection('utxos').remove({ + conn.collection('utxos').delete_many({ '$or': [{ '$and': [ {'transaction_id': unspent_output['transaction_id']}, @@ -258,7 +258,7 @@ def store_pre_commit_state(conn, state): commit_id = state['commit_id'] return conn.run( conn.collection('pre_commit') - .update({'commit_id': commit_id}, state, upsert=True) + .replace_one({'commit_id': commit_id}, state, upsert=True) ) diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index f657b9e8..96d555bb 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -29,6 +29,7 @@ from bigchaindb.commands.utils import (configure_bigchaindb, from bigchaindb.log import setup_logging from bigchaindb.tendermint_utils import public_key_from_base64 from bigchaindb.commands.election_types import elections +from bigchaindb.version import __tm_supported_versions__ logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) @@ -280,6 +281,15 @@ def run_start(args): start(args) +def run_tendermint_version(args): + """Show the supported Tendermint version(s)""" + supported_tm_ver = { + 'description': 'BigchainDB supports the following Tendermint version(s)', + 'tendermint': __tm_supported_versions__, + } + print(json.dumps(supported_tm_ver, indent=4, sort_keys=True)) + + def create_parser(): parser = argparse.ArgumentParser( description='Control your BigchainDB node.', @@ -360,6 +370,9 @@ def create_parser(): action='store_true', help='Skip database initialization') + subparsers.add_parser('tendermint-version', + help='Show the Tendermint supported versions') + start_parser.add_argument('--experimental-parallel-validation', dest='experimental_parallel_validation', default=False, diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 2f9fa23f..f936bd91 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -20,6 +20,8 @@ from abci.types_pb2 import ( ) from bigchaindb import BigchainDB +from bigchaindb.version import __tm_supported_versions__ +from bigchaindb.utils import tendermint_version_is_compatible from bigchaindb.tendermint_utils import (decode_transaction, calculate_hash) from bigchaindb.lib import Block, PreCommitState @@ -115,6 +117,14 @@ class App(BaseApplication): self.abort_if_abci_chain_is_not_synced() + # Check if BigchainDB supports the Tendermint version + if not (hasattr(request, 'version') and tendermint_version_is_compatible(request.version)): + logger.error(f'Unsupported Tendermint version: {getattr(request, "version", "no version")}.' + f' Currently, BigchainDB only supports {__tm_supported_versions__}. Exiting!') + sys.exit(1) + + logger.info(f"Tendermint version: {request.version}") + r = ResponseInfo() block = self.bigchaindb.get_latest_block() if block: diff --git a/bigchaindb/utils.py b/bigchaindb/utils.py index 78777133..c2b6ef81 100644 --- a/bigchaindb/utils.py +++ b/bigchaindb/utils.py @@ -9,7 +9,8 @@ import multiprocessing as mp import json import setproctitle - +from packaging import version +from bigchaindb.version import __tm_supported_versions__ from bigchaindb.tendermint_utils import key_from_base64 from bigchaindb.common.crypto import key_pair_from_ed25519_key @@ -185,3 +186,23 @@ def load_node_key(path): priv_key = priv_validator['priv_key']['value'] hex_private_key = key_from_base64(priv_key) return key_pair_from_ed25519_key(hex_private_key) + + +def tendermint_version_is_compatible(running_tm_ver): + """ + Check Tendermint compatability with BigchainDB server + + :param running_tm_ver: Version number of the connected Tendermint instance + :type running_tm_ver: str + :return: True/False depending on the compatability with BigchainDB server + :rtype: bool + """ + + # Splitting because version can look like this e.g. 0.22.8-40d6dc2e + tm_ver = running_tm_ver.split('-') + if not tm_ver: + return False + for ver in __tm_supported_versions__: + if version.parse(ver) == version.parse(tm_ver[0]): + return True + return False diff --git a/bigchaindb/version.py b/bigchaindb/version.py index 3120eea1..277f5645 100644 --- a/bigchaindb/version.py +++ b/bigchaindb/version.py @@ -4,3 +4,5 @@ __version__ = '2.0.0b5' __short_version__ = '2.0b5' +# supported Tendermint version +__tm_supported_versions__ = ["0.22.8"] diff --git a/docs/server/source/server-reference/bigchaindb-cli.md b/docs/server/source/server-reference/bigchaindb-cli.md index b67ca296..44fa2687 100644 --- a/docs/server/source/server-reference/bigchaindb-cli.md +++ b/docs/server/source/server-reference/bigchaindb-cli.md @@ -158,4 +158,17 @@ $ bigchaindb election show ELECTION_ID status= ``` -The election data is the same set of arguments used in the `election new` command that originally triggered the election. `status` takes three possible values, `ongoing`, if the election has not yet reached a 2/3 majority, `concluded`, if the election reached the 2/3 majority needed to pass, or `inconclusive`, if the validator set changed while the election was in process, rendering it undecidable. \ No newline at end of file +The election data is the same set of arguments used in the `election new` command that originally triggered the election. `status` takes three possible values, `ongoing`, if the election has not yet reached a 2/3 majority, `concluded`, if the election reached the 2/3 majority needed to pass, or `inconclusive`, if the validator set changed while the election was in process, rendering it undecidable. + +## bigchaindb tendermint-version + +Show the Tendermint versions supported by BigchainDB server. +```bash +$ bigchaindb tendermint-version +{ + "description": "BigchainDB supports the following Tendermint version(s)", + "tendermint": [ + "0.22.8" + ] +} +``` diff --git a/tests/backend/localmongodb/test_queries.py b/tests/backend/localmongodb/test_queries.py index dd8621d4..6a7ae2c4 100644 --- a/tests/backend/localmongodb/test_queries.py +++ b/tests/backend/localmongodb/test_queries.py @@ -58,7 +58,7 @@ def test_write_assets(): cursor = conn.db.assets.find({}, projection={'_id': False})\ .sort('id', pymongo.ASCENDING) - assert cursor.count() == 3 + assert cursor.collection.count_documents({}) == 3 assert list(cursor) == assets[:-1] @@ -180,7 +180,7 @@ def test_write_metadata(): cursor = conn.db.metadata.find({}, projection={'_id': False})\ .sort('id', pymongo.ASCENDING) - assert cursor.count() == 3 + assert cursor.collection.count_documents({}) == 3 assert list(cursor) == metadata @@ -244,7 +244,7 @@ def test_store_block(): transactions=[]) query.store_block(conn, block._asdict()) cursor = conn.db.blocks.find({}, projection={'_id': False}) - assert cursor.count() == 1 + assert cursor.collection.count_documents({}) == 1 def test_get_block(): @@ -267,14 +267,14 @@ def test_delete_zero_unspent_outputs(db_context, utxoset): unspent_outputs, utxo_collection = utxoset delete_res = query.delete_unspent_outputs(db_context.conn) assert delete_res is None - assert utxo_collection.count() == 3 - assert utxo_collection.find( + assert utxo_collection.count_documents({}) == 3 + assert utxo_collection.count_documents( {'$or': [ {'transaction_id': 'a', 'output_index': 0}, {'transaction_id': 'b', 'output_index': 0}, {'transaction_id': 'a', 'output_index': 1}, ]} - ).count() == 3 + ) == 3 def test_delete_one_unspent_outputs(db_context, utxoset): @@ -282,15 +282,15 @@ def test_delete_one_unspent_outputs(db_context, utxoset): unspent_outputs, utxo_collection = utxoset delete_res = query.delete_unspent_outputs(db_context.conn, unspent_outputs[0]) - assert delete_res['n'] == 1 - assert utxo_collection.find( + assert delete_res.raw_result['n'] == 1 + assert utxo_collection.count_documents( {'$or': [ {'transaction_id': 'a', 'output_index': 1}, {'transaction_id': 'b', 'output_index': 0}, ]} - ).count() == 2 - assert utxo_collection.find( - {'transaction_id': 'a', 'output_index': 0}).count() == 0 + ) == 2 + assert utxo_collection.count_documents( + {'transaction_id': 'a', 'output_index': 0}) == 0 def test_delete_many_unspent_outputs(db_context, utxoset): @@ -298,22 +298,22 @@ def test_delete_many_unspent_outputs(db_context, utxoset): unspent_outputs, utxo_collection = utxoset delete_res = query.delete_unspent_outputs(db_context.conn, *unspent_outputs[::2]) - assert delete_res['n'] == 2 - assert utxo_collection.find( + assert delete_res.raw_result['n'] == 2 + assert utxo_collection.count_documents( {'$or': [ {'transaction_id': 'a', 'output_index': 0}, {'transaction_id': 'b', 'output_index': 0}, ]} - ).count() == 0 - assert utxo_collection.find( - {'transaction_id': 'a', 'output_index': 1}).count() == 1 + ) == 0 + assert utxo_collection.count_documents( + {'transaction_id': 'a', 'output_index': 1}) == 1 def test_store_zero_unspent_output(db_context, utxo_collection): from bigchaindb.backend import query res = query.store_unspent_outputs(db_context.conn) assert res is None - assert utxo_collection.count() == 0 + assert utxo_collection.count_documents({}) == 0 def test_store_one_unspent_output(db_context, @@ -322,10 +322,10 @@ def test_store_one_unspent_output(db_context, res = query.store_unspent_outputs(db_context.conn, unspent_output_1) assert res.acknowledged assert len(res.inserted_ids) == 1 - assert utxo_collection.find( + assert utxo_collection.count_documents( {'transaction_id': unspent_output_1['transaction_id'], 'output_index': unspent_output_1['output_index']} - ).count() == 1 + ) == 1 def test_store_many_unspent_outputs(db_context, @@ -334,15 +334,15 @@ def test_store_many_unspent_outputs(db_context, res = query.store_unspent_outputs(db_context.conn, *unspent_outputs) assert res.acknowledged assert len(res.inserted_ids) == 3 - assert utxo_collection.find( + assert utxo_collection.count_documents( {'transaction_id': unspent_outputs[0]['transaction_id']} - ).count() == 3 + ) == 3 def test_get_unspent_outputs(db_context, utxoset): from bigchaindb.backend import query cursor = query.get_unspent_outputs(db_context.conn) - assert cursor.count() == 3 + assert cursor.collection.count_documents({}) == 3 retrieved_utxoset = list(cursor) unspent_outputs, utxo_collection = utxoset assert retrieved_utxoset == list( @@ -361,7 +361,7 @@ def test_store_pre_commit_state(db_context): query.store_pre_commit_state(db_context.conn, state._asdict()) cursor = db_context.conn.db.pre_commit.find({'commit_id': 'test'}, projection={'_id': False}) - assert cursor.count() == 1 + assert cursor.collection.count_documents({}) == 1 def test_get_pre_commit_state(db_context): @@ -372,7 +372,7 @@ def test_get_pre_commit_state(db_context): height=3, transactions=[]) - db_context.conn.db.pre_commit.insert(state._asdict()) + db_context.conn.db.pre_commit.insert_one(state._asdict()) resp = query.get_pre_commit_state(db_context.conn, 'test2') assert resp == state._asdict() diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index d02a8fde..a96e7b63 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -16,7 +16,7 @@ def test_init_creates_db_tables_and_indexes(): init_database() - collection_names = conn.conn[dbname].collection_names() + collection_names = conn.conn[dbname].list_collection_names() assert set(collection_names) == { 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'pre_commit', 'validators', 'elections', 'abci_chains', @@ -57,7 +57,7 @@ def test_init_database_is_graceful_if_db_exists(): dbname = bigchaindb.config['database']['name'] # The db is set up by the fixtures - assert dbname in conn.conn.database_names() + assert dbname in conn.conn.list_database_names() init_database() @@ -75,7 +75,7 @@ def test_create_tables(): schema.create_database(conn, dbname) schema.create_tables(conn, dbname) - collection_names = conn.conn[dbname].collection_names() + collection_names = conn.conn[dbname].list_collection_names() assert set(collection_names) == { 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'validators', 'elections', 'pre_commit', 'abci_chains', @@ -115,6 +115,6 @@ def test_drop(dummy_db): from bigchaindb.backend import schema conn = backend.connect() - assert dummy_db in conn.conn.database_names() + assert dummy_db in conn.conn.list_database_names() schema.drop_database(conn, dummy_db) - assert dummy_db not in conn.conn.database_names() + assert dummy_db not in conn.conn.list_database_names() diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index a9cb9d6a..38e251d8 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -11,7 +11,6 @@ from argparse import Namespace import pytest from bigchaindb import ValidatorElection -from tests.conftest import node_keys def test_make_sure_we_dont_remove_any_command(): @@ -30,6 +29,7 @@ def test_make_sure_we_dont_remove_any_command(): assert parser.parse_args(['election', 'approve', 'ELECTION_ID', '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command assert parser.parse_args(['election', 'show', 'ELECTION_ID']).command + assert parser.parse_args(['tendermint-version']).command @patch('bigchaindb.commands.utils.start') @@ -226,6 +226,9 @@ def test_calling_main(start_mock, monkeypatch): subparsers.add_parser.assert_any_call('drop', help='Drop the database') subparsers.add_parser.assert_any_call('start', help='Start BigchainDB') + subparsers.add_parser.assert_any_call('tendermint-version', + help='Show the Tendermint supported ' + 'versions') assert start_mock.called is True @@ -468,11 +471,21 @@ def test_election_approve_called_with_bad_key(caplog, b, bad_validator_path, new 'the eligible voters in this election.' +def test_bigchain_tendermint_version(capsys): + from bigchaindb.commands.bigchaindb import run_tendermint_version + + args = Namespace(config=None) + _, _ = capsys.readouterr() + run_tendermint_version(args) + output_config = json.loads(capsys.readouterr()[0]) + from bigchaindb.version import __tm_supported_versions__ + assert len(output_config["tendermint"]) == len(__tm_supported_versions__) + assert sorted(output_config["tendermint"]) == sorted(__tm_supported_versions__) + + def mock_get_validators(height): - keys = node_keys() - pub_key = list(keys.keys())[0] return [ - {'public_key': {'value': pub_key, + {'public_key': {'value': "zL/DasvKulXZzhSNFwx4cLRXKkSM9GPK7Y0nZ4FEylM=", 'type': 'ed25519-base64'}, 'voting_power': 10} ] diff --git a/tests/conftest.py b/tests/conftest.py index fb9615b1..15cfcf01 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -198,16 +198,6 @@ def alice(): return generate_key_pair() -@pytest.fixture -def alice_privkey(alice): - return alice.private_key - - -@pytest.fixture -def alice_pubkey(alice): - return alice.public_key - - @pytest.fixture def bob(): from bigchaindb.common.crypto import generate_key_pair @@ -300,10 +290,10 @@ def inputs(user_pk, b, alice): for height in range(1, 4): transactions = [ Transaction.create( - [alice_pubkey(alice)], + [alice.public_key], [([user_pk], 1)], metadata={'msg': random.random()}, - ).sign([alice_privkey(alice)]) + ).sign([alice.private_key]) for _ in range(10) ] tx_ids = [tx.id for tx in transactions] diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index 5875ed9b..d62b4786 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -26,6 +26,7 @@ from bigchaindb.core import (CodeTypeOk, from bigchaindb.lib import Block from bigchaindb.upsert_validator.validator_utils import new_validator_set from bigchaindb.tendermint_utils import public_key_to_base64 +from bigchaindb.version import __tm_supported_versions__ pytestmark = pytest.mark.bdb @@ -168,7 +169,7 @@ def test_init_chain_recognizes_new_chain_after_migration(b): def test_info(b): - r = RequestInfo() + r = RequestInfo(version=__tm_supported_versions__[0]) app = App(b) res = app.info(r) diff --git a/tests/tendermint/test_integration.py b/tests/tendermint/test_integration.py index 490da16b..da373442 100644 --- a/tests/tendermint/test_integration.py +++ b/tests/tendermint/test_integration.py @@ -11,6 +11,7 @@ import pytest from abci.server import ProtocolHandler from abci.encoding import read_messages +from bigchaindb.version import __tm_supported_versions__ from io import BytesIO @@ -24,7 +25,8 @@ def test_app(b, init_chain_request): app = App(b) p = ProtocolHandler(app) - data = p.process('info', types.Request(info=types.RequestInfo(version='2'))) + data = p.process('info', + types.Request(info=types.RequestInfo(version=__tm_supported_versions__[0]))) res = next(read_messages(BytesIO(data), types.Response)) assert res assert res.info.last_block_app_hash == b'' @@ -139,3 +141,14 @@ def test_post_transaction_responses(tendermint_ws_url, b): code, message = b.write_transaction(double_spend, mode) assert code == 500 assert message == 'Transaction validation failed' + + +@pytest.mark.bdb +def test_exit_when_tm_ver_not_supported(b): + from bigchaindb import App + + app = App(b) + p = ProtocolHandler(app) + + with pytest.raises(SystemExit): + p.process('info', types.Request(info=types.RequestInfo(version='2'))) diff --git a/tests/tendermint/test_lib.py b/tests/tendermint/test_lib.py index 3eec8903..701ab3ca 100644 --- a/tests/tendermint/test_lib.py +++ b/tests/tendermint/test_lib.py @@ -173,12 +173,12 @@ def test_update_utxoset(b, signed_create_tx, signed_transfer_tx, db_context): mongo_client = MongoClient(host=db_context.host, port=db_context.port) b.update_utxoset(signed_create_tx) utxoset = mongo_client[db_context.name]['utxos'] - assert utxoset.count() == 1 + assert utxoset.count_documents({}) == 1 utxo = utxoset.find_one() assert utxo['transaction_id'] == signed_create_tx.id assert utxo['output_index'] == 0 b.update_utxoset(signed_transfer_tx) - assert utxoset.count() == 1 + assert utxoset.count_documents({}) == 1 utxo = utxoset.find_one() assert utxo['transaction_id'] == signed_transfer_tx.id assert utxo['output_index'] == 0 @@ -195,7 +195,7 @@ def test_store_transaction(mocker, b, signed_create_tx, b.store_bulk_transactions([signed_create_tx]) # mongo_client = MongoClient(host=db_context.host, port=db_context.port) # utxoset = mongo_client[db_context.name]['utxos'] - # assert utxoset.count() == 1 + # assert utxoset.count_documents({}) == 1 # utxo = utxoset.find_one() # assert utxo['transaction_id'] == signed_create_tx.id # assert utxo['output_index'] == 0 @@ -217,7 +217,7 @@ def test_store_transaction(mocker, b, signed_create_tx, mocked_store_metadata.reset_mock() mocked_store_transaction.reset_mock() b.store_bulk_transactions([signed_transfer_tx]) - # assert utxoset.count() == 1 + # assert utxoset.count_documents({}) == 1 # utxo = utxoset.find_one() # assert utxo['transaction_id'] == signed_transfer_tx.id # assert utxo['output_index'] == 0 @@ -245,7 +245,7 @@ def test_store_bulk_transaction(mocker, b, signed_create_tx, b.store_bulk_transactions((signed_create_tx,)) # mongo_client = MongoClient(host=db_context.host, port=db_context.port) # utxoset = mongo_client[db_context.name]['utxos'] - # assert utxoset.count() == 1 + # assert utxoset.count_documents({}) == 1 # utxo = utxoset.find_one() # assert utxo['transaction_id'] == signed_create_tx.id # assert utxo['output_index'] == 0 @@ -266,7 +266,7 @@ def test_store_bulk_transaction(mocker, b, signed_create_tx, mocked_store_metadata.reset_mock() mocked_store_transactions.reset_mock() b.store_bulk_transactions((signed_transfer_tx,)) - # assert utxoset.count() == 1 + # assert utxoset.count_documents({}) == 1 # utxo = utxoset.find_one() # assert utxo['transaction_id'] == signed_transfer_tx.id # assert utxo['output_index'] == 0 @@ -288,51 +288,51 @@ def test_delete_zero_unspent_outputs(b, utxoset): unspent_outputs, utxo_collection = utxoset delete_res = b.delete_unspent_outputs() assert delete_res is None - assert utxo_collection.count() == 3 - assert utxo_collection.find( + assert utxo_collection.count_documents({}) == 3 + assert utxo_collection.count_documents( {'$or': [ {'transaction_id': 'a', 'output_index': 0}, {'transaction_id': 'b', 'output_index': 0}, {'transaction_id': 'a', 'output_index': 1}, ]} - ).count() == 3 + ) == 3 @pytest.mark.bdb def test_delete_one_unspent_outputs(b, utxoset): unspent_outputs, utxo_collection = utxoset delete_res = b.delete_unspent_outputs(unspent_outputs[0]) - assert delete_res['n'] == 1 - assert utxo_collection.find( + assert delete_res.raw_result['n'] == 1 + assert utxo_collection.count_documents( {'$or': [ {'transaction_id': 'a', 'output_index': 1}, {'transaction_id': 'b', 'output_index': 0}, ]} - ).count() == 2 - assert utxo_collection.find( - {'transaction_id': 'a', 'output_index': 0}).count() == 0 + ) == 2 + assert utxo_collection.count_documents( + {'transaction_id': 'a', 'output_index': 0}) == 0 @pytest.mark.bdb def test_delete_many_unspent_outputs(b, utxoset): unspent_outputs, utxo_collection = utxoset delete_res = b.delete_unspent_outputs(*unspent_outputs[::2]) - assert delete_res['n'] == 2 - assert utxo_collection.find( + assert delete_res.raw_result['n'] == 2 + assert utxo_collection.count_documents( {'$or': [ {'transaction_id': 'a', 'output_index': 0}, {'transaction_id': 'b', 'output_index': 0}, ]} - ).count() == 0 - assert utxo_collection.find( - {'transaction_id': 'a', 'output_index': 1}).count() == 1 + ) == 0 + assert utxo_collection.count_documents( + {'transaction_id': 'a', 'output_index': 1}) == 1 @pytest.mark.bdb def test_store_zero_unspent_output(b, utxo_collection): res = b.store_unspent_outputs() assert res is None - assert utxo_collection.count() == 0 + assert utxo_collection.count_documents({}) == 0 @pytest.mark.bdb @@ -340,10 +340,10 @@ def test_store_one_unspent_output(b, unspent_output_1, utxo_collection): res = b.store_unspent_outputs(unspent_output_1) assert res.acknowledged assert len(res.inserted_ids) == 1 - assert utxo_collection.find( + assert utxo_collection.count_documents( {'transaction_id': unspent_output_1['transaction_id'], 'output_index': unspent_output_1['output_index']} - ).count() == 1 + ) == 1 @pytest.mark.bdb @@ -351,9 +351,9 @@ def test_store_many_unspent_outputs(b, unspent_outputs, utxo_collection): res = b.store_unspent_outputs(*unspent_outputs) assert res.acknowledged assert len(res.inserted_ids) == 3 - assert utxo_collection.find( + assert utxo_collection.count_documents( {'transaction_id': unspent_outputs[0]['transaction_id']} - ).count() == 3 + ) == 3 def test_get_utxoset_merkle_root_when_no_utxo(b): From 3cb0f8e2ab532418503c44d4996323cb1c6acf02 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 14 Sep 2018 16:25:36 +0200 Subject: [PATCH 35/77] Update the list of MongoDB collections in query.rst (#2543) --- docs/root/source/query.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/root/source/query.rst b/docs/root/source/query.rst index 6ea9130f..2ffe2543 100644 --- a/docs/root/source/query.rst +++ b/docs/root/source/query.rst @@ -61,8 +61,10 @@ For example, if you're on a machine that's running a default BigchainDB node, th > use bigchain switched to db bigchain > show collections + abci_chains assets blocks + elections metadata pre_commit transactions From e6893632dc0a74c8bb92f10ba410ffb16fd194ca Mon Sep 17 00:00:00 2001 From: Zachary Bowen Date: Mon, 17 Sep 2018 13:59:57 +0200 Subject: [PATCH 36/77] Create migration election class (#2535). * Problem: We need a way to synchronize a halt to block production to allow for upgrades across breaking changes. * Solution: Created `MigrationElection`. * Problem: Need documentation for `migration` elections. * Solution: Updated the docs. * Problem: `MigrationElection` needs 'new' CLI method. * Solution: Updated the definition of `election` to include the new `migration` type. * Problem: The way `end_block` checks for concluded elections assumes there is only one type of election (so we can't conclude an `upsert-validator` and a `chain-migration` at the same height). * Solution: Re-engineered the code in `Elections` to conclude multiple elections in the same block. If more than one election change the validator set, only one of them is applied. * Problem: Tendermint change to store validator changes at height h+2 will break `Election.get_status`. * Solution: Reworked `get_validator_change` to look at only the latest block height or less. --- bigchaindb/__init__.py | 2 + bigchaindb/commands/bigchaindb.py | 60 ++++--- bigchaindb/commands/election_types.py | 9 + bigchaindb/common/schema/__init__.py | 3 + ...saction_chain_migration_election_v2.0.yaml | 44 +++++ .../common/schema/transaction_v2.0.yaml | 1 + bigchaindb/core.py | 15 +- bigchaindb/elections/election.py | 57 ++++--- .../migrations/chain_migration_election.py | 15 ++ .../upsert_validator/validator_election.py | 14 +- .../source/server-reference/bigchaindb-cli.md | 31 +++- tests/commands/test_commands.py | 40 ++++- tests/conftest.py | 140 ++++++++++++++- tests/elections/__init__.py | 0 tests/elections/test_election.py | 64 +++++++ tests/migrations/test_migration_election.py | 9 + tests/upsert_validator/conftest.py | 72 +------- .../test_upsert_validator_vote.py | 161 ++++++++---------- .../test_validator_election.py | 18 +- tests/utils.py | 19 +++ 20 files changed, 551 insertions(+), 223 deletions(-) create mode 100644 bigchaindb/common/schema/transaction_chain_migration_election_v2.0.yaml create mode 100644 bigchaindb/migrations/chain_migration_election.py create mode 100644 tests/elections/__init__.py create mode 100644 tests/elections/test_election.py create mode 100644 tests/migrations/test_migration_election.py diff --git a/bigchaindb/__init__.py b/bigchaindb/__init__.py index e1762f5d..223d7a0e 100644 --- a/bigchaindb/__init__.py +++ b/bigchaindb/__init__.py @@ -7,6 +7,7 @@ import logging from bigchaindb.log import DEFAULT_LOGGING_CONFIG as log_config from bigchaindb.lib import BigchainDB # noqa +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection from bigchaindb.version import __version__ # noqa from bigchaindb.core import App # noqa @@ -99,4 +100,5 @@ from bigchaindb.elections.vote import Vote # noqa Transaction.register_type(Transaction.CREATE, models.Transaction) Transaction.register_type(Transaction.TRANSFER, models.Transaction) Transaction.register_type(ValidatorElection.OPERATION, ValidatorElection) +Transaction.register_type(ChainMigrationElection.OPERATION, ChainMigrationElection) Transaction.register_type(Vote.OPERATION, Vote) diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index 96d555bb..f9588dcd 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -13,6 +13,7 @@ import copy import json import sys +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection from bigchaindb.utils import load_node_key from bigchaindb.common.exceptions import (DatabaseDoesNotExist, ValidationError) @@ -115,6 +116,31 @@ def run_election_new(args, bigchain): globals()[f'run_election_new_{args.election_type}'](args, bigchain) +def create_new_election(sk, bigchain, election_class, data): + + try: + key = load_node_key(sk) + voters = election_class.recipients(bigchain) + election = election_class.generate([key.public_key], + voters, + data, None).sign([key.private_key]) + election.validate(bigchain) + except ValidationError as e: + logger.error(e) + return False + except FileNotFoundError as fd_404: + logger.error(fd_404) + return False + + resp = bigchain.write_transaction(election, 'broadcast_tx_commit') + if resp == (202, ''): + logger.info('[SUCCESS] Submitted proposal with id: {}'.format(election.id)) + return election.id + else: + logger.error('Failed to commit election proposal') + return False + + def run_election_new_upsert_validator(args, bigchain): """Initiates an election to add/update/remove a validator to an existing BigchainDB network @@ -136,27 +162,21 @@ def run_election_new_upsert_validator(args, bigchain): 'node_id': args.node_id } - try: - key = load_node_key(args.sk) - voters = ValidatorElection.recipients(bigchain) - election = ValidatorElection.generate([key.public_key], - voters, - new_validator, None).sign([key.private_key]) - election.validate(bigchain) - except ValidationError as e: - logger.error(e) - return False - except FileNotFoundError as fd_404: - logger.error(fd_404) - return False + return create_new_election(args.sk, bigchain, ValidatorElection, new_validator) - resp = bigchain.write_transaction(election, 'broadcast_tx_commit') - if resp == (202, ''): - logger.info('[SUCCESS] Submitted proposal with id: {}'.format(election.id)) - return election.id - else: - logger.error('Failed to commit election proposal') - return False + +def run_election_new_chain_migration(args, bigchain): + """Initiates an election to halt block production + + :param args: dict + args = { + 'sk': the path to the private key of the node calling the election (str) + } + :param bigchain: an instance of BigchainDB + :return: election_id or `False` in case of failure + """ + + return create_new_election(args.sk, bigchain, ChainMigrationElection, {}) def run_election_approve(args, bigchain): diff --git a/bigchaindb/commands/election_types.py b/bigchaindb/commands/election_types.py index b57522f6..4745b85d 100644 --- a/bigchaindb/commands/election_types.py +++ b/bigchaindb/commands/election_types.py @@ -16,5 +16,14 @@ elections = { 'help': 'Path to the private key of the election initiator.' } } + }, + 'chain-migration': { + 'help': 'Call for a halt to block production to allow for a version change across breaking changes.', + 'args': { + '--private-key': { + 'dest': 'sk', + 'help': 'Path to the private key of the election initiator.' + } + } } } diff --git a/bigchaindb/common/schema/__init__.py b/bigchaindb/common/schema/__init__.py index 25943675..6280b7b9 100644 --- a/bigchaindb/common/schema/__init__.py +++ b/bigchaindb/common/schema/__init__.py @@ -37,6 +37,9 @@ _, TX_SCHEMA_TRANSFER = _load_schema('transaction_transfer_' + _, TX_SCHEMA_VALIDATOR_ELECTION = _load_schema('transaction_validator_election_' + TX_SCHEMA_VERSION) +_, TX_SCHEMA_CHAIN_MIGRATION_ELECTION = _load_schema('transaction_chain_migration_election_' + + TX_SCHEMA_VERSION) + _, TX_SCHEMA_VOTE = _load_schema('transaction_vote_' + TX_SCHEMA_VERSION) diff --git a/bigchaindb/common/schema/transaction_chain_migration_election_v2.0.yaml b/bigchaindb/common/schema/transaction_chain_migration_election_v2.0.yaml new file mode 100644 index 00000000..40fa1a7b --- /dev/null +++ b/bigchaindb/common/schema/transaction_chain_migration_election_v2.0.yaml @@ -0,0 +1,44 @@ +# Copyright BigchainDB GmbH and BigchainDB contributors +# SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) +# Code is Apache-2.0 and docs are CC-BY-4.0 + +--- +"$schema": "http://json-schema.org/draft-04/schema#" +type: object +title: Chain Migration Election Schema - Propose a halt in block production to allow for a version change +required: +- operation +- asset +- outputs +properties: + operation: + type: string + value: "CHAIN_MIGRATION_ELECTION" + asset: + additionalProperties: false + properties: + data: + additionalProperties: false + properties: + seed: + type: string + required: + - data + outputs: + type: array + items: + "$ref": "#/definitions/output" +definitions: + output: + type: object + properties: + condition: + type: object + required: + - uri + properties: + uri: + type: string + pattern: "^ni:///sha-256;([a-zA-Z0-9_-]{0,86})[?]\ + (fpt=ed25519-sha-256(&)?|cost=[0-9]+(&)?|\ + subtypes=ed25519-sha-256(&)?){2,3}$" diff --git a/bigchaindb/common/schema/transaction_v2.0.yaml b/bigchaindb/common/schema/transaction_v2.0.yaml index 562c0d86..6c056f7f 100644 --- a/bigchaindb/common/schema/transaction_v2.0.yaml +++ b/bigchaindb/common/schema/transaction_v2.0.yaml @@ -63,6 +63,7 @@ definitions: - CREATE - TRANSFER - VALIDATOR_ELECTION + - CHAIN_MIGRATION_ELECTION - VOTE asset: type: object diff --git a/bigchaindb/core.py b/bigchaindb/core.py index f936bd91..8be3cdaf 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -20,13 +20,13 @@ from abci.types_pb2 import ( ) from bigchaindb import BigchainDB +from bigchaindb.elections.election import Election from bigchaindb.version import __tm_supported_versions__ from bigchaindb.utils import tendermint_version_is_compatible from bigchaindb.tendermint_utils import (decode_transaction, calculate_hash) from bigchaindb.lib import Block, PreCommitState from bigchaindb.backend.query import PRE_COMMIT_ID -from bigchaindb.upsert_validator import ValidatorElection import bigchaindb.upsert_validator.validator_utils as vutils from bigchaindb.events import EventTypes, Event @@ -219,15 +219,12 @@ class App(BaseApplication): else: self.block_txn_hash = block['app_hash'] - # Check if the current block concluded any validator elections and - # update the locally tracked validator set - validator_update = ValidatorElection.approved_update(self.bigchaindb, - self.new_height, - self.block_transactions) - update = [validator_update] if validator_update else [] + # Process all concluded elections in the current block and get any update to the validator set + update = Election.approved_elections(self.bigchaindb, + self.new_height, + self.block_transactions) - # Store pre-commit state to recover in case there is a crash - # during `commit` + # Store pre-commit state to recover in case there is a crash during `commit` pre_commit_state = PreCommitState(commit_id=PRE_COMMIT_ID, height=self.new_height, transactions=self.block_txn_ids) diff --git a/bigchaindb/elections/election.py b/bigchaindb/elections/election.py index b78e1456..ca0a4be3 100644 --- a/bigchaindb/elections/election.py +++ b/bigchaindb/elections/election.py @@ -1,6 +1,7 @@ # Copyright BigchainDB GmbH and BigchainDB contributors # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 +from collections import defaultdict import base58 from uuid import uuid4 @@ -33,17 +34,18 @@ class Election(Transaction): INCONCLUSIVE = 'inconclusive' # Vote ratio to approve an election ELECTION_THRESHOLD = 2 / 3 + CHANGES_VALIDATOR_SET = True @classmethod - def get_validator_change(cls, bigchain, height=None): - """Return the latest change to the validator set + def get_validator_change(cls, bigchain): + """Return the validator set from the most recent approved block :return: { 'height': , - 'validators': , - 'election_id': + 'validators': } """ + height = bigchain.get_latest_block()['height'] return bigchain.get_validator_change(height) @classmethod @@ -242,24 +244,41 @@ class Election(Transaction): return response @classmethod - def approved_update(cls, bigchain, new_height, txns): - votes = {} - for txn in txns: - if not isinstance(txn, Vote): + def approved_elections(cls, bigchain, new_height, txns): + elections = defaultdict(list) + for tx in txns: + if not isinstance(tx, Vote): + continue + election_id = tx.asset['id'] + elections[election_id].append(tx) + + validator_set_updated = False + validator_set_change = [] + for election_id, votes in elections.items(): + election = Election.has_concluded(bigchain, election_id, votes, new_height) + + if not election: continue - election_id = txn.asset['id'] - election_votes = votes.get(election_id, []) - election_votes.append(txn) - votes[election_id] = election_votes + if election.makes_validator_set_change(): + if validator_set_updated: + continue + validator_set_change.append(election.get_validator_set_change(bigchain, new_height)) + validator_set_updated = True - election = cls.has_concluded(bigchain, election_id, election_votes, new_height) - # Once an election concludes any other conclusion for the same - # or any other election is invalidated - if election: - cls.store_election_results(bigchain, election, new_height) - return cls.on_approval(bigchain, election, new_height) - return None + election.on_approval(bigchain, election, new_height) + + return validator_set_change + + def makes_validator_set_change(self): + return self.CHANGES_VALIDATOR_SET + + def get_validator_set_change(self, bigchain, new_height): + if self.makes_validator_set_change(): + return self.change_validator_set(bigchain, new_height) + + def change_validator_set(self, bigchain, new_height): + raise NotImplementedError @classmethod def on_approval(cls, bigchain, election, new_height): diff --git a/bigchaindb/migrations/chain_migration_election.py b/bigchaindb/migrations/chain_migration_election.py new file mode 100644 index 00000000..e1e783d0 --- /dev/null +++ b/bigchaindb/migrations/chain_migration_election.py @@ -0,0 +1,15 @@ +from bigchaindb.common.schema import TX_SCHEMA_CHAIN_MIGRATION_ELECTION +from bigchaindb.elections.election import Election + + +class ChainMigrationElection(Election): + + OPERATION = 'CHAIN_MIGRATION_ELECTION' + CREATE = OPERATION + ALLOWED_OPERATIONS = (OPERATION,) + TX_SCHEMA_CUSTOM = TX_SCHEMA_CHAIN_MIGRATION_ELECTION + CHANGES_VALIDATOR_SET = False + + @classmethod + def on_approval(cls, bigchain, election, new_height): + bigchain.migrate_abci_chain() diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index 3daf22eb..856b31a1 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -4,7 +4,7 @@ from bigchaindb.common.exceptions import InvalidPowerChange from bigchaindb.elections.election import Election -from bigchaindb.common.schema import (TX_SCHEMA_VALIDATOR_ELECTION) +from bigchaindb.common.schema import TX_SCHEMA_VALIDATOR_ELECTION from .validator_utils import (new_validator_set, encode_validator, validate_asset_public_key) @@ -36,14 +36,18 @@ class ValidatorElection(Election): super(ValidatorElection, cls).validate_schema(tx) validate_asset_public_key(tx['asset']['data']['public_key']) - @classmethod - def on_approval(cls, bigchain, election, new_height): + def change_validator_set(self, bigchain, new_height): # The new validator set comes into effect from height = new_height+1 - validator_updates = [election.asset['data']] + # (upcoming changes to Tendermint will change this to height = new_height+2) + validator_updates = [self.asset['data']] curr_validator_set = bigchain.get_validators(new_height) updated_validator_set = new_validator_set(curr_validator_set, validator_updates) updated_validator_set = [v for v in updated_validator_set if v['voting_power'] > 0] bigchain.store_validator_set(new_height+1, updated_validator_set) - return encode_validator(election.asset['data']) + return encode_validator(self.asset['data']) + + @classmethod + def on_approval(cls, bigchain, election, new_height): + pass diff --git a/docs/server/source/server-reference/bigchaindb-cli.md b/docs/server/source/server-reference/bigchaindb-cli.md index 44fa2687..82870445 100644 --- a/docs/server/source/server-reference/bigchaindb-cli.md +++ b/docs/server/source/server-reference/bigchaindb-cli.md @@ -91,6 +91,10 @@ Election management is broken into several subcommands. Below is the command lin Create a new election which proposes a change to your BigChainDB network. +If the command succeeds, it will create an election and return an `election_id`. + +**NOTE**: The election proposal consists of vote tokens allocated to each current validator as per their voting power. Validators then cast their votes to approve the election by spending their vote tokens, (see the documentation on `election approve`). + There are multiple types of election, which each take different parameters. Below is a short description of each type of election, as well as their command line syntax and the return value. ###### election new upsert-validator @@ -106,7 +110,9 @@ $ bigchaindb election new upsert-validator E_PUBKEY E_POWER E_NODE_ID --private- - `E_PUBKEY`: Public key of the node to be added/updated/removed. - `E_POWER`: The new power for the `E_PUBKEY`. NOTE, if power is set to `0` then `E_PUBKEY` will be removed from the validator set when the election concludes. - `E_NODE_ID`: Node id of `E_PUBKEY`. The node operator of `E_PUBKEY` can generate the node id via `tendermint show_node_id`. -- `--private-key`: The path to Tendermint's private key which can be generally found at `/home/user/.tendermint/config/priv_validator.json`. For example, to add a new validator, provide the public key and node id for some node not already in the validator set, along with whatever voting power you'd like them to have. To remove an existing validator, provide their public key and node id, and set `E_POWER` to `0`. Please note that the private key provided here is of the node which is generating this election i.e. +- `--private-key`: The path to Tendermint's private key which can be generally found at `/home/user/.tendermint/config/priv_validator.json`. + +For example, to add a new validator, provide the public key and node id for some node not already in the validator set, along with whatever voting power you'd like them to have. To remove an existing validator, provide their public key and node id, and set `E_POWER` to `0`. Please note that the private key provided here is of the node which is generating this election i.e. NOTE: A change to the validator set can only be proposed by one of the exisitng validators. @@ -118,10 +124,29 @@ $ bigchaindb election new upsert-validator HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52 [SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa ``` -If the command succeeds, it will create an election and return an `election_id`. A successful execution of the above command **doesn't** imply that the validator set will be immediately updated but rather it means the proposal has been succcessfully accepted by the network. Once the `election_id` has been generated the node operator should share this `election_id` with other validators in the network and urge them to approve the proposal. Note that the node operator should themsleves also approve the proposal. +A successful execution of the above command **doesn't** imply that the validator set will be immediately updated but rather it means the proposal has been succcessfully accepted by the network. Once the `election_id` has been generated the node operator should share this `election_id` with other validators in the network and urge them to approve the proposal. Note that the node operator should themsleves also approve the proposal. -**NOTE**: The election proposal consists of vote tokens allocated to each current validator as per their voting power. Validators then cast their votes to approve the change to the validator set by spending their vote tokens. +###### election new migration + +Create an election to halt block production, to allow for a version change across breaking changes. + + +```bash +$ bigchaindb election new migration --private-key PATH_TO_YOUR_PRIVATE_KEY +[SUCCESS] Submitted proposal with id: +``` + +- `--private-key`: The path to Tendermint's private key which can be generally found at `/home/user/.tendermint/config/priv_validator.json`. + +Example usage, + +```bash +$ bigchaindb election new migration --private-key /home/user/.tendermint/config/priv_validator.json +[SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa +``` + +**NOTE** `migration` elections will halt block production at whichever blockheight they are approved. Once the election is concluded, the validators will need to restart their systems with a new `chain_id` to resume normal operations. #### election approve diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 38e251d8..bbeea227 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -26,6 +26,8 @@ def test_make_sure_we_dont_remove_any_command(): assert parser.parse_args(['start']).command assert parser.parse_args(['election', 'new', 'upsert-validator', 'TEMP_PUB_KEYPAIR', '10', 'TEMP_NODE_ID', '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command + assert parser.parse_args(['election', 'new', 'chain-migration', + '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command assert parser.parse_args(['election', 'approve', 'ELECTION_ID', '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command assert parser.parse_args(['election', 'show', 'ELECTION_ID']).command @@ -341,6 +343,42 @@ def test_election_new_upsert_validator_without_tendermint(caplog, b, priv_valida assert b.get_transaction(election_id) +@pytest.mark.abci +def test_election_new_chain_migration_with_tendermint(b, priv_validator_path, user_sk, validators): + from bigchaindb.commands.bigchaindb import run_election_new_chain_migration + + new_args = Namespace(action='new', + election_type='migration', + sk=priv_validator_path, + config={}) + + election_id = run_election_new_chain_migration(new_args, b) + + assert b.get_transaction(election_id) + + +@pytest.mark.bdb +def test_election_new_chain_migration_without_tendermint(caplog, b, priv_validator_path, user_sk): + from bigchaindb.commands.bigchaindb import run_election_new_chain_migration + + def mock_write(tx, mode): + b.store_bulk_transactions([tx]) + return (202, '') + + b.get_validators = mock_get_validators + b.write_transaction = mock_write + + args = Namespace(action='new', + election_type='migration', + sk=priv_validator_path, + config={}) + + with caplog.at_level(logging.INFO): + election_id = run_election_new_chain_migration(args, b) + assert caplog.records[0].msg == '[SUCCESS] Submitted proposal with id: ' + election_id + assert b.get_transaction(election_id) + + @pytest.mark.bdb def test_election_new_upsert_validator_invalid_election(caplog, b, priv_validator_path, user_sk): from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator @@ -415,7 +453,7 @@ def test_election_approve_without_tendermint(caplog, b, priv_validator_path, new b, election_id = call_election(b, new_validator, node_key) - # call run_upsert_validator_approve with args that point to the election + # call run_election_approve with args that point to the election args = Namespace(action='approve', election_id=election_id, sk=priv_validator_path, diff --git a/tests/conftest.py b/tests/conftest.py index 15cfcf01..4c271aad 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -20,15 +20,17 @@ from logging.config import dictConfig import pytest from pymongo import MongoClient +from bigchaindb import ValidatorElection from bigchaindb.common import crypto from bigchaindb.log import setup_logging +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection from bigchaindb.tendermint_utils import key_from_base64 -from bigchaindb.backend import schema +from bigchaindb.backend import schema, query from bigchaindb.common.crypto import (key_pair_from_ed25519_key, public_key_from_ed25519_key) from bigchaindb.common.exceptions import DatabaseDoesNotExist from bigchaindb.lib import Block - +from tests.utils import gen_vote TEST_DB_NAME = 'bigchain_test' @@ -242,6 +244,26 @@ def b(): return BigchainDB() +@pytest.fixture +def b_mock(b, network_validators): + b.get_validators = mock_get_validators(network_validators) + + return b + + +def mock_get_validators(network_validators): + def validator_set(height): + validators = [] + for public_key, power in network_validators.items(): + validators.append({ + 'public_key': {'type': 'ed25519-base64', 'value': public_key}, + 'voting_power': power + }) + return validators + + return validator_set + + @pytest.fixture def create_tx(alice, user_pk): from bigchaindb.models import Transaction @@ -674,3 +696,117 @@ def new_validator(): 'type': 'ed25519-base16'}, 'power': power, 'node_id': node_id} + + +@pytest.fixture +def valid_upsert_validator_election(b_mock, node_key, new_validator): + voters = ValidatorElection.recipients(b_mock) + return ValidatorElection.generate([node_key.public_key], + voters, + new_validator, None).sign([node_key.private_key]) + + +@pytest.fixture +def valid_upsert_validator_election_2(b_mock, node_key, new_validator): + voters = ValidatorElection.recipients(b_mock) + return ValidatorElection.generate([node_key.public_key], + voters, + new_validator, None).sign([node_key.private_key]) + + +@pytest.fixture +def valid_chain_migration_election(b_mock, node_key): + voters = ChainMigrationElection.recipients(b_mock) + return ChainMigrationElection.generate([node_key.public_key], + voters, + {}, None).sign([node_key.private_key]) + + +@pytest.fixture +def valid_chain_migration_election_2(b_mock, node_key): + voters = ChainMigrationElection.recipients(b_mock) + return ChainMigrationElection.generate([node_key.public_key], + voters, + {}, None).sign([node_key.private_key]) + + +@pytest.fixture +def ongoing_validator_election(b, valid_upsert_validator_election, ed25519_node_keys): + validators = b.get_validators(height=1) + genesis_validators = {'validators': validators, + 'height': 0, + 'election_id': None} + query.store_validator_set(b.connection, genesis_validators) + + b.store_bulk_transactions([valid_upsert_validator_election]) + block_1 = Block(app_hash='hash_1', height=1, transactions=[valid_upsert_validator_election.id]) + b.store_block(block_1._asdict()) + return valid_upsert_validator_election + + +@pytest.fixture +def ongoing_validator_election_2(b, valid_upsert_validator_election_2, ed25519_node_keys): + validators = b.get_validators(height=1) + genesis_validators = {'validators': validators, + 'height': 0, + 'election_id': None} + query.store_validator_set(b.connection, genesis_validators) + + b.store_bulk_transactions([valid_upsert_validator_election_2]) + block_1 = Block(app_hash='hash_2', height=1, transactions=[valid_upsert_validator_election_2.id]) + b.store_block(block_1._asdict()) + return valid_upsert_validator_election_2 + + +@pytest.fixture +def ongoing_chain_migration_election(b, valid_chain_migration_election, ed25519_node_keys): + + b.store_bulk_transactions([valid_chain_migration_election]) + block_1 = Block(app_hash='hash_1', height=1, transactions=[valid_chain_migration_election.id]) + b.store_block(block_1._asdict()) + return valid_chain_migration_election + + +@pytest.fixture +def ongoing_chain_migration_election_2(b, valid_chain_migration_election_2, ed25519_node_keys): + + b.store_bulk_transactions([valid_chain_migration_election_2]) + block_1 = Block(app_hash='hash_2', height=1, transactions=[valid_chain_migration_election_2.id]) + b.store_block(block_1._asdict()) + return valid_chain_migration_election_2 + + +@pytest.fixture +def validator_election_votes(b_mock, ongoing_validator_election, ed25519_node_keys): + voters = ValidatorElection.recipients(b_mock) + votes = generate_votes(ongoing_validator_election, voters, ed25519_node_keys) + return votes + + +@pytest.fixture +def validator_election_votes_2(b_mock, ongoing_validator_election_2, ed25519_node_keys): + voters = ValidatorElection.recipients(b_mock) + votes = generate_votes(ongoing_validator_election_2, voters, ed25519_node_keys) + return votes + + +@pytest.fixture +def chain_migration_election_votes(b_mock, ongoing_chain_migration_election, ed25519_node_keys): + voters = ChainMigrationElection.recipients(b_mock) + votes = generate_votes(ongoing_chain_migration_election, voters, ed25519_node_keys) + return votes + + +@pytest.fixture +def chain_migration_election_votes_2(b_mock, ongoing_chain_migration_election_2, ed25519_node_keys): + voters = ChainMigrationElection.recipients(b_mock) + votes = generate_votes(ongoing_chain_migration_election_2, voters, ed25519_node_keys) + return votes + + +def generate_votes(election, voters, keys): + votes = [] + for voter in range(len(voters)): + v = gen_vote(election, voter, keys) + votes.append(v) + return votes diff --git a/tests/elections/__init__.py b/tests/elections/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/elections/test_election.py b/tests/elections/test_election.py new file mode 100644 index 00000000..0e9da188 --- /dev/null +++ b/tests/elections/test_election.py @@ -0,0 +1,64 @@ +from unittest.mock import MagicMock + +import pytest + +from bigchaindb.elections.election import Election + + +@pytest.mark.bdb +def test_approved_elections_one_migration_one_upsert( + b, + ongoing_validator_election, validator_election_votes, + ongoing_chain_migration_election, chain_migration_election_votes +): + txns = validator_election_votes + \ + chain_migration_election_votes + mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) + mock_chain_migration.assert_called_once() + mock_store_validator.assert_called_once() + + +@pytest.mark.bdb +def test_approved_elections_one_migration_two_upsert( + b, + ongoing_validator_election, validator_election_votes, + ongoing_validator_election_2, validator_election_votes_2, + ongoing_chain_migration_election, chain_migration_election_votes +): + txns = validator_election_votes + \ + validator_election_votes_2 + \ + chain_migration_election_votes + mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) + mock_chain_migration.assert_called_once() + mock_store_validator.assert_called_once() + + +@pytest.mark.bdb +def test_approved_elections_two_migrations_one_upsert( + b, + ongoing_validator_election, validator_election_votes, + ongoing_chain_migration_election, chain_migration_election_votes, + ongoing_chain_migration_election_2, chain_migration_election_votes_2 +): + txns = validator_election_votes + \ + chain_migration_election_votes + \ + chain_migration_election_votes_2 + mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) + assert mock_chain_migration.call_count == 2 + mock_store_validator.assert_called_once() + + +def test_approved_elections_no_elections(b): + txns = [] + mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) + mock_chain_migration.assert_not_called() + mock_store_validator.assert_not_called() + + +def run_approved_elections(bigchain, txns): + mock_chain_migration = MagicMock() + mock_store_validator = MagicMock() + bigchain.migrate_abci_chain = mock_chain_migration + bigchain.store_validator_set = mock_store_validator + Election.approved_elections(bigchain, 1, txns) + return mock_chain_migration, mock_store_validator diff --git a/tests/migrations/test_migration_election.py b/tests/migrations/test_migration_election.py new file mode 100644 index 00000000..b811903b --- /dev/null +++ b/tests/migrations/test_migration_election.py @@ -0,0 +1,9 @@ +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection + + +def test_valid_migration_election(b_mock, node_key): + voters = ChainMigrationElection.recipients(b_mock) + election = ChainMigrationElection.generate([node_key.public_key], + voters, + {}, None).sign([node_key.private_key]) + assert election.validate(b_mock) diff --git a/tests/upsert_validator/conftest.py b/tests/upsert_validator/conftest.py index 9ab2dad8..58b5e6dd 100644 --- a/tests/upsert_validator/conftest.py +++ b/tests/upsert_validator/conftest.py @@ -5,42 +5,12 @@ from unittest.mock import patch import pytest -from bigchaindb import Vote from bigchaindb.backend.localmongodb import query -from bigchaindb.lib import Block from bigchaindb.upsert_validator import ValidatorElection @pytest.fixture -def b_mock(b, network_validators): - b.get_validators = mock_get_validators(network_validators) - - return b - - -def mock_get_validators(network_validators): - def validator_set(height): - validators = [] - for public_key, power in network_validators.items(): - validators.append({ - 'public_key': {'type': 'ed25519-base64', 'value': public_key}, - 'voting_power': power - }) - return validators - - return validator_set - - -@pytest.fixture -def valid_election(b_mock, node_key, new_validator): - voters = ValidatorElection.recipients(b_mock) - return ValidatorElection.generate([node_key.public_key], - voters, - new_validator, None).sign([node_key.private_key]) - - -@pytest.fixture -def valid_election_b(b, node_key, new_validator): +def valid_upsert_validator_election_b(b, node_key, new_validator): voters = ValidatorElection.recipients(b) return ValidatorElection.generate([node_key.public_key], voters, @@ -57,30 +27,16 @@ def fixed_seed_election(b_mock, node_key, new_validator): @pytest.fixture -def ongoing_election(b, valid_election, ed25519_node_keys): - validators = b.get_validators(height=1) - genesis_validators = {'validators': validators, - 'height': 0, - 'election_id': None} - query.store_validator_set(b.connection, genesis_validators) - - b.store_bulk_transactions([valid_election]) - block_1 = Block(app_hash='hash_1', height=1, transactions=[valid_election.id]) - b.store_block(block_1._asdict()) - return valid_election - - -@pytest.fixture -def concluded_election(b, ongoing_election, ed25519_node_keys): +def concluded_election(b, ongoing_validator_election, ed25519_node_keys): election_result = {'height': 2, - 'election_id': ongoing_election.id} + 'election_id': ongoing_validator_election.id} query.store_election_results(b.connection, election_result) - return ongoing_election + return ongoing_validator_election @pytest.fixture -def inconclusive_election(b, ongoing_election, new_validator): +def inconclusive_election(b, ongoing_validator_election, new_validator): validators = b.get_validators(height=1) validators[0]['voting_power'] = 15 validator_update = {'validators': validators, @@ -88,20 +44,4 @@ def inconclusive_election(b, ongoing_election, new_validator): 'election_id': 'some_other_election'} query.store_validator_set(b.connection, validator_update) - return ongoing_election - - -def vote(election, voter, keys, b): - election_input = election.to_inputs()[voter] - votes = election.outputs[voter].amount - public_key = election_input.owners_before[0] - key = keys[public_key] - - election_pub_key = ValidatorElection.to_public_key(election.id) - - v = Vote.generate([election_input], - [([election_pub_key], votes)], - election_id=election.id)\ - .sign([key.private_key]) - b.store_bulk_transactions([v]) - return v + return ongoing_validator_election diff --git a/tests/upsert_validator/test_upsert_validator_vote.py b/tests/upsert_validator/test_upsert_validator_vote.py index b2e23af7..3c3512d0 100644 --- a/tests/upsert_validator/test_upsert_validator_vote.py +++ b/tests/upsert_validator/test_upsert_validator_vote.py @@ -5,104 +5,105 @@ import pytest import codecs +from bigchaindb.elections.election import Election from bigchaindb.tendermint_utils import public_key_to_base64 from bigchaindb.upsert_validator import ValidatorElection from bigchaindb.common.exceptions import AmountError from bigchaindb.common.crypto import generate_key_pair from bigchaindb.common.exceptions import ValidationError from bigchaindb.elections.vote import Vote -from tests.utils import generate_block +from tests.utils import generate_block, gen_vote pytestmark = [pytest.mark.execute] @pytest.mark.bdb -def test_upsert_validator_valid_election_vote(b_mock, valid_election, ed25519_node_keys): - b_mock.store_bulk_transactions([valid_election]) +def test_upsert_validator_valid_election_vote(b_mock, valid_upsert_validator_election, ed25519_node_keys): + b_mock.store_bulk_transactions([valid_upsert_validator_election]) - input0 = valid_election.to_inputs()[0] - votes = valid_election.outputs[0].amount + input0 = valid_upsert_validator_election.to_inputs()[0] + votes = valid_upsert_validator_election.outputs[0].amount public_key0 = input0.owners_before[0] key0 = ed25519_node_keys[public_key0] - election_pub_key = ValidatorElection.to_public_key(valid_election.id) + election_pub_key = ValidatorElection.to_public_key(valid_upsert_validator_election.id) vote = Vote.generate([input0], [([election_pub_key], votes)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) assert vote.validate(b_mock) @pytest.mark.bdb -def test_upsert_validator_valid_non_election_vote(b_mock, valid_election, ed25519_node_keys): - b_mock.store_bulk_transactions([valid_election]) +def test_upsert_validator_valid_non_election_vote(b_mock, valid_upsert_validator_election, ed25519_node_keys): + b_mock.store_bulk_transactions([valid_upsert_validator_election]) - input0 = valid_election.to_inputs()[0] - votes = valid_election.outputs[0].amount + input0 = valid_upsert_validator_election.to_inputs()[0] + votes = valid_upsert_validator_election.outputs[0].amount public_key0 = input0.owners_before[0] key0 = ed25519_node_keys[public_key0] - election_pub_key = ValidatorElection.to_public_key(valid_election.id) + election_pub_key = ValidatorElection.to_public_key(valid_upsert_validator_election.id) # Ensure that threshold conditions are now allowed with pytest.raises(ValidationError): Vote.generate([input0], [([election_pub_key, key0.public_key], votes)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) @pytest.mark.bdb -def test_upsert_validator_delegate_election_vote(b_mock, valid_election, ed25519_node_keys): +def test_upsert_validator_delegate_election_vote(b_mock, valid_upsert_validator_election, ed25519_node_keys): alice = generate_key_pair() - b_mock.store_bulk_transactions([valid_election]) + b_mock.store_bulk_transactions([valid_upsert_validator_election]) - input0 = valid_election.to_inputs()[0] - votes = valid_election.outputs[0].amount + input0 = valid_upsert_validator_election.to_inputs()[0] + votes = valid_upsert_validator_election.outputs[0].amount public_key0 = input0.owners_before[0] key0 = ed25519_node_keys[public_key0] delegate_vote = Vote.generate([input0], [([alice.public_key], 3), ([key0.public_key], votes-3)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) assert delegate_vote.validate(b_mock) b_mock.store_bulk_transactions([delegate_vote]) - election_pub_key = ValidatorElection.to_public_key(valid_election.id) + election_pub_key = ValidatorElection.to_public_key(valid_upsert_validator_election.id) alice_votes = delegate_vote.to_inputs()[0] alice_casted_vote = Vote.generate([alice_votes], [([election_pub_key], 3)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([alice.private_key]) assert alice_casted_vote.validate(b_mock) key0_votes = delegate_vote.to_inputs()[1] key0_casted_vote = Vote.generate([key0_votes], [([election_pub_key], votes-3)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) assert key0_casted_vote.validate(b_mock) @pytest.mark.bdb -def test_upsert_validator_invalid_election_vote(b_mock, valid_election, ed25519_node_keys): - b_mock.store_bulk_transactions([valid_election]) +def test_upsert_validator_invalid_election_vote(b_mock, valid_upsert_validator_election, ed25519_node_keys): + b_mock.store_bulk_transactions([valid_upsert_validator_election]) - input0 = valid_election.to_inputs()[0] - votes = valid_election.outputs[0].amount + input0 = valid_upsert_validator_election.to_inputs()[0] + votes = valid_upsert_validator_election.outputs[0].amount public_key0 = input0.owners_before[0] key0 = ed25519_node_keys[public_key0] - election_pub_key = ValidatorElection.to_public_key(valid_election.id) + election_pub_key = ValidatorElection.to_public_key(valid_upsert_validator_election.id) vote = Vote.generate([input0], [([election_pub_key], votes+1)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) with pytest.raises(AmountError): @@ -110,113 +111,111 @@ def test_upsert_validator_invalid_election_vote(b_mock, valid_election, ed25519_ @pytest.mark.bdb -def test_valid_election_votes_received(b_mock, valid_election, ed25519_node_keys): +def test_valid_election_votes_received(b_mock, valid_upsert_validator_election, ed25519_node_keys): alice = generate_key_pair() - b_mock.store_bulk_transactions([valid_election]) - assert valid_election.get_commited_votes(b_mock) == 0 + b_mock.store_bulk_transactions([valid_upsert_validator_election]) + assert valid_upsert_validator_election.get_commited_votes(b_mock) == 0 - input0 = valid_election.to_inputs()[0] - votes = valid_election.outputs[0].amount + input0 = valid_upsert_validator_election.to_inputs()[0] + votes = valid_upsert_validator_election.outputs[0].amount public_key0 = input0.owners_before[0] key0 = ed25519_node_keys[public_key0] # delegate some votes to alice delegate_vote = Vote.generate([input0], [([alice.public_key], 4), ([key0.public_key], votes-4)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) b_mock.store_bulk_transactions([delegate_vote]) - assert valid_election.get_commited_votes(b_mock) == 0 + assert valid_upsert_validator_election.get_commited_votes(b_mock) == 0 - election_public_key = ValidatorElection.to_public_key(valid_election.id) + election_public_key = ValidatorElection.to_public_key(valid_upsert_validator_election.id) alice_votes = delegate_vote.to_inputs()[0] key0_votes = delegate_vote.to_inputs()[1] alice_casted_vote = Vote.generate([alice_votes], [([election_public_key], 2), ([alice.public_key], 2)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([alice.private_key]) assert alice_casted_vote.validate(b_mock) b_mock.store_bulk_transactions([alice_casted_vote]) # Check if the delegated vote is count as valid vote - assert valid_election.get_commited_votes(b_mock) == 2 + assert valid_upsert_validator_election.get_commited_votes(b_mock) == 2 key0_casted_vote = Vote.generate([key0_votes], [([election_public_key], votes-4)], - election_id=valid_election.id)\ + election_id=valid_upsert_validator_election.id)\ .sign([key0.private_key]) assert key0_casted_vote.validate(b_mock) b_mock.store_bulk_transactions([key0_casted_vote]) - assert valid_election.get_commited_votes(b_mock) == votes-2 + assert valid_upsert_validator_election.get_commited_votes(b_mock) == votes - 2 @pytest.mark.bdb -def test_valid_election_conclude(b_mock, valid_election, ed25519_node_keys): +def test_valid_election_conclude(b_mock, valid_upsert_validator_election, ed25519_node_keys): # Node 0: cast vote - tx_vote0 = gen_vote(valid_election, 0, ed25519_node_keys) + tx_vote0 = gen_vote(valid_upsert_validator_election, 0, ed25519_node_keys) # check if the vote is valid even before the election doesn't exist with pytest.raises(ValidationError): assert tx_vote0.validate(b_mock) # store election - b_mock.store_bulk_transactions([valid_election]) + b_mock.store_bulk_transactions([valid_upsert_validator_election]) # cannot conclude election as not votes exist - assert not ValidatorElection.has_concluded(b_mock, valid_election.id) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) # validate vote assert tx_vote0.validate(b_mock) - assert not ValidatorElection.has_concluded(b_mock, valid_election.id, [tx_vote0]) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote0]) b_mock.store_bulk_transactions([tx_vote0]) - assert not ValidatorElection.has_concluded(b_mock, valid_election.id) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) # Node 1: cast vote - tx_vote1 = gen_vote(valid_election, 1, ed25519_node_keys) + tx_vote1 = gen_vote(valid_upsert_validator_election, 1, ed25519_node_keys) # Node 2: cast vote - tx_vote2 = gen_vote(valid_election, 2, ed25519_node_keys) + tx_vote2 = gen_vote(valid_upsert_validator_election, 2, ed25519_node_keys) # Node 3: cast vote - tx_vote3 = gen_vote(valid_election, 3, ed25519_node_keys) + tx_vote3 = gen_vote(valid_upsert_validator_election, 3, ed25519_node_keys) assert tx_vote1.validate(b_mock) - assert not ValidatorElection.has_concluded(b_mock, valid_election.id, [tx_vote1]) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote1]) # 2/3 is achieved in the same block so the election can be.has_concludedd - assert ValidatorElection.has_concluded(b_mock, valid_election.id, [tx_vote1, tx_vote2]) + assert ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote1, tx_vote2]) b_mock.store_bulk_transactions([tx_vote1]) - assert not ValidatorElection.has_concluded(b_mock, valid_election.id) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) assert tx_vote2.validate(b_mock) assert tx_vote3.validate(b_mock) # conclusion can be triggered my different votes in the same block - assert ValidatorElection.has_concluded(b_mock, valid_election.id, [tx_vote2]) - assert ValidatorElection.has_concluded(b_mock, valid_election.id, [tx_vote2, tx_vote3]) + assert ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote2]) + assert ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote2, tx_vote3]) b_mock.store_bulk_transactions([tx_vote2]) # Once the blockchain records >2/3 of the votes the election is assumed to be.has_concludedd # so any invocation of `.has_concluded` for that election should return False - assert not ValidatorElection.has_concluded(b_mock, valid_election.id) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) # Vote is still valid but the election cannot be.has_concludedd as it it assmed that it has # been.has_concludedd before assert tx_vote3.validate(b_mock) - assert not ValidatorElection.has_concluded(b_mock, valid_election.id, [tx_vote3]) + assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote3]) @pytest.mark.abci def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): - import time - import requests if b.get_latest_block()['height'] == 0: generate_block(b) @@ -244,20 +243,17 @@ def test_upsert_validator(b, node_key, node_keys, ed25519_node_keys): new_validator, None).sign([node_key.private_key]) code, message = b.write_transaction(election, 'broadcast_tx_commit') assert code == 202 - time.sleep(3) - assert b.get_transaction(election.id) tx_vote = gen_vote(election, 0, ed25519_node_keys) assert tx_vote.validate(b) code, message = b.write_transaction(tx_vote, 'broadcast_tx_commit') assert code == 202 - time.sleep(3) - resp = requests.get(b.endpoint + 'validators') + resp = b.get_validators() validator_pub_keys = [] - for v in resp.json()['result']['validators']: - validator_pub_keys.append(v['pub_key']['value']) + for v in resp: + validator_pub_keys.append(v['public_key']['value']) assert (public_key64 in validator_pub_keys) new_validator_set = b.get_validators() @@ -293,19 +289,19 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): assert not ValidatorElection.has_concluded(b, election.id, [tx_vote0, tx_vote1]) assert ValidatorElection.has_concluded(b, election.id, [tx_vote0, tx_vote1, tx_vote2]) - assert not ValidatorElection.approved_update(b, 4, [tx_vote0]) - assert not ValidatorElection.approved_update(b, 4, [tx_vote0, tx_vote1]) + assert Election.approved_elections(b, 4, [tx_vote0]) == [] + assert Election.approved_elections(b, 4, [tx_vote0, tx_vote1]) == [] - update = ValidatorElection.approved_update(b, 4, [tx_vote0, tx_vote1, tx_vote2]) - assert update - update_public_key = codecs.encode(update.pub_key.data, 'base64').decode().rstrip('\n') + update = Election.approved_elections(b, 4, [tx_vote0, tx_vote1, tx_vote2]) + assert len(update) == 1 + update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 b.store_bulk_transactions([tx_vote0, tx_vote1]) - update = ValidatorElection.approved_update(b, 4, [tx_vote2]) - assert update - update_public_key = codecs.encode(update.pub_key.data, 'base64').decode().rstrip('\n') + update = Election.approved_elections(b, 4, [tx_vote2]) + assert len(update) == 1 + update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 # remove validator @@ -326,10 +322,9 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): b.store_bulk_transactions([tx_vote0, tx_vote1]) - update = ValidatorElection.approved_update(b, 9, [tx_vote2]) - if update: - update_public_key = codecs.encode(update.pub_key.data, 'base64').decode().rstrip('\n') - assert update + update = Election.approved_elections(b, 9, [tx_vote2]) + assert len(update) == 1 + update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 # assert that the public key is not a part of the current validator set @@ -340,22 +335,6 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): # ============================================================================ # Helper functions # ============================================================================ -def to_inputs(election, i, ed25519_node_keys): - input0 = election.to_inputs()[i] - votes = election.outputs[i].amount - public_key0 = input0.owners_before[0] - key0 = ed25519_node_keys[public_key0] - return (input0, votes, key0) - - -def gen_vote(election, i, ed25519_node_keys): - (input_i, votes_i, key_i) = to_inputs(election, i, ed25519_node_keys) - election_pub_key = ValidatorElection.to_public_key(election.id) - return Vote.generate([input_i], - [([election_pub_key], votes_i)], - election_id=election.id)\ - .sign([key_i.private_key]) - def reset_validator_set(b, node_keys, height): validators = [] diff --git a/tests/upsert_validator/test_validator_election.py b/tests/upsert_validator/test_validator_election.py index c1f56fff..205be554 100644 --- a/tests/upsert_validator/test_validator_election.py +++ b/tests/upsert_validator/test_validator_election.py @@ -111,9 +111,9 @@ def test_upsert_validator_invalid_election(b_mock, new_validator, node_key, fixe tx_election.validate(b_mock) -def test_get_status_ongoing(b, ongoing_election, new_validator): +def test_get_status_ongoing(b, ongoing_validator_election, new_validator): status = ValidatorElection.ONGOING - resp = ongoing_election.get_status(b) + resp = ongoing_validator_election.get_status(b) assert resp == status @@ -124,6 +124,9 @@ def test_get_status_concluded(b, concluded_election, new_validator): def test_get_status_inconclusive(b, inconclusive_election, new_validator): + def set_block_height_to_3(): + return {'height': 3} + def custom_mock_get_validators(height): if height >= 3: return [{'pub_key': {'data': 'zL/DasvKulXZzhSNFwx4cLRXKkSM9GPK7Y0nZ4FEylM=', @@ -153,18 +156,19 @@ def test_get_status_inconclusive(b, inconclusive_election, new_validator): 'voting_power': 8}] b.get_validators = custom_mock_get_validators + b.get_latest_block = set_block_height_to_3 status = ValidatorElection.INCONCLUSIVE resp = inconclusive_election.get_status(b) assert resp == status -def test_upsert_validator_show(caplog, ongoing_election, b): +def test_upsert_validator_show(caplog, ongoing_validator_election, b): from bigchaindb.commands.bigchaindb import run_election_show - election_id = ongoing_election.id - public_key = public_key_to_base64(ongoing_election.asset['data']['public_key']['value']) - power = ongoing_election.asset['data']['power'] - node_id = ongoing_election.asset['data']['node_id'] + election_id = ongoing_validator_election.id + public_key = public_key_to_base64(ongoing_validator_election.asset['data']['public_key']['value']) + power = ongoing_validator_election.asset['data']['power'] + node_id = ongoing_validator_election.asset['data']['node_id'] status = ValidatorElection.ONGOING show_args = Namespace(action='show', diff --git a/tests/utils.py b/tests/utils.py index 87c07a79..852be5e6 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -4,8 +4,10 @@ from functools import singledispatch +from bigchaindb import Vote from bigchaindb.backend.localmongodb.connection import LocalMongoDBConnection from bigchaindb.backend.schema import TABLES +from bigchaindb.elections.election import Election @singledispatch @@ -33,3 +35,20 @@ def generate_block(bigchain): code, message = bigchain.write_transaction(tx, 'broadcast_tx_commit') assert code == 202 time.sleep(2) + + +def to_inputs(election, i, ed25519_node_keys): + input0 = election.to_inputs()[i] + votes = election.outputs[i].amount + public_key0 = input0.owners_before[0] + key0 = ed25519_node_keys[public_key0] + return (input0, votes, key0) + + +def gen_vote(election, i, ed25519_node_keys): + (input_i, votes_i, key_i) = to_inputs(election, i, ed25519_node_keys) + election_pub_key = Election.to_public_key(election.id) + return Vote.generate([input_i], + [([election_pub_key], votes_i)], + election_id=election.id)\ + .sign([key_i.private_key]) From 8e5f77070712dd83cd0e2c5fb582ac0bf0e524d3 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Mon, 17 Sep 2018 14:06:48 +0200 Subject: [PATCH 37/77] List the "blocks" endpoint in response to HTTP info requests (#2545) * Problem: HTTP info endpoints don't include "blocks" Solution: Add "blocks" to the response to the HTTP info endpoints * Removed "blocks" endpoints from "Advanced Usage" because they are really not that advanced --- bigchaindb/web/views/info.py | 1 + docs/server/source/http-client-server-api.rst | 8 +------- tests/web/test_info.py | 2 ++ 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/bigchaindb/web/views/info.py b/bigchaindb/web/views/info.py index 4b44157b..fbbb0392 100644 --- a/bigchaindb/web/views/info.py +++ b/bigchaindb/web/views/info.py @@ -47,6 +47,7 @@ def get_api_v1_info(api_prefix): return { 'docs': ''.join(docs_url), 'transactions': '{}transactions/'.format(api_prefix), + 'blocks': '{}blocks/'.format(api_prefix), 'assets': '{}assets/'.format(api_prefix), 'outputs': '{}outputs/'.format(api_prefix), 'streams': websocket_root, diff --git a/docs/server/source/http-client-server-api.rst b/docs/server/source/http-client-server-api.rst index eadfe983..325320f1 100644 --- a/docs/server/source/http-client-server-api.rst +++ b/docs/server/source/http-client-server-api.rst @@ -616,14 +616,8 @@ Validators :statuscode 200: The query was executed successfully and validators set was returned. -Advanced Usage --------------------------------- - -The following endpoints are more advanced -and meant for debugging and transparency purposes. - Blocks -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------ .. http:get:: /api/v1/blocks/{block_height} diff --git a/tests/web/test_info.py b/tests/web/test_info.py index b91b6f60..41034e2f 100644 --- a/tests/web/test_info.py +++ b/tests/web/test_info.py @@ -16,6 +16,7 @@ def test_api_root_endpoint(client, wsserver_base_url): 'v1': { 'docs': ''.join(docs_url), 'transactions': '/api/v1/transactions/', + 'blocks': '/api/v1/blocks/', 'assets': '/api/v1/assets/', 'outputs': '/api/v1/outputs/', 'streams': '{}/api/v1/streams/valid_transactions'.format( @@ -38,6 +39,7 @@ def test_api_v1_endpoint(client, wsserver_base_url): api_v1_info = { 'docs': ''.join(docs_url), 'transactions': '/transactions/', + 'blocks': '/blocks/', 'assets': '/assets/', 'outputs': '/outputs/', 'streams': '{}/api/v1/streams/valid_transactions'.format( From e3e95836d076719488d5d6bf8f8148ff8a01ece8 Mon Sep 17 00:00:00 2001 From: Muawia Khan Date: Mon, 17 Sep 2018 14:10:22 +0200 Subject: [PATCH 38/77] Problem: bigchaindb not starting because of a module dependency (#2548) --- setup.py | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.py b/setup.py index a004178d..4181bddf 100644 --- a/setup.py +++ b/setup.py @@ -92,6 +92,7 @@ install_requires = [ 'aiohttp~=3.0', 'bigchaindb-abci==0.5.1', 'setproctitle~=1.1.0', + 'packaging~=17.0', ] setup( From 16355748ddda8e18b5691c7d31249c5e4aca559d Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Mon, 17 Sep 2018 14:39:01 +0200 Subject: [PATCH 39/77] Problem: Docs build fails (#2549) Solution: Create __init__.py so that migrations can be imported properly --- bigchaindb/migrations/__init__.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 bigchaindb/migrations/__init__.py diff --git a/bigchaindb/migrations/__init__.py b/bigchaindb/migrations/__init__.py new file mode 100644 index 00000000..e69de29b From eb0e387b18af977840e2c12cf22f230be4b94ad9 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Mon, 17 Sep 2018 14:43:03 +0200 Subject: [PATCH 40/77] Problem: CHANGELOG.md hasn't been updated since Aug. 1 (#2509) Last update to CHANGELOG.md before 2.0.0b6 --- CHANGELOG.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b8f3201a..d84acdd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,60 @@ For reference, the possible headings are: * **Known Issues** * **Notes** +## [2.0 Beta 6] - 2018-09-17 + +Tag name: v2.0.0b6 + +### Added + +* [New documentation about privacy and handling private data](https://docs.bigchaindb.com/en/latest/private-data.html). [Pull request #2437](https://github.com/bigchaindb/bigchaindb/pull/2437) +* New documentation about log rotation. Also rotate Tendermint logs if started using Monit. [Pull request #2528](https://github.com/bigchaindb/bigchaindb/pull/2528) +* Began implementing one of the migration strategies outlined in [BEP-42](https://github.com/bigchaindb/BEPs/tree/master/42). That involved creating a more general-purpose election process and commands. Pull requests [#2488](https://github.com/bigchaindb/bigchaindb/pull/2488), [#2495](https://github.com/bigchaindb/bigchaindb/pull/2495), [#2498](https://github.com/bigchaindb/bigchaindb/pull/2498), [#2515](https://github.com/bigchaindb/bigchaindb/pull/2515), [#2535](https://github.com/bigchaindb/bigchaindb/pull/2535) +* Used memoization to avoid doing some validation checks multiple times. [Pull request #2490](https://github.com/bigchaindb/bigchaindb/pull/2490) +* Created an all-in-one Docker image containing BigchainDB Server, Tendermint and MongoDB. It was created for a particular user and is not recommended for production use unless you really know what you're doing. [Pull request #2424](https://github.com/bigchaindb/bigchaindb/pull/2424) + +### Changed + +* The supported versions of Tendermint are now hard-wired into BigchainDB Server: it checks to see what version the connected Tendermint has, and if it's not compatible, BigchainDB Server exits with an error message. [Pull request #2541](https://github.com/bigchaindb/bigchaindb/pull/2541) +* The docs no longer say to install the highest version of Tendermint: they say to install a specific version. [Pull request #2524](https://github.com/bigchaindb/bigchaindb/pull/2524) +* The setup docs include more recommended settings for `config.toml`. [Pull request #2516](https://github.com/bigchaindb/bigchaindb/pull/2516) +* The process to add, remove or update the voting power of a validator at run time (using the `bigchaindb upsert-validator` subcommands) was completely changed and is now fully working. See [issue #2372](https://github.com/bigchaindb/bigchaindb/issues/2372) and all the pull requests it references. Pull requests [#2439](https://github.com/bigchaindb/bigchaindb/pull/2439) and [#2440](https://github.com/bigchaindb/bigchaindb/pull/2440) +* The license on the documentation was changed from CC-BY-SA-4 to CC-BY-4. [Pull request #2427](https://github.com/bigchaindb/bigchaindb/pull/2427) +* Re-activated and/or updated some unit tests that had been deacivated during the migration to Tendermint. Pull requests [#2390](https://github.com/bigchaindb/bigchaindb/pull/2390), [#2415](https://github.com/bigchaindb/bigchaindb/pull/2415), [#2452](https://github.com/bigchaindb/bigchaindb/pull/24), [#2456](https://github.com/bigchaindb/bigchaindb/pull/2456) +* Updated RapidJSON to a newer, faster version. [Pull request #2470](https://github.com/bigchaindb/bigchaindb/pull/2470) +* The Java driver is now officially supported. [Pull request #2478](https://github.com/bigchaindb/bigchaindb/pull/2478) +* The MongoDB indexes on transaction id and block height were changed to be [unique indexes](https://docs.mongodb.com/manual/core/index-unique/). [Pull request #2492](https://github.com/bigchaindb/bigchaindb/pull/2492) +* Updated the required `cryptoconditions` package to a newer one. [Pull request #2494](https://github.com/bigchaindb/bigchaindb/pull/2494) + +### Removed + +* Removed some old code and tests. Pull requests + [#2374](https://github.com/bigchaindb/bigchaindb/pull/2374), + [#2452](https://github.com/bigchaindb/bigchaindb/pull/2452), + [#2474](https://github.com/bigchaindb/bigchaindb/pull/2474), + [#2476](https://github.com/bigchaindb/bigchaindb/pull/2476), + [#2491](https://github.com/bigchaindb/bigchaindb/pull/2491) + +### Fixed + +* Fixed the Events API so that it only sends valid transactions to subscribers. Also changed how it works internally, so now it is more reliable. [Pull request #2529](https://github.com/bigchaindb/bigchaindb/pull/2529) +* Fixed a bug where MongoDB database initialization would abort if a collection already existed. [Pull request #2520](https://github.com/bigchaindb/bigchaindb/pull/2520) +* Fixed a unit test that was failing randomly. [Pull request #2423](https://github.com/bigchaindb/bigchaindb/pull/2423) +* Fixed the validator curl port. [Pull request #2447](https://github.com/bigchaindb/bigchaindb/pull/2447) +* Fixed an error in the docs about the HTTP POST /transactions endpoint. [Pull request #2481](https://github.com/bigchaindb/bigchaindb/pull/2481) +* Fixed a unit test that could loop forever. [Pull requqest #2486](https://github.com/bigchaindb/bigchaindb/pull/2486) +* Fixed a bug when validating a CREATE + TRANSFER. [Pull request #2487](https://github.com/bigchaindb/bigchaindb/pull/2487) +* Fixed the HTTP response when posting a transaction in commit mode. [Pull request #2510](https://github.com/bigchaindb/bigchaindb/pull/2510) +* Fixed a crash that happened when attempting to restart BigchainDB at Tendermint block height 1. [Pull request#2519](https://github.com/bigchaindb/bigchaindb/pull/2519) + +### External Contributors + +@danacr - [Pull request #2447](https://github.com/bigchaindb/bigchaindb/pull/2447) + +### Notes + +The docs section titled "Production Deployment Template" was renamed to "Kubernetes Deployment Template" and we no longer consider it the go-to deployment template. The "Simple Deployment Template" is simpler, easier to understand, and less expensive (unless you are with an organization that already has a big Kubernetes cluster). + ## [2.0 Beta 5] - 2018-08-01 Tag name: v2.0.0b5 From 528ba07c68f5ef64ad63b1ddaecca3171d55c7e4 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Mon, 17 Sep 2018 15:00:26 +0200 Subject: [PATCH 41/77] Last PR before 2.0.0b6 (#2550) * Update k8s files for 2.0.0b6 * Update version.py for 2.0.0b6 --- bigchaindb/version.py | 4 ++-- k8s/bigchaindb/bigchaindb-ss.yaml | 2 +- k8s/dev-setup/bigchaindb.yaml | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bigchaindb/version.py b/bigchaindb/version.py index 277f5645..b1761387 100644 --- a/bigchaindb/version.py +++ b/bigchaindb/version.py @@ -2,7 +2,7 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -__version__ = '2.0.0b5' -__short_version__ = '2.0b5' +__version__ = '2.0.0b6' +__short_version__ = '2.0b6' # supported Tendermint version __tm_supported_versions__ = ["0.22.8"] diff --git a/k8s/bigchaindb/bigchaindb-ss.yaml b/k8s/bigchaindb/bigchaindb-ss.yaml index 1371f884..b6ada5a3 100644 --- a/k8s/bigchaindb/bigchaindb-ss.yaml +++ b/k8s/bigchaindb/bigchaindb-ss.yaml @@ -158,7 +158,7 @@ spec: timeoutSeconds: 15 # BigchainDB container - name: bigchaindb - image: bigchaindb/bigchaindb:2.0.0-beta5 + image: bigchaindb/bigchaindb:2.0.0-beta6 imagePullPolicy: Always args: - start diff --git a/k8s/dev-setup/bigchaindb.yaml b/k8s/dev-setup/bigchaindb.yaml index f1d19615..7a2fa34c 100644 --- a/k8s/dev-setup/bigchaindb.yaml +++ b/k8s/dev-setup/bigchaindb.yaml @@ -38,7 +38,7 @@ spec: terminationGracePeriodSeconds: 10 containers: - name: bigchaindb - image: bigchaindb/bigchaindb:2.0.0-beta5 + image: bigchaindb/bigchaindb:2.0.0-beta6 imagePullPolicy: Always args: - start From cf6fa6b5539610df21e18700d0c2fa676d21486a Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Mon, 17 Sep 2018 15:36:38 +0200 Subject: [PATCH 42/77] Problem: Election type not casted for runnign command (#2551) Solution: Replace '-' with '_' before running election sub-command --- bigchaindb/commands/bigchaindb.py | 3 ++- tests/commands/test_commands.py | 10 +++++----- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index f9588dcd..58db7ac0 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -113,7 +113,8 @@ def run_election(args): def run_election_new(args, bigchain): - globals()[f'run_election_new_{args.election_type}'](args, bigchain) + election_type = args.election_type.replace('-', '_') + globals()[f'run_election_new_{election_type}'](args, bigchain) def create_new_election(sk, bigchain, election_class, data): diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index bbeea227..f9b14f89 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -306,7 +306,7 @@ def test_election_new_upsert_validator_with_tendermint(b, priv_validator_path, u from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator new_args = Namespace(action='new', - election_type='upsert_validator', + election_type='upsert-validator', public_key='HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY=', power=1, node_id='unique_node_id_for_test_upsert_validator_new_with_tendermint', @@ -330,7 +330,7 @@ def test_election_new_upsert_validator_without_tendermint(caplog, b, priv_valida b.write_transaction = mock_write args = Namespace(action='new', - election_type='upsert_validator', + election_type='upsert-validator', public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=', power=1, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', @@ -384,7 +384,7 @@ def test_election_new_upsert_validator_invalid_election(caplog, b, priv_validato from bigchaindb.commands.bigchaindb import run_election_new_upsert_validator args = Namespace(action='new', - election_type='upsert_validator', + election_type='upsert-validator', public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=', power=10, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', @@ -408,7 +408,7 @@ def test_election_new_upsert_validator_invalid_power(caplog, b, priv_validator_p b.write_transaction = mock_write b.get_validators = mock_get_validators args = Namespace(action='new', - election_type='upsert_validator', + election_type='upsert-validator', public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=', power=10, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', @@ -427,7 +427,7 @@ def test_election_approve_with_tendermint(b, priv_validator_path, user_sk, valid public_key = 'CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=' new_args = Namespace(action='new', - election_type='upsert_validator', + election_type='upsert-validator', public_key=public_key, power=1, node_id='fb7140f03a4ffad899fabbbf655b97e0321add66', From 126e90e7321f571dfbc7bc25b165b21ee6ebc9d1 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Tue, 18 Sep 2018 12:32:16 +0200 Subject: [PATCH 43/77] Adjust multiple elections conclusion. (#2553) - Do not conclude migration election if there is a migration in progress. - Rewrite election tests to not use mocks and assert many different things. - Record concluded elections in the `election` collection. --- bigchaindb/backend/localmongodb/query.py | 5 +- bigchaindb/elections/election.py | 42 ++-- .../migrations/chain_migration_election.py | 9 + tests/conftest.py | 51 +---- tests/elections/test_election.py | 179 +++++++++++++----- .../test_upsert_validator_vote.py | 33 ++-- tests/utils.py | 64 ++++++- 7 files changed, 236 insertions(+), 147 deletions(-) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index 75fa7aa5..6b0793ba 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -283,12 +283,11 @@ def store_validator_set(conn, validators_update): @register_query(LocalMongoDBConnection) def store_election_results(conn, election): - height = election['height'] return conn.run( conn.collection('elections').replace_one( - {'height': height}, + {'election_id': election['election_id']}, election, - upsert=True + upsert=True, ) ) diff --git a/bigchaindb/elections/election.py b/bigchaindb/elections/election.py index ca0a4be3..40861a52 100644 --- a/bigchaindb/elections/election.py +++ b/bigchaindb/elections/election.py @@ -186,27 +186,26 @@ class Election(Transaction): election_pk)) return self.count_votes(election_pk, txns, dict.get) - @classmethod - def has_concluded(cls, bigchain, election_id, current_votes=[], height=None): - """Check if the given `election_id` can be concluded or not - NOTE: - * Election is concluded iff the current validator set is exactly equal - to the validator set encoded in election outputs - * Election can concluded only if the current votes achieves a supermajority + def has_concluded(self, bigchain, current_votes=[], height=None): + """Check if the election can be concluded or not. + + * Elections can only be concluded if the current validator set + is exactly equal to the validator set encoded in the election outputs. + * Elections can be concluded only if the current votes form a supermajority. + + Custom elections may override this function and introduce additional checks. """ - election = bigchain.get_transaction(election_id) - if election: - election_pk = election.to_public_key(election.id) - votes_committed = election.get_commited_votes(bigchain, election_pk) - votes_current = election.count_votes(election_pk, current_votes) - current_validators = election.get_validators(bigchain, height) + election_pk = self.to_public_key(self.id) + votes_committed = self.get_commited_votes(bigchain, election_pk) + votes_current = self.count_votes(election_pk, current_votes) + current_validators = self.get_validators(bigchain, height) - if election.is_same_topology(current_validators, election.outputs): - total_votes = sum(current_validators.values()) - if (votes_committed < (2/3)*total_votes) and \ - (votes_committed + votes_current >= (2/3)*total_votes): - return election + if self.is_same_topology(current_validators, self.outputs): + total_votes = sum(current_validators.values()) + if (votes_committed < (2/3) * total_votes) and \ + (votes_committed + votes_current >= (2/3)*total_votes): + return True return False def get_status(self, bigchain): @@ -255,9 +254,11 @@ class Election(Transaction): validator_set_updated = False validator_set_change = [] for election_id, votes in elections.items(): - election = Election.has_concluded(bigchain, election_id, votes, new_height) + election = bigchain.get_transaction(election_id) + if election is None: + continue - if not election: + if not election.has_concluded(bigchain, votes, new_height): continue if election.makes_validator_set_change(): @@ -267,6 +268,7 @@ class Election(Transaction): validator_set_updated = True election.on_approval(bigchain, election, new_height) + election.store_election_results(bigchain, election, new_height) return validator_set_change diff --git a/bigchaindb/migrations/chain_migration_election.py b/bigchaindb/migrations/chain_migration_election.py index e1e783d0..8e80b979 100644 --- a/bigchaindb/migrations/chain_migration_election.py +++ b/bigchaindb/migrations/chain_migration_election.py @@ -10,6 +10,15 @@ class ChainMigrationElection(Election): TX_SCHEMA_CUSTOM = TX_SCHEMA_CHAIN_MIGRATION_ELECTION CHANGES_VALIDATOR_SET = False + def has_concluded(self, bigchaindb, *args, **kwargs): + chain = bigchaindb.get_latest_abci_chain() + if chain is not None and not chain['is_synced']: + # do not conclude the migration election if + # there is another migration in progress + return False + + return super().has_concluded(bigchaindb, *args, **kwargs) + @classmethod def on_approval(cls, bigchain, election, new_height): bigchain.migrate_abci_chain() diff --git a/tests/conftest.py b/tests/conftest.py index 4c271aad..90d54dea 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -23,7 +23,6 @@ from pymongo import MongoClient from bigchaindb import ValidatorElection from bigchaindb.common import crypto from bigchaindb.log import setup_logging -from bigchaindb.migrations.chain_migration_election import ChainMigrationElection from bigchaindb.tendermint_utils import key_from_base64 from bigchaindb.backend import schema, query from bigchaindb.common.crypto import (key_pair_from_ed25519_key, @@ -714,22 +713,6 @@ def valid_upsert_validator_election_2(b_mock, node_key, new_validator): new_validator, None).sign([node_key.private_key]) -@pytest.fixture -def valid_chain_migration_election(b_mock, node_key): - voters = ChainMigrationElection.recipients(b_mock) - return ChainMigrationElection.generate([node_key.public_key], - voters, - {}, None).sign([node_key.private_key]) - - -@pytest.fixture -def valid_chain_migration_election_2(b_mock, node_key): - voters = ChainMigrationElection.recipients(b_mock) - return ChainMigrationElection.generate([node_key.public_key], - voters, - {}, None).sign([node_key.private_key]) - - @pytest.fixture def ongoing_validator_election(b, valid_upsert_validator_election, ed25519_node_keys): validators = b.get_validators(height=1) @@ -758,24 +741,6 @@ def ongoing_validator_election_2(b, valid_upsert_validator_election_2, ed25519_n return valid_upsert_validator_election_2 -@pytest.fixture -def ongoing_chain_migration_election(b, valid_chain_migration_election, ed25519_node_keys): - - b.store_bulk_transactions([valid_chain_migration_election]) - block_1 = Block(app_hash='hash_1', height=1, transactions=[valid_chain_migration_election.id]) - b.store_block(block_1._asdict()) - return valid_chain_migration_election - - -@pytest.fixture -def ongoing_chain_migration_election_2(b, valid_chain_migration_election_2, ed25519_node_keys): - - b.store_bulk_transactions([valid_chain_migration_election_2]) - block_1 = Block(app_hash='hash_2', height=1, transactions=[valid_chain_migration_election_2.id]) - b.store_block(block_1._asdict()) - return valid_chain_migration_election_2 - - @pytest.fixture def validator_election_votes(b_mock, ongoing_validator_election, ed25519_node_keys): voters = ValidatorElection.recipients(b_mock) @@ -790,23 +755,9 @@ def validator_election_votes_2(b_mock, ongoing_validator_election_2, ed25519_nod return votes -@pytest.fixture -def chain_migration_election_votes(b_mock, ongoing_chain_migration_election, ed25519_node_keys): - voters = ChainMigrationElection.recipients(b_mock) - votes = generate_votes(ongoing_chain_migration_election, voters, ed25519_node_keys) - return votes - - -@pytest.fixture -def chain_migration_election_votes_2(b_mock, ongoing_chain_migration_election_2, ed25519_node_keys): - voters = ChainMigrationElection.recipients(b_mock) - votes = generate_votes(ongoing_chain_migration_election_2, voters, ed25519_node_keys) - return votes - - def generate_votes(election, voters, keys): votes = [] - for voter in range(len(voters)): + for voter, _ in enumerate(voters): v = gen_vote(election, voter, keys) votes.append(v) return votes diff --git a/tests/elections/test_election.py b/tests/elections/test_election.py index 0e9da188..0aed1816 100644 --- a/tests/elections/test_election.py +++ b/tests/elections/test_election.py @@ -1,64 +1,143 @@ -from unittest.mock import MagicMock - import pytest +from tests.utils import generate_election, generate_validators + +from bigchaindb.lib import Block from bigchaindb.elections.election import Election +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection +from bigchaindb.upsert_validator.validator_election import ValidatorElection @pytest.mark.bdb -def test_approved_elections_one_migration_one_upsert( - b, - ongoing_validator_election, validator_election_votes, - ongoing_chain_migration_election, chain_migration_election_votes -): - txns = validator_election_votes + \ - chain_migration_election_votes - mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) - mock_chain_migration.assert_called_once() - mock_store_validator.assert_called_once() +def test_approved_elections_concludes_all_elections(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + new_validator = generate_validators([1])[0] + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election']) + txs = [election] + total_votes = votes + + election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + + txs += [election] + total_votes += votes + + b.store_abci_chain(1, 'chain-X') + b.store_block(Block(height=1, + transactions=[tx.id for tx in txs], + app_hash='')._asdict()) + b.store_bulk_transactions(txs) + + Election.approved_elections(b, 1, total_votes) + + validators = b.get_validators() + assert len(validators) == 5 + assert new_validator['storage'] in validators + + chain = b.get_latest_abci_chain() + assert chain + assert chain == { + 'height': 2, + 'is_synced': False, + 'chain_id': 'chain-X-migrated-at-height-1', + } + + for tx in txs: + election = b.get_election(tx.id) + assert election @pytest.mark.bdb -def test_approved_elections_one_migration_two_upsert( - b, - ongoing_validator_election, validator_election_votes, - ongoing_validator_election_2, validator_election_votes_2, - ongoing_chain_migration_election, chain_migration_election_votes -): - txns = validator_election_votes + \ - validator_election_votes_2 + \ - chain_migration_election_votes - mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) - mock_chain_migration.assert_called_once() - mock_store_validator.assert_called_once() +def test_approved_elections_applies_only_one_validator_update(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + new_validator = generate_validators([1])[0] + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election']) + txs = [election] + total_votes = votes + + another_validator = generate_validators([1])[0] + + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + another_validator['election']) + txs += [election] + total_votes += votes + + b.store_block(Block(height=1, + transactions=[tx.id for tx in txs], + app_hash='')._asdict()) + b.store_bulk_transactions(txs) + + Election.approved_elections(b, 1, total_votes) + + validators = b.get_validators() + assert len(validators) == 5 + assert new_validator['storage'] in validators + assert another_validator['storage'] not in validators + + assert b.get_election(txs[0].id) + assert not b.get_election(txs[1].id) @pytest.mark.bdb -def test_approved_elections_two_migrations_one_upsert( - b, - ongoing_validator_election, validator_election_votes, - ongoing_chain_migration_election, chain_migration_election_votes, - ongoing_chain_migration_election_2, chain_migration_election_votes_2 -): - txns = validator_election_votes + \ - chain_migration_election_votes + \ - chain_migration_election_votes_2 - mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) - assert mock_chain_migration.call_count == 2 - mock_store_validator.assert_called_once() +def test_approved_elections_applies_only_one_migration(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + txs = [election] + total_votes = votes + + election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + + txs += [election] + total_votes += votes + + b.store_abci_chain(1, 'chain-X') + b.store_block(Block(height=1, + transactions=[tx.id for tx in txs], + app_hash='')._asdict()) + b.store_bulk_transactions(txs) + + Election.approved_elections(b, 1, total_votes) + chain = b.get_latest_abci_chain() + assert chain + assert chain == { + 'height': 2, + 'is_synced': False, + 'chain_id': 'chain-X-migrated-at-height-1', + } + + assert b.get_election(txs[0].id) + assert not b.get_election(txs[1].id) -def test_approved_elections_no_elections(b): - txns = [] - mock_chain_migration, mock_store_validator = run_approved_elections(b, txns) - mock_chain_migration.assert_not_called() - mock_store_validator.assert_not_called() - - -def run_approved_elections(bigchain, txns): - mock_chain_migration = MagicMock() - mock_store_validator = MagicMock() - bigchain.migrate_abci_chain = mock_chain_migration - bigchain.store_validator_set = mock_store_validator - Election.approved_elections(bigchain, 1, txns) - return mock_chain_migration, mock_store_validator +def test_approved_elections_gracefully_handles_empty_block(b): + Election.approved_elections(b, 1, []) diff --git a/tests/upsert_validator/test_upsert_validator_vote.py b/tests/upsert_validator/test_upsert_validator_vote.py index 3c3512d0..7c49bd77 100644 --- a/tests/upsert_validator/test_upsert_validator_vote.py +++ b/tests/upsert_validator/test_upsert_validator_vote.py @@ -168,14 +168,14 @@ def test_valid_election_conclude(b_mock, valid_upsert_validator_election, ed2551 # store election b_mock.store_bulk_transactions([valid_upsert_validator_election]) # cannot conclude election as not votes exist - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) + assert not valid_upsert_validator_election.has_concluded(b_mock) # validate vote assert tx_vote0.validate(b_mock) - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote0]) + assert not valid_upsert_validator_election.has_concluded(b_mock, [tx_vote0]) b_mock.store_bulk_transactions([tx_vote0]) - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) + assert not valid_upsert_validator_election.has_concluded(b_mock) # Node 1: cast vote tx_vote1 = gen_vote(valid_upsert_validator_election, 1, ed25519_node_keys) @@ -187,31 +187,31 @@ def test_valid_election_conclude(b_mock, valid_upsert_validator_election, ed2551 tx_vote3 = gen_vote(valid_upsert_validator_election, 3, ed25519_node_keys) assert tx_vote1.validate(b_mock) - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote1]) + assert not valid_upsert_validator_election.has_concluded(b_mock, [tx_vote1]) # 2/3 is achieved in the same block so the election can be.has_concludedd - assert ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote1, tx_vote2]) + assert valid_upsert_validator_election.has_concluded(b_mock, [tx_vote1, tx_vote2]) b_mock.store_bulk_transactions([tx_vote1]) - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) + assert not valid_upsert_validator_election.has_concluded(b_mock) assert tx_vote2.validate(b_mock) assert tx_vote3.validate(b_mock) # conclusion can be triggered my different votes in the same block - assert ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote2]) - assert ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote2, tx_vote3]) + assert valid_upsert_validator_election.has_concluded(b_mock, [tx_vote2]) + assert valid_upsert_validator_election.has_concluded(b_mock, [tx_vote2, tx_vote3]) b_mock.store_bulk_transactions([tx_vote2]) # Once the blockchain records >2/3 of the votes the election is assumed to be.has_concludedd # so any invocation of `.has_concluded` for that election should return False - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id) + assert not valid_upsert_validator_election.has_concluded(b_mock) # Vote is still valid but the election cannot be.has_concludedd as it it assmed that it has # been.has_concludedd before assert tx_vote3.validate(b_mock) - assert not ValidatorElection.has_concluded(b_mock, valid_upsert_validator_election.id, [tx_vote3]) + assert not valid_upsert_validator_election.has_concluded(b_mock, [tx_vote3]) @pytest.mark.abci @@ -285,9 +285,9 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): tx_vote1 = gen_vote(election, 1, ed25519_node_keys) tx_vote2 = gen_vote(election, 2, ed25519_node_keys) - assert not ValidatorElection.has_concluded(b, election.id, [tx_vote0]) - assert not ValidatorElection.has_concluded(b, election.id, [tx_vote0, tx_vote1]) - assert ValidatorElection.has_concluded(b, election.id, [tx_vote0, tx_vote1, tx_vote2]) + assert not election.has_concluded(b, [tx_vote0]) + assert not election.has_concluded(b, [tx_vote0, tx_vote1]) + assert election.has_concluded(b, [tx_vote0, tx_vote1, tx_vote2]) assert Election.approved_elections(b, 4, [tx_vote0]) == [] assert Election.approved_elections(b, 4, [tx_vote0, tx_vote1]) == [] @@ -297,13 +297,6 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 - b.store_bulk_transactions([tx_vote0, tx_vote1]) - - update = Election.approved_elections(b, 4, [tx_vote2]) - assert len(update) == 1 - update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') - assert update_public_key == public_key64 - # remove validator power = 0 new_validator = {'public_key': {'value': public_key, 'type': 'ed25519-base16'}, diff --git a/tests/utils.py b/tests/utils.py index 852be5e6..5b2e1ea6 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -2,12 +2,17 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 +import base58 +import base64 +import random + from functools import singledispatch -from bigchaindb import Vote from bigchaindb.backend.localmongodb.connection import LocalMongoDBConnection from bigchaindb.backend.schema import TABLES -from bigchaindb.elections.election import Election +from bigchaindb.common import crypto +from bigchaindb.elections.election import Election, Vote +from bigchaindb.tendermint_utils import key_to_base64 @singledispatch @@ -24,7 +29,6 @@ def flush_localmongo_db(connection, dbname): def generate_block(bigchain): from bigchaindb.common.crypto import generate_key_pair from bigchaindb.models import Transaction - import time alice = generate_key_pair() tx = Transaction.create([alice.public_key], @@ -34,7 +38,6 @@ def generate_block(bigchain): code, message = bigchain.write_transaction(tx, 'broadcast_tx_commit') assert code == 202 - time.sleep(2) def to_inputs(election, i, ed25519_node_keys): @@ -52,3 +55,56 @@ def gen_vote(election, i, ed25519_node_keys): [([election_pub_key], votes_i)], election_id=election.id)\ .sign([key_i.private_key]) + + +def generate_validators(powers): + """Generates an arbitrary number of validators with random public keys. + + The object under the `storage` key is in the format expected by DB. + + The object under the `eleciton` key is in the format expected by + the upsert validator election. + + `public_key`, `private_key` are in the format used for signing transactions. + + Args: + powers: A list of intergers representing the voting power to + assign to the corresponding validators. + """ + validators = [] + for power in powers: + kp = crypto.generate_key_pair() + validators.append({ + 'storage': { + 'public_key': { + 'value': key_to_base64(base58.b58decode(kp.public_key).hex()), + 'type': 'ed25519-base64', + }, + 'voting_power': power, + }, + 'election': { + 'node_id': f'node-{random.choice(range(100))}', + 'power': power, + 'public_key': { + 'value': base64.b16encode(base58.b58decode(kp.public_key)).decode('utf-8'), + 'type': 'ed25519-base16', + }, + }, + 'public_key': kp.public_key, + 'private_key': kp.private_key, + }) + return validators + + +def generate_election(b, cls, public_key, private_key, asset_data): + voters = cls.recipients(b) + election = cls.generate([public_key], + voters, + asset_data, + None).sign([private_key]) + + votes = [Vote.generate([election.to_inputs()[i]], + [([Election.to_public_key(election.id)], power)], + election.id) for i, (_, power) in enumerate(voters)] + + return election, votes From 2f6bbaeb4ba2409a3c0e21ba479c95b2c619ed7e Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 20 Sep 2018 09:54:34 +0200 Subject: [PATCH 44/77] Link to Alberto's tips from Troubleshooting page (#2562) because they are useful! --- .../server/source/simple-deployment-template/troubleshooting.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/server/source/simple-deployment-template/troubleshooting.md b/docs/server/source/simple-deployment-template/troubleshooting.md index 2d0311fd..b2d95722 100644 --- a/docs/server/source/simple-deployment-template/troubleshooting.md +++ b/docs/server/source/simple-deployment-template/troubleshooting.md @@ -26,4 +26,6 @@ addr_book_strict = false ## Other Problems +See the [Tendermint tips in the vrde/notes repository](https://github.com/vrde/notes/tree/master/tendermint). + If you're stuck, maybe [file a new issue on GitHub](https://github.com/bigchaindb/bigchaindb/issues/new). If your problem occurs often enough, we'll write about it here. From 407571ddf46405f8bdb623a6cb2be8931f6e6451 Mon Sep 17 00:00:00 2001 From: codegeschrei Date: Thu, 20 Sep 2018 10:31:05 +0200 Subject: [PATCH 45/77] Problem: send function has a deprecation warning in the driver (#2547) Solution: replace send with the matching new functions --- acceptance/python/src/test_basic.py | 4 ++-- acceptance/python/src/test_divisible_asset.py | 7 +++---- acceptance/python/src/test_double_spend.py | 2 +- acceptance/python/src/test_multiple_owners.py | 5 ++--- acceptance/python/src/test_naughty_strings.py | 2 +- acceptance/python/src/test_stream.py | 2 +- 6 files changed, 10 insertions(+), 12 deletions(-) diff --git a/acceptance/python/src/test_basic.py b/acceptance/python/src/test_basic.py index 8702ef65..a6f2b571 100644 --- a/acceptance/python/src/test_basic.py +++ b/acceptance/python/src/test_basic.py @@ -61,7 +61,7 @@ def test_basic(): bike_id = fulfilled_creation_tx['id'] # Now she is ready to send it to the BigchainDB Network. - sent_transfer_tx = bdb.transactions.send(fulfilled_creation_tx) + sent_transfer_tx = bdb.transactions.send_commit(fulfilled_creation_tx) # And just to be 100% sure, she also checks if she can retrieve # it from the BigchainDB node. @@ -107,7 +107,7 @@ def test_basic(): private_keys=alice.private_key) # She finally sends the transaction to a BigchainDB node. - sent_transfer_tx = bdb.transactions.send(fulfilled_transfer_tx) + sent_transfer_tx = bdb.transactions.send_commit(fulfilled_transfer_tx) # And just to be 100% sure, she also checks if she can retrieve # it from the BigchainDB node. diff --git a/acceptance/python/src/test_divisible_asset.py b/acceptance/python/src/test_divisible_asset.py index b9a725d9..221226ee 100644 --- a/acceptance/python/src/test_divisible_asset.py +++ b/acceptance/python/src/test_divisible_asset.py @@ -74,7 +74,7 @@ def test_divisible_assets(): prepared_token_tx, private_keys=alice.private_key) - bdb.transactions.send(fulfilled_token_tx, mode='commit') + bdb.transactions.send_commit(fulfilled_token_tx) # We store the `id` of the transaction to use it later on. bike_token_id = fulfilled_token_tx['id'] @@ -116,8 +116,7 @@ def test_divisible_assets(): prepared_transfer_tx, private_keys=bob.private_key) - sent_transfer_tx = bdb.transactions.send(fulfilled_transfer_tx, - mode='commit') + sent_transfer_tx = bdb.transactions.send_commit(fulfilled_transfer_tx) # First, Bob checks if the transaction was successful. assert bdb.transactions.retrieve( @@ -167,7 +166,7 @@ def test_divisible_assets(): # Remember Bob, last time you spent 3 tokens already, # so you only have 7 left. with pytest.raises(BadRequest) as error: - bdb.transactions.send(fulfilled_transfer_tx, mode='commit') + bdb.transactions.send_commit(fulfilled_transfer_tx) # Now Bob gets an error saying that the amount he wanted to spent is # higher than the amount of tokens he has left. diff --git a/acceptance/python/src/test_double_spend.py b/acceptance/python/src/test_double_spend.py index 2b9541b4..fd6da187 100644 --- a/acceptance/python/src/test_double_spend.py +++ b/acceptance/python/src/test_double_spend.py @@ -30,7 +30,7 @@ def test_double_create(): def send_and_queue(tx): try: - bdb.transactions.send(tx) + bdb.transactions.send_commit(tx) results.put('OK') except bigchaindb_driver.exceptions.TransportError as e: results.put('FAIL') diff --git a/acceptance/python/src/test_multiple_owners.py b/acceptance/python/src/test_multiple_owners.py index 89e93860..9eda2c87 100644 --- a/acceptance/python/src/test_multiple_owners.py +++ b/acceptance/python/src/test_multiple_owners.py @@ -64,7 +64,7 @@ def test_multiple_owners(): prepared_dw_tx, private_keys=[alice.private_key, bob.private_key]) - bdb.transactions.send(fulfilled_dw_tx, mode='commit') + bdb.transactions.send_commit(fulfilled_dw_tx) # We store the `id` of the transaction to use it later on. dw_id = fulfilled_dw_tx['id'] @@ -109,8 +109,7 @@ def test_multiple_owners(): prepared_transfer_tx, private_keys=[alice.private_key, bob.private_key]) - sent_transfer_tx = bdb.transactions.send(fulfilled_transfer_tx, - mode='commit') + sent_transfer_tx = bdb.transactions.send_commit(fulfilled_transfer_tx) # They check if the transaction was successful. assert bdb.transactions.retrieve( diff --git a/acceptance/python/src/test_naughty_strings.py b/acceptance/python/src/test_naughty_strings.py index df4cb7dc..13cd765e 100644 --- a/acceptance/python/src/test_naughty_strings.py +++ b/acceptance/python/src/test_naughty_strings.py @@ -54,7 +54,7 @@ def send_naughty_tx(asset, metadata): # The fulfilled tx gets sent to the BDB network try: - sent_transaction = bdb.transactions.send(fulfilled_transaction) + sent_transaction = bdb.transactions.send_commit(fulfilled_transaction) except BadRequest as e: sent_transaction = e diff --git a/acceptance/python/src/test_stream.py b/acceptance/python/src/test_stream.py index e8c12c5a..5421877b 100644 --- a/acceptance/python/src/test_stream.py +++ b/acceptance/python/src/test_stream.py @@ -100,7 +100,7 @@ def test_stream(): # transactions to be in the shared queue: this is a two phase test, # first we send a bunch of transactions, then we check if they are # valid (and, in this case, they should). - bdb.transactions.send(tx, mode='async') + bdb.transactions.send_async(tx) # The `id` of every sent transaction is then stored in a list. sent.append(tx['id']) From eb139fba000c2e96dbaa12e464e7d5bac544e0e4 Mon Sep 17 00:00:00 2001 From: Muawia Khan Date: Thu, 20 Sep 2018 14:30:30 +0200 Subject: [PATCH 46/77] Remove benchmark logging (#2565). * Problem: Benchmark logging is not needed. * Solution: Clean it up. --- bigchaindb/core.py | 10 ++----- bigchaindb/log.py | 30 +------------------ docs/server/source/appendices/log-rotation.md | 1 - .../source/server-reference/configuration.md | 4 +-- setup.py | 6 +--- tests/conftest.py | 5 ---- 6 files changed, 7 insertions(+), 49 deletions(-) diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 8be3cdaf..d7b98183 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -146,16 +146,13 @@ class App(BaseApplication): self.abort_if_abci_chain_is_not_synced() - logger.benchmark('CHECK_TX_INIT') logger.debug('check_tx: %s', raw_transaction) transaction = decode_transaction(raw_transaction) if self.bigchaindb.is_valid_transaction(transaction): logger.debug('check_tx: VALID') - logger.benchmark('CHECK_TX_END, tx_id:%s', transaction['id']) return ResponseCheckTx(code=CodeTypeOk) else: logger.debug('check_tx: INVALID') - logger.benchmark('CHECK_TX_END, tx_id:%s', transaction['id']) return ResponseCheckTx(code=CodeTypeError) def begin_block(self, req_begin_block): @@ -167,9 +164,9 @@ class App(BaseApplication): self.abort_if_abci_chain_is_not_synced() chain_shift = 0 if self.chain is None else self.chain['height'] - logger.benchmark('BEGIN BLOCK, height:%s, num_txs:%s', - req_begin_block.header.height + chain_shift, - req_begin_block.header.num_txs) + logger.debug('BEGIN BLOCK, height:%s, num_txs:%s', + req_begin_block.header.height + chain_shift, + req_begin_block.header.num_txs) self.block_txn_ids = [] self.block_transactions = [] @@ -253,7 +250,6 @@ class App(BaseApplication): logger.debug('Commit-ing new block with hash: apphash=%s ,' 'height=%s, txn ids=%s', data, self.new_height, self.block_txn_ids) - logger.benchmark('COMMIT_BLOCK, height:%s', self.new_height) if self.events_queue: event = Event(EventTypes.BLOCK_VALID, { diff --git a/bigchaindb/log.py b/bigchaindb/log.py index 6b6ba931..301feea3 100644 --- a/bigchaindb/log.py +++ b/bigchaindb/log.py @@ -11,8 +11,6 @@ import os DEFAULT_LOG_DIR = os.getcwd() -BENCHMARK_LOG_LEVEL = 15 - DEFAULT_LOGGING_CONFIG = { 'version': 1, @@ -29,11 +27,6 @@ DEFAULT_LOGGING_CONFIG = { 'format': ('[%(asctime)s] [%(levelname)s] (%(name)s) ' '%(message)s (%(processName)-10s - pid: %(process)d)'), 'datefmt': '%Y-%m-%d %H:%M:%S', - }, - 'benchmark': { - 'class': 'logging.Formatter', - 'format': ('%(asctime)s, %(levelname)s, %(message)s'), - 'datefmt': '%Y-%m-%d %H:%M:%S', } }, 'handlers': { @@ -59,31 +52,16 @@ DEFAULT_LOGGING_CONFIG = { 'backupCount': 5, 'formatter': 'file', 'level': logging.ERROR, - }, - 'benchmark': { - 'class': 'logging.handlers.RotatingFileHandler', - 'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb-benchmark.log'), - 'mode': 'w', - 'maxBytes': 209715200, - 'backupCount': 5, - 'formatter': 'benchmark', - 'level': BENCHMARK_LOG_LEVEL, } }, 'loggers': {}, 'root': { 'level': logging.DEBUG, - 'handlers': ['console', 'file', 'errors', 'benchmark'], + 'handlers': ['console', 'file', 'errors'], }, } -def benchmark(self, message, *args, **kws): - # Yes, logger takes its '*args' as 'args'. - if self.isEnabledFor(BENCHMARK_LOG_LEVEL): - self._log(BENCHMARK_LOG_LEVEL, message, args, **kws) - - def _normalize_log_level(level): try: return level.upper() @@ -104,11 +82,6 @@ def setup_logging(): """ - # Add a new logging level for logging benchmark - logging.addLevelName(BENCHMARK_LOG_LEVEL, 'BENCHMARK') - logging.BENCHMARK = BENCHMARK_LOG_LEVEL - logging.Logger.benchmark = benchmark - logging_configs = DEFAULT_LOGGING_CONFIG new_logging_configs = bigchaindb.config['log'] @@ -127,7 +100,6 @@ def setup_logging(): if 'level_logfile' in new_logging_configs: level = _normalize_log_level(new_logging_configs['level_logfile']) logging_configs['handlers']['file']['level'] = level - logging_configs['handlers']['benchmark']['level'] = level if 'fmt_console' in new_logging_configs: fmt = new_logging_configs['fmt_console'] diff --git a/docs/server/source/appendices/log-rotation.md b/docs/server/source/appendices/log-rotation.md index 25c0cd6e..95d52115 100644 --- a/docs/server/source/appendices/log-rotation.md +++ b/docs/server/source/appendices/log-rotation.md @@ -25,7 +25,6 @@ please refer to [MongoDB docs](https://docs.mongodb.com/v3.6/tutorial/rotate-log Log rotation is baked into BigchainDB server using the `logging` module. BigchainDB server logs information into the following files: - `bigchaindb.log` - `bigchaindb-errors.log` - - `bigchaindb-benchmark.log` These log files are created by default in the directory from where you run `bigchaindb start`, if you are using `monit`, from [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide, the default directory is: `$HOME/.bigchaindb-monit/logs` diff --git a/docs/server/source/server-reference/configuration.md b/docs/server/source/server-reference/configuration.md index 47d06de8..66d9f23f 100644 --- a/docs/server/source/server-reference/configuration.md +++ b/docs/server/source/server-reference/configuration.md @@ -292,7 +292,7 @@ defined by [Python](https://docs.python.org/3.6/library/logging.html#levels), but case-insensitive for the sake of convenience: ```text -"critical", "error", "warning", "info", "benchmark", "debug", "notset" +"critical", "error", "warning", "info", "debug", "notset" ``` ### log.level_logfile @@ -302,7 +302,7 @@ defined by [Python](https://docs.python.org/3.6/library/logging.html#levels), but case-insensitive for the sake of convenience: ```text -"critical", "error", "warning", "info", "benchmark", "debug", "notset" +"critical", "error", "warning", "info", "debug", "notset" ``` ### log.datefmt_console diff --git a/setup.py b/setup.py index 4181bddf..3bb2151c 100644 --- a/setup.py +++ b/setup.py @@ -71,10 +71,6 @@ tests_require = [ 'tox', ] + docs_require -benchmarks_require = [ - 'line-profiler==1.0', -] - install_requires = [ # TODO Consider not installing the db drivers, or putting them in extras. 'pymongo~=3.6', @@ -144,7 +140,7 @@ setup( tests_require=tests_require, extras_require={ 'test': tests_require, - 'dev': dev_require + tests_require + docs_require + benchmarks_require, + 'dev': dev_require + tests_require + docs_require, 'docs': docs_require, }, package_data={'bigchaindb.common.schema': ['*.yaml']}, diff --git a/tests/conftest.py b/tests/conftest.py index 90d54dea..e001465c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -22,7 +22,6 @@ from pymongo import MongoClient from bigchaindb import ValidatorElection from bigchaindb.common import crypto -from bigchaindb.log import setup_logging from bigchaindb.tendermint_utils import key_from_base64 from bigchaindb.backend import schema, query from bigchaindb.common.crypto import (key_pair_from_ed25519_key, @@ -108,10 +107,6 @@ def _configure_bigchaindb(request): config = config_utils.env_config(config) config_utils.set_config(config) - # NOTE: since we use a custom log level - # for benchmark logging we need to setup logging - setup_logging() - @pytest.fixture(scope='session') def _setup_database(_configure_bigchaindb): From 39be7a2fdfdd2ba0818c5598229fffbe6bbd9509 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 20 Sep 2018 17:17:33 +0200 Subject: [PATCH 47/77] Renamed "consensus" to "validation" where relevant (#2561) --- bigchaindb/README.md | 4 ++-- bigchaindb/config_utils.py | 22 +++++++++++----------- bigchaindb/core.py | 3 +-- bigchaindb/lib.py | 10 +++++----- bigchaindb/{consensus.py => validation.py} | 15 +++++++-------- tests/test_config_utils.py | 16 ++++++++-------- tests/test_core.py | 8 ++++---- 7 files changed, 38 insertions(+), 40 deletions(-) rename bigchaindb/{consensus.py => validation.py} (55%) diff --git a/bigchaindb/README.md b/bigchaindb/README.md index f3353108..61a0e8cd 100644 --- a/bigchaindb/README.md +++ b/bigchaindb/README.md @@ -18,9 +18,9 @@ The `BigchainDB` class is defined here. Most node-level operations and database `Block`, `Transaction`, and `Asset` classes are defined here. The classes mirror the block and transaction structure from the [documentation](https://docs.bigchaindb.com/projects/server/en/latest/data-models/index.html), but also include methods for validation and signing. -### [`consensus.py`](./consensus.py) +### [`validation.py`](./validation.py) -Base class for consensus methods (verification of votes, blocks, and transactions). The actual logic is mostly found in `transaction` and `block` models, defined in [`models.py`](./models.py). +Base class for validation methods (verification of votes, blocks, and transactions). The actual logic is mostly found in `transaction` and `block` models, defined in [`models.py`](./models.py). ### [`processes.py`](./processes.py) diff --git a/bigchaindb/config_utils.py b/bigchaindb/config_utils.py index 3836f437..3c2015f3 100644 --- a/bigchaindb/config_utils.py +++ b/bigchaindb/config_utils.py @@ -28,7 +28,7 @@ from bigchaindb.common import exceptions import bigchaindb -from bigchaindb.consensus import BaseConsensusRules +from bigchaindb.validation import BaseValidationRules # TODO: move this to a proper configuration file for logging logging.getLogger('requests').setLevel(logging.WARNING) @@ -258,38 +258,38 @@ def autoconfigure(filename=None, config=None, force=False): @lru_cache() -def load_consensus_plugin(name=None): - """Find and load the chosen consensus plugin. +def load_validation_plugin(name=None): + """Find and load the chosen validation plugin. Args: name (string): the name of the entry_point, as advertised in the setup.py of the providing package. Returns: - an uninstantiated subclass of ``bigchaindb.consensus.AbstractConsensusRules`` + an uninstantiated subclass of ``bigchaindb.validation.AbstractValidationRules`` """ if not name: - return BaseConsensusRules + return BaseValidationRules - # TODO: This will return the first plugin with group `bigchaindb.consensus` + # TODO: This will return the first plugin with group `bigchaindb.validation` # and name `name` in the active WorkingSet. # We should probably support Requirements specs in the config, e.g. - # consensus_plugin: 'my-plugin-package==0.0.1;default' + # validation_plugin: 'my-plugin-package==0.0.1;default' plugin = None - for entry_point in iter_entry_points('bigchaindb.consensus', name): + for entry_point in iter_entry_points('bigchaindb.validation', name): plugin = entry_point.load() # No matching entry_point found if not plugin: raise ResolutionError( - 'No plugin found in group `bigchaindb.consensus` with name `{}`'. + 'No plugin found in group `bigchaindb.validation` with name `{}`'. format(name)) # Is this strictness desireable? # It will probably reduce developer headaches in the wild. - if not issubclass(plugin, (BaseConsensusRules,)): + if not issubclass(plugin, (BaseValidationRules,)): raise TypeError('object of type "{}" does not implement `bigchaindb.' - 'consensus.BaseConsensusRules`'.format(type(plugin))) + 'validation.BaseValidationRules`'.format(type(plugin))) return plugin diff --git a/bigchaindb/core.py b/bigchaindb/core.py index d7b98183..34111c47 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -40,8 +40,7 @@ class App(BaseApplication): """Bridge between BigchainDB and Tendermint. The role of this class is to expose the BigchainDB - transactional logic to the Tendermint Consensus - State Machine. + transaction logic to Tendermint Core. """ def __init__(self, bigchaindb=None, events_queue=None): diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index 2dadcb90..974a7ac5 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -27,7 +27,7 @@ from bigchaindb.common.exceptions import (SchemaValidationError, DoubleSpend) from bigchaindb.tendermint_utils import encode_transaction, merkleroot from bigchaindb import exceptions as core_exceptions -from bigchaindb.consensus import BaseConsensusRules +from bigchaindb.validation import BaseValidationRules logger = logging.getLogger(__name__) @@ -64,12 +64,12 @@ class BigchainDB(object): self.tendermint_port = bigchaindb.config['tendermint']['port'] self.endpoint = 'http://{}:{}/'.format(self.tendermint_host, self.tendermint_port) - consensusPlugin = bigchaindb.config.get('consensus_plugin') + validationPlugin = bigchaindb.config.get('validation_plugin') - if consensusPlugin: - self.consensus = config_utils.load_consensus_plugin(consensusPlugin) + if validationPlugin: + self.validation = config_utils.load_validation_plugin(validationPlugin) else: - self.consensus = BaseConsensusRules + self.validation = BaseValidationRules self.connection = connection if connection else backend.connect(**bigchaindb.config['database']) diff --git a/bigchaindb/consensus.py b/bigchaindb/validation.py similarity index 55% rename from bigchaindb/consensus.py rename to bigchaindb/validation.py index a5a20efb..f3ec9dd6 100644 --- a/bigchaindb/consensus.py +++ b/bigchaindb/validation.py @@ -3,23 +3,22 @@ # Code is Apache-2.0 and docs are CC-BY-4.0 -class BaseConsensusRules(): - """Base consensus rules for Bigchain. +class BaseValidationRules(): + """Base validation rules for BigchainDB. - A consensus plugin must expose a class inheriting from this one via an entry_point. + A validation plugin must expose a class inheriting from this one via an entry_point. All methods listed below must be implemented. - """ @staticmethod - def validate_transaction(bigchain, transaction): + def validate_transaction(bigchaindb, transaction): """See :meth:`bigchaindb.models.Transaction.validate` for documentation. """ - return transaction.validate(bigchain) + return transaction.validate(bigchaindb) @staticmethod - def validate_block(bigchain, block): + def validate_block(bigchaindb, block): """See :meth:`bigchaindb.models.Block.validate` for documentation.""" - return block.validate(bigchain) + return block.validate(bigchaindb) diff --git a/tests/test_config_utils.py b/tests/test_config_utils.py index 72dc4e29..af83db5b 100644 --- a/tests/test_config_utils.py +++ b/tests/test_config_utils.py @@ -31,24 +31,24 @@ def test_bigchain_instance_is_initialized_when_conf_provided(): assert bigchaindb.config['CONFIGURED'] is True -def test_load_consensus_plugin_loads_default_rules_without_name(): +def test_load_validation_plugin_loads_default_rules_without_name(): from bigchaindb import config_utils - from bigchaindb.consensus import BaseConsensusRules + from bigchaindb.validation import BaseValidationRules - assert config_utils.load_consensus_plugin() == BaseConsensusRules + assert config_utils.load_validation_plugin() == BaseValidationRules -def test_load_consensus_plugin_raises_with_unknown_name(): +def test_load_validation_plugin_raises_with_unknown_name(): from pkg_resources import ResolutionError from bigchaindb import config_utils with pytest.raises(ResolutionError): - config_utils.load_consensus_plugin('bogus') + config_utils.load_validation_plugin('bogus') -def test_load_consensus_plugin_raises_with_invalid_subclass(monkeypatch): +def test_load_validation_plugin_raises_with_invalid_subclass(monkeypatch): # Monkeypatch entry_point.load to return something other than a - # ConsensusRules instance + # ValidationRules instance from bigchaindb import config_utils import time monkeypatch.setattr(config_utils, @@ -58,7 +58,7 @@ def test_load_consensus_plugin_raises_with_invalid_subclass(monkeypatch): with pytest.raises(TypeError): # Since the function is decorated with `lru_cache`, we need to # "miss" the cache using a name that has not been used previously - config_utils.load_consensus_plugin(str(time.time())) + config_utils.load_validation_plugin(str(time.time())) def test_load_events_plugins(monkeypatch): diff --git a/tests/test_core.py b/tests/test_core.py index 72e5bc1d..85fc7ec3 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -35,20 +35,20 @@ def config(request, monkeypatch): def test_bigchain_class_default_initialization(config): from bigchaindb import BigchainDB - from bigchaindb.consensus import BaseConsensusRules + from bigchaindb.validation import BaseValidationRules from bigchaindb.backend.connection import Connection bigchain = BigchainDB() assert isinstance(bigchain.connection, Connection) assert bigchain.connection.host == config['database']['host'] assert bigchain.connection.port == config['database']['port'] assert bigchain.connection.dbname == config['database']['name'] - assert bigchain.consensus == BaseConsensusRules + assert bigchain.validation == BaseValidationRules def test_bigchain_class_initialization_with_parameters(): from bigchaindb import BigchainDB from bigchaindb.backend import connect - from bigchaindb.consensus import BaseConsensusRules + from bigchaindb.validation import BaseValidationRules init_db_kwargs = { 'backend': 'localmongodb', 'host': 'this_is_the_db_host', @@ -61,7 +61,7 @@ def test_bigchain_class_initialization_with_parameters(): assert bigchain.connection.host == init_db_kwargs['host'] assert bigchain.connection.port == init_db_kwargs['port'] assert bigchain.connection.dbname == init_db_kwargs['name'] - assert bigchain.consensus == BaseConsensusRules + assert bigchain.validation == BaseValidationRules def test_get_spent_issue_1271(b, alice, bob, carol): From 24ca0b32a9f3711b1bb0d0f3aefda74644252946 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Fri, 21 Sep 2018 10:51:57 +0200 Subject: [PATCH 48/77] Separate pending and effective validator updates. (#2556) * Separate pending and effective validator updates. - Pending validator updates do not prevent elections from concluding. - ValidatorElection overrides has_conclude to not conclude when there is a pending update for the matching height. - Effective validator updates deem past elections inconclusive. * Problem: Looking for election block is inefficient. Solution: Record placed elections, update the records upon election conclusion. * Clarify the conclusion order in Election.process_blocks. * Insert election records in bulk. Otherwise, one can significantly slow nodes down by posting a whole bunch of unique elections. * Change get_election to use find_one. * Calculate total votes without making extra query. * Fix the pending valset check. * Fix election test setup. --- bigchaindb/backend/localmongodb/query.py | 23 ++- bigchaindb/backend/localmongodb/schema.py | 3 +- bigchaindb/backend/query.py | 13 +- bigchaindb/core.py | 9 +- bigchaindb/elections/election.py | 134 ++++++++++------- bigchaindb/lib.py | 16 +- .../migrations/chain_migration_election.py | 4 +- .../upsert_validator/validator_election.py | 28 ++-- tests/backend/localmongodb/test_schema.py | 49 +----- tests/conftest.py | 9 +- tests/elections/test_election.py | 141 +++++++++++++++--- tests/upsert_validator/conftest.py | 6 +- .../test_upsert_validator_vote.py | 8 +- 13 files changed, 272 insertions(+), 171 deletions(-) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index 6b0793ba..d193ebb9 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -282,16 +282,26 @@ def store_validator_set(conn, validators_update): @register_query(LocalMongoDBConnection) -def store_election_results(conn, election): +def store_election(conn, election_id, height, is_concluded): return conn.run( conn.collection('elections').replace_one( - {'election_id': election['election_id']}, - election, + {'election_id': election_id, + 'height': height}, + {'election_id': election_id, + 'height': height, + 'is_concluded': is_concluded}, upsert=True, ) ) +@register_query(LocalMongoDBConnection) +def store_elections(conn, elections): + return conn.run( + conn.collection('elections').insert_many(elections) + ) + + @register_query(LocalMongoDBConnection) def get_validator_set(conn, height=None): query = {} @@ -312,13 +322,12 @@ def get_validator_set(conn, height=None): def get_election(conn, election_id): query = {'election_id': election_id} - cursor = conn.run( + return conn.run( conn.collection('elections') - .find(query, projection={'_id': False}) + .find_one(query, projection={'_id': False}, + sort=[('height', DESCENDING)]) ) - return next(cursor, None) - @register_query(LocalMongoDBConnection) def get_asset_tokens_for_public_key(conn, asset_id, public_key): diff --git a/bigchaindb/backend/localmongodb/schema.py b/bigchaindb/backend/localmongodb/schema.py index 053cfc3d..157ff39d 100644 --- a/bigchaindb/backend/localmongodb/schema.py +++ b/bigchaindb/backend/localmongodb/schema.py @@ -45,7 +45,8 @@ INDEXES = { ('commit_id', dict(name='pre_commit_id', unique=True)), ], 'elections': [ - ('election_id', dict(name='election_id', unique=True)), + ([('height', DESCENDING), ('election_id', ASCENDING)], + dict(name='election_id_height', unique=True)), ], 'validators': [ ('height', dict(name='height', unique=True)), diff --git a/bigchaindb/backend/query.py b/bigchaindb/backend/query.py index d8f60320..0b5bbade 100644 --- a/bigchaindb/backend/query.py +++ b/bigchaindb/backend/query.py @@ -352,8 +352,15 @@ def store_validator_set(conn, validator_update): @singledispatch -def store_election_results(conn, election): - """Store election results""" +def store_election(conn, election_id, height, is_concluded): + """Store election record""" + + raise NotImplementedError + + +@singledispatch +def store_elections(conn, elections): + """Store election records in bulk""" raise NotImplementedError @@ -369,7 +376,7 @@ def get_validator_set(conn, height): @singledispatch def get_election(conn, election_id): - """Return a validator set change with the specified election_id + """Return the election record """ raise NotImplementedError diff --git a/bigchaindb/core.py b/bigchaindb/core.py index 34111c47..afe646bc 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -215,10 +215,9 @@ class App(BaseApplication): else: self.block_txn_hash = block['app_hash'] - # Process all concluded elections in the current block and get any update to the validator set - update = Election.approved_elections(self.bigchaindb, - self.new_height, - self.block_transactions) + validator_update = Election.process_block(self.bigchaindb, + self.new_height, + self.block_transactions) # Store pre-commit state to recover in case there is a crash during `commit` pre_commit_state = PreCommitState(commit_id=PRE_COMMIT_ID, @@ -226,7 +225,7 @@ class App(BaseApplication): transactions=self.block_txn_ids) logger.debug('Updating PreCommitState: %s', self.new_height) self.bigchaindb.store_pre_commit_state(pre_commit_state._asdict()) - return ResponseEndBlock(validator_updates=update) + return ResponseEndBlock(validator_updates=validator_update) def commit(self): """Store the new height and along with block hash.""" diff --git a/bigchaindb/elections/election.py b/bigchaindb/elections/election.py index 40861a52..e8b3256e 100644 --- a/bigchaindb/elections/election.py +++ b/bigchaindb/elections/election.py @@ -1,7 +1,7 @@ # Copyright BigchainDB GmbH and BigchainDB contributors # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -from collections import defaultdict +from collections import OrderedDict import base58 from uuid import uuid4 @@ -22,9 +22,13 @@ from bigchaindb.common.schema import (_validate_schema, class Election(Transaction): + """Represents election transactions. + + To implement a custom election, create a class deriving from this one + with OPERATION set to the election operation, ALLOWED_OPERATIONS + set to (OPERATION,), CREATE set to OPERATION. + """ - # NOTE: this transaction class extends create so the operation inheritance is achieved - # by setting an ELECTION_TYPE and renaming CREATE = ELECTION_TYPE and ALLOWED_OPERATIONS = (ELECTION_TYPE,) OPERATION = None # Custom validation schema TX_SCHEMA_CUSTOM = None @@ -34,7 +38,6 @@ class Election(Transaction): INCONCLUSIVE = 'inconclusive' # Vote ratio to approve an election ELECTION_THRESHOLD = 2 / 3 - CHANGES_VALIDATOR_SET = True @classmethod def get_validator_change(cls, bigchain): @@ -45,8 +48,10 @@ class Election(Transaction): 'validators': } """ - height = bigchain.get_latest_block()['height'] - return bigchain.get_validator_change(height) + latest_block = bigchain.get_latest_block() + if latest_block is None: + return None + return bigchain.get_validator_change(latest_block['height']) @classmethod def get_validators(cls, bigchain, height=None): @@ -186,49 +191,52 @@ class Election(Transaction): election_pk)) return self.count_votes(election_pk, txns, dict.get) - def has_concluded(self, bigchain, current_votes=[], height=None): + def has_concluded(self, bigchain, current_votes=[]): """Check if the election can be concluded or not. - * Elections can only be concluded if the current validator set - is exactly equal to the validator set encoded in the election outputs. + * Elections can only be concluded if the validator set has not changed + since the election was initiated. * Elections can be concluded only if the current votes form a supermajority. Custom elections may override this function and introduce additional checks. """ + if self.has_validator_set_changed(bigchain): + return False election_pk = self.to_public_key(self.id) votes_committed = self.get_commited_votes(bigchain, election_pk) votes_current = self.count_votes(election_pk, current_votes) - current_validators = self.get_validators(bigchain, height) - if self.is_same_topology(current_validators, self.outputs): - total_votes = sum(current_validators.values()) - if (votes_committed < (2/3) * total_votes) and \ - (votes_committed + votes_current >= (2/3)*total_votes): - return True + total_votes = sum(output.amount for output in self.outputs) + if (votes_committed < (2/3) * total_votes) and \ + (votes_committed + votes_current >= (2/3)*total_votes): + return True + return False def get_status(self, bigchain): - concluded = self.get_election(self.id, bigchain) - if concluded: + election = self.get_election(self.id, bigchain) + if election and election['is_concluded']: return self.CONCLUDED - latest_change = self.get_validator_change(bigchain) - latest_change_height = latest_change['height'] - election_height = bigchain.get_block_containing_tx(self.id)[0] + return self.INCONCLUSIVE if self.has_validator_set_changed(bigchain) else self.ONGOING - if latest_change_height >= election_height: - return self.INCONCLUSIVE - else: - return self.ONGOING + def has_validator_set_changed(self, bigchain): + latest_change = self.get_validator_change(bigchain) + if latest_change is None: + return False + + latest_change_height = latest_change['height'] + + election = self.get_election(self.id, bigchain) + + return latest_change_height > election['height'] def get_election(self, election_id, bigchain): - result = bigchain.get_election(election_id) - return result + return bigchain.get_election(election_id) - @classmethod - def store_election_results(cls, bigchain, election, height): - bigchain.store_election_results(height, election) + def store(self, bigchain, height, is_concluded): + bigchain.store_election(self.id, height, is_concluded) def show_election(self, bigchain): data = self.asset['data'] @@ -243,45 +251,61 @@ class Election(Transaction): return response @classmethod - def approved_elections(cls, bigchain, new_height, txns): - elections = defaultdict(list) + def process_block(cls, bigchain, new_height, txns): + """Looks for election and vote transactions inside the block, records + and processes elections. + + Every election is recorded in the database. + + Every vote has a chance to conclude the corresponding election. When + an election is concluded, the corresponding database record is + marked as such. + + Elections and votes are processed in the order in which they + appear in the block. Elections are concluded in the order of + appearance of their first votes in the block. + + For every election concluded in the block, calls its `on_approval` + method. The returned value of the last `on_approval`, if any, + is a validator set update to be applied in one of the following blocks. + + `on_approval` methods are implemented by elections of particular type. + The method may contain side effects but should be idempotent. To account + for other concluded elections, if it requires so, the method should + rely on the database state. + """ + # elections placed in this block + initiated_elections = [] + # elections voted for in this block and their votes + elections = OrderedDict() for tx in txns: + if isinstance(tx, Election): + initiated_elections.append({'election_id': tx.id, + 'height': new_height, + 'is_concluded': False}) if not isinstance(tx, Vote): continue election_id = tx.asset['id'] + if election_id not in elections: + elections[election_id] = [] elections[election_id].append(tx) - validator_set_updated = False - validator_set_change = [] + if initiated_elections: + bigchain.store_elections(initiated_elections) + + validator_update = None for election_id, votes in elections.items(): election = bigchain.get_transaction(election_id) if election is None: continue - if not election.has_concluded(bigchain, votes, new_height): + if not election.has_concluded(bigchain, votes): continue - if election.makes_validator_set_change(): - if validator_set_updated: - continue - validator_set_change.append(election.get_validator_set_change(bigchain, new_height)) - validator_set_updated = True + validator_update = election.on_approval(bigchain, new_height) + election.store(bigchain, new_height, is_concluded=True) - election.on_approval(bigchain, election, new_height) - election.store_election_results(bigchain, election, new_height) + return [validator_update] if validator_update else [] - return validator_set_change - - def makes_validator_set_change(self): - return self.CHANGES_VALIDATOR_SET - - def get_validator_set_change(self, bigchain, new_height): - if self.makes_validator_set_change(): - return self.change_validator_set(bigchain, new_height) - - def change_validator_set(self, bigchain, new_height): - raise NotImplementedError - - @classmethod - def on_approval(cls, bigchain, election, new_height): + def on_approval(self, bigchain, new_height): raise NotImplementedError diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index 974a7ac5..f38b0666 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -436,8 +436,7 @@ class BigchainDB(object): return [] if result is None else result['validators'] def get_election(self, election_id): - result = backend.query.get_election(self.connection, election_id) - return result + return backend.query.get_election(self.connection, election_id) def store_pre_commit_state(self, state): return backend.query.store_pre_commit_state(self.connection, state) @@ -481,13 +480,12 @@ class BigchainDB(object): self.store_abci_chain(block['height'] + 1, new_chain_id, False) - def store_election_results(self, height, election): - """Store election results - :param height: the block height at which the election concluded - :param election: a concluded election - """ - return backend.query.store_election_results(self.connection, {'height': height, - 'election_id': election.id}) + def store_election(self, election_id, height, is_concluded): + return backend.query.store_election(self.connection, election_id, + height, is_concluded) + + def store_elections(self, elections): + return backend.query.store_elections(self.connection, elections) Block = namedtuple('Block', ('app_hash', 'height', 'transactions')) diff --git a/bigchaindb/migrations/chain_migration_election.py b/bigchaindb/migrations/chain_migration_election.py index 8e80b979..129e6684 100644 --- a/bigchaindb/migrations/chain_migration_election.py +++ b/bigchaindb/migrations/chain_migration_election.py @@ -8,7 +8,6 @@ class ChainMigrationElection(Election): CREATE = OPERATION ALLOWED_OPERATIONS = (OPERATION,) TX_SCHEMA_CUSTOM = TX_SCHEMA_CHAIN_MIGRATION_ELECTION - CHANGES_VALIDATOR_SET = False def has_concluded(self, bigchaindb, *args, **kwargs): chain = bigchaindb.get_latest_abci_chain() @@ -19,6 +18,5 @@ class ChainMigrationElection(Election): return super().has_concluded(bigchaindb, *args, **kwargs) - @classmethod - def on_approval(cls, bigchain, election, new_height): + def on_approval(self, bigchain, *args, **kwargs): bigchain.migrate_abci_chain() diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index 856b31a1..27e0ff56 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -36,18 +36,28 @@ class ValidatorElection(Election): super(ValidatorElection, cls).validate_schema(tx) validate_asset_public_key(tx['asset']['data']['public_key']) - def change_validator_set(self, bigchain, new_height): - # The new validator set comes into effect from height = new_height+1 - # (upcoming changes to Tendermint will change this to height = new_height+2) + def has_concluded(self, bigchain, *args, **kwargs): + latest_block = bigchain.get_latest_block() + if latest_block is not None: + latest_block_height = latest_block['height'] + latest_validator_change = bigchain.get_validator_change()['height'] + + # TODO change to `latest_block_height + 3` when upgrading to Tendermint 0.24.0. + if latest_validator_change == latest_block_height + 2: + # do not conclude the election if there is a change assigned already + return False + + return super().has_concluded(bigchain, *args, **kwargs) + + def on_approval(self, bigchain, new_height): validator_updates = [self.asset['data']] curr_validator_set = bigchain.get_validators(new_height) updated_validator_set = new_validator_set(curr_validator_set, validator_updates) - updated_validator_set = [v for v in updated_validator_set if v['voting_power'] > 0] - bigchain.store_validator_set(new_height+1, updated_validator_set) - return encode_validator(self.asset['data']) + updated_validator_set = [v for v in updated_validator_set + if v['voting_power'] > 0] - @classmethod - def on_approval(cls, bigchain, election, new_height): - pass + # TODO change to `new_height + 2` when upgrading to Tendermint 0.24.0. + bigchain.store_validator_set(new_height + 1, updated_validator_set) + return encode_validator(self.asset['data']) diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index a96e7b63..c4f6669a 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -3,51 +3,6 @@ # Code is Apache-2.0 and docs are CC-BY-4.0 -def test_init_creates_db_tables_and_indexes(): - import bigchaindb - from bigchaindb import backend - from bigchaindb.backend.schema import init_database - - conn = backend.connect() - dbname = bigchaindb.config['database']['name'] - - # the db is set up by the fixture so we need to remove it - conn.conn.drop_database(dbname) - - init_database() - - collection_names = conn.conn[dbname].list_collection_names() - assert set(collection_names) == { - 'transactions', 'assets', 'metadata', 'blocks', 'utxos', 'pre_commit', - 'validators', 'elections', 'abci_chains', - } - - indexes = conn.conn[dbname]['assets'].index_information().keys() - assert set(indexes) == {'_id_', 'asset_id', 'text'} - - indexes = conn.conn[dbname]['transactions'].index_information().keys() - assert set(indexes) == { - '_id_', 'transaction_id', 'asset_id', 'outputs', 'inputs'} - - indexes = conn.conn[dbname]['blocks'].index_information().keys() - assert set(indexes) == {'_id_', 'height'} - - indexes = conn.conn[dbname]['utxos'].index_information().keys() - assert set(indexes) == {'_id_', 'utxo'} - - indexes = conn.conn[dbname]['pre_commit'].index_information().keys() - assert set(indexes) == {'_id_', 'pre_commit_id'} - - indexes = conn.conn[dbname]['validators'].index_information().keys() - assert set(indexes) == {'_id_', 'height'} - - indexes = conn.conn[dbname]['abci_chains'].index_information().keys() - assert set(indexes) == {'_id_', 'height', 'chain_id'} - - indexes = conn.conn[dbname]['elections'].index_information().keys() - assert set(indexes) == {'_id_', 'election_id'} - - def test_init_database_is_graceful_if_db_exists(): import bigchaindb from bigchaindb import backend @@ -102,8 +57,8 @@ def test_create_tables(): ('output_index', 1)] indexes = conn.conn[dbname]['elections'].index_information() - assert set(indexes.keys()) == {'_id_', 'election_id'} - assert indexes['election_id']['unique'] + assert set(indexes.keys()) == {'_id_', 'election_id_height'} + assert indexes['election_id_height']['unique'] indexes = conn.conn[dbname]['pre_commit'].index_information() assert set(indexes.keys()) == {'_id_', 'pre_commit_id'} diff --git a/tests/conftest.py b/tests/conftest.py index e001465c..a96d843f 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -712,12 +712,13 @@ def valid_upsert_validator_election_2(b_mock, node_key, new_validator): def ongoing_validator_election(b, valid_upsert_validator_election, ed25519_node_keys): validators = b.get_validators(height=1) genesis_validators = {'validators': validators, - 'height': 0, - 'election_id': None} + 'height': 0} query.store_validator_set(b.connection, genesis_validators) - b.store_bulk_transactions([valid_upsert_validator_election]) - block_1 = Block(app_hash='hash_1', height=1, transactions=[valid_upsert_validator_election.id]) + query.store_election(b.connection, valid_upsert_validator_election.id, 1, + is_concluded=False) + block_1 = Block(app_hash='hash_1', height=1, + transactions=[valid_upsert_validator_election.id]) b.store_block(block_1._asdict()) return valid_upsert_validator_election diff --git a/tests/elections/test_election.py b/tests/elections/test_election.py index 0aed1816..a8c5ce98 100644 --- a/tests/elections/test_election.py +++ b/tests/elections/test_election.py @@ -9,7 +9,7 @@ from bigchaindb.upsert_validator.validator_election import ValidatorElection @pytest.mark.bdb -def test_approved_elections_concludes_all_elections(b): +def test_process_block_concludes_all_elections(b): validators = generate_validators([1] * 4) b.store_validator_set(1, [v['storage'] for v in validators]) @@ -17,28 +17,30 @@ def test_approved_elections_concludes_all_elections(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] - election, votes = generate_election(b, - ValidatorElection, - public_key, private_key, - new_validator['election']) - txs = [election] - total_votes = votes election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, {}) + txs = [election] + total_votes = votes + + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election']) txs += [election] total_votes += votes b.store_abci_chain(1, 'chain-X') + Election.process_block(b, 1, txs) b.store_block(Block(height=1, transactions=[tx.id for tx in txs], app_hash='')._asdict()) b.store_bulk_transactions(txs) - Election.approved_elections(b, 1, total_votes) + Election.process_block(b, 2, total_votes) validators = b.get_validators() assert len(validators) == 5 @@ -53,12 +55,11 @@ def test_approved_elections_concludes_all_elections(b): } for tx in txs: - election = b.get_election(tx.id) - assert election + assert b.get_election(tx.id)['is_concluded'] @pytest.mark.bdb -def test_approved_elections_applies_only_one_validator_update(b): +def test_process_block_approves_only_one_validator_update(b): validators = generate_validators([1] * 4) b.store_validator_set(1, [v['storage'] for v in validators]) @@ -82,24 +83,123 @@ def test_approved_elections_applies_only_one_validator_update(b): txs += [election] total_votes += votes + Election.process_block(b, 1, txs) b.store_block(Block(height=1, transactions=[tx.id for tx in txs], app_hash='')._asdict()) b.store_bulk_transactions(txs) - Election.approved_elections(b, 1, total_votes) + Election.process_block(b, 2, total_votes) validators = b.get_validators() assert len(validators) == 5 assert new_validator['storage'] in validators assert another_validator['storage'] not in validators - assert b.get_election(txs[0].id) - assert not b.get_election(txs[1].id) + assert b.get_election(txs[0].id)['is_concluded'] + assert not b.get_election(txs[1].id)['is_concluded'] @pytest.mark.bdb -def test_approved_elections_applies_only_one_migration(b): +def test_process_block_approves_after_pending_validator_update(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + new_validator = generate_validators([1])[0] + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election']) + txs = [election] + total_votes = votes + + another_validator = generate_validators([1])[0] + + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + another_validator['election']) + txs += [election] + total_votes += votes + + election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + + txs += [election] + total_votes += votes + + b.store_abci_chain(1, 'chain-X') + Election.process_block(b, 1, txs) + b.store_block(Block(height=1, + transactions=[tx.id for tx in txs], + app_hash='')._asdict()) + b.store_bulk_transactions(txs) + + Election.process_block(b, 2, total_votes) + + validators = b.get_validators() + assert len(validators) == 5 + assert new_validator['storage'] in validators + assert another_validator['storage'] not in validators + + assert b.get_election(txs[0].id)['is_concluded'] + assert not b.get_election(txs[1].id)['is_concluded'] + assert b.get_election(txs[2].id)['is_concluded'] + + assert b.get_latest_abci_chain() == {'height': 2, + 'chain_id': 'chain-X-migrated-at-height-1', + 'is_synced': False} + + +@pytest.mark.bdb +def test_process_block_does_not_approve_after_validator_update(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + new_validator = generate_validators([1])[0] + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election']) + txs = [election] + total_votes = votes + + b.store_block(Block(height=1, + transactions=[tx.id for tx in txs], + app_hash='')._asdict()) + Election.process_block(b, 1, txs) + b.store_bulk_transactions(txs) + + second_election, second_votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + + Election.process_block(b, 2, total_votes + [second_election]) + + b.store_block(Block(height=2, + transactions=[v.id for v in total_votes + [second_election]], + app_hash='')._asdict()) + + b.store_abci_chain(1, 'chain-X') + Election.process_block(b, 3, second_votes) + + assert not b.get_election(second_election.id)['is_concluded'] + assert b.get_latest_abci_chain() == {'height': 1, + 'chain_id': 'chain-X', + 'is_synced': True} + + +@pytest.mark.bdb +def test_process_block_applies_only_one_migration(b): validators = generate_validators([1] * 4) b.store_validator_set(1, [v['storage'] for v in validators]) @@ -121,12 +221,13 @@ def test_approved_elections_applies_only_one_migration(b): total_votes += votes b.store_abci_chain(1, 'chain-X') + Election.process_block(b, 1, txs) b.store_block(Block(height=1, transactions=[tx.id for tx in txs], app_hash='')._asdict()) b.store_bulk_transactions(txs) - Election.approved_elections(b, 1, total_votes) + Election.process_block(b, 1, total_votes) chain = b.get_latest_abci_chain() assert chain assert chain == { @@ -135,9 +236,9 @@ def test_approved_elections_applies_only_one_migration(b): 'chain_id': 'chain-X-migrated-at-height-1', } - assert b.get_election(txs[0].id) - assert not b.get_election(txs[1].id) + assert b.get_election(txs[0].id)['is_concluded'] + assert not b.get_election(txs[1].id)['is_concluded'] -def test_approved_elections_gracefully_handles_empty_block(b): - Election.approved_elections(b, 1, []) +def test_process_block_gracefully_handles_empty_block(b): + Election.process_block(b, 1, []) diff --git a/tests/upsert_validator/conftest.py b/tests/upsert_validator/conftest.py index 58b5e6dd..c39b9ed3 100644 --- a/tests/upsert_validator/conftest.py +++ b/tests/upsert_validator/conftest.py @@ -28,10 +28,8 @@ def fixed_seed_election(b_mock, node_key, new_validator): @pytest.fixture def concluded_election(b, ongoing_validator_election, ed25519_node_keys): - election_result = {'height': 2, - 'election_id': ongoing_validator_election.id} - - query.store_election_results(b.connection, election_result) + query.store_election(b.connection, ongoing_validator_election.id, + 2, is_concluded=True) return ongoing_validator_election diff --git a/tests/upsert_validator/test_upsert_validator_vote.py b/tests/upsert_validator/test_upsert_validator_vote.py index 7c49bd77..eeac3e66 100644 --- a/tests/upsert_validator/test_upsert_validator_vote.py +++ b/tests/upsert_validator/test_upsert_validator_vote.py @@ -289,10 +289,10 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): assert not election.has_concluded(b, [tx_vote0, tx_vote1]) assert election.has_concluded(b, [tx_vote0, tx_vote1, tx_vote2]) - assert Election.approved_elections(b, 4, [tx_vote0]) == [] - assert Election.approved_elections(b, 4, [tx_vote0, tx_vote1]) == [] + assert Election.process_block(b, 4, [tx_vote0]) == [] + assert Election.process_block(b, 4, [tx_vote0, tx_vote1]) == [] - update = Election.approved_elections(b, 4, [tx_vote0, tx_vote1, tx_vote2]) + update = Election.process_block(b, 4, [tx_vote0, tx_vote1, tx_vote2]) assert len(update) == 1 update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 @@ -315,7 +315,7 @@ def test_get_validator_update(b, node_keys, node_key, ed25519_node_keys): b.store_bulk_transactions([tx_vote0, tx_vote1]) - update = Election.approved_elections(b, 9, [tx_vote2]) + update = Election.process_block(b, 9, [tx_vote2]) assert len(update) == 1 update_public_key = codecs.encode(update[0].pub_key.data, 'base64').decode().rstrip('\n') assert update_public_key == public_key64 From 40ea961d4a4896c8eb0a6d2a89ba391e9f4720b3 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Tue, 25 Sep 2018 10:46:30 +0200 Subject: [PATCH 49/77] Update elections documentation (#2563) * Improve election docs. Reference relevant BEPs. * Fix typos and headings (election docs). * Enumerate supported election types (election docs). * Clarify the BEP-42 ref (election docs). --- .../source/server-reference/bigchaindb-cli.md | 94 ++++++++++--------- 1 file changed, 50 insertions(+), 44 deletions(-) diff --git a/docs/server/source/server-reference/bigchaindb-cli.md b/docs/server/source/server-reference/bigchaindb-cli.md index 82870445..3d3bcaca 100644 --- a/docs/server/source/server-reference/bigchaindb-cli.md +++ b/docs/server/source/server-reference/bigchaindb-cli.md @@ -83,107 +83,113 @@ configuration file as documented under ## bigchaindb election -Manage elections to manage the BigChainDB network. The specifics of the election process are defined in [BEP-18](https://github.com/bigchaindb/BEPs/tree/master/18), please refer it for more details. +Manage elections to govern the BigchainDB network. The specifics of the election process are defined in [BEP-18](https://github.com/bigchaindb/BEPs/tree/master/18). -Election management is broken into several subcommands. Below is the command line syntax for each, +Election management is broken into several subcommands. Below is the command line syntax for each of them. -#### election new +### election new -Create a new election which proposes a change to your BigChainDB network. +Create a new election which proposes a change to the BigchainDB network. -If the command succeeds, it will create an election and return an `election_id`. +If the command succeeds, it will post an election transaction and output `election_id`. -**NOTE**: The election proposal consists of vote tokens allocated to each current validator as per their voting power. Validators then cast their votes to approve the election by spending their vote tokens, (see the documentation on `election approve`). +The election proposal consists of vote tokens allocated to every current validator proportional to his voting power. Validators spend their votes to approve the election using the [election-approve command](#election-approve). -There are multiple types of election, which each take different parameters. Below is a short description of each type of election, as well as their command line syntax and the return value. +Every election has a type. Currently supported types are `upsert-validator` and `chain-migration`. Their transaction operations are `VALIDATOR_ELECTION` and `CHAIN_MIGRATION` accordingly. See below for how to create an election of a particular type. -###### election new upsert-validator +Note that elections can only be proposed and approved by existing validators. -Create an election to add/update/remove a validator from the validator set. +#### election new upsert-validator + +Create an election to add, update, or remove a validator. ```bash -$ bigchaindb election new upsert-validator E_PUBKEY E_POWER E_NODE_ID --private-key PATH_TO_YOUR_PRIVATE_KEY -[SUCCESS] Submitted proposal with id: +$ bigchaindb election new upsert-validator --private-key ``` -- `E_PUBKEY`: Public key of the node to be added/updated/removed. -- `E_POWER`: The new power for the `E_PUBKEY`. NOTE, if power is set to `0` then `E_PUBKEY` will be removed from the validator set when the election concludes. -- `E_NODE_ID`: Node id of `E_PUBKEY`. The node operator of `E_PUBKEY` can generate the node id via `tendermint show_node_id`. -- `--private-key`: The path to Tendermint's private key which can be generally found at `/home/user/.tendermint/config/priv_validator.json`. +- `` is the public key of the node to be added/updated/removed. The encoding and type of the key have to match those specified in `genesis.json` in the supported Tendermint version. +- `` is the new power for the validator. To remove the validator, set the power to `0`. +- `` is the node identifier from Tendermint. A node operator can learn his node identifier by executing `tendermint show_node_id`. +- `` is the path to the private key of the validator who proposes the election. Tendermint places it at `.tendermint/config/priv_validator.json`. -For example, to add a new validator, provide the public key and node id for some node not already in the validator set, along with whatever voting power you'd like them to have. To remove an existing validator, provide their public key and node id, and set `E_POWER` to `0`. Please note that the private key provided here is of the node which is generating this election i.e. - - -NOTE: A change to the validator set can only be proposed by one of the exisitng validators. - -Example usage, +Example: ```bash $ bigchaindb election new upsert-validator HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY= 1 fb7140f03a4ffad899fabbbf655b97e0321add66 --private-key /home/user/.tendermint/config/priv_validator.json [SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa ``` -A successful execution of the above command **doesn't** imply that the validator set will be immediately updated but rather it means the proposal has been succcessfully accepted by the network. Once the `election_id` has been generated the node operator should share this `election_id` with other validators in the network and urge them to approve the proposal. Note that the node operator should themsleves also approve the proposal. +A successful execution of the above command does not imply the validator set has been updated but rather the proposal has been accepted by the network. +Once `election_id` has been generated, the proposer should share it with other validators of the network (e.g. via email) and ask them to approve the proposal. +Note that election proposers do not automatically approve elections by proposing them. -###### election new migration +For more details about how validator set changes work, refer to [BEP-21](https://github.com/bigchaindb/BEPs/tree/master/21). -Create an election to halt block production, to allow for a version change across breaking changes. +#### election new chain-migration + +Create an election to halt block production, to coordinate on making a Tendermint upgrade with a backwards-incompatible chain. ```bash -$ bigchaindb election new migration --private-key PATH_TO_YOUR_PRIVATE_KEY -[SUCCESS] Submitted proposal with id: +$ bigchaindb election new chain-migration --private-key ``` -- `--private-key`: The path to Tendermint's private key which can be generally found at `/home/user/.tendermint/config/priv_validator.json`. +- `` is the path to the private key of the validator who proposes the election. Tendermint places it at `.tendermint/config/priv_validator.json`. -Example usage, + +Example: ```bash $ bigchaindb election new migration --private-key /home/user/.tendermint/config/priv_validator.json [SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa ``` -**NOTE** `migration` elections will halt block production at whichever blockheight they are approved. Once the election is concluded, the validators will need to restart their systems with a new `chain_id` to resume normal operations. +Concluded chain migration elections halt block production at whichever block height they are approved. +Afterwards, validators are supposed to upgrade Tendermint, set new `chain_id`, `app_hash`, and `validators` (to learn these values, use the [election show](#election-show) command) in `genesis.json`, make and save a MongoDB dump, and restart the system. -#### election approve +For more details about how chain migrations work, refer to [Type 3 scenarios in BEP-42](https://github.com/bigchaindb/BEPs/tree/master/42). -Approve an election by voting for it. The proposal generated by executing `bigchaindb election new ...` can be approved by the validators using this command. The validator who is approving the proposal will spend all their votes i.e. if the validator has a network power of `10` then they will cast `10` votes for the proposal. +### election approve + +Approve an election by voting for it. The command places a `VOTE` transaction, spending all of the validator's vote tokens to the election address. -Below is the command line syntax and the return value, ```bash -$ bigchaindb election approve --private-key PATH_TO_YOUR_PRIVATE_KEY -[SUCCESS] Your vote has been submitted +$ bigchaindb election approve --private-key ``` -- `election_id` is the transaction id of the election the approval should be given for. -- `--private-key` should be the path to Tendermint's private key which can be generally found at `/home/user/.tendermint/config/priv_validator.json`. +- `election-id` is the election identifier the approval is given for. +- `` is the path to the private key of the validator who votes for the election. Tendermint places it at `.tendermint/config/priv_validator.json`. - Example usage, +Example: ```bash $ bigchaindb election approve 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa --private-key /home/user/.tendermint/config/priv_validator.json [SUCCESS] Your vote has been submitted ``` -If the command succeeds a message will be returned stating that the vote was submitted successfully. Once a proposal has been approved by sufficent validators (more than `2/3` of the total voting power) then the proposed change is applied to the network. For example, consider a network wherein the total power is `90` then the proposed changed is applied only after `60` (`2/3 * 90`) have been received. +Once a proposal has been approved by the sufficient amount of validators (contributing more than `2/3` of the total voting power), the proposed change is applied to the network. -#### election show +### election show -Retrieves information about an election initiated by `election new`. +Retrieves the information about elections. -Below is the command line syntax and the return value, ```bash -$ bigchaindb election show ELECTION_ID - +$ bigchaindb election show + status= ``` -The election data is the same set of arguments used in the `election new` command that originally triggered the election. `status` takes three possible values, `ongoing`, if the election has not yet reached a 2/3 majority, `concluded`, if the election reached the 2/3 majority needed to pass, or `inconclusive`, if the validator set changed while the election was in process, rendering it undecidable. +`status` has three possible values: + +- `ongoing`, if the election can be concluded but has not yet collected enough votes, +- `concluded`, if the election has been concluded, +- `inconclusive`, if the validator set changed while the election was in process, rendering it undecidable. + +After a chain migration is concluded, the `show` command also outputs `chain_id`, `app_hash`, and `validators` for `genesis.json` of the new chain. ## bigchaindb tendermint-version From cb74cb43d2ea287f7ee6821b65940b29775dc2a3 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Tue, 25 Sep 2018 11:38:29 +0200 Subject: [PATCH 50/77] Extend chain migration show command (#2558) * Extend `election show` with migration data. * Test chain-migration show. --- bigchaindb/commands/bigchaindb.py | 2 +- bigchaindb/commands/election_types.py | 2 + .../migrations/chain_migration_election.py | 23 ++++++ tests/commands/test_commands.py | 82 +++++++++++++++++++ 4 files changed, 108 insertions(+), 1 deletion(-) diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index 58db7ac0..ddc7a5f5 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -118,7 +118,6 @@ def run_election_new(args, bigchain): def create_new_election(sk, bigchain, election_class, data): - try: key = load_node_key(sk) voters = election_class.recipients(bigchain) @@ -362,6 +361,7 @@ def create_parser(): help='The election_id of the election.') approve_election_parser.add_argument('--private-key', dest='sk', + required=True, help='Path to the private key of the election initiator.') show_election_parser = election_subparser.add_parser('show', diff --git a/bigchaindb/commands/election_types.py b/bigchaindb/commands/election_types.py index 4745b85d..cfa58b9e 100644 --- a/bigchaindb/commands/election_types.py +++ b/bigchaindb/commands/election_types.py @@ -13,6 +13,7 @@ elections = { }, '--private-key': { 'dest': 'sk', + 'required': True, 'help': 'Path to the private key of the election initiator.' } } @@ -22,6 +23,7 @@ elections = { 'args': { '--private-key': { 'dest': 'sk', + 'required': True, 'help': 'Path to the private key of the election initiator.' } } diff --git a/bigchaindb/migrations/chain_migration_election.py b/bigchaindb/migrations/chain_migration_election.py index 129e6684..52f8b020 100644 --- a/bigchaindb/migrations/chain_migration_election.py +++ b/bigchaindb/migrations/chain_migration_election.py @@ -1,3 +1,5 @@ +import json + from bigchaindb.common.schema import TX_SCHEMA_CHAIN_MIGRATION_ELECTION from bigchaindb.elections.election import Election @@ -20,3 +22,24 @@ class ChainMigrationElection(Election): def on_approval(self, bigchain, *args, **kwargs): bigchain.migrate_abci_chain() + + def show_election(self, bigchain): + output = super().show_election(bigchain) + chain = bigchain.get_latest_abci_chain() + if chain is None or chain['is_synced']: + return output + + output += f'\nchain_id={chain["chain_id"]}' + block = bigchain.get_latest_block() + output += f'\napp_hash={block["app_hash"]}' + validators = [ + { + 'pub_key': { + 'type': 'tendermint/PubKeyEd25519', + 'value': k, + }, + 'power': v, + } for k, v in self.get_validators(bigchain).items() + ] + output += f'\nvalidators={json.dumps(validators, indent=4)}' + return output diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index f9b14f89..1512d7eb 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -11,6 +11,12 @@ from argparse import Namespace import pytest from bigchaindb import ValidatorElection +from bigchaindb.commands.bigchaindb import run_election_show +from bigchaindb.elections.election import Election +from bigchaindb.lib import Block +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection + +from tests.utils import generate_election, generate_validators def test_make_sure_we_dont_remove_any_command(): @@ -509,6 +515,82 @@ def test_election_approve_called_with_bad_key(caplog, b, bad_validator_path, new 'the eligible voters in this election.' +@pytest.mark.bdb +def test_chain_migration_election_show_shows_inconclusive(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + + election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + + assert not run_election_show(Namespace(election_id=election.id), b) + + Election.process_block(b, 1, [election]) + b.store_bulk_transactions([election]) + + assert run_election_show(Namespace(election_id=election.id), b) == \ + 'status=ongoing' + + b.store_block(Block(height=1, transactions=[], app_hash='')._asdict()) + b.store_validator_set(2, [v['storage'] for v in validators]) + + assert run_election_show(Namespace(election_id=election.id), b) == \ + 'status=ongoing' + + b.store_block(Block(height=2, transactions=[], app_hash='')._asdict()) + # TODO insert yet another block here when upgrading to Tendermint 0.22.4. + + assert run_election_show(Namespace(election_id=election.id), b) == \ + 'status=inconclusive' + + +@pytest.mark.bdb +def test_chain_migration_election_show_shows_concluded(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + + election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}) + + assert not run_election_show(Namespace(election_id=election.id), b) + + b.store_bulk_transactions([election]) + Election.process_block(b, 1, [election]) + + assert run_election_show(Namespace(election_id=election.id), b) == \ + 'status=ongoing' + + b.store_abci_chain(1, 'chain-X') + b.store_block(Block(height=1, + transactions=[v.id for v in votes], + app_hash='last_app_hash')._asdict()) + Election.process_block(b, 2, votes) + + assert run_election_show(Namespace(election_id=election.id), b) == \ + f'''status=concluded +chain_id=chain-X-migrated-at-height-1 +app_hash=last_app_hash +validators=[{''.join([f""" + {{ + "pub_key": {{ + "type": "tendermint/PubKeyEd25519", + "value": "{v['public_key']}" + }}, + "power": {v['storage']['voting_power']} + }}{',' if i + 1 != len(validators) else ''}""" for i, v in enumerate(validators)])} +]''' + + def test_bigchain_tendermint_version(capsys): from bigchaindb.commands.bigchaindb import run_tendermint_version From bedb1945a973914dbb377ee5d7f852f4e0a65034 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Tue, 25 Sep 2018 17:33:59 +0200 Subject: [PATCH 51/77] Added docs & link re/ adding/removing validators (#2570) --- .../server/source/simple-deployment-template/network-setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index a89c3ac4..308b9854 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -352,9 +352,9 @@ $ pgrep bigchaindb | xargs kill -9 If you started BigchainDB in the foreground, a `Ctrl + C` or `Ctrl + Z` would shut down BigchainDB. -## Member: Dynamically Add a New Member to the Network +## Member: Dynamically Add or Remove Validators -TBD. +One member can make a proposal to call an election to add a validator, remove a validator, or change the voting power of a validator. They then share the election/proposal ID with all the other members. Once more than 2/3 of the voting power votes yes, the proposed change comes into effect. The commands to create a new election/proposal, to approve an election/proposal, and to get the current status of an election/proposal can be found in the documentation about the [bigchaindb election](../server-reference/bigchaindb-cli.html#bigchaindb-election) subcommands. ## Log rotation From 78dafce146c6e0b1356a848d67738c6c5179aa35 Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Wed, 26 Sep 2018 10:17:39 +0200 Subject: [PATCH 52/77] Problem: Outputs API responds with incorrect outputs (#2567). * Problem: Outputs API doesn't respond with correct unspent/spent outputs Solution: Fix fastquery such that embedded document is queried properly * Problem: key order agnostic queries not implemented Solution: get_spent queries embedded documents which respect key order. This is not expected by the application hence the query should be altered to query any kind of key order * Problem: Mongo query for get_spent too complicated Solution: Simplify query using $elemMatch * Problem: No test for checking mixed spent outputs Solution: Add test for `get_spending_transactions` to check that correct matching is done when querying documents with multiple inputs * Problem: tranasction ids not assert when getting spent outputs Solution: assert tranasction ids --- bigchaindb/backend/localmongodb/query.py | 26 ++++++++----- tests/backend/localmongodb/test_queries.py | 43 +++++++++++++++++++++ tests/tendermint/test_fastquery.py | 44 ++++++++++++++++++++++ tests/tendermint/test_lib.py | 33 ++++++++++++++++ tests/test_core.py | 12 +++--- 5 files changed, 143 insertions(+), 15 deletions(-) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index d193ebb9..effbdacb 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -91,8 +91,11 @@ def get_assets(conn, asset_ids): @register_query(LocalMongoDBConnection) def get_spent(conn, transaction_id, output): - query = {'inputs.fulfills': {'transaction_id': transaction_id, - 'output_index': output}} + query = {'inputs': + {'$elemMatch': + {'$and': [{'fulfills.transaction_id': transaction_id}, + {'fulfills.output_index': output}]}}} + return conn.run( conn.collection('transactions') .find(query, {'_id': 0})) @@ -180,15 +183,18 @@ def get_owned_ids(conn, owner): @register_query(LocalMongoDBConnection) def get_spending_transactions(conn, inputs): + transaction_ids = [i['transaction_id'] for i in inputs] + output_indexes = [i['output_index'] for i in inputs] + query = {'inputs': + {'$elemMatch': + {'$and': + [ + {'fulfills.transaction_id': {'$in': transaction_ids}}, + {'fulfills.output_index': {'$in': output_indexes}} + ]}}} + cursor = conn.run( - conn.collection('transactions').aggregate([ - {'$match': { - 'inputs.fulfills': { - '$in': inputs, - }, - }}, - {'$project': {'_id': False}} - ])) + conn.collection('transactions').find(query, {'_id': False})) return cursor diff --git a/tests/backend/localmongodb/test_queries.py b/tests/backend/localmongodb/test_queries.py index 6a7ae2c4..d2fa58c1 100644 --- a/tests/backend/localmongodb/test_queries.py +++ b/tests/backend/localmongodb/test_queries.py @@ -234,6 +234,49 @@ def test_get_spending_transactions(user_pk, user_sk): assert txns == [tx2.to_dict(), tx4.to_dict()] +def test_get_spending_transactions_multiple_inputs(): + from bigchaindb.backend import connect, query + from bigchaindb.models import Transaction + from bigchaindb.common.crypto import generate_key_pair + conn = connect() + (alice_sk, alice_pk) = generate_key_pair() + (bob_sk, bob_pk) = generate_key_pair() + (carol_sk, carol_pk) = generate_key_pair() + + out = [([alice_pk], 9)] + tx1 = Transaction.create([alice_pk], out).sign([alice_sk]) + + inputs1 = tx1.to_inputs() + tx2 = Transaction.transfer([inputs1[0]], + [([alice_pk], 6), ([bob_pk], 3)], + tx1.id).sign([alice_sk]) + + inputs2 = tx2.to_inputs() + tx3 = Transaction.transfer([inputs2[0]], + [([bob_pk], 3), ([carol_pk], 3)], + tx1.id).sign([alice_sk]) + + inputs3 = tx3.to_inputs() + tx4 = Transaction.transfer([inputs2[1], inputs3[0]], + [([carol_pk], 6)], + tx1.id).sign([bob_sk]) + + txns = [deepcopy(tx.to_dict()) for tx in [tx1, tx2, tx3, tx4]] + conn.db.transactions.insert_many(txns) + + links = [ + ({'transaction_id': tx2.id, 'output_index': 0}, 1, [tx3.id]), + ({'transaction_id': tx2.id, 'output_index': 1}, 1, [tx4.id]), + ({'transaction_id': tx3.id, 'output_index': 0}, 1, [tx4.id]), + ({'transaction_id': tx3.id, 'output_index': 1}, 0, None), + ] + for l, num, match in links: + txns = list(query.get_spending_transactions(conn, [l])) + assert len(txns) == num + if len(txns): + assert [tx['id'] for tx in txns] == match + + def test_store_block(): from bigchaindb.backend import connect, query from bigchaindb.lib import Block diff --git a/tests/tendermint/test_fastquery.py b/tests/tendermint/test_fastquery.py index bf93850f..b83722e0 100644 --- a/tests/tendermint/test_fastquery.py +++ b/tests/tendermint/test_fastquery.py @@ -73,3 +73,47 @@ def test_filter_unspent_outputs(b, user_pk, user_sk): assert set(sp for sp in spents) == { inputs[0].fulfills, } + + +def test_outputs_query_key_order(b, user_pk, user_sk, user2_pk, user2_sk): + from bigchaindb import backend + from bigchaindb.backend import connect + + tx1 = Transaction.create([user_pk], + [([user_pk], 3), ([user_pk], 2), ([user_pk], 1)])\ + .sign([user_sk]) + b.store_bulk_transactions([tx1]) + + inputs = tx1.to_inputs() + tx2 = Transaction.transfer([inputs[1]], [([user2_pk], 2)], tx1.id).sign([user_sk]) + assert tx2.validate(b) + + tx2_dict = tx2.to_dict() + fulfills = tx2_dict['inputs'][0]['fulfills'] + tx2_dict['inputs'][0]['fulfills'] = {'transaction_id': fulfills['transaction_id'], + 'output_index': fulfills['output_index']} + backend.query.store_transactions(b.connection, [tx2_dict]) + + outputs = b.get_outputs_filtered(user_pk, spent=False) + assert len(outputs) == 2 + + outputs = b.get_outputs_filtered(user2_pk, spent=False) + assert len(outputs) == 1 + + # clean the transaction, metdata and asset collection + conn = connect() + conn.run(conn.collection('transactions').delete_many({})) + conn.run(conn.collection('metadata').delete_many({})) + conn.run(conn.collection('assets').delete_many({})) + + b.store_bulk_transactions([tx1]) + tx2_dict = tx2.to_dict() + tx2_dict['inputs'][0]['fulfills'] = {'output_index': fulfills['output_index'], + 'transaction_id': fulfills['transaction_id']} + + backend.query.store_transactions(b.connection, [tx2_dict]) + outputs = b.get_outputs_filtered(user_pk, spent=False) + assert len(outputs) == 2 + + outputs = b.get_outputs_filtered(user2_pk, spent=False) + assert len(outputs) == 1 diff --git a/tests/tendermint/test_lib.py b/tests/tendermint/test_lib.py index 701ab3ca..f85a3fb9 100644 --- a/tests/tendermint/test_lib.py +++ b/tests/tendermint/test_lib.py @@ -471,3 +471,36 @@ def test_migrate_abci_chain_generates_new_chains(b, chain, block_height, b.migrate_abci_chain() latest_chain = b.get_latest_abci_chain() assert latest_chain == expected + + +@pytest.mark.bdb +def test_get_spent_key_order(b, user_pk, user_sk, user2_pk, user2_sk): + from bigchaindb import backend + from bigchaindb.models import Transaction + from bigchaindb.common.crypto import generate_key_pair + from bigchaindb.common.exceptions import DoubleSpend + + alice = generate_key_pair() + bob = generate_key_pair() + + tx1 = Transaction.create([user_pk], + [([alice.public_key], 3), ([user_pk], 2)], + asset=None)\ + .sign([user_sk]) + b.store_bulk_transactions([tx1]) + + inputs = tx1.to_inputs() + tx2 = Transaction.transfer([inputs[1]], [([user2_pk], 2)], tx1.id).sign([user_sk]) + assert tx2.validate(b) + + tx2_dict = tx2.to_dict() + fulfills = tx2_dict['inputs'][0]['fulfills'] + tx2_dict['inputs'][0]['fulfills'] = {'output_index': fulfills['output_index'], + 'transaction_id': fulfills['transaction_id']} + + backend.query.store_transactions(b.connection, [tx2_dict]) + + tx3 = Transaction.transfer([inputs[1]], [([bob.public_key], 2)], tx1.id).sign([user_sk]) + + with pytest.raises(DoubleSpend): + tx3.validate(b) diff --git a/tests/test_core.py b/tests/test_core.py index 85fc7ec3..423f5c57 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -64,6 +64,7 @@ def test_bigchain_class_initialization_with_parameters(): assert bigchain.validation == BaseValidationRules +@pytest.mark.bdb def test_get_spent_issue_1271(b, alice, bob, carol): from bigchaindb.models import Transaction @@ -71,7 +72,7 @@ def test_get_spent_issue_1271(b, alice, bob, carol): [carol.public_key], [([carol.public_key], 8)], ).sign([carol.private_key]) - assert b.validate_transaction(tx_1) + assert tx_1.validate(b) b.store_bulk_transactions([tx_1]) tx_2 = Transaction.transfer( @@ -81,7 +82,7 @@ def test_get_spent_issue_1271(b, alice, bob, carol): ([carol.public_key], 4)], asset_id=tx_1.id, ).sign([carol.private_key]) - assert b.validate_transaction(tx_2) + assert tx_2.validate(b) b.store_bulk_transactions([tx_2]) tx_3 = Transaction.transfer( @@ -90,7 +91,7 @@ def test_get_spent_issue_1271(b, alice, bob, carol): ([carol.public_key], 3)], asset_id=tx_1.id, ).sign([carol.private_key]) - assert b.validate_transaction(tx_3) + assert tx_3.validate(b) b.store_bulk_transactions([tx_3]) tx_4 = Transaction.transfer( @@ -98,7 +99,7 @@ def test_get_spent_issue_1271(b, alice, bob, carol): [([bob.public_key], 3)], asset_id=tx_1.id, ).sign([alice.private_key]) - assert b.validate_transaction(tx_4) + assert tx_4.validate(b) b.store_bulk_transactions([tx_4]) tx_5 = Transaction.transfer( @@ -106,7 +107,8 @@ def test_get_spent_issue_1271(b, alice, bob, carol): [([alice.public_key], 2)], asset_id=tx_1.id, ).sign([bob.private_key]) - assert b.validate_transaction(tx_5) + assert tx_5.validate(b) + b.store_bulk_transactions([tx_5]) assert b.get_spent(tx_2.id, 0) == tx_5 From 72d7986a582ee62cafb129a9380010520465fb33 Mon Sep 17 00:00:00 2001 From: Vanshdeep Singh Date: Wed, 26 Sep 2018 16:33:23 +0200 Subject: [PATCH 53/77] Problem: 'packaging' version not pinned (#2571) Solution: Pin the version --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 3bb2151c..14c092dc 100644 --- a/setup.py +++ b/setup.py @@ -88,7 +88,7 @@ install_requires = [ 'aiohttp~=3.0', 'bigchaindb-abci==0.5.1', 'setproctitle~=1.1.0', - 'packaging~=17.0', + 'packaging~=18.0', ] setup( From 4f01dd4997e2f67f36bef9231b3d9cced80fee30 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 27 Sep 2018 13:30:58 +0200 Subject: [PATCH 54/77] See what happens if packaging~=17.0 in setup.py AND docs/*/requirements.txt (#2572) --- docs/contributing/requirements.txt | 1 + docs/root/requirements.txt | 1 + docs/server/requirements.txt | 1 + setup.py | 2 +- 4 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/contributing/requirements.txt b/docs/contributing/requirements.txt index d2406438..930608d1 100644 --- a/docs/contributing/requirements.txt +++ b/docs/contributing/requirements.txt @@ -2,3 +2,4 @@ Sphinx~=1.0 recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 wget +packaging~=17.0 diff --git a/docs/root/requirements.txt b/docs/root/requirements.txt index b092a32c..ec7c164a 100644 --- a/docs/root/requirements.txt +++ b/docs/root/requirements.txt @@ -3,3 +3,4 @@ recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 sphinxcontrib-napoleon>=0.4.4 sphinxcontrib-httpdomain>=1.5.0 +packaging~=17.0 diff --git a/docs/server/requirements.txt b/docs/server/requirements.txt index c9fa59e6..7871571a 100644 --- a/docs/server/requirements.txt +++ b/docs/server/requirements.txt @@ -5,3 +5,4 @@ sphinxcontrib-napoleon>=0.4.4 sphinxcontrib-httpdomain>=1.5.0 pyyaml~=3.12 aafigure>=0.6 +packaging~=17.0 \ No newline at end of file diff --git a/setup.py b/setup.py index 14c092dc..3bb2151c 100644 --- a/setup.py +++ b/setup.py @@ -88,7 +88,7 @@ install_requires = [ 'aiohttp~=3.0', 'bigchaindb-abci==0.5.1', 'setproctitle~=1.1.0', - 'packaging~=18.0', + 'packaging~=17.0', ] setup( From 54ac7adaa9e90460ed690828e0f846829667d5a0 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Thu, 27 Sep 2018 15:31:30 +0200 Subject: [PATCH 55/77] Changed packaging~=17.0 to 18.0 in four places (#2573) --- docs/contributing/requirements.txt | 2 +- docs/root/requirements.txt | 2 +- docs/server/requirements.txt | 2 +- setup.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/contributing/requirements.txt b/docs/contributing/requirements.txt index 930608d1..de6c0c1d 100644 --- a/docs/contributing/requirements.txt +++ b/docs/contributing/requirements.txt @@ -2,4 +2,4 @@ Sphinx~=1.0 recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 wget -packaging~=17.0 +packaging~=18.0 diff --git a/docs/root/requirements.txt b/docs/root/requirements.txt index ec7c164a..4ee8907d 100644 --- a/docs/root/requirements.txt +++ b/docs/root/requirements.txt @@ -3,4 +3,4 @@ recommonmark>=0.4.0 sphinx-rtd-theme>=0.1.9 sphinxcontrib-napoleon>=0.4.4 sphinxcontrib-httpdomain>=1.5.0 -packaging~=17.0 +packaging~=18.0 diff --git a/docs/server/requirements.txt b/docs/server/requirements.txt index 7871571a..467bc009 100644 --- a/docs/server/requirements.txt +++ b/docs/server/requirements.txt @@ -5,4 +5,4 @@ sphinxcontrib-napoleon>=0.4.4 sphinxcontrib-httpdomain>=1.5.0 pyyaml~=3.12 aafigure>=0.6 -packaging~=17.0 \ No newline at end of file +packaging~=18.0 diff --git a/setup.py b/setup.py index 3bb2151c..14c092dc 100644 --- a/setup.py +++ b/setup.py @@ -88,7 +88,7 @@ install_requires = [ 'aiohttp~=3.0', 'bigchaindb-abci==0.5.1', 'setproctitle~=1.1.0', - 'packaging~=17.0', + 'packaging~=18.0', ] setup( From 009cbe4e116fc37694efcdd6cd1cc41d768c65eb Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 28 Sep 2018 09:19:20 +0200 Subject: [PATCH 56/77] Copy-edited the log rotation page (docs) (#2546) * Copy-edited the log rotation page (docs) * Fix and add some hyperlinks re/ logging. * Add back the section/link to log-rotation.html * Correct the location of BigchainDB logs. Omit mention of bigchaindb-benchmark.log * Revive notes re BigchainDB Server log rotation --- docs/server/source/appendices/log-rotation.md | 54 ++++++++++--------- .../source/server-reference/configuration.md | 6 +-- .../network-setup.md | 5 +- .../troubleshooting.md | 1 + 4 files changed, 34 insertions(+), 32 deletions(-) diff --git a/docs/server/source/appendices/log-rotation.md b/docs/server/source/appendices/log-rotation.md index 95d52115..9161d5de 100644 --- a/docs/server/source/appendices/log-rotation.md +++ b/docs/server/source/appendices/log-rotation.md @@ -4,44 +4,46 @@ SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) Code is Apache-2.0 and docs are CC-BY-4.0 ---> -# Log rotation for a BigchainDB node +# Logging and Log Rotation -Each BigchainDB node comprises of 3 main services: - - BigchainDB server - - Tendermint - - MongoDB +Each BigchainDB node runs: -To run a BigchainDB test network/dev node, that is expected to run for relatively longer periods -of time, we need to consider the log rotation of these services i.e. we do not want the logs taking -up large amounts of storage and making the node unresponsive or get into bad state. +- MongoDB +- BigchainDB Server +- Tendermint -## Log rotation for MongoDB +When running a BigchainDB node for long periods +of time, we need to consider doing log rotation, i.e. we do not want the logs taking +up large amounts of storage and making the node unresponsive or getting it into a bad state. -Currently, we leave the log rotation of MongoDB to the BigchainDB administrator. For more notes on MongoDB log rotation -please refer to [MongoDB docs](https://docs.mongodb.com/v3.6/tutorial/rotate-log-files/). +## MongoDB Logging and Log Rotation -## Log rotation for BigchainDB +See the MongoDB docs about +[logging](https://docs.mongodb.com/v3.6/administration/monitoring/#monitoring-standard-loggging) +and [log rotation](https://docs.mongodb.com/v3.6/tutorial/rotate-log-files/). -Log rotation is baked into BigchainDB server using the `logging` module. BigchainDB server logs information into the following files: - - `bigchaindb.log` - - `bigchaindb-errors.log` +## BigchainDB Server Logging and Log Rotation -These log files are created by default in the directory from where you run `bigchaindb start`, if you are using `monit`, from -[How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide, the default directory is: `$HOME/.bigchaindb-monit/logs` +BigchainDB Server writes its logs to two files: normal logs and error logs. The names of those files, and their locations, are set as part of the BigchainDB configuration settings. The default names and locations are: -The logs for BigchainDB server are rotated when any of the above mentioned file exceeds `209715200 bytes (i.e. approximately 209 MB).`. +- `~/bigchaindb.log` +- `~/bigchaindb-errors.log` +Log rotation is baked into BigchainDB Server using Python's `logging` module. The logs for BigchainDB Server are rotated when any of the above mentioned files exceeds 209715200 bytes (i.e. approximately 209 MB). -## Log rotation for Tendermint +For more information, see the docs about [the BigchainDB Server configuration settings related to logging](../server-reference/configuration.html#log). -In order to set up log rotation of Tendermint, you will need to use the [Monit]( https://www.mmonit.com/monit) scripts provided by us. Covered in the [How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.md) guide. +## Tendermint Logging and Log Rotation -```bash -$ monit -d 1 -``` +Tendermint writes its logs to the files: -Monit monitors both Tendermint and BigchainDB processes as well as the Tendermint log files, `tendermint.out.log` and `tendermint.err.log`. Default location for these log files is: -`$HOME/.bigchaindb-monit/logs`. +- `tendermint.out.log` +- `tendermint.err.log` -Tendermint logs are rotated if any of the above mentioned log files exceeds `200 MB` in size. +If you started BigchainDB Server and Tendermint using Monit, as suggested by our guide on +[How to Set Up a BigchainDB Network](../simple-deployment-template/network-setup.html), +then the logs will be written to `$HOME/.bigchaindb-monit/logs/`. +Moreover, if you started BigchainDB Server and Tendermint using Monit, +then Monit monitors the Tendermint log files. +Tendermint logs are rotated if any of the above mentioned log files exceeds 200 MB. diff --git a/docs/server/source/server-reference/configuration.md b/docs/server/source/server-reference/configuration.md index 66d9f23f..d059b0a6 100644 --- a/docs/server/source/server-reference/configuration.md +++ b/docs/server/source/server-reference/configuration.md @@ -269,11 +269,11 @@ The full path to the file where logs should be written. The user running `bigchaindb` must have write access to the specified path. -**Log rotation:** Log files have a size limit of 200 MB +**Log rotation:** Log files have a size limit of about 200 MB and will be rotated up to five times. -For example, if we `log.file` is set to `"~/bigchain.log"`, then +For example, if `log.file` is set to `"~/bigchain.log"`, then logs would always be written to `bigchain.log`. Each time the file -`bigchain.log` reaches 200 MB it would be closed and renamed +`bigchain.log` reaches 200 MB it will be closed and renamed `bigchain.log.1`. If `bigchain.log.1` and `bigchain.log.2` already exist they would be renamed `bigchain.log.2` and `bigchain.log.3`. This pattern would be applied up to `bigchain.log.5` after which `bigchain.log.5` would be diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index 308b9854..7dd2c937 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -356,10 +356,9 @@ If you started BigchainDB in the foreground, a `Ctrl + C` or `Ctrl + Z` would sh One member can make a proposal to call an election to add a validator, remove a validator, or change the voting power of a validator. They then share the election/proposal ID with all the other members. Once more than 2/3 of the voting power votes yes, the proposed change comes into effect. The commands to create a new election/proposal, to approve an election/proposal, and to get the current status of an election/proposal can be found in the documentation about the [bigchaindb election](../server-reference/bigchaindb-cli.html#bigchaindb-election) subcommands. -## Log rotation - -Please go over the [document describing log rotation of a BigchainDB node](../appendices/log-rotation.md). +## Logging and Log Rotation +See the page in the Appendices about [logging and log rotation](../appendices/log-rotation.html). [bdb:software]: https://github.com/bigchaindb/bigchaindb/ [bdb:pypi]: https://pypi.org/project/BigchainDB/#history diff --git a/docs/server/source/simple-deployment-template/troubleshooting.md b/docs/server/source/simple-deployment-template/troubleshooting.md index b2d95722..3f8187e9 100644 --- a/docs/server/source/simple-deployment-template/troubleshooting.md +++ b/docs/server/source/simple-deployment-template/troubleshooting.md @@ -3,6 +3,7 @@ ## General Tips - Check the BigchainDB, Tendermint and MongoDB logs. + For help with that, see the page about [Logging and Log Rotation](../appendices/log-rotation.html). - Try Googling the error message. ## Resolving Tendermint Connectivity Problems From f4d6ccbacf2600ce3b997359033af47cada39924 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 28 Sep 2018 10:45:03 +0200 Subject: [PATCH 57/77] Added server.bind note re connecting via IPv6 & edited reverse proxy instructions. (#2575) --- .../server/source/server-reference/configuration.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/server/source/server-reference/configuration.md b/docs/server/source/server-reference/configuration.md index d059b0a6..57eb46ec 100644 --- a/docs/server/source/server-reference/configuration.md +++ b/docs/server/source/server-reference/configuration.md @@ -110,7 +110,18 @@ If (no environment variables were set and there's no local config file), or you `server.bind`, `server.loglevel` and `server.workers` are settings for the [Gunicorn HTTP server](http://gunicorn.org/), which is used to serve the [HTTP client-server API](../http-client-server-api.html). -`server.bind` is where to bind the Gunicorn HTTP server socket. It's a string. It can be any valid value for [Gunicorn's bind setting](http://docs.gunicorn.org/en/stable/settings.html#bind). If you want to allow IPv4 connections from anyone, on port 9984, use `0.0.0.0:9984`. In a production setting, we recommend you use Gunicorn behind a reverse proxy server. If Gunicorn and the reverse proxy are running on the same machine, then use `localhost:PORT` where PORT is _not_ 9984 (because the reverse proxy needs to listen on port 9984). Maybe use PORT=9983 in that case because we know 9983 isn't used. If Gunicorn and the reverse proxy are running on different machines, then use `A.B.C.D:9984` where A.B.C.D is the IP address of the reverse proxy. There's [more information about deploying behind a reverse proxy in the Gunicorn documentation](http://docs.gunicorn.org/en/stable/deploy.html). (They call it a proxy.) +`server.bind` is where to bind the Gunicorn HTTP server socket. It's a string. It can be any valid value for [Gunicorn's bind setting](http://docs.gunicorn.org/en/stable/settings.html#bind). For example: + +* If you want to allow IPv4 connections from anyone, on port 9984, use `0.0.0.0:9984` +* If you want to allow IPv6 connections from anyone, on port 9984, use `[::]:9984` + +In a production setting, we recommend you use Gunicorn behind a reverse proxy server such as NGINX. If Gunicorn and the reverse proxy are running on the same machine, then you can use `localhost:9984` (the default value), meaning Gunicorn will talk to the reverse proxy on port 9984. The reverse proxy could then be bound to port 80 (for HTTP) or port 443 (for HTTPS), so that external clients would connect using that port. For example: + +[External clients]---(port 443)---[NGINX]---(port 9984)---[Gunicorn / BigchainDB Server] + +If Gunicorn and the reverse proxy are running on different machines, then `server.bind` should be `hostname:9984`, where hostname is the IP address or [FQDN](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) of the reverse proxy. + +There's [more information about deploying behind a reverse proxy in the Gunicorn documentation](http://docs.gunicorn.org/en/stable/deploy.html). (They call it a proxy.) `server.loglevel` sets the log level of Gunicorn's Error log outputs. See [Gunicorn's documentation](http://docs.gunicorn.org/en/latest/settings.html#loglevel) From 0c4d6ec906539b78894982f51f029327c1c12691 Mon Sep 17 00:00:00 2001 From: Lev Berman Date: Fri, 28 Sep 2018 12:32:45 +0200 Subject: [PATCH 58/77] Implement crash recovery for elections. (#2566) 1. Save pre-commit state in the beginning of end block. 2. Provide an interface for custom elections to rollback on crash recovery. 3. Simplify pre-commit management. 4. Add crash recovery for updert-validator and chain migration elecitons. --- bigchaindb/backend/localmongodb/query.py | 30 ++++- bigchaindb/backend/localmongodb/schema.py | 2 +- bigchaindb/backend/query.py | 41 +++++-- bigchaindb/commands/bigchaindb.py | 14 +-- bigchaindb/core.py | 35 ++++-- bigchaindb/elections/election.py | 73 ++++++++--- bigchaindb/lib.py | 20 ++- .../migrations/chain_migration_election.py | 3 + .../upsert_validator/validator_election.py | 4 + tests/backend/localmongodb/test_queries.py | 19 +-- tests/backend/localmongodb/test_schema.py | 4 +- tests/commands/test_commands.py | 14 ++- tests/elections/test_election.py | 42 +++++-- tests/tendermint/test_core.py | 116 ++++++++++++++---- tests/tendermint/test_lib.py | 21 ---- tests/utils.py | 4 +- 16 files changed, 307 insertions(+), 135 deletions(-) diff --git a/bigchaindb/backend/localmongodb/query.py b/bigchaindb/backend/localmongodb/query.py index effbdacb..2cf68813 100644 --- a/bigchaindb/backend/localmongodb/query.py +++ b/bigchaindb/backend/localmongodb/query.py @@ -261,18 +261,15 @@ def get_unspent_outputs(conn, *, query=None): @register_query(LocalMongoDBConnection) def store_pre_commit_state(conn, state): - commit_id = state['commit_id'] return conn.run( conn.collection('pre_commit') - .replace_one({'commit_id': commit_id}, state, upsert=True) + .replace_one({}, state, upsert=True) ) @register_query(LocalMongoDBConnection) -def get_pre_commit_state(conn, commit_id): - return conn.run(conn.collection('pre_commit') - .find_one({'commit_id': commit_id}, - projection={'_id': False})) +def get_pre_commit_state(conn): + return conn.run(conn.collection('pre_commit').find_one()) @register_query(LocalMongoDBConnection) @@ -287,6 +284,13 @@ def store_validator_set(conn, validators_update): ) +@register_query(LocalMongoDBConnection) +def delete_validator_set(conn, height): + return conn.run( + conn.collection('validators').delete_many({'height': height}) + ) + + @register_query(LocalMongoDBConnection) def store_election(conn, election_id, height, is_concluded): return conn.run( @@ -308,6 +312,13 @@ def store_elections(conn, elections): ) +@register_query(LocalMongoDBConnection) +def delete_elections(conn, height): + return conn.run( + conn.collection('elections').delete_many({'height': height}) + ) + + @register_query(LocalMongoDBConnection) def get_validator_set(conn, height=None): query = {} @@ -360,6 +371,13 @@ def store_abci_chain(conn, height, chain_id, is_synced=True): ) +@register_query(LocalMongoDBConnection) +def delete_abci_chain(conn, height): + return conn.run( + conn.collection('abci_chains').delete_many({'height': height}) + ) + + @register_query(LocalMongoDBConnection) def get_latest_abci_chain(conn): return conn.run( diff --git a/bigchaindb/backend/localmongodb/schema.py b/bigchaindb/backend/localmongodb/schema.py index 157ff39d..e99d0492 100644 --- a/bigchaindb/backend/localmongodb/schema.py +++ b/bigchaindb/backend/localmongodb/schema.py @@ -42,7 +42,7 @@ INDEXES = { ('output_index', ASCENDING)], dict(name='utxo', unique=True)), ], 'pre_commit': [ - ('commit_id', dict(name='pre_commit_id', unique=True)), + ('height', dict(name='height', unique=True)), ], 'elections': [ ([('height', DESCENDING), ('election_id', ASCENDING)], diff --git a/bigchaindb/backend/query.py b/bigchaindb/backend/query.py index 0b5bbade..b6290cb5 100644 --- a/bigchaindb/backend/query.py +++ b/bigchaindb/backend/query.py @@ -8,9 +8,6 @@ from functools import singledispatch from bigchaindb.backend.exceptions import OperationError -VALIDATOR_UPDATE_ID = 'a_unique_id_string' -PRE_COMMIT_ID = 'a_unique_id_string' - @singledispatch def store_asset(connection, asset): @@ -316,12 +313,11 @@ def get_unspent_outputs(connection, *, query=None): @singledispatch -def store_pre_commit_state(connection, commit_id, state): - """Store pre-commit state in a document with `id` as `commit_id`. +def store_pre_commit_state(connection, state): + """Store pre-commit state. Args: - commit_id (string): `id` of document where `state` should be stored. - state (dict): commit state. + state (dict): pre-commit state. Returns: The result of the operation. @@ -331,14 +327,11 @@ def store_pre_commit_state(connection, commit_id, state): @singledispatch -def get_pre_commit_state(connection, commit_id): - """Get pre-commit state where `id` is `commit_id`. - - Args: - commit_id (string): `id` of document where `state` should be stored. +def get_pre_commit_state(connection): + """Get pre-commit state. Returns: - Document with `id` as `commit_id` + Document representing the pre-commit state. """ raise NotImplementedError @@ -351,6 +344,13 @@ def store_validator_set(conn, validator_update): raise NotImplementedError +@singledispatch +def delete_validator_set(conn, height): + """Delete the validator set at the given height.""" + + raise NotImplementedError + + @singledispatch def store_election(conn, election_id, height, is_concluded): """Store election record""" @@ -365,6 +365,13 @@ def store_elections(conn, elections): raise NotImplementedError +@singledispatch +def delete_elections(conn, height): + """Delete all election records at the given height""" + + raise NotImplementedError + + @singledispatch def get_validator_set(conn, height): """Get validator set for a given `height`, if `height` is not specified @@ -403,6 +410,14 @@ def store_abci_chain(conn, height, chain_id, is_synced=True): Args: is_synced: True if the chain is known by both ABCI client and server """ + + raise NotImplementedError + + +@singledispatch +def delete_abci_chain(conn, height): + """Delete the ABCI chain at the given height.""" + raise NotImplementedError diff --git a/bigchaindb/commands/bigchaindb.py b/bigchaindb/commands/bigchaindb.py index ddc7a5f5..639b3584 100644 --- a/bigchaindb/commands/bigchaindb.py +++ b/bigchaindb/commands/bigchaindb.py @@ -13,6 +13,7 @@ import copy import json import sys +from bigchaindb.core import rollback from bigchaindb.migrations.chain_migration_election import ChainMigrationElection from bigchaindb.utils import load_node_key from bigchaindb.common.exceptions import (DatabaseDoesNotExist, @@ -22,8 +23,6 @@ import bigchaindb from bigchaindb import (backend, ValidatorElection, BigchainDB) from bigchaindb.backend import schema -from bigchaindb.backend import query -from bigchaindb.backend.query import PRE_COMMIT_ID from bigchaindb.commands import utils from bigchaindb.commands.utils import (configure_bigchaindb, input_on_stderr) @@ -270,16 +269,7 @@ def run_drop(args): def run_recover(b): - pre_commit = query.get_pre_commit_state(b.connection, PRE_COMMIT_ID) - - # Initially the pre-commit collection would be empty - if pre_commit: - latest_block = query.get_latest_block(b.connection) - - # NOTE: the pre-commit state can only be ahead of the commited state - # by 1 block - if latest_block and (latest_block['height'] < pre_commit['height']): - query.delete_transactions(b.connection, pre_commit['transactions']) + rollback(b) @configure_bigchaindb diff --git a/bigchaindb/core.py b/bigchaindb/core.py index afe646bc..331cf22c 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -25,8 +25,7 @@ from bigchaindb.version import __tm_supported_versions__ from bigchaindb.utils import tendermint_version_is_compatible from bigchaindb.tendermint_utils import (decode_transaction, calculate_hash) -from bigchaindb.lib import Block, PreCommitState -from bigchaindb.backend.query import PRE_COMMIT_ID +from bigchaindb.lib import Block import bigchaindb.upsert_validator.validator_utils as vutils from bigchaindb.events import EventTypes, Event @@ -207,6 +206,14 @@ class App(BaseApplication): height = request_end_block.height + chain_shift self.new_height = height + + # store pre-commit state to recover in case there is a crash during + # `end_block` or `commit` + logger.debug(f'Updating pre-commit state: {self.new_height}') + pre_commit_state = dict(height=self.new_height, + transactions=self.block_txn_ids) + self.bigchaindb.store_pre_commit_state(pre_commit_state) + block_txn_hash = calculate_hash(self.block_txn_ids) block = self.bigchaindb.get_latest_block() @@ -219,12 +226,6 @@ class App(BaseApplication): self.new_height, self.block_transactions) - # Store pre-commit state to recover in case there is a crash during `commit` - pre_commit_state = PreCommitState(commit_id=PRE_COMMIT_ID, - height=self.new_height, - transactions=self.block_txn_ids) - logger.debug('Updating PreCommitState: %s', self.new_height) - self.bigchaindb.store_pre_commit_state(pre_commit_state._asdict()) return ResponseEndBlock(validator_updates=validator_update) def commit(self): @@ -257,3 +258,21 @@ class App(BaseApplication): self.events_queue.put(event) return ResponseCommit(data=data) + + +def rollback(b): + pre_commit = b.get_pre_commit_state() + + if pre_commit is None: + # the pre_commit record is first stored in the first `end_block` + return + + latest_block = b.get_latest_block() + if latest_block is None: + logger.error('Found precommit state but no blocks!') + sys.exit(1) + + # NOTE: the pre-commit state is always at most 1 block ahead of the commited state + if latest_block['height'] < pre_commit['height']: + Election.rollback(b, pre_commit['height'], pre_commit['transactions']) + b.delete_transactions(pre_commit['transactions']) diff --git a/bigchaindb/elections/election.py b/bigchaindb/elections/election.py index e8b3256e..470717f0 100644 --- a/bigchaindb/elections/election.py +++ b/bigchaindb/elections/election.py @@ -250,6 +250,30 @@ class Election(Transaction): return response + @classmethod + def _get_initiated_elections(cls, height, txns): + elections = [] + for tx in txns: + if not isinstance(tx, Election): + continue + + elections.append({'election_id': tx.id, 'height': height, + 'is_concluded': False}) + return elections + + @classmethod + def _get_votes(cls, txns): + elections = OrderedDict() + for tx in txns: + if not isinstance(tx, Vote): + continue + + election_id = tx.asset['id'] + if election_id not in elections: + elections[election_id] = [] + elections[election_id].append(tx) + return elections + @classmethod def process_block(cls, bigchain, new_height, txns): """Looks for election and vote transactions inside the block, records @@ -274,25 +298,15 @@ class Election(Transaction): for other concluded elections, if it requires so, the method should rely on the database state. """ - # elections placed in this block - initiated_elections = [] - # elections voted for in this block and their votes - elections = OrderedDict() - for tx in txns: - if isinstance(tx, Election): - initiated_elections.append({'election_id': tx.id, - 'height': new_height, - 'is_concluded': False}) - if not isinstance(tx, Vote): - continue - election_id = tx.asset['id'] - if election_id not in elections: - elections[election_id] = [] - elections[election_id].append(tx) + # elections initiated in this block + initiated_elections = cls._get_initiated_elections(new_height, txns) if initiated_elections: bigchain.store_elections(initiated_elections) + # elections voted for in this block and their votes + elections = cls._get_votes(txns) + validator_update = None for election_id, votes in elections.items(): election = bigchain.get_transaction(election_id) @@ -307,5 +321,34 @@ class Election(Transaction): return [validator_update] if validator_update else [] + @classmethod + def rollback(cls, bigchain, new_height, txn_ids): + """Looks for election and vote transactions inside the block and + cleans up the database artifacts possibly created in `process_blocks`. + + Part of the `end_block`/`commit` crash recovery. + """ + + # delete election records for elections initiated at this height and + # elections concluded at this height + bigchain.delete_elections(new_height) + + txns = [bigchain.get_transaction(tx_id) for tx_id in txn_ids] + + elections = cls._get_votes(txns) + for election_id in elections: + election = bigchain.get_transaction(election_id) + election.on_rollback(bigchain, new_height) + def on_approval(self, bigchain, new_height): + """Override to update the database state according to the + election rules. Consider the current database state to account for + other concluded elections, if required. + """ + raise NotImplementedError + + def on_rollback(self, bigchain, new_height): + """Override to clean up the database artifacts possibly created + in `on_approval`. Part of the `end_block`/`commit` crash recovery. + """ raise NotImplementedError diff --git a/bigchaindb/lib.py b/bigchaindb/lib.py index f38b0666..ce7091d3 100644 --- a/bigchaindb/lib.py +++ b/bigchaindb/lib.py @@ -143,6 +143,9 @@ class BigchainDB(object): backend.query.store_assets(self.connection, assets) return backend.query.store_transactions(self.connection, txns) + def delete_transactions(self, txs): + return backend.query.delete_transactions(self.connection, txs) + def update_utxoset(self, transaction): """Update the UTXO set given ``transaction``. That is, remove the outputs that the given ``transaction`` spends, and add the @@ -251,6 +254,9 @@ class BigchainDB(object): return transaction + def get_transactions(self, txn_ids): + return backend.query.get_transactions(self.connection, txn_ids) + def get_transactions_filtered(self, asset_id, operation=None): """Get a list of transactions filtered on some criteria """ @@ -438,6 +444,9 @@ class BigchainDB(object): def get_election(self, election_id): return backend.query.get_election(self.connection, election_id) + def get_pre_commit_state(self): + return backend.query.get_pre_commit_state(self.connection) + def store_pre_commit_state(self, state): return backend.query.store_pre_commit_state(self.connection, state) @@ -449,10 +458,16 @@ class BigchainDB(object): return backend.query.store_validator_set(self.connection, {'height': height, 'validators': validators}) + def delete_validator_set(self, height): + return backend.query.delete_validator_set(self.connection, height) + def store_abci_chain(self, height, chain_id, is_synced=True): return backend.query.store_abci_chain(self.connection, height, chain_id, is_synced) + def delete_abci_chain(self, height): + return backend.query.delete_abci_chain(self.connection, height) + def get_latest_abci_chain(self): return backend.query.get_latest_abci_chain(self.connection) @@ -487,7 +502,8 @@ class BigchainDB(object): def store_elections(self, elections): return backend.query.store_elections(self.connection, elections) + def delete_elections(self, height): + return backend.query.delete_elections(self.connection, height) + Block = namedtuple('Block', ('app_hash', 'height', 'transactions')) - -PreCommitState = namedtuple('PreCommitState', ('commit_id', 'height', 'transactions')) diff --git a/bigchaindb/migrations/chain_migration_election.py b/bigchaindb/migrations/chain_migration_election.py index 52f8b020..119392c2 100644 --- a/bigchaindb/migrations/chain_migration_election.py +++ b/bigchaindb/migrations/chain_migration_election.py @@ -43,3 +43,6 @@ class ChainMigrationElection(Election): ] output += f'\nvalidators={json.dumps(validators, indent=4)}' return output + + def on_rollback(self, bigchain, new_height): + bigchain.delete_abci_chain(new_height) diff --git a/bigchaindb/upsert_validator/validator_election.py b/bigchaindb/upsert_validator/validator_election.py index 27e0ff56..202f73fe 100644 --- a/bigchaindb/upsert_validator/validator_election.py +++ b/bigchaindb/upsert_validator/validator_election.py @@ -61,3 +61,7 @@ class ValidatorElection(Election): # TODO change to `new_height + 2` when upgrading to Tendermint 0.24.0. bigchain.store_validator_set(new_height + 1, updated_validator_set) return encode_validator(self.asset['data']) + + def on_rollback(self, bigchaindb, new_height): + # TODO change to `new_height + 2` when upgrading to Tendermint 0.24.0. + bigchaindb.delete_validator_set(new_height + 1) diff --git a/tests/backend/localmongodb/test_queries.py b/tests/backend/localmongodb/test_queries.py index d2fa58c1..030470a0 100644 --- a/tests/backend/localmongodb/test_queries.py +++ b/tests/backend/localmongodb/test_queries.py @@ -395,13 +395,10 @@ def test_get_unspent_outputs(db_context, utxoset): def test_store_pre_commit_state(db_context): from bigchaindb.backend import query - from bigchaindb.lib import PreCommitState - state = PreCommitState(commit_id='test', - height=3, - transactions=[]) + state = dict(height=3, transactions=[]) - query.store_pre_commit_state(db_context.conn, state._asdict()) + query.store_pre_commit_state(db_context.conn, state) cursor = db_context.conn.db.pre_commit.find({'commit_id': 'test'}, projection={'_id': False}) assert cursor.collection.count_documents({}) == 1 @@ -409,15 +406,11 @@ def test_store_pre_commit_state(db_context): def test_get_pre_commit_state(db_context): from bigchaindb.backend import query - from bigchaindb.lib import PreCommitState - state = PreCommitState(commit_id='test2', - height=3, - transactions=[]) - - db_context.conn.db.pre_commit.insert_one(state._asdict()) - resp = query.get_pre_commit_state(db_context.conn, 'test2') - assert resp == state._asdict() + state = dict(height=3, transactions=[]) + db_context.conn.db.pre_commit.insert_one(state) + resp = query.get_pre_commit_state(db_context.conn) + assert resp == state def test_validator_update(): diff --git a/tests/backend/localmongodb/test_schema.py b/tests/backend/localmongodb/test_schema.py index c4f6669a..84cfdaaf 100644 --- a/tests/backend/localmongodb/test_schema.py +++ b/tests/backend/localmongodb/test_schema.py @@ -61,8 +61,8 @@ def test_create_tables(): assert indexes['election_id_height']['unique'] indexes = conn.conn[dbname]['pre_commit'].index_information() - assert set(indexes.keys()) == {'_id_', 'pre_commit_id'} - assert indexes['pre_commit_id']['unique'] + assert set(indexes.keys()) == {'_id_', 'height'} + assert indexes['height']['unique'] def test_drop(dummy_db): diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index 1512d7eb..6ce4c804 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -259,8 +259,7 @@ def test_recover_db_on_start(mock_run_recover, def test_run_recover(b, alice, bob): from bigchaindb.commands.bigchaindb import run_recover from bigchaindb.models import Transaction - from bigchaindb.lib import Block, PreCommitState - from bigchaindb.backend.query import PRE_COMMIT_ID + from bigchaindb.lib import Block from bigchaindb.backend import query tx1 = Transaction.create([alice.public_key], @@ -288,8 +287,7 @@ def test_run_recover(b, alice, bob): b.store_block(block9) # create a pre_commit state which is ahead of the commit state - pre_commit_state = PreCommitState(commit_id=PRE_COMMIT_ID, height=10, - transactions=[tx2.id])._asdict() + pre_commit_state = dict(height=10, transactions=[tx2.id]) b.store_pre_commit_state(pre_commit_state) run_recover(b) @@ -522,11 +520,13 @@ def test_chain_migration_election_show_shows_inconclusive(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) assert not run_election_show(Namespace(election_id=election.id), b) @@ -556,11 +556,13 @@ def test_chain_migration_election_show_shows_concluded(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) assert not run_election_show(Namespace(election_id=election.id), b) diff --git a/tests/elections/test_election.py b/tests/elections/test_election.py index a8c5ce98..bb70ff19 100644 --- a/tests/elections/test_election.py +++ b/tests/elections/test_election.py @@ -17,11 +17,13 @@ def test_process_block_concludes_all_elections(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) txs = [election] total_votes = votes @@ -29,7 +31,8 @@ def test_process_block_concludes_all_elections(b): election, votes = generate_election(b, ValidatorElection, public_key, private_key, - new_validator['election']) + new_validator['election'], + voter_keys) txs += [election] total_votes += votes @@ -67,10 +70,13 @@ def test_process_block_approves_only_one_validator_update(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] + election, votes = generate_election(b, ValidatorElection, public_key, private_key, - new_validator['election']) + new_validator['election'], + voter_keys) txs = [election] total_votes = votes @@ -79,7 +85,8 @@ def test_process_block_approves_only_one_validator_update(b): election, votes = generate_election(b, ValidatorElection, public_key, private_key, - another_validator['election']) + another_validator['election'], + voter_keys) txs += [election] total_votes += votes @@ -109,10 +116,13 @@ def test_process_block_approves_after_pending_validator_update(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] + election, votes = generate_election(b, ValidatorElection, public_key, private_key, - new_validator['election']) + new_validator['election'], + voter_keys) txs = [election] total_votes = votes @@ -121,14 +131,16 @@ def test_process_block_approves_after_pending_validator_update(b): election, votes = generate_election(b, ValidatorElection, public_key, private_key, - another_validator['election']) + another_validator['election'], + voter_keys) txs += [election] total_votes += votes election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) txs += [election] total_votes += votes @@ -165,10 +177,13 @@ def test_process_block_does_not_approve_after_validator_update(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] + election, votes = generate_election(b, ValidatorElection, public_key, private_key, - new_validator['election']) + new_validator['election'], + voter_keys) txs = [election] total_votes = votes @@ -181,7 +196,8 @@ def test_process_block_does_not_approve_after_validator_update(b): second_election, second_votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) Election.process_block(b, 2, total_votes + [second_election]) @@ -205,17 +221,21 @@ def test_process_block_applies_only_one_migration(b): public_key = validators[0]['public_key'] private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] + election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) txs = [election] total_votes = votes election, votes = generate_election(b, ChainMigrationElection, public_key, private_key, - {}) + {}, + voter_keys) txs += [election] total_votes += votes diff --git a/tests/tendermint/test_core.py b/tests/tendermint/test_core.py index d62b4786..9605e7f1 100644 --- a/tests/tendermint/test_core.py +++ b/tests/tendermint/test_core.py @@ -22,12 +22,18 @@ from bigchaindb.backend.localmongodb import query from bigchaindb.common.crypto import generate_key_pair from bigchaindb.core import (CodeTypeOk, CodeTypeError, + rollback ) +from bigchaindb.elections.election import Election from bigchaindb.lib import Block +from bigchaindb.migrations.chain_migration_election import ChainMigrationElection +from bigchaindb.upsert_validator.validator_election import ValidatorElection from bigchaindb.upsert_validator.validator_utils import new_validator_set from bigchaindb.tendermint_utils import public_key_to_base64 from bigchaindb.version import __tm_supported_versions__ +from tests.utils import generate_election, generate_validators + pytestmark = pytest.mark.bdb @@ -347,39 +353,45 @@ def test_deliver_transfer_tx__double_spend_fails(b, init_chain_request): assert result.code == CodeTypeError -# The test below has to re-written one election conclusion logic has been implemented -@pytest.mark.skip def test_end_block_return_validator_updates(b, init_chain_request): - from bigchaindb import App - from bigchaindb.backend import query - from bigchaindb.core import encode_validator - from bigchaindb.backend.query import VALIDATOR_UPDATE_ID - app = App(b) app.init_chain(init_chain_request) begin_block = RequestBeginBlock() app.begin_block(begin_block) - validator = {'pub_key': {'type': 'ed25519', - 'data': 'B0E42D2589A455EAD339A035D6CE1C8C3E25863F268120AA0162AD7D003A4014'}, - 'power': 10} - validator_update = {'validator': validator, - 'update_id': VALIDATOR_UPDATE_ID} - query.store_validator_update(b.connection, validator_update) + # generate a block containing a concluded validator election + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) - resp = app.end_block(RequestEndBlock(height=99)) - assert resp.validator_updates[0] == encode_validator(validator) + new_validator = generate_validators([1])[0] - updates = b.approved_update() - assert not updates + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] + + election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election'], + voter_keys) + b.store_block(Block(height=1, transactions=[election.id], + app_hash='')._asdict()) + b.store_bulk_transactions([election]) + Election.process_block(b, 1, [election]) + + app.block_transactions = votes + + resp = app.end_block(RequestEndBlock(height=2)) + assert resp.validator_updates[0].power == new_validator['election']['power'] + expected = bytes.fromhex(new_validator['election']['public_key']['value']) + assert expected == resp.validator_updates[0].pub_key.data def test_store_pre_commit_state_in_end_block(b, alice, init_chain_request): from bigchaindb import App from bigchaindb.backend import query from bigchaindb.models import Transaction - from bigchaindb.backend.query import PRE_COMMIT_ID tx = Transaction.create([alice.public_key], [([alice.public_key], 1)], @@ -394,16 +406,14 @@ def test_store_pre_commit_state_in_end_block(b, alice, init_chain_request): app.deliver_tx(encode_tx_to_bytes(tx)) app.end_block(RequestEndBlock(height=99)) - resp = query.get_pre_commit_state(b.connection, PRE_COMMIT_ID) - assert resp['commit_id'] == PRE_COMMIT_ID + resp = query.get_pre_commit_state(b.connection) assert resp['height'] == 99 assert resp['transactions'] == [tx.id] app.begin_block(begin_block) app.deliver_tx(encode_tx_to_bytes(tx)) app.end_block(RequestEndBlock(height=100)) - resp = query.get_pre_commit_state(b.connection, PRE_COMMIT_ID) - assert resp['commit_id'] == PRE_COMMIT_ID + resp = query.get_pre_commit_state(b.connection) assert resp['height'] == 100 assert resp['transactions'] == [tx.id] @@ -413,12 +423,70 @@ def test_store_pre_commit_state_in_end_block(b, alice, init_chain_request): app.begin_block(begin_block) app.deliver_tx(encode_tx_to_bytes(tx)) app.end_block(RequestEndBlock(height=1)) - resp = query.get_pre_commit_state(b.connection, PRE_COMMIT_ID) - assert resp['commit_id'] == PRE_COMMIT_ID + resp = query.get_pre_commit_state(b.connection) assert resp['height'] == 101 assert resp['transactions'] == [tx.id] +def test_rollback_pre_commit_state_after_crash(b): + validators = generate_validators([1] * 4) + b.store_validator_set(1, [v['storage'] for v in validators]) + b.store_block(Block(height=1, transactions=[], app_hash='')._asdict()) + + public_key = validators[0]['public_key'] + private_key = validators[0]['private_key'] + voter_keys = [v['private_key'] for v in validators] + + migration_election, votes = generate_election(b, + ChainMigrationElection, + public_key, private_key, + {}, + voter_keys) + + total_votes = votes + txs = [migration_election, *votes] + + new_validator = generate_validators([1])[0] + validator_election, votes = generate_election(b, + ValidatorElection, + public_key, private_key, + new_validator['election'], + voter_keys) + + total_votes += votes + txs += [validator_election, *votes] + + b.store_bulk_transactions(txs) + b.store_abci_chain(2, 'new_chain') + b.store_validator_set(2, [v['storage'] for v in validators]) + # TODO change to `4` when upgrading to Tendermint 0.22.4. + b.store_validator_set(3, [new_validator['storage']]) + b.store_election(migration_election.id, 2, is_concluded=False) + b.store_election(validator_election.id, 2, is_concluded=True) + + # no pre-commit state + rollback(b) + + for tx in txs: + assert b.get_transaction(tx.id) + assert b.get_latest_abci_chain() + assert len(b.get_validator_change()['validators']) == 1 + assert b.get_election(migration_election.id) + assert b.get_election(validator_election.id) + + b.store_pre_commit_state({'height': 2, 'transactions': [tx.id for tx in txs]}) + + rollback(b) + + for tx in txs: + assert not b.get_transaction(tx.id) + assert not b.get_latest_abci_chain() + assert len(b.get_validator_change()['validators']) == 4 + assert len(b.get_validator_change(2)['validators']) == 4 + assert not b.get_election(migration_election.id) + assert not b.get_election(validator_election.id) + + def test_new_validator_set(b): node1 = {'public_key': {'type': 'ed25519-base64', 'value': 'FxjS2/8AFYoIUqF6AcePTc87qOT7e4WGgH+sGCpTUDQ='}, diff --git a/tests/tendermint/test_lib.py b/tests/tendermint/test_lib.py index f85a3fb9..5bf41d23 100644 --- a/tests/tendermint/test_lib.py +++ b/tests/tendermint/test_lib.py @@ -147,27 +147,6 @@ def test_post_transaction_invalid_mode(b): b.write_transaction(tx, 'nope') -@pytest.mark.skip -@pytest.mark.bdb -def test_validator_updates(b, validator_pub_key): - from bigchaindb.backend import query - from bigchaindb.backend.query import VALIDATOR_UPDATE_ID - - # create a validator update object - validator = {'pub_key': {'type': 'ed25519', - 'data': validator_pub_key}, - 'power': 10} - validator_update = {'validator': validator, - 'update_id': VALIDATOR_UPDATE_ID} - query.store_validator_update(b.connection, validator_update) - - updates = b.approved_updates() - assert updates == validator_update['validator'] - - b.delete_validator_update() - assert not b.approved_updates() - - @pytest.mark.bdb def test_update_utxoset(b, signed_create_tx, signed_transfer_tx, db_context): mongo_client = MongoClient(host=db_context.host, port=db_context.port) diff --git a/tests/utils.py b/tests/utils.py index 5b2e1ea6..2fb89892 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -96,7 +96,7 @@ def generate_validators(powers): return validators -def generate_election(b, cls, public_key, private_key, asset_data): +def generate_election(b, cls, public_key, private_key, asset_data, voter_keys): voters = cls.recipients(b) election = cls.generate([public_key], voters, @@ -106,5 +106,7 @@ def generate_election(b, cls, public_key, private_key, asset_data): votes = [Vote.generate([election.to_inputs()[i]], [([Election.to_public_key(election.id)], power)], election.id) for i, (_, power) in enumerate(voters)] + for key, v in zip(voter_keys, votes): + v.sign([key]) return election, votes From 3da13eda3d07140ea96e6961bafd95a4fcf65c40 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 28 Sep 2018 13:23:04 +0200 Subject: [PATCH 59/77] Last PR before releasing 2.0.0b7 (#2576) * Updated changelog for 2.0.0b7 * Change 2.0.0b3 to b7 in network-setup.md * Updated k8s yamls for 2.0.0b7 release * Updated version.py for 2.0.0b7 release --- CHANGELOG.md | 25 +++++++++++++++++++ bigchaindb/version.py | 4 +-- .../network-setup.md | 6 ++--- k8s/bigchaindb/bigchaindb-ss.yaml | 2 +- k8s/dev-setup/bigchaindb.yaml | 2 +- 5 files changed, 32 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d84acdd9..6bfc2fbd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,31 @@ For reference, the possible headings are: * **Known Issues** * **Notes** +## [2.0 Beta 7] - 2018-09-28 + +Tag name: v2.0.0b7 + +### Added + +Completed the implementation of chain-migration elections (BEP-42). Pull requests [#2553](https://github.com/bigchaindb/bigchaindb/pull/2553), [#2556](https://github.com/bigchaindb/bigchaindb/pull/2556), [#2558](https://github.com/bigchaindb/bigchaindb/pull/2558), [#2563](https://github.com/bigchaindb/bigchaindb/pull/2563) and [#2566](https://github.com/bigchaindb/bigchaindb/pull/2566) + +### Changed + +* Code that used the Python driver's (deprecated) transactions.send() method now uses its transactions.send_commit() method instead. [Pull request #2547](https://github.com/bigchaindb/bigchaindb/pull/2547) +* Code that implied pluggable "consensus" now implies pluggable transaction "validation" (a more accurate word). [Pull request #2561](https://github.com/bigchaindb/bigchaindb/pull/2561) + +### Removed + +Benchmark logs. [Pull request #2565](https://github.com/bigchaindb/bigchaindb/pull/2565) + +### Fixed + +A bug caused by an incorrect MongoDB query. [Pull request #2567](https://github.com/bigchaindb/bigchaindb/pull/2567) + +### Notes + +There's now better documentation about logs, log rotation, and the `server.bind` config setting. Pull requests [#2546](https://github.com/bigchaindb/bigchaindb/pull/2546) and [#2575](https://github.com/bigchaindb/bigchaindb/pull/2575) + ## [2.0 Beta 6] - 2018-09-17 Tag name: v2.0.0b6 diff --git a/bigchaindb/version.py b/bigchaindb/version.py index b1761387..f7d03f69 100644 --- a/bigchaindb/version.py +++ b/bigchaindb/version.py @@ -2,7 +2,7 @@ # SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) # Code is Apache-2.0 and docs are CC-BY-4.0 -__version__ = '2.0.0b6' -__short_version__ = '2.0b6' +__version__ = '2.0.0b7' +__short_version__ = '2.0b7' # supported Tendermint version __tm_supported_versions__ = ["0.22.8"] diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index 7dd2c937..c3057895 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -62,11 +62,11 @@ sudo apt install -y python3-pip libssl-dev # Ubuntu 16.04, and other Linux distros, may require other packages or more packages ``` -Now install the latest version of BigchainDB. You can find the latest version by going to the [BigchainDB project release history page on PyPI][bdb:pypi]. For example, to install version 2.0.0b3, you would do: +Now install the latest version of BigchainDB. You can find the latest version by going to the [BigchainDB project release history page on PyPI][bdb:pypi]. For example, to install version 2.0.0b7, you would do: ``` -# Change 2.0.0b3 to the latest version as explained above: -sudo pip3 install bigchaindb==2.0.0b3 +# Change 2.0.0b7 to the latest version as explained above: +sudo pip3 install bigchaindb==2.0.0b7 ``` Check that you installed the correct version of BigchainDB Server using `bigchaindb --version`. diff --git a/k8s/bigchaindb/bigchaindb-ss.yaml b/k8s/bigchaindb/bigchaindb-ss.yaml index b6ada5a3..9821b258 100644 --- a/k8s/bigchaindb/bigchaindb-ss.yaml +++ b/k8s/bigchaindb/bigchaindb-ss.yaml @@ -158,7 +158,7 @@ spec: timeoutSeconds: 15 # BigchainDB container - name: bigchaindb - image: bigchaindb/bigchaindb:2.0.0-beta6 + image: bigchaindb/bigchaindb:2.0.0-beta7 imagePullPolicy: Always args: - start diff --git a/k8s/dev-setup/bigchaindb.yaml b/k8s/dev-setup/bigchaindb.yaml index 7a2fa34c..7ad0a1ff 100644 --- a/k8s/dev-setup/bigchaindb.yaml +++ b/k8s/dev-setup/bigchaindb.yaml @@ -38,7 +38,7 @@ spec: terminationGracePeriodSeconds: 10 containers: - name: bigchaindb - image: bigchaindb/bigchaindb:2.0.0-beta6 + image: bigchaindb/bigchaindb:2.0.0-beta7 imagePullPolicy: Always args: - start From 5c1511b4792908d7e50754dff2fd5865896a1272 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Fri, 28 Sep 2018 13:35:33 +0200 Subject: [PATCH 60/77] Add help with `make release` to release process --- RELEASE_PROCESS.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/RELEASE_PROCESS.md b/RELEASE_PROCESS.md index 3c24dedf..157272c2 100644 --- a/RELEASE_PROCESS.md +++ b/RELEASE_PROCESS.md @@ -60,7 +60,10 @@ The following steps are what we do to release a new version of _BigchainDB Serve 1. Click "Publish release" to publish the release on GitHub. 1. On your local computer, make sure you're on the `master` branch and that it's up-to-date with the `master` branch in the bigchaindb/bigchaindb repository (e.g. `git pull upstream master`). We're going to use that to push a new `bigchaindb` package to PyPI. 1. Make sure you have a `~/.pypirc` file containing credentials for PyPI. -1. Do `make release` to build and publish the new `bigchaindb` package on PyPI. +1. Do `make release` to build and publish the new `bigchaindb` package on PyPI. For this step you need to have `twine` installed. If you get an error like `Makefile:135: recipe for target 'clean-pyc' failed` then try doing + ```text + sudo chown -R $(whoami):$(whoami) . + ``` 1. [Log in to readthedocs.org](https://readthedocs.org/accounts/login/) and go to the **BigchainDB Server** project, then: - Go to Admin --> Advanced Settings and make sure that "Default branch:" (i.e. what "latest" points to) From c1993c37873599bbffd090956bb12d20731b7910 Mon Sep 17 00:00:00 2001 From: Troy McConaghy Date: Tue, 9 Oct 2018 12:27:06 +0200 Subject: [PATCH 61/77] Revised Simple Deployment Template: added NGINX instructions (#2578) --- .../deploy-a-machine.md | 69 ++++++ .../simple-deployment-template/index.rst | 24 +- .../network-setup.md | 227 +++--------------- .../set-up-nginx.md | 42 ++++ .../set-up-node-software.md | 104 ++++++++ .../source/simple-deployment-template/tips.md | 59 +++++ nginx/nginx.conf | 125 ++++++++++ 7 files changed, 454 insertions(+), 196 deletions(-) create mode 100644 docs/server/source/simple-deployment-template/deploy-a-machine.md create mode 100644 docs/server/source/simple-deployment-template/set-up-nginx.md create mode 100644 docs/server/source/simple-deployment-template/set-up-node-software.md create mode 100644 docs/server/source/simple-deployment-template/tips.md create mode 100644 nginx/nginx.conf diff --git a/docs/server/source/simple-deployment-template/deploy-a-machine.md b/docs/server/source/simple-deployment-template/deploy-a-machine.md new file mode 100644 index 00000000..764229d0 --- /dev/null +++ b/docs/server/source/simple-deployment-template/deploy-a-machine.md @@ -0,0 +1,69 @@ + + +# Deploy a Machine for Your BigchainDB Node + +The first step is to deploy a machine for your BigchainDB node. +It might be a virtual machine (VM) or a real machine. +If you follow this simple deployment template, all your node's +software will run on that one machine. + +We don't make any assumptions about _where_ you run the machine. +It might be in Azure, AWS, your data center or a Raspberry Pi. + +## IP Addresses + +The following instructions assume all the nodes +in the network (including yours) have public IP addresses. +(A BigchainDB network _can_ be run inside a private network, +using private IP addresses, but we don't cover that here.) + +## Operating System + +**Use Ubuntu 18.04 or Ubuntu Server 18.04 as the operating system.** + +Similar instructions will work on other versions of Ubuntu, +and other recent Debian-like Linux distros, +but you may have to change the names of the packages, +or install more packages. + +## Network Security Group + +If your machine is in AWS or Azure, for example, _and_ +you want users to connect to BigchainDB via HTTPS, +then you should configure its network security group +to allow all incoming and outgoing traffic for: + +* TCP on port 22 (SSH) +* TCP on port 80 (HTTP) +* TCP on port 443 (HTTPS) +* Any protocol on port 26656 (Tendermint P2P) + +If you don't care about HTTPS, then forget about port 443, +and replace port 80 with port 9984 (the default BigchainDB HTTP port). + +## Update Your System + +SSH into your machine and update all its OS-level packages: + +``` +sudo apt update +sudo apt full-upgrade +``` + +## Node Security + +If you're going to use your node in production, +then you should take additional steps to secure it. +We don't cover that here; there are many books and websites +about securing Linux machines. + +## DNS Setup + +* Register a domain name for your BigchainDB node, such as `example.com` +* Pick a subdomain of that domain for your BigchainDB node, such as `bnode.example.com` +* Create a DNS "A Record" pointing your chosen subdomain (such as `bnode.example.com`) + at your machine's IP address. diff --git a/docs/server/source/simple-deployment-template/index.rst b/docs/server/source/simple-deployment-template/index.rst index 9ee713c2..55f277da 100644 --- a/docs/server/source/simple-deployment-template/index.rst +++ b/docs/server/source/simple-deployment-template/index.rst @@ -8,14 +8,34 @@ Simple Deployment Template ========================== -This section describes *one* way to deploy a BigchainDB network. -You can modify it as you see fit. +This section describes *one* way to deploy a BigchainDB network +(i.e. a set of connected BigchainDB nodes). +You can modify this simple deployment template as you see fit. It's "simple" in the sense that each BigchainDB node is installed and run on a single virtual machine (or real machine). We also have a :ref:`kubernetes-deployment-template` (not simple). +**Note 1:** These instructions will also work for a "network" with only one node. +If you want your network to be able to handle the failure or misbehavior +of one node, then your network must have at least four nodes. +Nodes can be added or removed from a network after is it up and running. + +**Note 2:** You might not need to set up your own network yet. +You should start by creating a proof-of-concept app that writes to +[the BigchainDB Testnet](https://testnet.bigchaindb.com/), +and if that goes well, then you can look into setting up your own network. + +**Note 3:** If you want to set up a node or network +so that you can contribute to developing and testing the BigchainDB code, +then see +[the docs about contributing to BigchainDB](https://docs.bigchaindb.com/projects/contributing/en/latest/index.html). + .. toctree:: :maxdepth: 1 + deploy-a-machine + set-up-nginx + set-up-node-software network-setup + tips troubleshooting diff --git a/docs/server/source/simple-deployment-template/network-setup.md b/docs/server/source/simple-deployment-template/network-setup.md index c3057895..79bb4385 100644 --- a/docs/server/source/simple-deployment-template/network-setup.md +++ b/docs/server/source/simple-deployment-template/network-setup.md @@ -6,132 +6,23 @@ Code is Apache-2.0 and docs are CC-BY-4.0 # How to Set Up a BigchainDB Network -Note 1: These instructions will also work for a "network" with only one node. +Until now, everything could be done by a node operator, by themselves. +Now the node operators, also called **Members**, must share some information +with each other, so they can form a network. -Note 2: You might not need to set up your own network yet. You should start by creating a proof-of-concept app that writes to [the BigchainDB Testnet](https://testnet.bigchaindb.com/), and if that goes well, then you can look into setting up your own network. +There is one special Member who helps coordinate everyone: the **Coordinator**. -Note 3: If you want to set up a node or network so that you can contribute to developing and testing the BigchainDB code, then see [the docs about contributing to BigchainDB](https://docs.bigchaindb.com/projects/contributing/en/latest/index.html). +## Member: Share hostname, pub_key.value and node_id -
+Each BigchainDB node is identified by its: -The process to create a network is both *social* and *technical*: *social* because someone (that we will call **Coordinator**) needs to find at least three other **Members** willing to join the network, and coordinate the effort; *technical* because each member of the network needs to set up a machine running BigchainDB. (Note: a Coordinator is a Member as well.) +* `hostname`, i.e. the node's DNS subdomain, such as `bnode.example.com`, or its IP address, such as `46.145.17.32` +* Tendermint `pub_key.value` +* Tendermint `node_id` -A **BigchainDB Network** (or just *Network*) is a set of **4 or more BigchainDB Nodes** (or *Nodes*). Every Node is independently managed by a Member, and runs an instance of the [BigchainDB Server software][bdb:software]. At the **Genesis** of a Network, there **MUST** be at least **4** Nodes ready to connect. After the Genesis, a Network can dynamically add new Nodes or remove old Nodes. - -A Network will stop working if more than one third of the Nodes are down or faulty _in any way_. The bigger a Network, the more failures it can handle. A Network of size 4 can tolerate only 1 failure, so if 3 out of 4 Nodes are online, everything will work as expected. Eventually, the Node that was offline will automatically sync with the others. - -## Before We Start - -This tutorial assumes you have basic knowledge on how to manage a GNU/Linux machine. - -**Please note: The commands on this page work on Ubuntu 18.04. Similar commands will work on other versions of Ubuntu, and other recent Debian-like Linux distros, but you may have to change the names of the packages, or install more packages.** - -We don't make any assumptions about **where** you run the Node. -You can run BigchainDB Server on a Virtual Machine on the cloud, on a machine in your data center, or even on a Raspberry Pi. Just make sure that your Node is reachable by the other Nodes. Here's a **non-exhaustive list of examples**: - -- **good**: all Nodes running in the cloud using public IPs. -- **bad**: some Nodes running in the cloud using public IPs, some Nodes in a private network. -- **good**: all Nodes running in a private network. - -The rule of thumb is: if Nodes can ping each other, then you are good to go. - -The next sections are labelled with **Member** or **Coordinator**, depending on who should follow the instructions. Remember, a Coordinator is also a Member. - -## Member: Set Up a Node - -Every Member in the Network **must** set up its own Node. The process consists of installing three components, BigchainDB Server, Tendermint Core, and MongoDB, and configuring the firewall. - -**Important note on security: it's up to the Member to harden their system.** - -### Install the Required Software - -Make sure your system is up to date. - -``` -sudo apt update -sudo apt full-upgrade -``` - -#### Install BigchainDB Server - -BigchainDB Server requires **Python 3.6+**, so make sure your system has it. Install the required packages: - -``` -# For Ubuntu 18.04: -sudo apt install -y python3-pip libssl-dev -# Ubuntu 16.04, and other Linux distros, may require other packages or more packages -``` - -Now install the latest version of BigchainDB. You can find the latest version by going to the [BigchainDB project release history page on PyPI][bdb:pypi]. For example, to install version 2.0.0b7, you would do: - -``` -# Change 2.0.0b7 to the latest version as explained above: -sudo pip3 install bigchaindb==2.0.0b7 -``` - -Check that you installed the correct version of BigchainDB Server using `bigchaindb --version`. - -#### Install (and Start) MongoDB - -Install a recent version of MongoDB. BigchainDB Server requires version 3.4 or newer. - -``` -sudo apt install mongodb -``` - -If you install MongoDB using the above command (which installs the `mongodb` package), it also configures MongoDB, starts MongoDB (in the background), and installs a MongoDB startup script (so that MongoDB will be started automatically when the machine is restarted). - -Note: The `mongodb` package is _not_ the official MongoDB package from MongoDB the company. If you want to install the official MongoDB package, please see [the MongoDB documentation](https://docs.mongodb.com/manual/installation/). Note that installing the official package _doesn't_ also start MongoDB. - -#### Install Tendermint - -The version of BigchainDB Server described in these docs only works well with Tendermint 0.22.8 (not a higher version number). Install that: - -``` -sudo apt install -y unzip -wget https://github.com/tendermint/tendermint/releases/download/v0.22.8/tendermint_0.22.8_linux_amd64.zip -unzip tendermint_0.22.8_linux_amd64.zip -rm tendermint_0.22.8_linux_amd64.zip -sudo mv tendermint /usr/local/bin -``` - -### Set Up the Firewall - -Make sure to accept inbound connections on ports `9984`, `9985`, and `26656`. You might also want to add port `22` so that you can continue to access the machine via SSH. - -``` -sudo ufw allow 22/tcp -sudo ufw allow 9984/tcp -sudo ufw allow 9985/tcp -sudo ufw allow 26656/tcp -sudo ufw enable -``` - -Some cloud providers, like Microsoft Azure, require you to change "security groups" (virtual firewalls) using their portal or other APIs (such as their CLI). - -## Member: Configure BigchainDB Server - -To configure BigchainDB Server, run: - -``` -bigchaindb configure -``` - -The first question is ``API Server bind? (default `localhost:9984`)``. To expose the API to the public, bind the API Server to `0.0.0.0:9984`. Unless you have specific needs, you can keep the default value for all other questions. - -## Member: Generate the Private Key and Node id - -A Node is identified by the triplet ``. - -As a Member, it's your duty to create and store securely your private key, and share your `hostname`, `node_id`, and `public_key` with the other members of the network. - -To generate all of that, run: - -``` -tendermint init -``` - -The `public_key` is stored in the file `.tendermint/config/priv_validator.json`, and it should look like: +The Tendermint `pub_key.value` is stored +in the file `$HOME/.tendermint/config/priv_validator.json`. +That file should look like: ```json { @@ -150,7 +41,7 @@ The `public_key` is stored in the file `.tendermint/config/priv_validator.json`, } ``` -To extract your `node_id`, run the command: +To get your Tendermint `node_id`, run the command: ``` tendermint show_node_id @@ -158,15 +49,13 @@ tendermint show_node_id An example `node_id` is `9b989cd5ac65fec52652a457aed6f5fd200edc22`. -An example hostname is `charlie5.cloudservers.company.com`. You can also use a public IP addres, like `46.145.17.32`, instead of a hostname, but make sure that IP address won't change. +**Share your `hostname`, `pub_key.value` and `node_id` with all other Members.** -Share the `node_id`, `pub_key.value` and hostname of your Node with all other Members. +## Coordinator: Create & Share the genesis.json File -**Important note on security: each Member should take extra steps to verify the public keys they receive from the other Members have not been tampered with, e.g. a key signing party would be one way.** - -## Coordinator: Initialize the Network - -At this point the Coordinator should have received the data from all the Members, and should combine them in the `.tendermint/config/genesis.json` file: +At this point the Coordinator should have received the data +from all the Members, and should combine them in the file +`$HOME/.tendermint/config/genesis.json`: ```json { @@ -225,9 +114,12 @@ At this point the Coordinator should have received the data from all the Members } ``` -**Note:** `consensus_params` in the `genesis.json` are default values for Tendermint consensus. +**Note:** The above `consensus_params` in the `genesis.json` +are default values. -The new `genesis.json` file contains the data that describes the Network. The key `name` is the Member's moniker; it can be any valid string, but put something human-readable like `"Alice's Node Shop"`. +The new `genesis.json` file contains the data that describes the Network. +The key `name` is the Member's moniker; it can be any valid string, +but put something human-readable like `"Alice's Node Shop"`. At this point, the Coordinator must share the new `genesis.json` file with all Members. @@ -235,11 +127,13 @@ At this point, the Coordinator must share the new `genesis.json` file with all M At this point the Member should have received the `genesis.json` file. -**Important note on security: each Member should verify that the `genesis.json` file contains the correct public keys.** +The Member must copy the `genesis.json` file +into their local `$HOME/.tendermint/config` directory. +Every Member now shares the same `chain_id` and `genesis_time` (used to identify the Network), +and the same list of `validators`. -The Member must copy the `genesis.json` file in the local `.tendermint/config` directory. Every Member now shares the same `chain_id`, `genesis_time`, used to identify the Network, and the same list of `validators`. - -The Member must edit the `.tendermint/config/config.toml` file and make the following changes: +Each Member must edit their `$HOME/.tendermint/config/config.toml` file +and make the following changes: ``` moniker = "Name of our node" @@ -256,6 +150,9 @@ recv_rate = 102400000 recheck = false ``` +Note: The list of `persistent_peers` doesn't have to include all nodes +in the network. + ## Member: Start MongoDB If you installed MongoDB using `sudo apt install mongodb`, then MongoDB should already be running in the background. You can check using `systemctl status mongodb`. @@ -300,65 +197,7 @@ If you want to start and manage the BigchainDB and Tendermint processes yourself ## How Others Can Access Your Node -If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL `http://hostname:9984` (where hostname is something like `bdb7.canada.vmsareus.net` or `17.122.200.76`). That is, anyone can interact with your node using the [BigchainDB HTTP API](http-client-server-api.html) exposed at that address. The most common way to do that is to use one of the [BigchainDB Drivers](./drivers-clients/index.html). - -## Refreshing Your Node - -If you want to refresh your node back to a fresh empty state, then your best bet is to terminate it and deploy a new virtual machine, but if that's not an option, then you can: - -- drop the `bigchain` database in MongoDB using `bigchaindb drop` (but that only works if MongoDB is running) -- reset Tendermint using `tendermint unsafe_reset_all` -- delete the directory `$HOME/.tendermint` - -## Shutting down BigchainDB - -If you want to stop/kill BigchainDB, you can do so by sending `SIGINT`, `SIGQUIT` or `SIGTERM` to the running BigchainDB -process(es). Depending on how you started BigchainDB i.e. foreground or background. e.g. you started BigchainDB in the background as mentioned above in the guide: - -```bash -$ nohup bigchaindb start 2>&1 > bigchaindb.log & - -$ # Check the PID of the main BigchainDB process -$ ps -ef | grep bigchaindb - *