Problem statement:
BigchainDB v2.0.0b9 has been around for quite a while. Recently we have updated
Tendermint supported version to v0.31.5 which has incompatible blockchain.
Despite the fact that we have defined instructions on chain migration, no one
expected to migrate to incompatible chain within patch version range. So there
is a demand for Tendermint v0.22.8 compatibility among BigchainDB users.
Work has been done:
bigchaindb-abci package was upgraded to support multiple API versions.
New configuration field stating tendermint version was added.
Signed-off-by: David Dashyan <mail@davie.li>
1. Save pre-commit state in the beginning of end block.
2. Provide an interface for custom elections to rollback on crash recovery.
3. Simplify pre-commit management.
4. Add crash recovery for updert-validator and chain migration elecitons.
* Problem: Outputs API doesn't respond with correct unspent/spent outputs
Solution: Fix fastquery such that embedded document is queried properly
* Problem: key order agnostic queries not implemented
Solution: get_spent queries embedded documents which respect key order. This is
not expected by the application hence the query should be altered to query any
kind of key order
* Problem: Mongo query for get_spent too complicated
Solution: Simplify query using $elemMatch
* Problem: No test for checking mixed spent outputs
Solution: Add test for `get_spending_transactions` to check that correct
matching is done when querying documents with multiple inputs
* Problem: tranasction ids not assert when getting spent outputs
Solution: assert tranasction ids
- Hard-wire the supported Tendermint version(s) right in the code of BigchainDB Server. Check the version of Tendermint and disconnect if Tendermint version is an unsupported one.
- Expose a CLI command bigchaindb tendermint-version to display the supported Tendermint versions.
- PR also takes care the long list of warnings we get when we run tests.
- Updated deprecated pymongo methods
- Do not call pytest fixtures directly.
- Also added the doc for the new cli command
Solution: Stop getting events from the Tendermint events API. Get the
event right before returning from the COMMIT phase of Tendermint, and
publish it to the events queue.
* Problem: `ValidatorElection` and `MigrationElection` need to inherit from a common `Election` class.
Solution: Factored the common logic out of `ValidatorElection` and moved it to `Election` parent class.
* Problem: No need to store different types of elections in their own tables
Solution: Remove `DB_TABLE` property from `Election` class.
Solution: Created the `elections` table with secondary_index `election_id`.
* Problem: `UpsertValidatorVote` can be generalized to just be `Vote`
Solution: Renamed, refactored and moved the `Vote` class to tie in with the more general `Election` base class.
* Problem: `election_id` is not unique if two elections have the same properties.
Solution: Added a random `uuid4` seed to enforce uniqueness.
* Problem: we have unused and outdated fixtures
Solution: clean up fixtures and tests accordingly
* Problem: there are still unused fixtures
Solution: remove Merlin keys
* Problem: There are unnecessary markers
Solution: remove the tendermint marker for tests
* Problem: The commit mode behaves incorrectly.
Solution: Parse the Tendermint response properly. The functionality was disabled in https://github.com/bigchaindb/bigchaindb/pull/2235/files#diff-c6511560546a7dc577e7e647b5bfdaceL68 and was not fixed since then.
* Add a test case for the sync mode.
* Do not strictly expect deliver_tx in the response.
* Fix post_mock in web/test_transactions.py.
* Check for the error field first.
* Problem: No good way to check for val set absence.
Solution: Make get_validator_set/get_validators return None/[] when there are no validators yet.
* Problem: Incompatible ABCI chain upgrades.
Solution: Record known chains and sync through InitChain. Triggering the migration and adjusting other ABCI endpoints will follow.
* Problem: We need to store the `election_id` as part of the `validator_update` so we can efficiently check which election was resposible for the change
Solution: Added the parameter to `store_validator_set` and aligned the tests
* Problem: Logic for `upsert-validator show` is convoluted
Solution: Rewrote the function to be much simpler
* Problem: Need a uniqueness constraint for election_id wrt validator changes
Solution: Added a new key to the db schema
Solution: if a TRANSFER transaction is validated after a CREATE
transaction, the system crashes with `AttributeError: 'NoneType' object
has no attribute 'txid'`.
This happens because querying `get_spent` checks the attributes `txid`
and `output` of `input.fulfills` for every transaction in the current
buffer (`current_transactions`). For a CREATE, `input.fulfills` is None,
so the check would fail.
The solution is to check if `input.fulfills` is defined. If not, then
the current transaction cannot spend any output, so we can safely skip
it.
* Problem: several tests are skipped
Solution: activate or remove tests
* Problem: store_transaction is deprecated
Solution: replace it with store_bulk_transaction
* Problem: we don't test the transaction split
Solution: undelete `test_store_transaction`
* Problem: failing tests
Solution: merge master and change deprecated store_transaction
* Problem: Source files contain no license info
Solution: Add comments with SPDX license info to source files
* Python 3 files don't need # -*- coding: utf-8 -*-
* Problem: Validator set not tracked by BigchainDB
Solution: BigchainDB depends on tendermint's RPC API to get the validator set
which is not avaiable during replay so the validators set should be tracked
inside BigchainDB
* Problem: Unclear code and documentation
Solution: Fix decode_validator and docs strings
* Problem: Doc strings missing
Solution: Add doc string for store_validato_set
* Problem: core.py contains an unused class, `Bigchain`
Solution: Remove core.py. Refactor BigchainDB Class to remove inheritance from Bigchain.
* Problem: core.py contains an unused class, `Bigchain`
Solution: Remove core.py. Refactor BigchainDB Class to remove inheritance from Bigchain.
* Fixed flake8 complaint about too many blank lines
* Attempting to fix Sphinx docs. This may result in some redundant commits, as I don't know what I'm doing, and I can't experiment without running the CI...
Sorry in advance!
* Attempting to fix Sphinx docs. This may result in some redundant commits, as I don't know what I'm doing, and I can't experiment without running the CI...
Sorry in advance!
* Updating from master changed BigchainDB.process_post_response to a private method, so I had to align with that.
* Fixed a couple stale references to bigchaindb.Bigchain in docstrings
* Missed a reference to `Bigchain` in a patch call...
* Problem: BigchainDB class should be part of project root
Solution: Removed the /tendermint directory and moved its contents to project root
* Problem: Flake8 complained that imports were not at the top of the file
Solution: Had to play around with the order of imports to avoid cyclic dependencies, but its working and style compliant now
* Problem: Stale reference to /tendermint directory in the index
Solution: Removed the references to /tendermint
* Problem: Flake8 complaining of unused import in __init__.py
Solution: The import is there so I can import App directly from bigchaindb, rather than from bigchaindb.core (which I think improves code readability. I added a # noqa to silence Flake8.
* Problem: Stale references to `bigchaindb.tendermint.BigchainDB` in the rst files cause Sphinx to fail
Solution: Updated the autodoc files to use `bigchaindb.BigchainDB` instead
* Problem: Stale reference to the `tendermint` directory in an @patch in a disabled test
Solution: Updated the @patch for completeness
* Problem: BigchainDB class should be part of project root
Solution: Removed the /tendermint directory and moved its contents to project root
* Problem: Flake8 complained that imports were not at the top of the file
Solution: Had to play around with the order of imports to avoid cyclic dependencies, but its working and style compliant now
* Problem: Stale reference to /tendermint directory in the index
Solution: Removed the references to /tendermint
* Problem: Flake8 complaining of unused import in __init__.py
Solution: The import is there so I can import App directly from bigchaindb, rather than from bigchaindb.core (which I think improves code readability. I added a # noqa to silence Flake8.
* Problem: Stale references to `bigchaindb.tendermint.BigchainDB` in the rst files cause Sphinx to fail
Solution: Updated the autodoc files to use `bigchaindb.BigchainDB` instead
* Problem: Stale reference to the `tendermint` directory in an @patch in a disabled test
Solution: Updated the @patch for completeness
* Problem: 2 implementations of fastquery exist.
Solution: Remove the old deprecated implementation. Update the tests.
* Problem: There are still 3 outdated fastquery tests.
Solution: Fix the tests.
* Problem: API returns 404 for existing empty blocks
Solution: Empty blocks are not store in MongoDB because Tendermint does
not notify BigchainDB about them. In case a user requests a block that
is not in our MongoDB, we check if the requested height is less than or
equal to latest_block_height. If that's the case, we return an empty
block. If the requested height is greater than latest_block_height, then
we return 404.
* Address Lev's comment
* address Lev and Vansh comment
* Problem: Amount error is not tested on the HTTP level.
Solution: A failed web test to reproduce the problem.
* Problem: Invalid transaction posted to Tendermint
Solution: Pass the exception to HTTP POST handler function
* Problem: DoubleSpend and CriticalDoubleSpend not differentiated
Solution: Handle these exceptions differently in `get_spent`
* Problem: No test for checking exception DoubleSpend and CriticalDoubleSpend
Solution: Add necessary tests
* Problem: find doesn't raise IndexError
Solution: Remove exception handling for IndexError
* Problem: Upsert validator not migrated to Tendermint 0.19.0
Solution: Migrate the same to Tendermint 0.19.0
* Problem: There are no tests for public key encoding/decoding
Solution: Add necessary tests
* Replace double quotes with single quotes (flake8)
* Test event subscriber to tendermint via ws
* Problem: Tendermint is not part of stack for CI
Solution: Add Tendermint to stack for CI. For simplicity's sake
docker-compose is being used.
