Merge branch 'create-dynamic-upsert-validator-commands' of github.com:z-bowen/bigchaindb into create-dynamic-upsert-validator-commands

HOW_TO_HANDLE_PULL_REQUESTS.md

@@ -1,80 +1,3 @@
-# How to Handle Pull Requests
+# How to Handle External Pull Requests
 
-This document is for whoever has the ability to merge pull requests in the Git repositories associated with BigchainDB.
+See [BEP-16](https://github.com/bigchaindb/BEPs/tree/master/16).
-
-If the pull request is from an employee of BigchainDB GmbH, then you can ignore this document.
-
-If the pull request is from someone who is _not_ an employee of BigchainDB, then:
-
-A. Have they agreed to the Individual Contributor Agreement in the past? There's a list of them in [a Google Spreadsheet that's accessible to all bigchaindb.com accounts](https://docs.google.com/spreadsheets/d/1VhekO6lgk1ZPx8dSjriucy4UinaU9pIdPQ5JXKcbD_Y/edit?usp=sharing). If yes, then you can merge the PR and ignore the rest of this document.
-
-B. Do they belong to a company or organization which agreed to the Entity Contributor Agreement in the past, and will they be contributing on behalf of that company or organization? (See the Google Spreadsheet link in A.) If yes, then you can merge the PR and ignore the rest of this document.
-
-C. Did they make a pull request to one of the bigchaindb repositories on GitHub (e.g. bigchaindb/bigchaindb)? If you're not sure, or you can't find one, then respond with an email of the form:
-
-Dear [NAME OF PERSON WHO AGREED TO THE CLA]
-
-According to the email copied below, you agreed to the BigchainDB Contributor License Agreement (CLA).
-
-Did you intend to do that? If no, then feel free to ignore this email and we'll pretend it never happened.
-
-If you did intend to do that, then do you intend to make a pull request in a BigchainDB repository? Maybe you already did? If so, can you please point me to the pull request in question?
-
-Sincerely,
-[INSERT YOUR NAME HERE]
-
-D. Otherwise, go to the pull request in question and post a comment using this template:
-
-Hi @nameofuser
-
-Before we can merge this pull request, we need you or your organization to agree to one of our contributor agreements. One of the big concerns for people using and developing open source software is that someone who contributed to the code might claim the code infringes on their copyright or patent. To guard against this, we ask all our contributors to sign a Contributor License Agreement. This gives us the right to use the code contributed and any patents the contribution relies on. It also gives us and our users comfort that they won't be sued for using open source software. We know it's a hassle, but it makes the project more reliable in the long run. Thank you for your understanding and your contribution!
-
-If you are contributing on behalf of yourself (and not on behalf of your employer or another organization you are part of) then you should:
-
-1. Go to: https://www.bigchaindb.com/cla/
-2. Read the Individual Contributor Agreement
-3. Fill in the form "For Individuals"
-4. Check the box to agree
-5. Click the SEND button
-
-If you're contributing as an employee, and/or you want all employees of your employing organization to be covered by our contributor agreement, then someone in your organization with the authority to enter agreements on behalf of all employees must do the following:
-
-1. Go to: https://www.bigchaindb.com/cla/
-2. Read the Entity Contributor Agreement
-3. Fill in the form "For Organizations”
-4. Check the box to agree
-5. Click the SEND button
-
-We will email you (or your employer) with further instructions.
-
-(END OF COMMENT)
-
-Once they click SEND, we (BigchainDB) will get an email with the information in the form. (Troy gets those emails for sure, I'm not sure who else.) The next step is to send an email to the email address submitted in the form, saying something like (where the stuff in [square brackets] should be replaced):
-
-Hi [NAME],
-
-The next step is for you to copy the following block of text into the comments of Pull Request #[NN] on GitHub:
-
-BEGIN BLOCK
-
-This is to confirm that I agreed to and accepted the BigchainDB [Entity/Individual] Contributor Agreement at https://www.bigchaindb.com/cla/ and to represent and warrant that I have authority to do so.
-
-[Insert long random string here. One good source of those is https://www.grc.com/passwords.htm ]
-
-END BLOCK
-
-(END OF EMAIL)
-
-The next step is to wait for them to copy that comment into the comments of the indicated pull request. Once they do so, it's safe to merge the pull request.
-
-## How to Handle CLA Agreement Emails with No Associated Pull Request
-
-Reply with an email like this:
-
-Hi [First Name],
-
-Today I got an email (copied below) to tell me that you agreed to the BigchainDB Contributor License Agreement. Did you intend to do that?
-
-If no, then you can ignore this email.
-
-If yes, then there's another step to connect your email address with your GitHub account. To do that, you must first create a pull request in one of the BigchainDB repositories on GitHub. Once you've done that, please reply to this email with a link to the pull request. Then I'll send you a special block of text to paste into the comments on that pull request.

README.md

@@ -2,9 +2,8 @@
 (including pre-release versions) from PyPI,
 so show the latest GitHub release instead.
 --->
-<!--- Codecov isn't working for us lately, so comment it out for now:
+
 [![Codecov branch](https://img.shields.io/codecov/c/github/bigchaindb/bigchaindb/master.svg)](https://codecov.io/github/bigchaindb/bigchaindb?branch=master)
---->
 [![Latest release](https://img.shields.io/github/release/bigchaindb/bigchaindb/all.svg)](https://github.com/bigchaindb/bigchaindb/releases)
 [![Status on PyPI](https://img.shields.io/pypi/status/bigchaindb.svg)](https://pypi.org/project/BigchainDB/)
 [![Travis branch](https://img.shields.io/travis/bigchaindb/bigchaindb/master.svg)](https://travis-ci.org/bigchaindb/bigchaindb)

bigchaindb/events.py

@@ -67,7 +67,7 @@ class Exchange:
         """
 
         try:
-            self.started_queue.get_nowait()
+            self.started_queue.get(timeout=1)
             raise RuntimeError('Cannot create a new subscriber queue while Exchange is running.')
         except Empty:
             pass

bigchaindb/lib.py

@@ -34,18 +34,6 @@ class BigchainDB(object):
     Create, read, sign, write transactions to the database
     """
 
-    BLOCK_INVALID = 'invalid'
-    """return if a block is invalid"""
-
-    BLOCK_VALID = TX_VALID = 'valid'
-    """return if a block is valid, or tx is in valid block"""
-
-    BLOCK_UNDECIDED = TX_UNDECIDED = 'undecided'
-    """return if block is undecided, or tx is in undecided block"""
-
-    TX_IN_BACKLOG = 'backlog'
-    """return if transaction is in backlog"""
-
     def __init__(self, connection=None):
         """Initialize the Bigchain instance
 
@@ -251,7 +239,7 @@ class BigchainDB(object):
             return backend.query.delete_unspent_outputs(
                                         self.connection, *unspent_outputs)
 
-    def get_transaction(self, transaction_id, include_status=False):
+    def get_transaction(self, transaction_id):
         transaction = backend.query.get_transaction(self.connection, transaction_id)
 
         if transaction:
@@ -269,9 +257,6 @@ class BigchainDB(object):
 
             transaction = Transaction.from_dict(transaction)
 
-        if include_status:
-            return transaction, self.TX_VALID if transaction else None
-        else:
         return transaction
 
     def get_transactions_filtered(self, asset_id, operation=None):
@@ -280,9 +265,7 @@ class BigchainDB(object):
         txids = backend.query.get_txids_filtered(self.connection, asset_id,
                                                  operation)
         for txid in txids:
-            tx, status = self.get_transaction(txid, True)
+            yield self.get_transaction(txid)
-            if status == self.TX_VALID:
-                yield tx
 
     def get_outputs_filtered(self, owner, spent=None):
         """Get a list of output links filtered on some criteria
@@ -421,17 +404,9 @@ class BigchainDB(object):
         Returns:
             iter: An iterator of assets that match the text search.
         """
-        objects = backend.query.text_search(self.connection, search, limit=limit,
+        return backend.query.text_search(self.connection, search, limit=limit,
                                          table=table)
 
-        # TODO: This is not efficient. There may be a more efficient way to
-        #       query by storing block ids with the assets and using fastquery.
-        #       See https://github.com/bigchaindb/bigchaindb/issues/1496
-        for obj in objects:
-            tx, status = self.get_transaction(obj['id'], True)
-            if status == self.TX_VALID:
-                yield obj
-
     def get_assets(self, asset_ids):
         """Return a list of assets that match the asset_ids
 

bigchaindb/models.py

@@ -10,15 +10,12 @@ class Transaction(Transaction):
 
     def validate(self, bigchain, current_transactions=[]):
         """Validate transaction spend
-
         Args:
             bigchain (BigchainDB): an instantiated bigchaindb.BigchainDB object.
-
         Returns:
             The transaction (Transaction) if the transaction is valid else it
             raises an exception describing the reason why the transaction is
             invalid.
-
         Raises:
             ValidationError: If the transaction is invalid
         """

bigchaindb/web/views/transactions.py

@@ -29,9 +29,9 @@ class TransactionApi(Resource):
         pool = current_app.config['bigchain_pool']
 
         with pool() as bigchain:
-            tx, status = bigchain.get_transaction(tx_id, include_status=True)
+            tx = bigchain.get_transaction(tx_id)
 
-        if not tx or status is not bigchain.TX_VALID:
+        if not tx:
             return make_error(404)
 
         return tx.to_dict()

docs/root/source/immutable.md

@@ -2,13 +2,14 @@
 
 The word _immutable_ means "unchanging over time or unable to be changed." For example, the decimal digits of π are immutable (3.14159…).
 
-The blockchain community often describes blockchains as “immutable.” If we interpret that word literally, it means that blockchain data is unchangeable or permanent, which is absurd. The data _can_ be changed. For example, a plague might drive humanity extinct; the data would then get corrupted over time due to water damage, thermal noise, and the general increase of entropy. In the case of Bitcoin, nothing so drastic is required: a 51% attack will suffice.
+The blockchain community often describes blockchains as “immutable.” If we interpret that word literally, it means that blockchain data is unchangeable or permanent, which is absurd. The data _can_ be changed. For example, a plague might drive humanity extinct; the data would then get corrupted over time due to water damage, thermal noise, and the general increase of entropy.
 
 It’s true that blockchain data is more difficult to change (or delete) than usual. It's more than just "tamper-resistant" (which implies intent), blockchain data also resists random changes that can happen without any intent, such as data corruption on a hard drive. Therefore, in the context of blockchains, we interpret the word “immutable” to mean *practically* immutable, for all intents and purposes. (Linguists would say that the word “immutable” is a _term of art_ in the blockchain community.)
 
-Blockchain data can achieve immutability in several ways:
+Blockchain data can be made immutable in several ways:
 
-1. **Replication.** All data is replicated (copied) to several different places. The replication factor can be set by the consortium. The higher the replication factor, the more difficult it becomes to change or delete all replicas.
+1. **No APIs for changing or deleting data.** Blockchain software usually doesn't expose any APIs for changing or deleting the data stored in the blockchain. BigchainDB has no such APIs. This doesn't prevent changes or deletions from happening in _other_ ways; it's just one line of defense.
+1. **Replication.** All data is replicated (copied) to several different places. The higher the replication factor, the more difficult it becomes to change or delete all replicas.
 1. **Internal watchdogs.** All nodes monitor all changes and if some unallowed change happens, then appropriate action can be taken.
 1. **External watchdogs.** A consortium may opt to have trusted third-parties to monitor and audit their data, looking for irregularities. For a consortium with publicly-readable data, the public can act as an auditor.
 1. **Economic incentives.** Some blockchain systems make it very expensive to change old stored data. Examples include proof-of-work and proof-of-stake systems. BigchainDB doesn't use explicit incentives like those.
@@ -17,5 +18,3 @@ Blockchain data can achieve immutability in several ways:
 1. **Full or partial backups** may be recorded from time to time, possibly on magnetic tape storage, other blockchains, printouts, etc.
 1. **Strong security.** Node owners can adopt and enforce strong security policies.
 1. **Node diversity.** Diversity makes it so that no one thing (e.g. natural disaster or operating system bug) can compromise enough of the nodes. See [the section on the kinds of node diversity](diversity.html).
-
-Some of these things come "for free" as part of the BigchainDB software, and others require some extra effort from the consortium and node owners.

docs/server/source/glossary.rst

@@ -1,19 +0,0 @@
-Glossary
-========
-
-.. glossary::
-   :sorted:
-
-   associative array
-      A collection of key/value (or name/value) pairs
-      such that each possible key appears at most once
-      in the collection.
-      In JavaScript (and JSON), all objects behave as associative arrays
-      with string-valued keys.
-      In Python and .NET, associative arrays are called *dictionaries*.
-      In Java and Go, they are called *maps*.
-      In Ruby, they are called *hashes*.
-      See also: Wikipedia's articles for
-      `Associative array <https://en.wikipedia.org/wiki/Associative_array>`_
-      and
-      `Comparison of programming languages (associative array) <https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(associative_array)>`_

docs/server/source/http-client-server-api.rst

@@ -301,7 +301,7 @@ Assets
 
         Currently this endpoint is only supported if using MongoDB.
 
-.. http:get:: /api/v1/assets?search={search}
+.. http:get:: /api/v1/assets/?search={search}
 
     Return all assets that match a given text search.
     
@@ -310,6 +310,10 @@ Assets
        The ``id`` of the asset
        is the same ``id`` of the CREATE transaction that created the asset.
 
+    .. note::
+    
+       You can use ``assets/?search`` or ``assets?search``.
+
     If no assets match the text search it returns an empty list.
 
     If the text string is empty or the server does not support text search,
@@ -425,6 +429,10 @@ Transaction Metadata
        The ``id`` of the metadata
        is the same ``id`` of the transaction where it was defined.
 
+    .. note::
+    
+       You can use ``metadata/?search`` or ``metadata?search``.
+
     If no metadata objects match the text search it returns an empty list.
 
     If the text string is empty or the server does not support text search,

docs/server/source/index.rst

@@ -18,5 +18,4 @@ BigchainDB Server Documentation
    data-models/index
    k8s-deployment-template/index
    release-notes
-   glossary
    appendices/index

tests/db/test_bigchain_api.py

@@ -280,7 +280,6 @@ class TestBigchainApi(object):
 
     @pytest.mark.usefixtures('inputs')
     def test_write_transaction(self, b, user_pk, user_sk):
-        from bigchaindb import BigchainDB
         from bigchaindb.models import Transaction
 
         input_tx = b.get_owned_ids(user_pk).pop()
@@ -294,7 +293,6 @@ class TestBigchainApi(object):
         tx_from_db, status = b.get_transaction(tx.id, include_status=True)
 
         assert tx_from_db.to_dict() == tx.to_dict()
-        assert status == BigchainDB.TX_IN_BACKLOG
 
     @pytest.mark.usefixtures('inputs')
     def test_read_transaction(self, b, user_pk, user_sk):
@@ -315,7 +313,6 @@ class TestBigchainApi(object):
         response, status = b.get_transaction(tx.id, include_status=True)
         # add validity information, which will be returned
         assert tx.to_dict() == response.to_dict()
-        assert status == b.TX_UNDECIDED
 
     @pytest.mark.usefixtures('inputs')
     def test_read_transaction_invalid_block(self, b, user_pk, user_sk):
@@ -342,35 +339,6 @@ class TestBigchainApi(object):
         # and a copy of the tx is not in the backlog
         assert response is None
 
-    @pytest.mark.usefixtures('inputs')
-    def test_read_transaction_invalid_block_and_backlog(self, b, user_pk, user_sk):
-        from bigchaindb.models import Transaction
-
-        input_tx = b.get_owned_ids(user_pk).pop()
-        input_tx = b.get_transaction(input_tx.txid)
-        inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_pk], 1)],
-                                  asset_id=input_tx.id)
-        tx = tx.sign([user_sk])
-
-        # Make sure there's a copy of tx in the backlog
-        b.write_transaction(tx)
-
-        # create block
-        block = b.create_block([tx])
-        b.write_block(block)
-
-        # vote the block invalid
-        vote = b.vote(block.id, b.get_last_voted_block().id, False)
-        b.write_vote(vote)
-
-        # a copy of the tx is both in the backlog and in an invalid
-        # block, so get_transaction should return a transaction,
-        # and a status of TX_IN_BACKLOG
-        response, status = b.get_transaction(tx.id, include_status=True)
-        assert tx.to_dict() == response.to_dict()
-        assert status == b.TX_IN_BACKLOG
-
     @pytest.mark.usefixtures('inputs')
     def test_genesis_block(self, b):
         from bigchaindb.backend import query

tests/test_core.py

@@ -1,6 +1,5 @@
 import pytest
 
-
 pytestmark = pytest.mark.tendermint
 
 
@@ -21,6 +20,10 @@ def config(request, monkeypatch):
             'connection_timeout': 5000,
             'max_tries': 3
         },
+        'tendermint': {
+            'host': 'localhost',
+            'port': 26657,
+        },
         'CONFIGURED': True,
     }
 
@@ -29,7 +32,6 @@ def config(request, monkeypatch):
     return config
 
 
-@pytest.mark.skipif(reason='will be fixed in another PR')
 def test_bigchain_class_default_initialization(config):
     from bigchaindb import BigchainDB
     from bigchaindb.consensus import BaseConsensusRules
@@ -42,8 +44,7 @@ def test_bigchain_class_default_initialization(config):
     assert bigchain.consensus == BaseConsensusRules
 
 
-@pytest.mark.skipif(reason='will be fixed in another PR')
+def test_bigchain_class_initialization_with_parameters():
-def test_bigchain_class_initialization_with_parameters(config):
     from bigchaindb import BigchainDB
     from bigchaindb.backend import connect
     from bigchaindb.consensus import BaseConsensusRules
@@ -54,7 +55,7 @@ def test_bigchain_class_initialization_with_parameters(config):
         'name': 'this_is_the_db_name',
     }
     connection = connect(**init_db_kwargs)
-    bigchain = BigchainDB(connection=connection, **init_db_kwargs)
+    bigchain = BigchainDB(connection=connection)
     assert bigchain.connection == connection
     assert bigchain.connection.host == init_db_kwargs['host']
     assert bigchain.connection.port == init_db_kwargs['port']
@@ -62,20 +63,6 @@ def test_bigchain_class_initialization_with_parameters(config):
     assert bigchain.consensus == BaseConsensusRules
 
 
-@pytest.mark.skipif(reason='will be fixed in another PR')
-def test_get_blocks_status_containing_tx(monkeypatch):
-    from bigchaindb.backend import query as backend_query
-    from bigchaindb import BigchainDB
-    blocks = [
-        {'id': 1}, {'id': 2}
-    ]
-    monkeypatch.setattr(backend_query, 'get_blocks_status_from_transaction', lambda x: blocks)
-    monkeypatch.setattr(BigchainDB, 'block_election_status', lambda x, y, z: BigchainDB.BLOCK_VALID)
-    bigchain = BigchainDB(public_key='pubkey', private_key='privkey')
-    with pytest.raises(Exception):
-        bigchain.get_blocks_status_containing_tx('txid')
-
-
 @pytest.mark.genesis
 def test_get_spent_issue_1271(b, alice, bob, carol):
     from bigchaindb.models import Transaction

tests/test_txlist.py

@@ -4,46 +4,27 @@ This test module defines it's own fixture which is used by all the tests.
 """
 import pytest
 
+pytestmark = pytest.mark.tendermint
+
 
 @pytest.fixture
-def txlist(b, user_pk, user2_pk, user_sk, user2_sk, genesis_block):
+def txlist(b, user_pk, user2_pk, user_sk, user2_sk):
     from bigchaindb.models import Transaction
-    prev_block_id = genesis_block.id
 
-    # Create first block with CREATE transactions
+    # Create two CREATE transactions
     create1 = Transaction.create([user_pk], [([user2_pk], 6)]) \
         .sign([user_sk])
+
     create2 = Transaction.create([user2_pk],
                                  [([user2_pk], 5), ([user_pk], 5)]) \
         .sign([user2_sk])
-    block1 = b.create_block([create1, create2])
-    b.write_block(block1)
 
-    # Create second block with TRANSFER transactions
+    # Create a TRANSFER transactions
     transfer1 = Transaction.transfer(create1.to_inputs(),
                                      [([user_pk], 8)],
                                      create1.id).sign([user2_sk])
-    block2 = b.create_block([transfer1])
-    b.write_block(block2)
 
-    # Create block with double spend
+    b.store_bulk_transactions([create1, create2, transfer1])
-    tx_doublespend = Transaction.transfer(create1.to_inputs(), [([user_pk], 9)],
-                                          create1.id).sign([user2_sk])
-    block_doublespend = b.create_block([tx_doublespend])
-    b.write_block(block_doublespend)
-
-    # Vote on all the blocks
-    prev_block_id = genesis_block.id
-    for bid in [block1.id, block2.id]:
-        vote = b.vote(bid, prev_block_id, True)
-        prev_block_id = bid
-        b.write_vote(vote)
-
-    # Create undecided block
-    untx = Transaction.create([user_pk], [([user2_pk], 7)]) \
-        .sign([user_sk])
-    block_undecided = b.create_block([untx])
-    b.write_block(block_undecided)
 
     return type('', (), {
         'create1': create1,
@@ -54,8 +35,8 @@ def txlist(b, user_pk, user2_pk, user_sk, user2_sk, genesis_block):
 @pytest.mark.bdb
 def test_get_txlist_by_asset(b, txlist):
     res = b.get_transactions_filtered(txlist.create1.id)
-    assert set(tx.id for tx in res) == set([txlist.transfer1.id,
+    assert sorted(set(tx.id for tx in res)) == sorted(
-                                            txlist.create1.id])
+        set([txlist.transfer1.id, txlist.create1.id]))
 
 
 @pytest.mark.bdb

tests/web/test_transactions.py

@@ -402,30 +402,6 @@ def test_transactions_get_list_bad(client):
         assert client.get(url).status_code == 400
 
 
-@pytest.mark.tendermint
-def test_return_only_valid_transaction(client):
-    from bigchaindb import BigchainDB
-
-    def get_transaction_patched(status):
-        def inner(self, tx_id, include_status):
-            return {}, status
-        return inner
-
-    # NOTE: `get_transaction` only returns a transaction if it's included in an
-    #       UNDECIDED or VALID block, as well as transactions from the backlog.
-    #       As the endpoint uses `get_transaction`, we don't have to test
-    #       against invalid transactions here.
-    with patch('bigchaindb.BigchainDB.get_transaction',
-               get_transaction_patched(BigchainDB.TX_UNDECIDED)):
-        url = '{}{}'.format(TX_ENDPOINT, '123')
-        assert client.get(url).status_code == 404
-
-    with patch('bigchaindb.BigchainDB.get_transaction',
-               get_transaction_patched(BigchainDB.TX_IN_BACKLOG)):
-        url = '{}{}'.format(TX_ENDPOINT, '123')
-        assert client.get(url).status_code == 404
-
-
 @pytest.mark.tendermint
 @patch('requests.post')
 @pytest.mark.parametrize('mode', [