Merge remote-tracking branch 'ssh_upstream/master' into 21/conclude-election

Dockerfile-all-in-one

@@ -21,7 +21,7 @@ RUN apk --update add sudo bash \
 RUN apk --update add mongodb monit
 
 # Install Tendermint
-RUN wget https://github.com/tendermint/tendermint/releases/download/v${TM_VERSION}-autodraft/tendermint_${TM_VERSION}_linux_amd64.zip \
+RUN wget https://github.com/tendermint/tendermint/releases/download/v${TM_VERSION}/tendermint_${TM_VERSION}_linux_amd64.zip \
     && unzip tendermint_${TM_VERSION}_linux_amd64.zip \
     && mv tendermint /usr/local/bin/ \
     && rm tendermint_${TM_VERSION}_linux_amd64.zip

Dockerfile-dev

@@ -34,4 +34,3 @@ COPY . /usr/src/app/
 WORKDIR /usr/src/app
 RUN pip install --no-cache-dir --process-dependency-links -e .[dev]
 RUN bigchaindb -y configure
-

HOW_TO_HANDLE_PULL_REQUESTS.md

@@ -1,80 +1,3 @@
-# How to Handle Pull Requests
+# How to Handle External Pull Requests
 
-This document is for whoever has the ability to merge pull requests in the Git repositories associated with BigchainDB.
+See [BEP-16](https://github.com/bigchaindb/BEPs/tree/master/16).
-
-If the pull request is from an employee of BigchainDB GmbH, then you can ignore this document.
-
-If the pull request is from someone who is _not_ an employee of BigchainDB, then:
-
-A. Have they agreed to the Individual Contributor Agreement in the past? There's a list of them in [a Google Spreadsheet that's accessible to all bigchaindb.com accounts](https://docs.google.com/spreadsheets/d/1VhekO6lgk1ZPx8dSjriucy4UinaU9pIdPQ5JXKcbD_Y/edit?usp=sharing). If yes, then you can merge the PR and ignore the rest of this document.
-
-B. Do they belong to a company or organization which agreed to the Entity Contributor Agreement in the past, and will they be contributing on behalf of that company or organization? (See the Google Spreadsheet link in A.) If yes, then you can merge the PR and ignore the rest of this document.
-
-C. Did they make a pull request to one of the bigchaindb repositories on GitHub (e.g. bigchaindb/bigchaindb)? If you're not sure, or you can't find one, then respond with an email of the form:
-
-Dear [NAME OF PERSON WHO AGREED TO THE CLA]
-
-According to the email copied below, you agreed to the BigchainDB Contributor License Agreement (CLA).
-
-Did you intend to do that? If no, then feel free to ignore this email and we'll pretend it never happened.
-
-If you did intend to do that, then do you intend to make a pull request in a BigchainDB repository? Maybe you already did? If so, can you please point me to the pull request in question?
-
-Sincerely,
-[INSERT YOUR NAME HERE]
-
-D. Otherwise, go to the pull request in question and post a comment using this template:
-
-Hi @nameofuser
-
-Before we can merge this pull request, we need you or your organization to agree to one of our contributor agreements. One of the big concerns for people using and developing open source software is that someone who contributed to the code might claim the code infringes on their copyright or patent. To guard against this, we ask all our contributors to sign a Contributor License Agreement. This gives us the right to use the code contributed and any patents the contribution relies on. It also gives us and our users comfort that they won't be sued for using open source software. We know it's a hassle, but it makes the project more reliable in the long run. Thank you for your understanding and your contribution!
-
-If you are contributing on behalf of yourself (and not on behalf of your employer or another organization you are part of) then you should:
-
-1. Go to: https://www.bigchaindb.com/cla/
-2. Read the Individual Contributor Agreement
-3. Fill in the form "For Individuals"
-4. Check the box to agree
-5. Click the SEND button
-
-If you're contributing as an employee, and/or you want all employees of your employing organization to be covered by our contributor agreement, then someone in your organization with the authority to enter agreements on behalf of all employees must do the following:
-
-1. Go to: https://www.bigchaindb.com/cla/
-2. Read the Entity Contributor Agreement
-3. Fill in the form "For Organizations”
-4. Check the box to agree
-5. Click the SEND button
-
-We will email you (or your employer) with further instructions.
-
-(END OF COMMENT)
-
-Once they click SEND, we (BigchainDB) will get an email with the information in the form. (Troy gets those emails for sure, I'm not sure who else.) The next step is to send an email to the email address submitted in the form, saying something like (where the stuff in [square brackets] should be replaced):
-
-Hi [NAME],
-
-The next step is for you to copy the following block of text into the comments of Pull Request #[NN] on GitHub:
-
-BEGIN BLOCK
-
-This is to confirm that I agreed to and accepted the BigchainDB [Entity/Individual] Contributor Agreement at https://www.bigchaindb.com/cla/ and to represent and warrant that I have authority to do so.
-
-[Insert long random string here. One good source of those is https://www.grc.com/passwords.htm ]
-
-END BLOCK
-
-(END OF EMAIL)
-
-The next step is to wait for them to copy that comment into the comments of the indicated pull request. Once they do so, it's safe to merge the pull request.
-
-## How to Handle CLA Agreement Emails with No Associated Pull Request
-
-Reply with an email like this:
-
-Hi [First Name],
-
-Today I got an email (copied below) to tell me that you agreed to the BigchainDB Contributor License Agreement. Did you intend to do that?
-
-If no, then you can ignore this email.
-
-If yes, then there's another step to connect your email address with your GitHub account. To do that, you must first create a pull request in one of the BigchainDB repositories on GitHub. Once you've done that, please reply to this email with a link to the pull request. Then I'll send you a special block of text to paste into the comments on that pull request.

README.md

@@ -2,9 +2,8 @@
 (including pre-release versions) from PyPI,
 so show the latest GitHub release instead.
 --->
-<!--- Codecov isn't working for us lately, so comment it out for now:
+
 [![Codecov branch](https://img.shields.io/codecov/c/github/bigchaindb/bigchaindb/master.svg)](https://codecov.io/github/bigchaindb/bigchaindb?branch=master)
---->
 [![Latest release](https://img.shields.io/github/release/bigchaindb/bigchaindb/all.svg)](https://github.com/bigchaindb/bigchaindb/releases)
 [![Status on PyPI](https://img.shields.io/pypi/status/bigchaindb.svg)](https://pypi.org/project/BigchainDB/)
 [![Travis branch](https://img.shields.io/travis/bigchaindb/bigchaindb/master.svg)](https://travis-ci.org/bigchaindb/bigchaindb)

bigchaindb/commands/bigchaindb.py

@@ -9,19 +9,19 @@ import copy
 import json
 import sys
 
+from bigchaindb.utils import load_node_key
 from bigchaindb.common.exceptions import (DatabaseAlreadyExists,
                                           DatabaseDoesNotExist,
-                                          MultipleValidatorOperationError)
+                                          OperationError)
 import bigchaindb
-from bigchaindb import backend
+from bigchaindb import backend, ValidatorElection, BigchainDB
 from bigchaindb.backend import schema
 from bigchaindb.backend import query
-from bigchaindb.backend.query import VALIDATOR_UPDATE_ID, PRE_COMMIT_ID
+from bigchaindb.backend.query import PRE_COMMIT_ID
 from bigchaindb.commands import utils
 from bigchaindb.commands.utils import (configure_bigchaindb,
                                        input_on_stderr)
 from bigchaindb.log import setup_logging
-from bigchaindb.tendermint_utils import public_key_from_base64
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
@@ -95,21 +95,48 @@ def run_configure(args):
 
 @configure_bigchaindb
 def run_upsert_validator(args):
-    """Store validators which should be synced with Tendermint"""
+    """Initiate and manage elections to change the validator set"""
 
-    b = bigchaindb.BigchainDB()
+    b = BigchainDB()
-    public_key = public_key_from_base64(args.public_key)
+
-    validator = {'pub_key': {'type': 'ed25519',
+    # Call the function specified by args.action, as defined above
-                             'data': public_key},
+    globals()[f'run_upsert_validator_{args.action}'](args, b)
-                 'power': args.power}
+
-    validator_update = {'validator': validator,
+
-                        'update_id': VALIDATOR_UPDATE_ID}
+def run_upsert_validator_new(args, bigchain):
-    try:
+    """Initiates an election to add/update/remove a validator to an existing BigchainDB network
-        query.store_validator_update(b.connection, validator_update)
+
-    except MultipleValidatorOperationError:
+    :param args: dict
-        logger.error('A validator update is pending to be applied. '
+        args = {
-                     'Please re-try after the current update has '
+        'public_key': the public key of the proposed peer, (str)
-                     'been processed.')
+        'power': the proposed validator power for the new peer, (str)
+        'node_id': the node_id of the new peer (str)
+        'sk': the path to the private key of the node calling the election (str)
+        }
+    :param bigchain: an instance of BigchainDB
+    :return: election_id (tx_id)
+    :raises: OperationError if the write transaction fails for any reason
+    """
+
+    new_validator = {
+        'public_key': args.public_key,
+        'power': args.power,
+        'node_id': args.node_id
+    }
+
+    key = load_node_key(args.sk)
+
+    voters = ValidatorElection.recipients(bigchain)
+
+    election = ValidatorElection.generate([key.public_key],
+                                          voters,
+                                          new_validator, None).sign([key.private_key])
+    election.validate(bigchain)
+    resp = bigchain.write_transaction(election, 'broadcast_tx_commit')
+    if resp == (202, ''):
+        return election.id
+    else:
+        raise OperationError('Failed to commit election')
 
 
 def _run_init():
@@ -208,16 +235,30 @@ def create_parser():
                                help='The backend to use. It can only be '
                                '"localmongodb", currently.')
 
+    # parser for managing validator elections
     validator_parser = subparsers.add_parser('upsert-validator',
-                                             help='Add/update/delete a validator')
+                                             help='Add/update/delete a validator.')
 
-    validator_parser.add_argument('public_key',
+    validator_subparser = validator_parser.add_subparsers(title='Action',
-                                  help='Public key of the validator.')
+                                                          dest='action')
 
-    validator_parser.add_argument('power',
+    new_election_parser = validator_subparser.add_parser('new',
+                                                         help='Calls a new election.')
+
+    new_election_parser.add_argument('public_key',
+                                     help='Public key of the validator to be added/updated/removed.')
+
+    new_election_parser.add_argument('power',
                                      type=int,
-                                  help='Voting power of the validator. '
+                                     help='The proposed power for the validator. '
-                                  'Setting it to 0 will delete the validator.')
+                                          'Setting to 0 will remove the validator.')
+
+    new_election_parser.add_argument('node_id',
+                                     help='The node_id of the validator.')
+
+    new_election_parser.add_argument('--private-key',
+                                     dest='sk',
+                                     help='Path to the private key of the election initiator.')
 
     # parsers for showing/exporting config values
     subparsers.add_parser('show-config',

bigchaindb/events.py

@@ -67,7 +67,7 @@ class Exchange:
         """
 
         try:
-            self.started_queue.get_nowait()
+            self.started_queue.get(timeout=1)
             raise RuntimeError('Cannot create a new subscriber queue while Exchange is running.')
         except Empty:
             pass

bigchaindb/lib.py

@@ -36,18 +36,6 @@ class BigchainDB(object):
     Create, read, sign, write transactions to the database
     """
 
-    BLOCK_INVALID = 'invalid'
-    """return if a block is invalid"""
-
-    BLOCK_VALID = TX_VALID = 'valid'
-    """return if a block is valid, or tx is in valid block"""
-
-    BLOCK_UNDECIDED = TX_UNDECIDED = 'undecided'
-    """return if block is undecided, or tx is in undecided block"""
-
-    TX_IN_BACKLOG = 'backlog'
-    """return if transaction is in backlog"""
-
     def __init__(self, connection=None):
         """Initialize the Bigchain instance
 
@@ -253,7 +241,7 @@ class BigchainDB(object):
             return backend.query.delete_unspent_outputs(
                                         self.connection, *unspent_outputs)
 
-    def get_transaction(self, transaction_id, include_status=False):
+    def get_transaction(self, transaction_id):
         transaction = backend.query.get_transaction(self.connection, transaction_id)
 
         if transaction:
@@ -271,9 +259,6 @@ class BigchainDB(object):
 
             transaction = Transaction.from_dict(transaction)
 
-        if include_status:
-            return transaction, self.TX_VALID if transaction else None
-        else:
         return transaction
 
     def get_transactions_filtered(self, asset_id, operation=None):
@@ -282,9 +267,7 @@ class BigchainDB(object):
         txids = backend.query.get_txids_filtered(self.connection, asset_id,
                                                  operation)
         for txid in txids:
-            tx, status = self.get_transaction(txid, True)
+            yield self.get_transaction(txid)
-            if status == self.TX_VALID:
-                yield tx
 
     def get_outputs_filtered(self, owner, spent=None):
         """Get a list of output links filtered on some criteria
@@ -423,17 +406,9 @@ class BigchainDB(object):
         Returns:
             iter: An iterator of assets that match the text search.
         """
-        objects = backend.query.text_search(self.connection, search, limit=limit,
+        return backend.query.text_search(self.connection, search, limit=limit,
                                          table=table)
 
-        # TODO: This is not efficient. There may be a more efficient way to
-        #       query by storing block ids with the assets and using fastquery.
-        #       See https://github.com/bigchaindb/bigchaindb/issues/1496
-        for obj in objects:
-            tx, status = self.get_transaction(obj['id'], True)
-            if status == self.TX_VALID:
-                yield obj
-
     def get_assets(self, asset_ids):
         """Return a list of assets that match the asset_ids
 

bigchaindb/models.py

@@ -10,15 +10,12 @@ class Transaction(Transaction):
 
     def validate(self, bigchain, current_transactions=[]):
         """Validate transaction spend
-
         Args:
             bigchain (BigchainDB): an instantiated bigchaindb.BigchainDB object.
-
         Returns:
             The transaction (Transaction) if the transaction is valid else it
             raises an exception describing the reason why the transaction is
             invalid.
-
         Raises:
             ValidationError: If the transaction is invalid
         """

bigchaindb/utils.py

@@ -2,9 +2,13 @@ import contextlib
 import threading
 import queue
 import multiprocessing as mp
+import json
 
 import setproctitle
 
+from bigchaindb.tendermint_utils import key_from_base64
+from bigchaindb.common.crypto import key_pair_from_ed25519_key
+
 
 class ProcessGroup(object):
 
@@ -31,7 +35,8 @@ class ProcessGroup(object):
 class Process(mp.Process):
     """Wrapper around multiprocessing.Process that uses
     setproctitle to set the name of the process when running
-    the target task."""
+    the target task.
+    """
 
     def run(self):
         setproctitle.setproctitle(self.name)
@@ -167,3 +172,12 @@ class Lazy:
 
         self.stack = []
         return last
+
+
+# Load Tendermint's public and private key from the file path
+def load_node_key(path):
+    with open(path) as json_data:
+        priv_validator = json.load(json_data)
+        priv_key = priv_validator['priv_key']['value']
+        hex_private_key = key_from_base64(priv_key)
+        return key_pair_from_ed25519_key(hex_private_key)

bigchaindb/web/views/transactions.py

@@ -29,9 +29,9 @@ class TransactionApi(Resource):
         pool = current_app.config['bigchain_pool']
 
         with pool() as bigchain:
-            tx, status = bigchain.get_transaction(tx_id, include_status=True)
+            tx = bigchain.get_transaction(tx_id)
 
-        if not tx or status is not bigchain.TX_VALID:
+        if not tx:
             return make_error(404)
 
         return tx.to_dict()

docs/contributing/source/dev-setup-coding-and-contribution-process/run-node-as-processes.md

@@ -23,7 +23,7 @@ Find [the version number of the latest Tendermint release](https://github.com/te
 
 ```bash
 $ sudo apt install -y unzip
-$ wget https://github.com/tendermint/tendermint/releases/download/v0.22.8-autodraft/tendermint_0.22.8_linux_amd64.zip
+$ wget https://github.com/tendermint/tendermint/releases/download/v0.22.8/tendermint_0.22.8_linux_amd64.zip
 $ unzip tendermint_0.22.8_linux_amd64.zip
 $ rm tendermint_0.22.8_linux_amd64.zip
 $ sudo mv tendermint /usr/local/bin

docs/root/source/immutable.md

@@ -2,13 +2,14 @@
 
 The word _immutable_ means "unchanging over time or unable to be changed." For example, the decimal digits of π are immutable (3.14159…).
 
-The blockchain community often describes blockchains as “immutable.” If we interpret that word literally, it means that blockchain data is unchangeable or permanent, which is absurd. The data _can_ be changed. For example, a plague might drive humanity extinct; the data would then get corrupted over time due to water damage, thermal noise, and the general increase of entropy. In the case of Bitcoin, nothing so drastic is required: a 51% attack will suffice.
+The blockchain community often describes blockchains as “immutable.” If we interpret that word literally, it means that blockchain data is unchangeable or permanent, which is absurd. The data _can_ be changed. For example, a plague might drive humanity extinct; the data would then get corrupted over time due to water damage, thermal noise, and the general increase of entropy.
 
 It’s true that blockchain data is more difficult to change (or delete) than usual. It's more than just "tamper-resistant" (which implies intent), blockchain data also resists random changes that can happen without any intent, such as data corruption on a hard drive. Therefore, in the context of blockchains, we interpret the word “immutable” to mean *practically* immutable, for all intents and purposes. (Linguists would say that the word “immutable” is a _term of art_ in the blockchain community.)
 
-Blockchain data can achieve immutability in several ways:
+Blockchain data can be made immutable in several ways:
 
-1. **Replication.** All data is replicated (copied) to several different places. The replication factor can be set by the consortium. The higher the replication factor, the more difficult it becomes to change or delete all replicas.
+1. **No APIs for changing or deleting data.** Blockchain software usually doesn't expose any APIs for changing or deleting the data stored in the blockchain. BigchainDB has no such APIs. This doesn't prevent changes or deletions from happening in _other_ ways; it's just one line of defense.
+1. **Replication.** All data is replicated (copied) to several different places. The higher the replication factor, the more difficult it becomes to change or delete all replicas.
 1. **Internal watchdogs.** All nodes monitor all changes and if some unallowed change happens, then appropriate action can be taken.
 1. **External watchdogs.** A consortium may opt to have trusted third-parties to monitor and audit their data, looking for irregularities. For a consortium with publicly-readable data, the public can act as an auditor.
 1. **Economic incentives.** Some blockchain systems make it very expensive to change old stored data. Examples include proof-of-work and proof-of-stake systems. BigchainDB doesn't use explicit incentives like those.
@@ -17,5 +18,3 @@ Blockchain data can achieve immutability in several ways:
 1. **Full or partial backups** may be recorded from time to time, possibly on magnetic tape storage, other blockchains, printouts, etc.
 1. **Strong security.** Node owners can adopt and enforce strong security policies.
 1. **Node diversity.** Diversity makes it so that no one thing (e.g. natural disaster or operating system bug) can compromise enough of the nodes. See [the section on the kinds of node diversity](diversity.html).
-
-Some of these things come "for free" as part of the BigchainDB software, and others require some extra effort from the consortium and node owners.

docs/server/source/appendices/all-in-one-bigchaindb.md

@@ -15,7 +15,7 @@ This image contains all the services required for a BigchainDB node i.e.
 *This image is to help quick deployment for early adopters, for a more standard approach please refer to one of our deployment guides:*
 
 - [BigchainDB developer setup guides](https://docs.bigchaindb.com/projects/contributing/en/latest/dev-setup-coding-and-contribution-process/index.html).
-- [BigchainDB with Kubernetes](http://docs.bigchaindb.com/projects/server/en/latest/production-deployment-template/index.html).
+- [BigchainDB with Kubernetes](http://docs.bigchaindb.com/projects/server/en/latest/k8s-deployment-template/index.html).
 
 ## Prerequisite(s)
 - [Docker](https://docs.docker.com/engine/installation/)

docs/server/source/clusters.md

@@ -2,7 +2,6 @@
 
 A **BigchainDB Cluster** is a set of connected **BigchainDB Nodes**, managed by a **BigchainDB Consortium** (i.e. an organization). Those terms are defined in the [BigchainDB Terminology page](https://docs.bigchaindb.com/en/latest/terminology.html).
 
-
 ## Consortium Structure & Governance
 
 The consortium might be a company, a foundation, a cooperative, or [some other form of organization](https://en.wikipedia.org/wiki/Organizational_structure).
@@ -13,13 +12,6 @@ This documentation doesn't explain how to create a consortium, nor does it outli
 It's worth noting that the decentralization of a BigchainDB cluster depends,
 to some extent, on the decentralization of the associated consortium. See the pages about [decentralization](https://docs.bigchaindb.com/en/latest/decentralized.html) and [node diversity](https://docs.bigchaindb.com/en/latest/diversity.html).
 
-
-## Relevant Technical Documentation
-
-Anyone building or managing a BigchainDB cluster may be interested
-in [our production deployment template](production-deployment-template/index.html).
-
-
 ## Cluster DNS Records and SSL Certificates
 
 We now describe how *we* set up the external (public-facing) DNS records for a BigchainDB cluster. Your consortium may opt to do it differently.
@@ -30,14 +22,12 @@ There were several goals:
 * There should be no sharing of SSL certificates among BigchainDB node operators.
 * Optional: Allow clients to connect to a "random" BigchainDB node in the cluster at one particular domain (or subdomain).
 
-
 ### Node Operator Responsibilities
 
 1. Register a domain (or use one that you already have) for your BigchainDB node. You can use a subdomain if you like. For example, you might opt to use `abc-org73.net`, `api.dynabob8.io` or `figmentdb3.ninja`.
 2. Get an SSL certificate for your domain or subdomain, and properly install it in your node (e.g. in your NGINX instance).
 3. Create a DNS A Record mapping your domain or subdomain to the public IP address of your node (i.e. the one that serves the BigchainDB HTTP API).
 
-
 ### Consortium Responsibilities
 
 Optional: The consortium managing the BigchainDB cluster could register a domain name and set up CNAME records mapping that domain name (or one of its subdomains) to each of the nodes in the cluster. For example, if the consortium registered `bdbcluster.io`, they could set up CNAME records like the following:

docs/server/source/drivers-clients/index.rst

@@ -6,17 +6,18 @@ Libraries and Tools Maintained by the BigchainDB Team
 
 * `Python Driver <https://docs.bigchaindb.com/projects/py-driver/en/latest/index.html>`_
 * `JavaScript / Node.js Driver <https://github.com/bigchaindb/js-bigchaindb-driver>`_
+* `Java driver <https://github.com/bigchaindb/java-bigchaindb-driver>`_
 
 Community-Driven Libraries and Tools
 ------------------------------------
 
-.. note::
+.. warning::
 
    Some of these projects are a work in progress,
    but may still be useful.
+   Others might not work with the latest version of BigchainDB.
 
 * `Haskell transaction builder <https://github.com/bigchaindb/bigchaindb-hs>`_
 * `Go driver <https://github.com/zbo14/envoke/blob/master/bigchain/bigchain.go>`_
-* `Java driver <https://github.com/authenteq/java-bigchaindb-driver>`_
 * `Ruby driver <https://github.com/LicenseRocks/bigchaindb_ruby>`_
 * `Ruby library for preparing/signing transactions and submitting them or querying a BigchainDB node (MIT licensed) <https://rubygems.org/gems/bigchaindb>`_

docs/server/source/glossary.rst

@@ -1,19 +0,0 @@
-Glossary
-========
-
-.. glossary::
-   :sorted:
-
-   associative array
-      A collection of key/value (or name/value) pairs
-      such that each possible key appears at most once
-      in the collection.
-      In JavaScript (and JSON), all objects behave as associative arrays
-      with string-valued keys.
-      In Python and .NET, associative arrays are called *dictionaries*.
-      In Java and Go, they are called *maps*.
-      In Ruby, they are called *hashes*.
-      See also: Wikipedia's articles for
-      `Associative array <https://en.wikipedia.org/wiki/Associative_array>`_
-      and
-      `Comparison of programming languages (associative array) <https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(associative_array)>`_

docs/server/source/http-client-server-api.rst

@@ -301,7 +301,7 @@ Assets
 
         Currently this endpoint is only supported if using MongoDB.
 
-.. http:get:: /api/v1/assets?search={search}
+.. http:get:: /api/v1/assets/?search={search}
 
     Return all assets that match a given text search.
     
@@ -310,6 +310,10 @@ Assets
        The ``id`` of the asset
        is the same ``id`` of the CREATE transaction that created the asset.
 
+    .. note::
+    
+       You can use ``assets/?search`` or ``assets?search``.
+
     If no assets match the text search it returns an empty list.
 
     If the text string is empty or the server does not support text search,
@@ -425,6 +429,10 @@ Transaction Metadata
        The ``id`` of the metadata
        is the same ``id`` of the transaction where it was defined.
 
+    .. note::
+    
+       You can use ``metadata/?search`` or ``metadata?search``.
+
     If no metadata objects match the text search it returns an empty list.
 
     If the text string is empty or the server does not support text search,

docs/server/source/index.rst

@@ -10,13 +10,12 @@ BigchainDB Server Documentation
    simple-network-setup
    production-nodes/index
    clusters
-   production-deployment-template/index
    dev-and-test/index
    server-reference/index
    http-client-server-api
    events/index
    drivers-clients/index
    data-models/index
+   k8s-deployment-template/index
    release-notes
-   glossary
    appendices/index

docs/server/source/k8s-deployment-template/architecture.rst

@@ -1,13 +1,25 @@
-Architecture of a BigchainDB Node
+Architecture of a BigchainDB Node Running in a Kubernetes Cluster
-==================================
+=================================================================
 
-A BigchainDB Production deployment is hosted on a Kubernetes cluster and includes:
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
+If you deploy a BigchainDB node into a Kubernetes cluster
+as described in these docs, it will include:
 
 * NGINX, OpenResty, BigchainDB, MongoDB and Tendermint
   `Kubernetes Services <https://kubernetes.io/docs/concepts/services-networking/service/>`_.
-* NGINX, OpenResty, BigchainDB and MongoDB Monitoring Agent.
+* NGINX, OpenResty, BigchainDB and MongoDB Monitoring Agent
   `Kubernetes Deployments <https://kubernetes.io/docs/concepts/workloads/controllers/deployment/>`_.
-* MongoDB and Tendermint `Kubernetes StatefulSet <https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/>`_.
+* MongoDB and Tendermint `Kubernetes StatefulSets <https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/>`_.
 * Third party services like `3scale <https://3scale.net>`_,
   `MongoDB Cloud Manager <https://cloud.mongodb.com>`_ and the
   `Azure Operations Management Suite

docs/server/source/k8s-deployment-template/bigchaindb-network-on-kubernetes.rst

@@ -3,6 +3,17 @@
 Kubernetes Template: Deploying a BigchainDB network
 ===================================================
 
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
 This page describes how to deploy a static BigchainDB + Tendermint network.
 
 If you want to deploy a stand-alone BigchainDB node in a BigchainDB cluster,

docs/server/source/k8s-deployment-template/cloud-manager.rst

@@ -41,7 +41,7 @@ Configure MongoDB Cloud Manager for Monitoring
     
   * If you have authentication enabled, select the option to enable
     authentication and specify the authentication mechanism as per your
-    deployment. The default BigchainDB production deployment currently
+    deployment. The default BigchainDB Kubernetes deployment template currently
     supports ``X.509 Client Certificate`` as the authentication mechanism.
     
   * If you have TLS enabled, select the option to enable TLS/SSL for MongoDB

docs/server/source/k8s-deployment-template/index.rst

@@ -0,0 +1,40 @@
+Kubernetes Deployment Template
+==============================
+
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
+This section outlines a way to deploy a BigchainDB node (or BigchainDB cluster)
+on Microsoft Azure using Kubernetes.
+You may choose to use it as a template or reference for your own deployment,
+but *we make no claim that it is suitable for your purposes*.
+Feel free change things to suit your needs or preferences.
+
+
+.. toctree::
+   :maxdepth: 1
+
+   workflow
+   ca-installation
+   server-tls-certificate
+   client-tls-certificate
+   revoke-tls-certificate
+   template-kubernetes-azure
+   node-on-kubernetes
+   node-config-map-and-secrets
+   log-analytics
+   cloud-manager
+   easy-rsa
+   upgrade-on-kubernetes
+   bigchaindb-network-on-kubernetes
+   tectonic-azure
+   troubleshoot
+   architecture

docs/server/source/k8s-deployment-template/node-config-map-and-secrets.rst

@@ -3,6 +3,17 @@
 How to Configure a BigchainDB Node
 ==================================
 
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
 This page outlines the steps to set a bunch of configuration settings
 in your BigchainDB node.
 They are pushed to the Kubernetes cluster in two files,

docs/server/source/k8s-deployment-template/node-on-kubernetes.rst

@@ -3,7 +3,18 @@
 Kubernetes Template: Deploy a Single BigchainDB Node
 ====================================================
 
-This page describes how to deploy a BigchainDB + Tendermint node
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
+This page describes how to deploy a BigchainDB node
 using `Kubernetes <https://kubernetes.io/>`_.
 It assumes you already have a running Kubernetes cluster.
 
@@ -29,7 +40,7 @@ If you don't have that file, then you need to get it.
 
 **Azure.** If you deployed your Kubernetes cluster on Azure
 using the Azure CLI 2.0 (as per :doc:`our template
-<../production-deployment-template/template-kubernetes-azure>`),
+<../k8s-deployment-template/template-kubernetes-azure>`),
 then you can get the ``~/.kube/config`` file using:
 
 .. code:: bash
@@ -277,7 +288,7 @@ The first thing to do is create the Kubernetes storage classes.
 First, you need an Azure storage account.
 If you deployed your Kubernetes cluster on Azure
 using the Azure CLI 2.0
-(as per :doc:`our template <../production-deployment-template/template-kubernetes-azure>`),
+(as per :doc:`our template <../k8s-deployment-template/template-kubernetes-azure>`),
 then the `az acs create` command already created a
 storage account in the same location and resource group
 as your Kubernetes cluster.
@@ -289,7 +300,7 @@ in the same data center.
 Premium storage is higher-cost and higher-performance.
 It uses solid state drives (SSD).
 
-We recommend using Premium storage for our production template.
+We recommend using Premium storage with our Kubernetes deployment template.
 Create a `storage account <https://docs.microsoft.com/en-us/azure/storage/common/storage-create-storage-account>`_
 for Premium storage and associate it with your Azure resource group.
 For future reference, the command to create a storage account is
@@ -372,7 +383,7 @@ but it should become "Bound" fairly quickly.
         $ kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
 
     For notes on recreating a private volume form a released Azure disk resource consult
-    :doc:`the page about cluster troubleshooting <../production-deployment-template/troubleshoot>`.
+    :doc:`the page about cluster troubleshooting <../k8s-deployment-template/troubleshoot>`.
 
 .. _start-kubernetes-stateful-set-mongodb:
 
@@ -569,7 +580,7 @@ Step 19(Optional): Configure the MongoDB Cloud Manager
 ------------------------------------------------------
 
 Refer to the
-:doc:`documentation <../production-deployment-template/cloud-manager>`
+:doc:`documentation <../k8s-deployment-template/cloud-manager>`
 for details on how to configure the MongoDB Cloud Manager to enable
 monitoring and backup.
 
@@ -749,4 +760,4 @@ verify that your node or cluster works as expected.
 
 Next, you can set up log analytics and monitoring, by following our templates:
 
-* :doc:`../production-deployment-template/log-analytics`.
+* :doc:`../k8s-deployment-template/log-analytics`.

docs/server/source/k8s-deployment-template/tectonic-azure.rst

@@ -1,6 +1,17 @@
 Walkthrough: Deploy a Kubernetes Cluster on Azure using Tectonic by CoreOS
 ==========================================================================
 
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
 A BigchainDB node can be run inside a `Kubernetes <https://kubernetes.io/>`_
 cluster.
 This page describes one way to deploy a Kubernetes cluster on Azure using Tectonic.

docs/server/source/k8s-deployment-template/template-kubernetes-azure.rst

@@ -1,6 +1,17 @@
 Template: Deploy a Kubernetes Cluster on Azure
 ==============================================
 
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
 A BigchainDB node can be run inside a `Kubernetes <https://kubernetes.io/>`_
 cluster.
 This page describes one way to deploy a Kubernetes cluster on Azure.

docs/server/source/k8s-deployment-template/upgrade-on-kubernetes.rst

@@ -1,6 +1,17 @@
 Kubernetes Template: Upgrade all Software in a BigchainDB Node
 ==============================================================
 
+.. note::
+
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
 This page outlines how to upgrade all the software associated
 with a BigchainDB node running on Kubernetes,
 including host operating systems, Docker, Kubernetes,

docs/server/source/k8s-deployment-template/workflow.rst

@@ -3,9 +3,19 @@
 Overview
 ========
 
-This page summarizes the steps *we* go through
+.. note::
-to set up a production BigchainDB cluster.
+
-We are constantly improving them.
+   A highly-available Kubernetes cluster requires at least five virtual machines
+   (three for the master and two for your app's containers).
+   Therefore we don't recommend using Kubernetes to run a BigchainDB node
+   if that's the only thing the Kubernetes cluster will be running.
+   Instead, see **How to Set Up a BigchainDB Network**.
+   If your organization already *has* a big Kubernetes cluster running many containers,
+   and your organization has people who know Kubernetes,
+   then this Kubernetes deployment template might be helpful.
+
+This page summarizes some steps to go through
+to set up a BigchainDB cluster.
 You can modify them to suit your needs.
 
 .. _generate-the-blockchain-id-and-genesis-time:
@@ -113,13 +123,13 @@ and set it equal to your secret token, e.g.
 
 
 3. Deploy a Kubernetes cluster for your BigchainDB node. We have some instructions for how to
-:doc:`Deploy a Kubernetes cluster on Azure <../production-deployment-template/template-kubernetes-azure>`.
+:doc:`Deploy a Kubernetes cluster on Azure <../k8s-deployment-template/template-kubernetes-azure>`.
 
 .. warning::
 
    In theory, you can deploy your BigchainDB node to any Kubernetes cluster, but there can be differences
    between different Kubernetes clusters, especially if they are running different versions of Kubernetes.
-   We tested this Production Deployment Template on Azure ACS in February 2018 and at that time
+   We tested this Kubernetes Deployment Template on Azure ACS in February 2018 and at that time
    ACS was deploying a **Kubernetes 1.7.7** cluster. If you can force your cluster to have that version of Kubernetes,
    then you'll increase the likelihood that everything will work in your cluster.
 

docs/server/source/production-deployment-template/index.rst

@@ -1,31 +0,0 @@
-Production Deployment Template
-==============================
-
-This section outlines how *we* deploy production BigchainDB,
-integrated with Tendermint(backend for BFT consensus),
-clusters on Microsoft Azure using
-Kubernetes. We improve it constantly.
-You may choose to use it as a template or reference for your own deployment,
-but *we make no claim that it is suitable for your purposes*.
-Feel free change things to suit your needs or preferences.
-
-
-.. toctree::
-   :maxdepth: 1
-
-   workflow
-   ca-installation
-   server-tls-certificate
-   client-tls-certificate
-   revoke-tls-certificate
-   template-kubernetes-azure
-   node-on-kubernetes
-   node-config-map-and-secrets
-   log-analytics
-   cloud-manager
-   easy-rsa
-   upgrade-on-kubernetes
-   bigchaindb-network-on-kubernetes
-   tectonic-azure
-   troubleshoot
-   architecture

docs/server/source/server-reference/bigchaindb-cli.md

@@ -80,22 +80,27 @@ configuration file as documented under
 **This is an experimental feature. Users are advised not to use it in production.**
 
 
-Add, update, or remove a validator from the validators set of the local node. The command implements [3/UPSERT-VALIDATORS](https://github.com/bigchaindb/BEPs/tree/master/3), check it out if you need more details on how this is orchestrated.
+Manage elections to add, update, or remove a validator from the validators set of the local node. The upsert-validator subcommands implement [BEP-21](https://github.com/bigchaindb/BEPs/tree/master/21). Check it out if you need more details on how this is orchestrated.
 
-Below is the command line syntax,
+Election management is broken into several subcommands. Below is the command line syntax for each,
+
+#### upsert-validator new
+
+Calls a new election, proposing a change to the validator set.
+
+Below is the command line syntax and the return value,
 
 ```bash
-$ bigchaindb upsert-validator PUBLIC_KEY_OF_VALIDATOR POWER
+$ bigchaindb upsert-validator new E_PUBKEY E_POWER E_NODE_ID --private-key PATH_TO_YOUR_PRIVATE_KEY
+<election_id>
 ```
 
+Here, `E_PUBKEY`, `E_POWER`, and `E_NODE_ID` are the public key, proposed power, and node id of the validator being voted on. `--private-key` should be the path to wherever the private key for your validator node is stored, (*not* the private key itself.). For example, to add a new validator, provide the public key and node id for some node not already in the validator set, along with whatever voting power you'd like them to have. To remove an existing validator, provide their public key and node id, and set `E_POWER` to `0`.
+
 Example usage,
 
 ```bash
-$ bigchaindb upsert-validator B0E42D2589A455EAD339A035D6CE1C8C3E25863F268120AA0162AD7D003A4014 10
+$ bigchaindb upsert-validator new B0E42D2589A455EAD339A035D6CE1C8C3E25863F268120AA0162AD7D003A4014 1 12345 --private-key /home/user/.tendermint/config/priv_validator.json
 ```
 
-If the command is returns without any error then a request to update the validator set has been successfully submitted. So, even if the command has been successfully executed it doesn't imply that the validator set has been updated. In order to check whether the change has been applied, the node operator can execute `curl http://node_ip:9985/api/v1/validators` which will list the current validators set. Refer to the [validators](/http-client-server-api.html#validators) section of the HTTP API docs for more detail.
+If the command succeeds, it will create an election and return an `election_id`. Elections consist of one vote token per voting power, issued to the members of the validator set. Validators can cast their votes to approve the change to the validator set by spending their vote tokens. The status of the election can be monitored by providing the `election_id` to the `show` subcommand.
-
-Note:
-- When `POWER`is set to `0` then the validator will be removed from the validator set.
-- Upsert requests are handled once per block i.e. the validators set is updated once a new block is committed. So, the node operator is not allowed to submit a new upsert request until the current request has been processed. Furthermore, if Tendermint is started with `--consensus.create_empty_blocks=false`, and there are no new incoming transactions then the validators set update is delayed until any new transactions are received and a new block can be committed.

tests/commands/test_commands.py

@@ -18,7 +18,8 @@ def test_make_sure_we_dont_remove_any_command():
     assert parser.parse_args(['init']).command
     assert parser.parse_args(['drop']).command
     assert parser.parse_args(['start']).command
-    assert parser.parse_args(['upsert-validator', 'TEMP_PUB_KEYPAIR', '10']).command
+    assert parser.parse_args(['upsert-validator', 'new', 'TEMP_PUB_KEYPAIR', '10', 'TEMP_NODE_ID',
+                              '--private-key', 'TEMP_PATH_TO_PRIVATE_KEY']).command
 
 
 @pytest.mark.tendermint
@@ -341,15 +342,68 @@ class MockResponse():
         return {'result': {'latest_block_height': self.height}}
 
 
+# @pytest.mark.execute
+# @patch('bigchaindb.lib.BigchainDB.get_validators')
+# @pytest.mark.abci
 @pytest.mark.skip
-@patch('bigchaindb.config_utils.autoconfigure')
+def test_upsert_validator_new_with_tendermint(b, priv_validator_path, user_sk, monkeypatch):
-@patch('bigchaindb.backend.query.store_validator_update')
+    """WIP: Will be fixed and activated in the next PR
+    """
+    from bigchaindb.commands.bigchaindb import run_upsert_validator_new
+    import time
+
+    time.sleep(3)
+
+    def mock_get():
+        return [
+            {'pub_key': {'value': 'zL/DasvKulXZzhSNFwx4cLRXKkSM9GPK7Y0nZ4FEylM=',
+                         'type': 'tendermint/PubKeyEd25519'},
+             'voting_power': 10}
+        ]
+
+    # b.get_validators = mock_get
+    # mock_get_validators = mock_get
+    # monkeypatch.setattr('requests.get', mock_get)
+
+    proposer_key = b.get_validators()[0]['pub_key']['value']
+
+    args = Namespace(action='new',
+                     public_key=proposer_key,
+                     power=1,
+                     node_id='12345',
+                     sk=priv_validator_path,
+                     config={})
+    resp = run_upsert_validator_new(args, b)
+    time.sleep(3)
+
+    assert b.get_transaction(resp)
+
+
 @pytest.mark.tendermint
-def test_upsert_validator(mock_autoconfigure, mock_store_validator_update):
+@pytest.mark.bdb
-    from bigchaindb.commands.bigchaindb import run_upsert_validator
+def test_upsert_validator_new_without_tendermint(b, priv_validator_path, user_sk, monkeypatch):
+    from bigchaindb.commands.bigchaindb import run_upsert_validator_new
 
-    args = Namespace(public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=',
+    def mock_get():
-                     power='10', config={})
+        return [
-    run_upsert_validator(args)
+            {'pub_key': {'data': 'zL/DasvKulXZzhSNFwx4cLRXKkSM9GPK7Y0nZ4FEylM=',
+                         'type': 'tendermint/PubKeyEd25519'},
+             'voting_power': 10}
+        ]
 
-    assert mock_store_validator_update.called
+    def mock_write(tx, mode):
+        b.store_transaction(tx)
+        return (202, '')
+
+    b.get_validators = mock_get
+    b.write_transaction = mock_write
+
+    args = Namespace(action='new',
+                     public_key='CJxdItf4lz2PwEf4SmYNAu/c/VpmX39JEgC5YpH7fxg=',
+                     power=1,
+                     node_id='12345',
+                     sk=priv_validator_path,
+                     config={})
+    resp = run_upsert_validator_new(args, b)
+
+    assert b.get_transaction(resp)

tests/conftest.py

@@ -4,10 +4,11 @@ Tasks:
 1. setup test database before starting the tests
 2. delete test database after running the tests
 """
-
+import json
 import os
 import copy
 import random
+import tempfile
 from collections import namedtuple
 from logging import getLogger
 from logging.config import dictConfig
@@ -668,3 +669,27 @@ def node_keys():
             '83VINXdj2ynOHuhvSZz5tGuOE5oYzIi0mEximkX1KYMlt/Csu8JUjA4+by2Pz3fqSLshhuYYeM+IpvqcBl6BEA==',
             'PecJ58SaNRsWJZodDmqjpCWqG6btdwXFHLyE40RYlYM=':
             'uz8bYgoL4rHErWT1gjjrnA+W7bgD/uDQWSRKDmC8otc95wnnxJo1GxYlmh0OaqOkJaobpu13BcUcvITjRFiVgw=='}
+
+
+@pytest.fixture(scope='session')
+def priv_validator_path(node_keys):
+    (public_key, private_key) = list(node_keys.items())[0]
+    priv_validator = {
+        'address': '84F787D95E196DC5DE5F972666CFECCA36801426',
+        'pub_key': {
+            'type': 'AC26791624DE60',
+            'value': public_key
+        },
+        'last_height': 0,
+        'last_round': 0,
+        'last_step': 0,
+        'priv_key': {
+            'type': '954568A3288910',
+            'value': private_key
+        }
+    }
+    fd, path = tempfile.mkstemp()
+    socket = os.fdopen(fd, 'w')
+    json.dump(priv_validator, socket)
+    socket.close()
+    return path

tests/db/test_bigchain_api.py

@@ -280,7 +280,6 @@ class TestBigchainApi(object):
 
     @pytest.mark.usefixtures('inputs')
     def test_write_transaction(self, b, user_pk, user_sk):
-        from bigchaindb import BigchainDB
         from bigchaindb.models import Transaction
 
         input_tx = b.get_owned_ids(user_pk).pop()
@@ -294,7 +293,6 @@ class TestBigchainApi(object):
         tx_from_db, status = b.get_transaction(tx.id, include_status=True)
 
         assert tx_from_db.to_dict() == tx.to_dict()
-        assert status == BigchainDB.TX_IN_BACKLOG
 
     @pytest.mark.usefixtures('inputs')
     def test_read_transaction(self, b, user_pk, user_sk):
@@ -315,7 +313,6 @@ class TestBigchainApi(object):
         response, status = b.get_transaction(tx.id, include_status=True)
         # add validity information, which will be returned
         assert tx.to_dict() == response.to_dict()
-        assert status == b.TX_UNDECIDED
 
     @pytest.mark.usefixtures('inputs')
     def test_read_transaction_invalid_block(self, b, user_pk, user_sk):
@@ -342,35 +339,6 @@ class TestBigchainApi(object):
         # and a copy of the tx is not in the backlog
         assert response is None
 
-    @pytest.mark.usefixtures('inputs')
-    def test_read_transaction_invalid_block_and_backlog(self, b, user_pk, user_sk):
-        from bigchaindb.models import Transaction
-
-        input_tx = b.get_owned_ids(user_pk).pop()
-        input_tx = b.get_transaction(input_tx.txid)
-        inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_pk], 1)],
-                                  asset_id=input_tx.id)
-        tx = tx.sign([user_sk])
-
-        # Make sure there's a copy of tx in the backlog
-        b.write_transaction(tx)
-
-        # create block
-        block = b.create_block([tx])
-        b.write_block(block)
-
-        # vote the block invalid
-        vote = b.vote(block.id, b.get_last_voted_block().id, False)
-        b.write_vote(vote)
-
-        # a copy of the tx is both in the backlog and in an invalid
-        # block, so get_transaction should return a transaction,
-        # and a status of TX_IN_BACKLOG
-        response, status = b.get_transaction(tx.id, include_status=True)
-        assert tx.to_dict() == response.to_dict()
-        assert status == b.TX_IN_BACKLOG
-
     @pytest.mark.usefixtures('inputs')
     def test_genesis_block(self, b):
         from bigchaindb.backend import query

tests/test_core.py

@@ -1,6 +1,5 @@
 import pytest
 
-
 pytestmark = pytest.mark.tendermint
 
 
@@ -21,6 +20,10 @@ def config(request, monkeypatch):
             'connection_timeout': 5000,
             'max_tries': 3
         },
+        'tendermint': {
+            'host': 'localhost',
+            'port': 26657,
+        },
         'CONFIGURED': True,
     }
 
@@ -29,7 +32,6 @@ def config(request, monkeypatch):
     return config
 
 
-@pytest.mark.skipif(reason='will be fixed in another PR')
 def test_bigchain_class_default_initialization(config):
     from bigchaindb import BigchainDB
     from bigchaindb.consensus import BaseConsensusRules
@@ -42,8 +44,7 @@ def test_bigchain_class_default_initialization(config):
     assert bigchain.consensus == BaseConsensusRules
 
 
-@pytest.mark.skipif(reason='will be fixed in another PR')
+def test_bigchain_class_initialization_with_parameters():
-def test_bigchain_class_initialization_with_parameters(config):
     from bigchaindb import BigchainDB
     from bigchaindb.backend import connect
     from bigchaindb.consensus import BaseConsensusRules
@@ -54,7 +55,7 @@ def test_bigchain_class_initialization_with_parameters(config):
         'name': 'this_is_the_db_name',
     }
     connection = connect(**init_db_kwargs)
-    bigchain = BigchainDB(connection=connection, **init_db_kwargs)
+    bigchain = BigchainDB(connection=connection)
     assert bigchain.connection == connection
     assert bigchain.connection.host == init_db_kwargs['host']
     assert bigchain.connection.port == init_db_kwargs['port']
@@ -62,20 +63,6 @@ def test_bigchain_class_initialization_with_parameters(config):
     assert bigchain.consensus == BaseConsensusRules
 
 
-@pytest.mark.skipif(reason='will be fixed in another PR')
-def test_get_blocks_status_containing_tx(monkeypatch):
-    from bigchaindb.backend import query as backend_query
-    from bigchaindb import BigchainDB
-    blocks = [
-        {'id': 1}, {'id': 2}
-    ]
-    monkeypatch.setattr(backend_query, 'get_blocks_status_from_transaction', lambda x: blocks)
-    monkeypatch.setattr(BigchainDB, 'block_election_status', lambda x, y, z: BigchainDB.BLOCK_VALID)
-    bigchain = BigchainDB(public_key='pubkey', private_key='privkey')
-    with pytest.raises(Exception):
-        bigchain.get_blocks_status_containing_tx('txid')
-
-
 @pytest.mark.genesis
 def test_get_spent_issue_1271(b, alice, bob, carol):
     from bigchaindb.models import Transaction

tests/test_txlist.py

@@ -4,46 +4,27 @@ This test module defines it's own fixture which is used by all the tests.
 """
 import pytest
 
+pytestmark = pytest.mark.tendermint
+
 
 @pytest.fixture
-def txlist(b, user_pk, user2_pk, user_sk, user2_sk, genesis_block):
+def txlist(b, user_pk, user2_pk, user_sk, user2_sk):
     from bigchaindb.models import Transaction
-    prev_block_id = genesis_block.id
 
-    # Create first block with CREATE transactions
+    # Create two CREATE transactions
     create1 = Transaction.create([user_pk], [([user2_pk], 6)]) \
         .sign([user_sk])
+
     create2 = Transaction.create([user2_pk],
                                  [([user2_pk], 5), ([user_pk], 5)]) \
         .sign([user2_sk])
-    block1 = b.create_block([create1, create2])
-    b.write_block(block1)
 
-    # Create second block with TRANSFER transactions
+    # Create a TRANSFER transactions
     transfer1 = Transaction.transfer(create1.to_inputs(),
                                      [([user_pk], 8)],
                                      create1.id).sign([user2_sk])
-    block2 = b.create_block([transfer1])
-    b.write_block(block2)
 
-    # Create block with double spend
+    b.store_bulk_transactions([create1, create2, transfer1])
-    tx_doublespend = Transaction.transfer(create1.to_inputs(), [([user_pk], 9)],
-                                          create1.id).sign([user2_sk])
-    block_doublespend = b.create_block([tx_doublespend])
-    b.write_block(block_doublespend)
-
-    # Vote on all the blocks
-    prev_block_id = genesis_block.id
-    for bid in [block1.id, block2.id]:
-        vote = b.vote(bid, prev_block_id, True)
-        prev_block_id = bid
-        b.write_vote(vote)
-
-    # Create undecided block
-    untx = Transaction.create([user_pk], [([user2_pk], 7)]) \
-        .sign([user_sk])
-    block_undecided = b.create_block([untx])
-    b.write_block(block_undecided)
 
     return type('', (), {
         'create1': create1,
@@ -54,8 +35,8 @@ def txlist(b, user_pk, user2_pk, user_sk, user2_sk, genesis_block):
 @pytest.mark.bdb
 def test_get_txlist_by_asset(b, txlist):
     res = b.get_transactions_filtered(txlist.create1.id)
-    assert set(tx.id for tx in res) == set([txlist.transfer1.id,
+    assert sorted(set(tx.id for tx in res)) == sorted(
-                                            txlist.create1.id])
+        set([txlist.transfer1.id, txlist.create1.id]))
 
 
 @pytest.mark.bdb

tests/web/test_transactions.py

@@ -402,30 +402,6 @@ def test_transactions_get_list_bad(client):
         assert client.get(url).status_code == 400
 
 
-@pytest.mark.tendermint
-def test_return_only_valid_transaction(client):
-    from bigchaindb import BigchainDB
-
-    def get_transaction_patched(status):
-        def inner(self, tx_id, include_status):
-            return {}, status
-        return inner
-
-    # NOTE: `get_transaction` only returns a transaction if it's included in an
-    #       UNDECIDED or VALID block, as well as transactions from the backlog.
-    #       As the endpoint uses `get_transaction`, we don't have to test
-    #       against invalid transactions here.
-    with patch('bigchaindb.BigchainDB.get_transaction',
-               get_transaction_patched(BigchainDB.TX_UNDECIDED)):
-        url = '{}{}'.format(TX_ENDPOINT, '123')
-        assert client.get(url).status_code == 404
-
-    with patch('bigchaindb.BigchainDB.get_transaction',
-               get_transaction_patched(BigchainDB.TX_IN_BACKLOG)):
-        url = '{}{}'.format(TX_ENDPOINT, '123')
-        assert client.get(url).status_code == 404
-
-
 @pytest.mark.tendermint
 @patch('requests.post')
 @pytest.mark.parametrize('mode', [