Mirroristas/bigchaindb
2
0
Fork 0
mirror of https://github.com/bigchaindb/bigchaindb.git synced 2024-10-13 13:34:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into implement-connection-run-for-mongodb

Browse Source
This commit is contained in:
vrde 2017-02-02 19:37:14 +01:00
commit 702d74a584
No known key found for this signature in database
GPG Key ID: 6581C7C39B3D397D
49 changed files with 869 additions and 241 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
Dockerfile
View File

@ -11,9 +11,12 @@ RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
ENV LANG en_US.UTF-8
RUN apt-get -y install python3 python3-pip libffi-dev
RUN pip3 install --upgrade pip
RUN pip3 install --upgrade setuptools
# The `apt-get update` command executed with the install instructions should
# not use a locally cached storage layer. Force update the cache again.
# https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run
RUN apt-get update && apt-get -y install python3 python3-pip libffi-dev \
&& pip3 install --upgrade pip \
&& pip3 install --upgrade setuptools
RUN mkdir -p /usr/src/app

11
Makefile
View File

@ -65,12 +65,11 @@ coverage: ## check code coverage quickly with the default Python
$(BROWSER) htmlcov/index.html
docs: ## generate Sphinx HTML documentation, including API docs
rm -f docs/bigchaindb.rst
rm -f docs/modules.rst
sphinx-apidoc -o docs/ bigchaindb
$(MAKE) -C docs clean
$(MAKE) -C docs html
$(BROWSER) docs/_build/html/index.html
$(MAKE) -C docs/root clean
$(MAKE) -C docs/root html
$(MAKE) -C docs/server clean
$(MAKE) -C docs/server html
$(BROWSER) docs/root/_build/html/index.html
servedocs: docs ## compile the docs watching for changes
watchmedo shell-command -p '*.rst' -c '$(MAKE) -C docs html' -R -D .

59
Release_Process.md
View File

@ -1,23 +1,52 @@
# Our Release Process
This is a summary of the steps we go through to release a new version of BigchainDB Server.
The release process for BigchainDB server differs slightly depending on whether it's a minor or a patch release.
1. Update the `CHANGELOG.md` file
1. Update the version numbers in `bigchaindb/version.py`. Note that we try to use [semantic versioning](http://semver.org/) (i.e. MAJOR.MINOR.PATCH)
1. Go to the [bigchaindb/bigchaindb Releases page on GitHub](https://github.com/bigchaindb/bigchaindb/releases)
and click the "Draft a new release" button
1. Name the tag something like v0.7.0
1. The target should be a specific commit: the one when the update of `bigchaindb/version.py` got merged into master
1. The release title should be something like v0.7.0
1. The description should be copied from the `CHANGELOG.md` file updated above
1. Generate and send the latest `bigchaindb` package to PyPI. Dimi and Sylvain can do this, maybe others
1. Login to readthedocs.org as a maintainer of the BigchainDB Server docs.
Go to Admin --> Versions and under **Choose Active Versions**, make sure that the new version's tag is
"Active" and "Public"
BigchainDB follows [semantic versioning](http://semver.org/) (i.e. MAJOR.MINOR.PATCH), taking into account
that [major version 0.x does not export a stable API](http://semver.org/#spec-item-4).
After the release:
## Minor release
1. Update `bigchaindb/version.py` again, to be something like 0.8.0.dev (with a dev on the end).
A minor release is preceeded by a feature freeze and created from the 'master' branch. This is a summary of the steps we go through to release a new minor version of BigchainDB Server.
1. Update the `CHANGELOG.md` file in master
1. Create and checkout a new branch for the release, named after the minor version, without preceeding 'v', ie: `git checkout -b 0.9`
1. Commit changes and push new branch to Github
1. Follow steps outlined in [Common Steps](#common-steps)
1. In 'master' branch, Edit `bigchaindb/version.py`, increment the minor version to the next planned release ie: `0.10.0.dev'.
This is so people reading the latest docs will know that they're for the latest (master branch)
version of BigchainDB Server, not the docs at the time of the most recent release (which are also
available).
Congratulations, you have released BigchainDB!
## Patch release
A patch release is similar to a minor release, but piggybacks on an existing minor release branch:
1. Check out the minor release branch
1. Apply the changes you want, ie using `git cherry-pick`.
1. Update the `CHANGELOG.md` file
1. Increment the patch version in `bigchaindb/version.py`, ie: "0.9.1"
1. Follow steps outlined in [Common Steps](#common-steps)
## Common steps
These steps are common between minor and patch releases:
1. Go to the [bigchaindb/bigchaindb Releases page on GitHub](https://github.com/bigchaindb/bigchaindb/releases)
and click the "Draft a new release" button
1. Fill in the details:
- Tag version: version number preceeded by 'v', ie: "v0.9.1"
- Target: the release branch that was just pushed
- Title: Same as tag name
- Description: The body of the changelog entry (Added, Changed etc)
1. Publish the release on Github
1. Generate the release tarball with `python setup.py sdist`. Upload the release to Pypi.
1. Login to readthedocs.org as a maintainer of the BigchainDB Server docs.
Go to Admin --> Versions and under **Choose Active Versions**, make sure that the new version's tag is
"Active" and "Public", and make sure the new version's branch
(without the 'v' in front) is _not_ active
1. Also in readthedocs.org, go to Admin --> Advanced Settings
and make sure that "Default branch:" (i.e. what "latest" points to)
is set to the new release's tag, e.g. `v0.9.1`. (Don't miss the 'v' in front.)

29
bigchaindb/__init__.py
View File

@ -5,6 +5,25 @@ import os
# PORT_NUMBER = reduce(lambda x, y: x * y, map(ord, 'BigchainDB')) % 2**16
# basically, the port number is 9984
_database_rethinkdb = {
'backend': os.environ.get('BIGCHAINDB_DATABASE_BACKEND', 'rethinkdb'),
'host': os.environ.get('BIGCHAINDB_DATABASE_HOST', 'localhost'),
'port': int(os.environ.get('BIGCHAINDB_DATABASE_PORT', 28015)),
'name': os.environ.get('BIGCHAINDB_DATABASE_NAME', 'bigchain'),
}
_database_mongodb = {
'backend': os.environ.get('BIGCHAINDB_DATABASE_BACKEND', 'mongodb'),
'host': os.environ.get('BIGCHAINDB_DATABASE_HOST', 'localhost'),
'port': int(os.environ.get('BIGCHAINDB_DATABASE_PORT', 27017)),
'name': os.environ.get('BIGCHAINDB_DATABASE_NAME', 'bigchain'),
'replicaset': os.environ.get('BIGCHAINDB_DATABASE_REPLICASET', 'bigchain-rs'),
}
_database_map = {
'mongodb': _database_mongodb,
'rethinkdb': _database_rethinkdb
}
config = {
'server': {
@ -14,13 +33,9 @@ config = {
'workers': None, # if none, the value will be cpu_count * 2 + 1
'threads': None, # if none, the value will be cpu_count * 2 + 1
},
'database': {
'backend': os.environ.get('BIGCHAINDB_DATABASE_BACKEND', 'rethinkdb'),
'host': os.environ.get('BIGCHAINDB_DATABASE_HOST', 'localhost'),
'port': int(os.environ.get('BIGCHAINDB_DATABASE_PORT', 28015)),
'name': os.environ.get('BIGCHAINDB_DATABASE_NAME', 'bigchain'),
'replicaset': os.environ.get('BIGCHAINDB_DATABASE_REPLICASET', 'bigchain-rs'),
},
'database': _database_map[
os.environ.get('BIGCHAINDB_DATABASE_BACKEND', 'rethinkdb')
],
'keypair': {
'public': None,
'private': None,

12
bigchaindb/backend/admin.py
View File

@ -20,3 +20,15 @@ def set_shards(connection, *, shards):
@singledispatch
def set_replicas(connection, *, replicas):
raise NotImplementedError
@singledispatch
def add_replicas(connection, replicas):
raise NotImplementedError('This command is specific to the '
'MongoDB backend.')
@singledispatch
def remove_replicas(connection, replicas):
raise NotImplementedError('This command is specific to the '
'MongoDB backend.')

8
bigchaindb/backend/connection.py
View File

@ -41,6 +41,12 @@ def connect(backend=None, host=None, port=None, name=None, replicaset=None):
host = host or bigchaindb.config['database']['host']
port = port or bigchaindb.config['database']['port']
dbname = name or bigchaindb.config['database']['name']
# Not sure how to handle this here. This setting is only relevant for
# mongodb.
# I added **kwargs for both RethinkDBConnection and MongoDBConnection
# to handle these these additional args. In case of RethinkDBConnection
# it just does not do anything with it.
replicaset = replicaset or bigchaindb.config['database'].get('replicaset')
try:
module_name, _, class_name = BACKENDS[backend].rpartition('.')
@ -52,7 +58,7 @@ def connect(backend=None, host=None, port=None, name=None, replicaset=None):
raise ConfigurationError('Error loading backend `{}`'.format(backend)) from exc
logger.debug('Connection: {}'.format(Class))
return Class(host, port, dbname)
return Class(host, port, dbname, replicaset=replicaset)
class Connection:

2
bigchaindb/backend/mongodb/__init__.py
View File

@ -16,7 +16,7 @@ generic backend interfaces to the implementations in this module.
"""
# Register the single dispatched modules on import.
from bigchaindb.backend.mongodb import schema, query, changefeed # noqa
from bigchaindb.backend.mongodb import admin, schema, query, changefeed # noqa
# MongoDBConnection should always be accessed via
# ``bigchaindb.backend.connect()``.

86
bigchaindb/backend/mongodb/admin.py Normal file
View File

@ -0,0 +1,86 @@
"""Database configuration functions."""
import logging
from pymongo.errors import OperationFailure
from bigchaindb.backend import admin
from bigchaindb.backend.utils import module_dispatch_registrar
from bigchaindb.backend.exceptions import OperationError
from bigchaindb.backend.mongodb.connection import MongoDBConnection
logger = logging.getLogger(__name__)
register_admin = module_dispatch_registrar(admin)
@register_admin(MongoDBConnection)
def add_replicas(connection, replicas):
"""Add a set of replicas to the replicaset
Args:
connection (:class:`~bigchaindb.backend.connection.Connection`):
A connection to the database.
replicas (:obj:`list` of :obj:`str`): replica addresses in the
form "hostname:port".
Raises:
OperationError: If the reconfiguration fails due to a MongoDB
:exc:`OperationFailure`
"""
# get current configuration
conf = connection.conn.admin.command('replSetGetConfig')
# MongoDB does not automatically add an id for the members so we need
# to choose one that does not exist yet. The safest way is to use
# incrementing ids, so we first check what is the highest id already in
# the set and continue from there.
cur_id = max([member['_id'] for member in conf['config']['members']])
# add the nodes to the members list of the replica set
for replica in replicas:
cur_id += 1
conf['config']['members'].append({'_id': cur_id, 'host': replica})
# increase the configuration version number
# when reconfiguring, mongodb expects a version number higher than the one
# it currently has
conf['config']['version'] += 1
# apply new configuration
try:
connection.conn.admin.command('replSetReconfig', conf['config'])
except OperationFailure as exc:
raise OperationError(exc.details['errmsg'])
@register_admin(MongoDBConnection)
def remove_replicas(connection, replicas):
"""Remove a set of replicas from the replicaset
Args:
connection (:class:`~bigchaindb.backend.connection.Connection`):
A connection to the database.
replicas (:obj:`list` of :obj:`str`): replica addresses in the
form "hostname:port".
Raises:
OperationError: If the reconfiguration fails due to a MongoDB
:exc:`OperationFailure`
"""
# get the current configuration
conf = connection.conn.admin.command('replSetGetConfig')
# remove the nodes from the members list in the replica set
conf['config']['members'] = list(
filter(lambda member: member['host'] not in replicas,
conf['config']['members'])
)
# increase the configuration version number
conf['config']['version'] += 1
# apply new configuration
try:
connection.conn.admin.command('replSetReconfig', conf['config'])
except OperationFailure as exc:
raise OperationError(exc.details['errmsg'])

15
bigchaindb/backend/mongodb/connection.py
View File

@ -125,8 +125,8 @@ def initialize_replica_set(host, port, connection_timeout):
# `backend.connect` will connect you to a replica set but this fails if
# you try to connect to a replica set that is not yet initialized
conn = pymongo.MongoClient(host=host,
port=port,
serverselectiontimeoutms=connection_timeout)
port=port,
serverselectiontimeoutms=connection_timeout)
_check_replica_set(conn)
host = '{}:{}'.format(bigchaindb.config['database']['host'],
bigchaindb.config['database']['port'])
@ -164,16 +164,15 @@ def _check_replica_set(conn):
repl_set_name = repl_opts.get('replSetName', None) or repl_opts['replSet']
except KeyError:
raise ConfigurationError('mongod was not started with'
' the replSet option.')
' the replSet option.')
bdb_repl_set_name = bigchaindb.config['database']['replicaset']
if repl_set_name != bdb_repl_set_name:
raise ConfigurationError('The replicaset configuration of '
'bigchaindb (`{}`) needs to match '
'the replica set name from MongoDB'
' (`{}`)'
.format(bdb_repl_set_name,
repl_set_name))
'bigchaindb (`{}`) needs to match '
'the replica set name from MongoDB'
' (`{}`)'.format(bdb_repl_set_name,
repl_set_name))
def _wait_for_replica_set_initialization(conn):

16
bigchaindb/backend/mongodb/query.py
View File

@ -4,8 +4,6 @@ from time import time
from itertools import chain
from pymongo import ReturnDocument
from pymongo import errors
from bigchaindb import backend
from bigchaindb.common.exceptions import CyclicBlockchainError
@ -162,6 +160,10 @@ def get_asset_by_id(conn, asset_id):
def get_spent(conn, transaction_id, output):
cursor = conn.run(
conn.collection('bigchain').aggregate([
{'$match': {
'block.transactions.inputs.fulfills.txid': transaction_id,
'block.transactions.inputs.fulfills.output': output
}},
{'$unwind': '$block.transactions'},
{'$match': {
'block.transactions.inputs.fulfills.txid': transaction_id,
@ -176,14 +178,10 @@ def get_spent(conn, transaction_id, output):
@register_query(MongoDBConnection)
def get_owned_ids(conn, owner):
cursor = conn.run(
conn.collection('bigchain')
.aggregate([
conn.collection('bigchain').aggregate([
{'$match': {'block.transactions.outputs.public_keys': owner}},
{'$unwind': '$block.transactions'},
{'$match': {
'block.transactions.outputs.public_keys': {
'$elemMatch': {'$eq': owner}
}
}}
{'$match': {'block.transactions.outputs.public_keys': owner}}
]))
# we need to access some nested fields before returning so lets use a
# generator to avoid having to read all records on the cursor at this point

12
bigchaindb/backend/mongodb/schema.py
View File

@ -63,6 +63,18 @@ def create_bigchain_secondary_index(conn, dbname):
.create_index('block.transactions.transaction.asset.id',
name='asset_id')
# secondary index on the public keys of outputs
conn.conn[dbname]['bigchain']\
.create_index('block.transactions.outputs.public_keys',
name='outputs')
# secondary index on inputs/transaction links (txid, output)
conn.conn[dbname]['bigchain']\
.create_index([
('block.transactions.inputs.fulfills.txid', ASCENDING),
('block.transactions.inputs.fulfills.output', ASCENDING),
], name='inputs')
def create_backlog_secondary_index(conn, dbname):
logger.info('Create `backlog` secondary index.')

2
bigchaindb/backend/rethinkdb/connection.py
View File

@ -17,7 +17,7 @@ class RethinkDBConnection(Connection):
more times to run the query or open a connection.
"""
def __init__(self, host, port, dbname, max_tries=3):
def __init__(self, host, port, dbname, max_tries=3, **kwargs):
"""Create a new :class:`~.RethinkDBConnection` instance.
See :meth:`.Connection.__init__` for

15
bigchaindb/backend/rethinkdb/query.py
View File

@ -111,21 +111,22 @@ def _get_asset_create_tx_query(asset_id):
@register_query(RethinkDBConnection)
def get_spent(connection, transaction_id, output):
# TODO: use index!
return connection.run(
r.table('bigchain', read_mode=READ_MODE)
.concat_map(lambda doc: doc['block']['transactions'])
.filter(lambda transaction: transaction['inputs'].contains(
lambda input: input['fulfills'] == {'txid': transaction_id, 'output': output})))
.get_all([transaction_id, output], index='inputs')
.concat_map(lambda doc: doc['block']['transactions'])
.filter(lambda transaction: transaction['inputs'].contains(
lambda input_: input_['fulfills'] == {'txid': transaction_id, 'output': output})))
@register_query(RethinkDBConnection)
def get_owned_ids(connection, owner):
# TODO: use index!
return connection.run(
r.table('bigchain', read_mode=READ_MODE)
.concat_map(lambda doc: doc['block']['transactions'])
.filter(lambda tx: tx['outputs'].contains(
.get_all(owner, index='outputs')
.distinct()
.concat_map(lambda doc: doc['block']['transactions'])
.filter(lambda tx: tx['outputs'].contains(
lambda c: c['public_keys'].contains(owner))))

25
bigchaindb/backend/rethinkdb/schema.py
View File

@ -66,6 +66,31 @@ def create_bigchain_secondary_index(connection, dbname):
.table('bigchain')
.index_create('asset_id', r.row['block']['transactions']['asset']['id'], multi=True))
# secondary index on the public keys of outputs
# the last reduce operation is to return a flatten list of public_keys
# without it we would need to match exactly the public_keys list.
# For instance querying for `pk1` would not match documents with
# `public_keys: [pk1, pk2, pk3]`
connection.run(
r.db(dbname)
.table('bigchain')
.index_create('outputs',
r.row['block']['transactions']
.concat_map(lambda tx: tx['outputs']['public_keys'])
.reduce(lambda l, r: l + r), multi=True))
# secondary index on inputs/transaction links (txid, output)
connection.run(
r.db(dbname)
.table('bigchain')
.index_create('inputs',
r.row['block']['transactions']
.concat_map(lambda tx: tx['inputs']['fulfills'])
.with_fields('txid', 'output')
.map(lambda fulfills: [fulfills['txid'],
fulfills['output']]),
multi=True))
# wait for rethinkdb to finish creating secondary indexes
connection.run(
r.db(dbname)

70
bigchaindb/commands/bigchain.py
View File

@ -22,7 +22,8 @@ from bigchaindb.models import Transaction
from bigchaindb.utils import ProcessGroup
from bigchaindb import backend
from bigchaindb.backend import schema
from bigchaindb.backend.admin import set_replicas, set_shards
from bigchaindb.backend.admin import (set_replicas, set_shards, add_replicas,
remove_replicas)
from bigchaindb.backend.exceptions import (OperationError,
ConnectionError)
from bigchaindb.commands import utils
@ -87,6 +88,11 @@ def run_configure(args, skip_if_exists=False):
conf['keypair']['private'], conf['keypair']['public'] = \
crypto.generate_key_pair()
# select the correct config defaults based on the backend
print('Generating default configuration for backend {}'
.format(args.backend))
conf['database'] = bigchaindb._database_map[args.backend]
if not args.yes:
for key in ('bind', ):
val = conf['server'][key]
@ -264,6 +270,32 @@ def run_set_replicas(args):
logger.warn(e)
def run_add_replicas(args):
# Note: This command is specific to MongoDB
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
conn = backend.connect()
try:
add_replicas(conn, args.replicas)
except (OperationError, NotImplementedError) as e:
logger.warn(e)
else:
logger.info('Added {} to the replicaset.'.format(args.replicas))
def run_remove_replicas(args):
# Note: This command is specific to MongoDB
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
conn = backend.connect()
try:
remove_replicas(conn, args.replicas)
except (OperationError, NotImplementedError) as e:
logger.warn(e)
else:
logger.info('Removed {} from the replicaset.'.format(args.replicas))
def create_parser():
parser = argparse.ArgumentParser(
description='Control your BigchainDB node.',
@ -287,9 +319,13 @@ def create_parser():
dest='command')
# parser for writing a config file
subparsers.add_parser('configure',
help='Prepare the config file '
'and create the node keypair')
config_parser = subparsers.add_parser('configure',
help='Prepare the config file '
'and create the node keypair')
config_parser.add_argument('backend',
choices=['rethinkdb', 'mongodb'],
help='The backend to use. It can be either '
'rethinkdb or mongodb.')
# parsers for showing/exporting config values
subparsers.add_parser('show-config',
@ -325,6 +361,32 @@ def create_parser():
type=int, default=1,
help='Number of replicas (i.e. the replication factor)')
# parser for adding nodes to the replica set
add_replicas_parser = subparsers.add_parser('add-replicas',
help='Add a set of nodes to the '
'replica set. This command '
'is specific to the MongoDB'
' backend.')
add_replicas_parser.add_argument('replicas', nargs='+',
type=utils.mongodb_host,
help='A list of space separated hosts to '
'add to the replicaset. Each host '
'should be in the form `host:port`.')
# parser for removing nodes from the replica set
rm_replicas_parser = subparsers.add_parser('remove-replicas',
help='Remove a set of nodes from the '
'replica set. This command '
'is specific to the MongoDB'
' backend.')
rm_replicas_parser.add_argument('replicas', nargs='+',
type=utils.mongodb_host,
help='A list of space separated hosts to '
'remove from the replicaset. Each host '
'should be in the form `host:port`.')
load_parser = subparsers.add_parser('load',
help='Write transactions to the backlog')

31
bigchaindb/commands/utils.py
View File

@ -3,14 +3,15 @@ for ``argparse.ArgumentParser``.
"""
import argparse
from bigchaindb.common.exceptions import StartupError
import multiprocessing as mp
import subprocess
import rethinkdb as r
from pymongo import uri_parser
import bigchaindb
from bigchaindb import backend
from bigchaindb.common.exceptions import StartupError
from bigchaindb.version import __version__
@ -95,6 +96,34 @@ def start(parser, argv, scope):
return func(args)
def mongodb_host(host):
"""Utility function that works as a type for mongodb ``host`` args.
This function validates the ``host`` args provided by to the
``add-replicas`` and ``remove-replicas`` commands and checks if each arg
is in the form "host:port"
Args:
host (str): A string containing hostname and port (e.g. "host:port")
Raises:
ArgumentTypeError: if it fails to parse the argument
"""
# check if mongodb can parse the host
try:
hostname, port = uri_parser.parse_host(host, default_port=None)
except ValueError as exc:
raise argparse.ArgumentTypeError(exc.args[0])
# we do require the port to be provided.
if port is None or hostname == '':
raise argparse.ArgumentTypeError('expected host in the form '
'`host:port`. Got `{}` instead.'
.format(host))
return host
base_parser = argparse.ArgumentParser(add_help=False, prog='bigchaindb')
base_parser.add_argument('-c', '--config',

6
bigchaindb/common/transaction.py
View File

@ -482,8 +482,8 @@ class Transaction(object):
Args:
tx_signers (:obj:`list` of :obj:`str`): A list of keys that
represent the signers of the CREATE Transaction.
recipients (:obj:`list` of :obj:`str`): A list of keys that
represent the recipients of the outputs of this
recipients (:obj:`list` of :obj:`tuple`): A list of
([keys],amount) that represent the recipients of this
Transaction.
metadata (dict): The metadata to be stored along with the
Transaction.
@ -549,7 +549,7 @@ class Transaction(object):
inputs (:obj:`list` of :class:`~bigchaindb.common.transaction.
Input`): Converted `Output`s, intended to
be used as inputs in the transfer to generate.
recipients (:obj:`list` of :obj:`str`): A list of
recipients (:obj:`list` of :obj:`tuple`): A list of
([keys],amount) that represent the recipients of this
Transaction.
asset_id (str): The asset ID of the asset to be transferred in

5
bigchaindb/models.py
View File

@ -202,11 +202,6 @@ class Block(object):
OperationError: If a non-federation node signed the Block.
InvalidSignature: If a Block's signature is invalid.
"""
# First, make sure this node hasn't already voted on this block
if bigchain.has_previous_vote(self.id, self.voters):
return self
# Check if the block was created by a federation node
possible_voters = (bigchain.nodes_except_me + [bigchain.me])
if self.node_pubkey not in possible_voters:

2
docker-compose.yml
View File

@ -5,7 +5,7 @@ services:
image: mongo:3.4.1
ports:
- "27017"
command: mongod --replSet=rs0
command: mongod --replSet=bigchain-rs
rdb:
image: rethinkdb

1
docs/server/generate_http_server_api_documentation.py
View File

@ -198,6 +198,7 @@ def main():
"6qHyZew94NMmUTYyHnkZsB8cxJYuRNEiEpXHe1ih9QX3",
"AdDuyrTyjrDt935YnFu4VBCVDhHtY2Y6rcy7x2TFeiRi"
]
res_data['public_key'] = 'NC8c8rYcAhyKVpx1PCV65CBmyq4YUbLysy3Rqrg8L8mz'
ctx['index'] = pretty_json(res_data)
# API index

2
docs/server/source/appendices/install-latest-pip.md
View File

@ -7,7 +7,7 @@ pip -V
If it says that `pip` isn't installed, or it says `pip` is associated with a Python version less than 3.4, then you must install a `pip` version associated with Python 3.4+. In the following instructions, we call it `pip3` but you may be able to use `pip` if that refers to the same thing. See [the `pip` installation instructions](https://pip.pypa.io/en/stable/installing/).
On Ubuntu 14.04, we found that this works:
On Ubuntu 16.04, we found that this works:
```text
sudo apt-get install python3-pip
```

4
docs/server/source/appendices/install-os-level-deps.md
View File

@ -2,13 +2,13 @@
BigchainDB Server has some OS-level dependencies that must be installed.
On Ubuntu 14.04 and 16.04, we found that the following was enough:
On Ubuntu 16.04, we found that the following was enough:
```text
sudo apt-get update
sudo apt-get install g++ python3-dev libffi-dev
```
On Fedora 23 and 24, we found that the following was enough:
On Fedora 23–25, we found that the following was enough:
```text
sudo dnf update
sudo dnf install gcc-c++ redhat-rpm-config python3-devel libffi-devel

2
docs/server/source/appendices/install-with-lxd.md
View File

@ -1,5 +1,7 @@
# Installing BigchainDB on LXC containers using LXD
**Note: This page was contributed by an external contributor and is not actively maintained. We include it in case someone is interested.**
You can visit this link to install LXD (instructions here): [LXD Install](https://linuxcontainers.org/lxd/getting-started-cli/)
(assumption is that you are using Ubuntu 14.04 for host/container)

4
docs/server/source/appendices/ntp-notes.md
View File

@ -23,9 +23,9 @@ If your BigchainDB node is running on an Amazon Linux instance (i.e. a Linux ins
That said, you should check _which_ NTP daemon is installed. Is it recent? Is it configured securely?
## Ubuntu's ntp Package
## The Ubuntu ntp Packages
The [Ubuntu 14.04 (Trusty Tahr) package `ntp`](https://launchpad.net/ubuntu/trusty/+source/ntp) is based on the reference implementation of an NTP daemon (i.e. `ntpd`).
The [Ubuntu `ntp` packages](https://launchpad.net/ubuntu/+source/ntp) are based on the reference implementation of NTP.
The following commands will uninstall the `ntp` and `ntpdate` packages, install the latest `ntp` package (which _might not be based on the latest ntpd code_), and start the NTP daemon (a local NTP server). (`ntpdate` is not reinstalled because it's [deprecated](https://askubuntu.com/questions/297560/ntpd-vs-ntpdate-pros-and-cons) and you shouldn't use it.)
```text

2
docs/server/source/appendices/run-with-docker.md
View File

@ -21,7 +21,7 @@ be stored in a file on your host machine at `~/bigchaindb_docker/.bigchaindb`:
```text
docker run --rm -v "$HOME/bigchaindb_docker:/data" -ti \
bigchaindb/bigchaindb -y configure
bigchaindb/bigchaindb -y configure rethinkdb
Generating keypair
Configuration written to /data/.bigchaindb
Ready to go!

22
docs/server/source/cloud-deployment-starter-templates/template-ansible.md
View File

@ -2,12 +2,12 @@
If you didn't read the introduction to the [cloud deployment starter templates](index.html), please do that now. The main point is that they're not for deploying a production node; they can be used as a starting point.
This page explains how to use [Ansible](https://www.ansible.com/) to install, configure and run all the software needed to run a one-machine BigchainDB node on a server running Ubuntu 14.04.
This page explains how to use [Ansible](https://www.ansible.com/) to install, configure and run all the software needed to run a one-machine BigchainDB node on a server running Ubuntu 16.04.
## Install Ansible
The Ansible documentation has [installation instructions](https://docs.ansible.com/ansible/intro_installation.html). Note the control machine requirements: at the time of writing, Ansible required Python 2.6 or 2.7. (Support for Python 3 [is a goal of Ansible 2.2](https://github.com/ansible/ansible/issues/15976#issuecomment-221264089).)
The Ansible documentation has [installation instructions](https://docs.ansible.com/ansible/intro_installation.html). Note the control machine requirements: at the time of writing, Ansible required Python 2.6 or 2.7. ([Python 3 support is coming](https://docs.ansible.com/ansible/python_3_support.html): "Ansible 2.2 features a tech preview of Python 3 support." and the latest version, as of January 31, 2017, was 2.2.1.0. For now, it's probably best to use it with Python 2.)
For example, you could create a special Python 2.x virtualenv named `ansenv` and then install Ansible in it:
```text
@ -19,9 +19,9 @@ pip install ansible
## About Our Example Ansible Playbook
Our example Ansible playbook installs, configures and runs a basic BigchainDB node on an Ubuntu 14.04 machine. That playbook is in `.../bigchaindb/ntools/one-m/ansible/one-m-node.yml`.
Our example Ansible playbook installs, configures and runs a basic BigchainDB node on an Ubuntu 16.04 machine. That playbook is in `.../bigchaindb/ntools/one-m/ansible/one-m-node.yml`.
When you run the playbook (as per the instructions below), it ensures all the necessary software is installed, configured and running. It can be used to get a BigchainDB node set up on a bare Ubuntu 14.04 machine, but it can also be used to ensure that everything is okay on a running BigchainDB node. (If you run the playbook against a host where everything is okay, then it won't change anything on that host.)
When you run the playbook (as per the instructions below), it ensures all the necessary software is installed, configured and running. It can be used to get a BigchainDB node set up on a bare Ubuntu 16.04 machine, but it can also be used to ensure that everything is okay on a running BigchainDB node. (If you run the playbook against a host where everything is okay, then it won't change anything on that host.)
## Create an Ansible Inventory File
@ -39,7 +39,15 @@ echo "192.0.2.128" > hosts
but replace `192.0.2.128` with the IP address of the host.
## Run the Ansible Playbook
## Run the Ansible Playbook(s)
The latest Ubuntu 16.04 AMIs from Canonical don't include Python 2 (which is required by Ansible), so the first step is to run a small Ansible playbook to install Python 2 on the managed node:
```text
# cd to the directory .../bigchaindb/ntools/one-m/ansible
ansible-playbook -i hosts --private-key ~/.ssh/<key-name> install-python2.yml
```
where `<key-name>` should be replaced by the name of the SSH private key you created earlier (for SSHing to the host machine at your cloud hosting provider).
The next step is to run the Ansible playbook named `one-m-node.yml`:
```text
@ -47,14 +55,12 @@ The next step is to run the Ansible playbook named `one-m-node.yml`:
ansible-playbook -i hosts --private-key ~/.ssh/<key-name> one-m-node.yml
```
where `<key-name>` should be replaced by the name of the SSH private key you created earlier (for SSHing to the host machine at your cloud hosting provider).
What did you just do? Running that playbook ensures all the software necessary for a one-machine BigchainDB node is installed, configured, and running properly. You can run that playbook on a regular schedule to ensure that the system stays properly configured. If something is okay, it does nothing; it only takes action when something is not as-desired.
## Some Notes on the One-Machine Node You Just Got Running
* It ensures that the installed version of RethinkDB is `2.3.4~0trusty`. You can change that by changing the installation task.
* It ensures that the installed version of RethinkDB is the latest. You can change that by changing the installation task.
* It uses a very basic RethinkDB configuration file based on `bigchaindb/ntools/one-m/ansible/roles/rethinkdb/templates/rethinkdb.conf.j2`.
* If you edit the RethinkDB configuration file, then running the Ansible playbook will **not** restart RethinkDB for you. You must do that manually. (You can stop RethinkDB using `sudo /etc/init.d/rethinkdb stop`; run the playbook to get RethinkDB started again. This assumes you're using init.d, which is what the Ansible playbook assumes. If you want to use systemd, you'll have to edit the playbook accordingly, and stop RethinkDB using `sudo systemctl stop rethinkdb@<name_instance>`.)
* It generates and uses a default BigchainDB configuration file, which it stores in `~/.bigchaindb` (the default location).

4
docs/server/source/cloud-deployment-starter-templates/template-terraform-aws.md
View File

@ -2,7 +2,7 @@
If you didn't read the introduction to the [cloud deployment starter templates](index.html), please do that now. The main point is that they're not for deploying a production node; they can be used as a starting point.
This page explains a way to use [Terraform](https://www.terraform.io/) to provision an Ubuntu machine (i.e. an EC2 instance with Ubuntu 14.04) and other resources on [AWS](https://aws.amazon.com/). That machine can then be used to host a one-machine BigchainDB node.
This page explains a way to use [Terraform](https://www.terraform.io/) to provision an Ubuntu machine (i.e. an EC2 instance with Ubuntu 16.04) and other resources on [AWS](https://aws.amazon.com/). That machine can then be used to host a one-machine BigchainDB node.
## Install Terraform
@ -65,7 +65,7 @@ terraform apply
Terraform will report its progress as it provisions all the resources. Once it's done, you can go to the Amazon EC2 web console and see the instance, its security group, its elastic IP, and its attached storage volumes (one for the root directory and one for RethinkDB storage).
At this point, there is no software installed on the instance except for Ubuntu 14.04 and whatever else came with the Amazon Machine Image (AMI) specified in the Terraform configuration (files).
At this point, there is no software installed on the instance except for Ubuntu 16.04 and whatever else came with the Amazon Machine Image (AMI) specified in the Terraform configuration (files).
The next step is to install, configure and run all the necessary software for a BigchainDB node. You could use [our example Ansible playbook](template-ansible.html) to do that.

6
docs/server/source/clusters-feds/aws-testing-cluster.md
View File

@ -14,11 +14,11 @@ We use some Bash and Python scripts to launch several instances (virtual servers
## Python Setup
The instructions that follow have been tested on Ubuntu 14.04, but may also work on similar distros or operating systems.
The instructions that follow have been tested on Ubuntu 16.04. Similar instructions should work on similar Linux distros.
**Note: Our Python scripts for deploying to AWS use Python 2 because Fabric doesn't work with Python 3.**
You must install the Python package named `fabric`, but it depends on the `cryptography` package, and that depends on some OS-level packages. On Ubuntu 14.04, you can install those OS-level packages using:
You must install the Python package named `fabric`, but it depends on the `cryptography` package, and that depends on some OS-level packages. On Ubuntu 16.04, you can install those OS-level packages using:
```text
sudo apt-get install build-essential libssl-dev libffi-dev python-dev
```
@ -72,7 +72,7 @@ One way to monitor a BigchainDB cluster is to use the monitoring setup described
You can deploy a monitoring server on AWS. To do that, go to the AWS EC2 Console and launch an instance:
1. Choose an AMI: select Ubuntu Server 14.04 LTS.
1. Choose an AMI: select Ubuntu Server 16.04 LTS.
2. Choose an Instance Type: a t2.micro will suffice.
3. Configure Instance Details: you can accept the defaults, but feel free to change them.
4. Add Storage: A "Root" volume type should already be included. You _could_ store monitoring data there (e.g. in a folder named `/influxdb-data`) but we will attach another volume and store the monitoring data there instead. Select "Add New Volume" and an EBS volume type.

88
docs/server/source/dev-and-test/setup-run-node.md
View File

@ -7,25 +7,27 @@ The BigchainDB core dev team develops BigchainDB on recent Ubuntu and Fedora dis
## Option A: Using a Local Dev Machine
First, read through the BigchainDB [CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md). It outlines the steps to setup a machine for developing and testing BigchainDB.
Read through the BigchainDB [CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md). It outlines the steps to setup a machine for developing and testing BigchainDB.
Next, create a default BigchainDB config file (in `$HOME/.bigchaindb`):
### With RethinkDB
Create a default BigchainDB config file (in `$HOME/.bigchaindb`):
```text
bigchaindb -y configure
$ bigchaindb -y configure rethinkdb
```
Note: [The BigchainDB CLI](../server-reference/bigchaindb-cli.html) and the [BigchainDB Configuration Settings](../server-reference/configuration.html) are documented elsewhere. (Click the links.)
Start RethinkDB using:
```text
rethinkdb
$ rethinkdb
```
You can verify that RethinkDB is running by opening the RethinkDB web interface in your web browser. It should be at [http://localhost:8080/](http://localhost:8080/).
To run BigchainDB Server, do:
```text
bigchaindb start
$ bigchaindb start
```
You can [run all the unit tests](running-unit-tests.html) to test your installation.
@ -33,13 +35,37 @@ You can [run all the unit tests](running-unit-tests.html) to test your installat
The BigchainDB [CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md) has more details about how to contribute.
## Option B: Using a Dev Machine on Cloud9
### With MongoDB
Ian Worrall of [Encrypted Labs](http://www.encryptedlabs.com/) wrote a document (PDF) explaining how to set up a BigchainDB (Server) dev machine on Cloud9:
Create a default BigchainDB config file (in `$HOME/.bigchaindb`):
```text
$ bigchaindb -y configure mongodb
```
[Download that document from GitHub](https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/docs/server/source/_static/cloud9.pdf)
Note: [The BigchainDB CLI](../server-reference/bigchaindb-cli.html) and the [BigchainDB Configuration Settings](../server-reference/configuration.html) are documented elsewhere. (Click the links.)
## Option C: Using a Local Dev Machine and Docker
Start MongoDB __3.4+__ using:
```text
$ mongod --replSet=bigchain-rs
```
You can verify that MongoDB is running correctly by checking the output of the
previous command for the line:
```text
waiting for connections on port 27017
```
To run BigchainDB Server, do:
```text
$ bigchaindb start
```
You can [run all the unit tests](running-unit-tests.html) to test your installation.
The BigchainDB [CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md) has more details about how to contribute.
## Option B: Using a Local Dev Machine and Docker
You need to have recent versions of [Docker Engine](https://docs.docker.com/engine/installation/)
and (Docker) [Compose](https://docs.docker.com/compose/install/).
@ -50,6 +76,8 @@ Build the images:
docker-compose build
```
### Docker with RethinkDB
**Note**: If you're upgrading BigchainDB and have previously already built the images, you may need
to rebuild them after the upgrade to install any new dependencies.
@ -62,7 +90,7 @@ docker-compose up -d rdb
The RethinkDB web interface should be accessible at <http://localhost:58080/>.
Depending on which platform, and/or how you are running docker, you may need
to change `localhost` for the `ip` of the machine that is running docker. As a
dummy example, if the `ip` of that machine was `0.0.0.0`, you would accees the
dummy example, if the `ip` of that machine was `0.0.0.0`, you would access the
web interface at: <http://0.0.0.0:58080/>.
Start a BigchainDB node:
@ -83,6 +111,40 @@ If you wish to run the tests:
docker-compose run --rm bdb py.test -v -n auto
```
### Docker with MongoDB
Start MongoDB:
```bash
docker-compose up -d mdb
```
MongoDB should now be up and running. You can check the port binding for the
MongoDB driver port using:
```bash
$ docker-compose port mdb 27017
```
Start a BigchainDB node:
```bash
docker-compose up -d bdb-mdb
```
You can monitor the logs:
```bash
docker-compose logs -f bdb-mdb
```
If you wish to run the tests:
```bash
docker-compose run --rm bdb-mdb py.test -v --database-backend=mongodb
```
### Accessing the HTTP API
A quick check to make sure that the BigchainDB server API is operational:
```bash
@ -123,3 +185,9 @@ root:
```bash
curl 0.0.0.0:32772
```
## Option C: Using a Dev Machine on Cloud9
Ian Worrall of [Encrypted Labs](http://www.encryptedlabs.com/) wrote a document (PDF) explaining how to set up a BigchainDB (Server) dev machine on Cloud9:
[Download that document from GitHub](https://raw.githubusercontent.com/bigchaindb/bigchaindb/master/docs/server/source/_static/cloud9.pdf)

10
docs/server/source/drivers-clients/example-apps.rst
View File

@ -1,10 +0,0 @@
Example Apps
============
.. warning::
There are some example BigchainDB apps (i.e. apps which use BigchainDB) in the GitHub repository named `bigchaindb-examples <https://github.com/bigchaindb/bigchaindb-examples>`_. They were created before there was much of an HTTP API, so instead of communicating with a BigchainDB node via the HTTP API, they communicate directly with the node using the BigchainDB Python server API and the RethinkDB Python Driver. That's not how a real production app would work. The HTTP API is getting better, and we recommend using it to communicate with BigchainDB nodes.
Moreover, because of changes to the BigchainDB Server code, some of the examples in the bigchaindb-examples repo might not work anymore, or they might not work as expected.
In the future, we hope to create a set of examples using the HTTP API (or wrappers of it, such as the Python Driver API).

1
docs/server/source/drivers-clients/index.rst
View File

@ -14,4 +14,3 @@ your choice, and then use the HTTP API directly to post transactions.
http-client-server-api
The Python Driver <https://docs.bigchaindb.com/projects/py-driver/en/latest/index.html>
Transaction CLI <https://docs.bigchaindb.com/projects/cli/en/latest/>
example-apps

2
docs/server/source/nodes/node-requirements.md
View File

@ -9,7 +9,7 @@ Note: This section will be broken apart into several pages, e.g. NTP requirement
* BigchainDB Server requires Python 3.4+ and Python 3.4+ [will run on any modern OS](https://docs.python.org/3.4/using/index.html).
* BigchaindB Server uses the Python `multiprocessing` package and [some functionality in the `multiprocessing` package doesn't work on OS X](https://docs.python.org/3.4/library/multiprocessing.html#multiprocessing.Queue.qsize). You can still use Mac OS X if you use Docker or a virtual machine.
The BigchainDB core dev team uses Ubuntu 14.04, Ubuntu 16.04, Fedora 23, and Fedora 24.
The BigchainDB core dev team uses recent LTS versions of Ubuntu and recent versions of Fedora.
We don't test BigchainDB on Windows or Mac OS X, but you can try.

18
docs/server/source/nodes/setup-run-node.md
View File

@ -94,21 +94,7 @@ If you're testing or developing BigchainDB on a stand-alone node, then you shoul
## Install BigchainDB Server
BigchainDB Server has some OS-level dependencies that must be installed.
On Ubuntu 14.04, we found that the following was enough:
```text
sudo apt-get update
sudo apt-get install g++ python3-dev libffi-dev
```
On Fedora 23, we found that the following was enough (tested in February 2015):
```text
sudo dnf update
sudo dnf install gcc-c++ redhat-rpm-config python3-devel libffi-devel
```
(If you're using a version of Fedora before version 22, you may have to use `yum` instead of `dnf`.)
First, [install the OS-level dependencies of BigchainDB Server (link)](../appendices/install-os-level-deps.html).
With OS-level dependencies installed, you can install BigchainDB Server with `pip` or from source.
@ -122,7 +108,7 @@ pip -V
If it says that `pip` isn't installed, or it says `pip` is associated with a Python version less than 3.4, then you must install a `pip` version associated with Python 3.4+. In the following instructions, we call it `pip3` but you may be able to use `pip` if that refers to the same thing. See [the `pip` installation instructions](https://pip.pypa.io/en/stable/installing/).
On Ubuntu 14.04, we found that this works:
On Ubuntu 16.04, we found that this works:
```text
sudo apt-get install python3-pip
```

43
docs/server/source/quickstart.md
View File

@ -2,34 +2,55 @@
This page has instructions to set up a single stand-alone BigchainDB node for learning or experimenting. Instructions for other cases are [elsewhere](introduction.html). We will assume you're using Ubuntu 16.04 or similar. If you're not using Linux, then you might try [running BigchainDB with Docker](appendices/run-with-docker.html).
A. [Install RethinkDB Server](https://rethinkdb.com/docs/install/ubuntu/)
A. Install the database backend.
B. Open a Terminal and run RethinkDB Server with the command:
[Install RethinkDB Server](https://rethinkdb.com/docs/install/ubuntu/) or
[Install MongoDB Server 3.4+](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/)
B. Run the database backend. Open a Terminal and run the command:
with RethinkDB
```text
rethinkdb
$ rethinkdb
```
with MongoDB __3.4+__
```text
$ mongod --replSet=bigchain-rs
```
C. Ubuntu 16.04 already has Python 3.5, so you don't need to install it, but you do need to install some other things:
```text
sudo apt-get update
sudo apt-get install g++ python3-dev libffi-dev
$ sudo apt-get update
$ sudo apt-get install g++ python3-dev libffi-dev
```
D. Get the latest version of pip and setuptools:
```text
sudo apt-get install python3-pip
sudo pip3 install --upgrade pip setuptools
$ sudo apt-get install python3-pip
$ sudo pip3 install --upgrade pip setuptools
```
E. Install the `bigchaindb` Python package from PyPI:
```text
sudo pip3 install bigchaindb
$ sudo pip3 install bigchaindb
```
F. Configure and run BigchainDB Server:
F. Configure the BigchainDB Server: and run BigchainDB Server:
with RethinkDB
```text
bigchaindb -y configure
bigchaindb start
$ bigchaindb -y configure rethinkdb
```
with MongoDB
```text
$ bigchaindb -y configure mongodb
```
G. Run the BigchainDB Server:
```text
$ bigchaindb start
```
That's it!

32
docs/server/source/server-reference/bigchaindb-cli.md
View File

@ -15,18 +15,22 @@ Show the version number. `bigchaindb -v` does the same thing.
## bigchaindb configure
Generate a local config file (which can be used to set some or all [BigchainDB node configuration settings](configuration.html)). It will auto-generate a public-private keypair and then ask you for the values of other configuration settings. If you press Enter for a value, it will use the default value.
Generate a local configuration file (which can be used to set some or all [BigchainDB node configuration settings](configuration.html)). It will auto-generate a public-private keypair and then ask you for the values of other configuration settings. If you press Enter for a value, it will use the default value.
Since BigchainDB supports multiple databases you need to always specify the
database backend that you want to use. At this point only two database backends
are supported: `rethinkdb` and `mongodb`.
If you use the `-c` command-line option, it will generate the file at the specified path:
```text
bigchaindb -c path/to/new_config.json configure
bigchaindb -c path/to/new_config.json configure rethinkdb
```
If you don't use the `-c` command-line option, the file will be written to `$HOME/.bigchaindb` (the default location where BigchainDB looks for a config file, if one isn't specified).
If you use the `-y` command-line option, then there won't be any interactive prompts: it will just generate a keypair and use the default values for all the other configuration settings.
```text
bigchaindb -y configure
bigchaindb -y configure rethinkdb
```
@ -83,3 +87,25 @@ Set the number of replicas (of each shard) in the underlying datastore. For exam
```text
$ bigchaindb set-replicas 3
```
## bigchaindb add-replicas
This command is specific to MongoDB so it will only run if BigchainDB is
configured with `mongodb` as the backend.
This command is used to add nodes to a BigchainDB cluster. It accepts a list of
space separated hosts in the form _hostname:port_:
```text
$ bigchaindb add-replicas server1.com:27017 server2.com:27017 server3.com:27017
```
## bigchaindb remove-replicas
This command is specific to MongoDB so it will only run if BigchainDB is
configured with `mongodb` as the backend.
This command is used to remove nodes from a BigchainDB cluster. It accepts a
list of space separated hosts in the form _hostname:port_:
```text
$ bigchaindb remove-replicas server1.com:27017 server2.com:27017 server3.com:27017
```

15
ntools/one-m/ansible/install-python2.yml Normal file
View File

@ -0,0 +1,15 @@
---
# This playbook ensures Python 2 is installed on the managed node.
# This is inspired by https://gist.github.com/gwillem/4ba393dceb55e5ae276a87300f6b8e6f
- hosts: all
gather_facts: false
remote_user: ubuntu
pre_tasks:
- name: Install Python 2
raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
become: true
# action: setup will gather facts after python2 has been installed
- action: setup

18
ntools/one-m/ansible/roles/bigchaindb/tasks/main.yml
View File

@ -10,22 +10,24 @@
apt: name={{item}} state=latest update_cache=yes
become: true
with_items:
- make
- git
- g++
- python3-dev
- libffi-dev
- python3-setuptools # mainly for easy_install3, which is used to get latest pip3
# This should make both pip and pip3 be pip version >=8.1.2 (python 3.4).
# See the comments about this below.
- name: Ensure the latest pip/pip3 is installed, using easy_install3
easy_install: executable=easy_install3 name=pip state=latest
become: true
- python3-dev
- python3-pip
- python3-setuptools
- name: Ensure the latest setuptools (Python package) is installed
pip: executable=pip3 name=setuptools state=latest
become: true
# This should make both pip and pip3 be pip version >=8.1.2 (python 3.4).
# See the comments about this below.
#- name: Ensure the latest pip/pip3 is installed, using easy_install3
# easy_install: executable=easy_install3 name=pip state=latest
# become: true
- name: Install BigchainDB from PyPI using sudo pip3 install bigchaindb
pip: executable=pip3 name=bigchaindb state=latest
become: true

4
ntools/one-m/ansible/roles/db_storage/tasks/main.yml
View File

@ -12,12 +12,14 @@
# To better understand the /etc/fstab fields/columns, see:
# http://man7.org/linux/man-pages/man5/fstab.5.html
# https://tinyurl.com/jmmsyon = the soure code of the mount module
# Note: It seems the "nobootwait" option is gone in Ubuntu 16.04. See
# https://askubuntu.com/questions/786928/ubuntu-16-04-fstab-fails-with-nobootwait
- name: Ensure /data dir exists and is mounted + update /etc/fstab
mount:
name=/data
src=/dev/xvdp
fstype=ext4
opts="defaults,nofail,nobootwait"
opts="defaults,nofail"
dump=0
passno=2
state=mounted

9
ntools/one-m/ansible/roles/rethinkdb/tasks/main.yml
View File

@ -2,11 +2,12 @@
# ansible/roles/rethinkdb/tasks/main.yml
# Note: the .list extension will be added to the rethinkdb filename automatically
# Note: xenial is the $DISTRIB_CODENAME for Ubuntu 16.04
- name: >
Ensure RethinkDB's APT repository for Ubuntu trusty is present
Ensure RethinkDB's APT repository for Ubuntu xenial is present
in /etc/apt/sources.list.d/rethinkdb.list
apt_repository:
repo='deb http://download.rethinkdb.com/apt trusty main'
repo='deb http://download.rethinkdb.com/apt xenial main'
filename=rethinkdb
state=present
become: true
@ -15,8 +16,8 @@
apt_key: url=http://download.rethinkdb.com/apt/pubkey.gpg state=present
become: true
- name: Ensure the Ubuntu package rethinkdb 2.3.4~0trusty is installed
apt: name=rethinkdb=2.3.4~0trusty state=present update_cache=yes
- name: Ensure the latest rethinkdb package is installed
apt: name=rethinkdb state=latest update_cache=yes
become: true
- name: Ensure the /data directory's owner and group are both 'rethinkdb'

19
ntools/one-m/aws/amis.tf
View File

@ -2,19 +2,20 @@
# even though the contents are the same.
# This file has the mapping from region --> AMI name.
#
# These are all Ubuntu 14.04 LTS AMIs
# These are all Ubuntu 16.04 LTS AMIs
# with Arch = amd64, Instance Type = hvm:ebs-ssd
# from https://cloud-images.ubuntu.com/locator/ec2/
# as of Jan. 31, 2017
variable "amis" {
type = "map"
default = {
eu-west-1 = "ami-55452e26"
eu-central-1 = "ami-b1cf39de"
us-east-1 = "ami-8e0b9499"
us-west-2 = "ami-547b3834"
ap-northeast-1 = "ami-49d31328"
ap-southeast-1 = "ami-5e429c3d"
ap-southeast-2 = "ami-25f3c746"
sa-east-1 = "ami-97980efb"
eu-west-1 = "ami-d8f4deab"
eu-central-1 = "ami-5aee2235"
us-east-1 = "ami-6edd3078"
us-west-2 = "ami-7c803d1c"
ap-northeast-1 = "ami-eb49358c"
ap-southeast-1 = "ami-b1943fd2"
ap-southeast-2 = "ami-fe71759d"
sa-east-1 = "ami-7379e31f"
}
}

108
tests/backend/mongodb/test_admin.py Normal file
View File

@ -0,0 +1,108 @@
"""Tests for the :mod:`bigchaindb.backend.mongodb.admin` module."""
import copy
from unittest import mock
import pytest
from pymongo.database import Database
from pymongo.errors import OperationFailure
@pytest.fixture
def mock_replicaset_config():
return {
'config': {
'_id': 'bigchain-rs',
'members': [
{
'_id': 0,
'arbiterOnly': False,
'buildIndexes': True,
'hidden': False,
'host': 'localhost:27017',
'priority': 1.0,
'slaveDelay': 0,
'tags': {},
'votes': 1
}
],
'version': 1
}
}
@pytest.fixture
def connection():
from bigchaindb.backend import connect
connection = connect()
# connection is a lazy object. It only actually creates a connection to
# the database when its first used.
# During the setup of a MongoDBConnection some `Database.command` are
# executed to make sure that the replica set is correctly initialized.
# Here we force the the connection setup so that all required
# `Database.command` are executed before we mock them it in the tests.
connection._connect()
return connection
def test_add_replicas(mock_replicaset_config, connection):
from bigchaindb.backend.admin import add_replicas
expected_config = copy.deepcopy(mock_replicaset_config)
expected_config['config']['members'] += [
{'_id': 1, 'host': 'localhost:27018'},
{'_id': 2, 'host': 'localhost:27019'}
]
expected_config['config']['version'] += 1
with mock.patch.object(Database, 'command') as mock_command:
mock_command.return_value = mock_replicaset_config
add_replicas(connection, ['localhost:27018', 'localhost:27019'])
mock_command.assert_called_with('replSetReconfig',
expected_config['config'])
def test_add_replicas_raises(mock_replicaset_config, connection):
from bigchaindb.backend.admin import add_replicas
from bigchaindb.backend.exceptions import OperationError
with mock.patch.object(Database, 'command') as mock_command:
mock_command.side_effect = [
mock_replicaset_config,
OperationFailure(error=1, details={'errmsg': ''})
]
with pytest.raises(OperationError):
add_replicas(connection, ['localhost:27018'])
def test_remove_replicas(mock_replicaset_config, connection):
from bigchaindb.backend.admin import remove_replicas
expected_config = copy.deepcopy(mock_replicaset_config)
expected_config['config']['version'] += 1
# add some hosts to the configuration to remove
mock_replicaset_config['config']['members'] += [
{'_id': 1, 'host': 'localhost:27018'},
{'_id': 2, 'host': 'localhost:27019'}
]
with mock.patch.object(Database, 'command') as mock_command:
mock_command.return_value = mock_replicaset_config
remove_replicas(connection, ['localhost:27018', 'localhost:27019'])
mock_command.assert_called_with('replSetReconfig',
expected_config['config'])
def test_remove_replicas_raises(mock_replicaset_config, connection):
from bigchaindb.backend.admin import remove_replicas
from bigchaindb.backend.exceptions import OperationError
with mock.patch.object(Database, 'command') as mock_command:
mock_command.side_effect = [
mock_replicaset_config,
OperationFailure(error=1, details={'errmsg': ''})
]
with pytest.raises(OperationError):
remove_replicas(connection, ['localhost:27018'])

8
tests/backend/mongodb/test_schema.py
View File

@ -21,8 +21,8 @@ def test_init_creates_db_tables_and_indexes():
assert sorted(collection_names) == ['backlog', 'bigchain', 'votes']
indexes = conn.conn[dbname]['bigchain'].index_information().keys()
assert sorted(indexes) == ['_id_', 'asset_id', 'block_timestamp',
'transaction_id']
assert sorted(indexes) == ['_id_', 'asset_id', 'block_timestamp', 'inputs',
'outputs', 'transaction_id']
indexes = conn.conn[dbname]['backlog'].index_information().keys()
assert sorted(indexes) == ['_id_', 'assignee__transaction_timestamp',
@ -81,8 +81,8 @@ def test_create_secondary_indexes():
# Bigchain table
indexes = conn.conn[dbname]['bigchain'].index_information().keys()
assert sorted(indexes) == ['_id_', 'asset_id', 'block_timestamp',
'transaction_id']
assert sorted(indexes) == ['_id_', 'asset_id', 'block_timestamp', 'inputs',
'outputs', 'transaction_id']
# Backlog table
indexes = conn.conn[dbname]['backlog'].index_information().keys()

4
tests/backend/rethinkdb/test_schema.py
View File

@ -85,6 +85,10 @@ def test_create_secondary_indexes():
'transaction_id')) is True
assert conn.run(r.db(dbname).table('bigchain').index_list().contains(
'asset_id')) is True
assert conn.run(r.db(dbname).table('bigchain').index_list().contains(
'inputs')) is True
assert conn.run(r.db(dbname).table('bigchain').index_list().contains(
'outputs')) is True
# Backlog table
assert conn.run(r.db(dbname).table('backlog').index_list().contains(

28
tests/backend/test_generics.py
View File

@ -1,6 +1,3 @@
from importlib import import_module
from unittest.mock import patch
from pytest import mark, raises
@ -69,34 +66,13 @@ def test_changefeed_class(changefeed_class_func_name, args_qty):
changefeed_class_func(None, *range(args_qty))
@mark.parametrize('db,conn_cls', (
('mongodb', 'MongoDBConnection'),
('rethinkdb', 'RethinkDBConnection'),
))
@patch('bigchaindb.backend.schema.create_indexes',
autospec=True, return_value=None)
@patch('bigchaindb.backend.schema.create_tables',
autospec=True, return_value=None)
@patch('bigchaindb.backend.schema.create_database',
autospec=True, return_value=None)
def test_init_database(mock_create_database, mock_create_tables,
mock_create_indexes, db, conn_cls):
from bigchaindb.backend.schema import init_database
conn = getattr(
import_module('bigchaindb.backend.{}.connection'.format(db)),
conn_cls,
)('host', 'port', 'dbname')
init_database(connection=conn, dbname='mickeymouse')
mock_create_database.assert_called_once_with(conn, 'mickeymouse')
mock_create_tables.assert_called_once_with(conn, 'mickeymouse')
mock_create_indexes.assert_called_once_with(conn, 'mickeymouse')
@mark.parametrize('admin_func_name,kwargs', (
('get_config', {'table': None}),
('reconfigure', {'table': None, 'shards': None, 'replicas': None}),
('set_shards', {'shards': None}),
('set_replicas', {'replicas': None}),
('add_replicas', {'replicas': None}),
('remove_replicas', {'replicas': None}),
))
def test_admin(admin_func_name, kwargs):
from bigchaindb.backend import admin

113
tests/commands/test_commands.py
View File

@ -1,6 +1,6 @@
import json
from unittest.mock import Mock, patch
from argparse import Namespace
from argparse import Namespace, ArgumentTypeError
import copy
import pytest
@ -12,7 +12,8 @@ def test_make_sure_we_dont_remove_any_command():
parser = create_parser()
assert parser.parse_args(['configure']).command
assert parser.parse_args(['configure', 'rethinkdb']).command
assert parser.parse_args(['configure', 'mongodb']).command
assert parser.parse_args(['show-config']).command
assert parser.parse_args(['export-my-pubkey']).command
assert parser.parse_args(['init']).command
@ -21,6 +22,8 @@ def test_make_sure_we_dont_remove_any_command():
assert parser.parse_args(['set-shards', '1']).command
assert parser.parse_args(['set-replicas', '1']).command
assert parser.parse_args(['load']).command
assert parser.parse_args(['add-replicas', 'localhost:27017']).command
assert parser.parse_args(['remove-replicas', 'localhost:27017']).command
def test_start_raises_if_command_not_implemented():
@ -31,8 +34,8 @@ def test_start_raises_if_command_not_implemented():
with pytest.raises(NotImplementedError):
# Will raise because `scope`, the third parameter,
# doesn't contain the function `run_configure`
utils.start(parser, ['configure'], {})
# doesn't contain the function `run_start`
utils.start(parser, ['start'], {})
def test_start_raises_if_no_arguments_given():
@ -204,7 +207,7 @@ def test_run_configure_when_config_does_not_exist(monkeypatch,
from bigchaindb.commands.bigchain import run_configure
monkeypatch.setattr('os.path.exists', lambda path: False)
monkeypatch.setattr('builtins.input', lambda: '\n')
args = Namespace(config='foo', yes=True)
args = Namespace(config='foo', backend='rethinkdb', yes=True)
return_value = run_configure(args)
assert return_value is None
@ -228,6 +231,36 @@ def test_run_configure_when_config_does_exist(monkeypatch,
assert value == {}
@pytest.mark.parametrize('backend', (
'rethinkdb',
'mongodb',
))
def test_run_configure_with_backend(backend, monkeypatch, mock_write_config):
import bigchaindb
from bigchaindb.commands.bigchain import run_configure
value = {}
def mock_write_config(new_config, filename=None):
value['return'] = new_config
monkeypatch.setattr('os.path.exists', lambda path: False)
monkeypatch.setattr('builtins.input', lambda: '\n')
monkeypatch.setattr('bigchaindb.config_utils.write_config',
mock_write_config)
args = Namespace(config='foo', backend=backend, yes=True)
expected_config = bigchaindb.config
run_configure(args)
# update the expected config with the correct backend and keypair
backend_conf = getattr(bigchaindb, '_database_' + backend)
expected_config.update({'database': backend_conf,
'keypair': value['return']['keypair']})
assert value['return'] == expected_config
@patch('bigchaindb.common.crypto.generate_key_pair',
return_value=('private_key', 'public_key'))
@pytest.mark.usefixtures('ignore_local_config_file')
@ -345,3 +378,73 @@ def test_calling_main(start_mock, base_parser_mock, parse_args_mock,
'distributed equally to all '
'the processes')
assert start_mock.called is True
@pytest.mark.usefixtures('ignore_local_config_file')
@patch('bigchaindb.commands.bigchain.add_replicas')
def test_run_add_replicas(mock_add_replicas):
from bigchaindb.commands.bigchain import run_add_replicas
from bigchaindb.backend.exceptions import OperationError
args = Namespace(config=None, replicas=['localhost:27017'])
# test add_replicas no raises
mock_add_replicas.return_value = None
assert run_add_replicas(args) is None
assert mock_add_replicas.call_count == 1
mock_add_replicas.reset_mock()
# test add_replicas with `OperationError`
mock_add_replicas.side_effect = OperationError()
assert run_add_replicas(args) is None
assert mock_add_replicas.call_count == 1
mock_add_replicas.reset_mock()
# test add_replicas with `NotImplementedError`
mock_add_replicas.side_effect = NotImplementedError()
assert run_add_replicas(args) is None
assert mock_add_replicas.call_count == 1
mock_add_replicas.reset_mock()
@pytest.mark.usefixtures('ignore_local_config_file')
@patch('bigchaindb.commands.bigchain.remove_replicas')
def test_run_remove_replicas(mock_remove_replicas):
from bigchaindb.commands.bigchain import run_remove_replicas
from bigchaindb.backend.exceptions import OperationError
args = Namespace(config=None, replicas=['localhost:27017'])
# test add_replicas no raises
mock_remove_replicas.return_value = None
assert run_remove_replicas(args) is None
assert mock_remove_replicas.call_count == 1
mock_remove_replicas.reset_mock()
# test add_replicas with `OperationError`
mock_remove_replicas.side_effect = OperationError()
assert run_remove_replicas(args) is None
assert mock_remove_replicas.call_count == 1
mock_remove_replicas.reset_mock()
# test add_replicas with `NotImplementedError`
mock_remove_replicas.side_effect = NotImplementedError()
assert run_remove_replicas(args) is None
assert mock_remove_replicas.call_count == 1
mock_remove_replicas.reset_mock()
def test_mongodb_host_type():
from bigchaindb.commands.utils import mongodb_host
# bad port provided
with pytest.raises(ArgumentTypeError):
mongodb_host('localhost:11111111111')
# no port information provided
with pytest.raises(ArgumentTypeError):
mongodb_host('localhost')
# bad host provided
with pytest.raises(ArgumentTypeError):
mongodb_host(':27017')

13
tests/conftest.py
View File

@ -109,26 +109,23 @@ def _restore_dbs(request):
@pytest.fixture(scope='session')
def _configure_bigchaindb(request):
import bigchaindb
from bigchaindb import config_utils
test_db_name = TEST_DB_NAME
# Put a suffix like _gw0, _gw1 etc on xdist processes
xdist_suffix = getattr(request.config, 'slaveinput', {}).get('slaveid')
if xdist_suffix:
test_db_name = '{}_{}'.format(TEST_DB_NAME, xdist_suffix)
backend = request.config.getoption('--database-backend')
config = {
'database': {
'name': test_db_name,
'backend': request.config.getoption('--database-backend'),
},
'database': bigchaindb._database_map[backend],
'keypair': {
'private': '31Lb1ZGKTyHnmVK3LUMrAUrPNfd4sE2YyBt3UA4A25aA',
'public': '4XYfCbabAWVUCbjTmRTFEu2sc3dFEdkse4r6X498B1s8',
}
}
# FIXME
if config['database']['backend'] == 'mongodb':
# not a great way to do this
config['database']['port'] = 27017
config['database']['name'] = test_db_name
config_utils.set_config(config)

18
tests/db/test_bigchain_api.py
View File

@ -1222,3 +1222,21 @@ def test_cant_spend_same_input_twice_in_tx(b, genesis_block):
assert b.is_valid_transaction(tx_transfer_signed) is False
with pytest.raises(DoubleSpend):
tx_transfer_signed.validate(b)
@pytest.mark.bdb
def test_transaction_unicode(b):
from bigchaindb.common.utils import serialize
from bigchaindb.models import Transaction
# http://www.fileformat.info/info/unicode/char/1f37a/index.htm
beer_python = {'beer': '\N{BEER MUG}'}
beer_json = '{"beer":"\N{BEER MUG}"}'
tx = (Transaction.create([b.me], [([b.me], 100)], beer_python)
).sign([b.me_private])
block = b.create_block([tx])
b.write_block(block)
assert b.get_block(block.id) == block.to_dict()
assert block.validate(b) == block
assert beer_json in serialize(block.to_dict())

84
tests/test_config_utils.py
View File

@ -10,24 +10,32 @@ ORIGINAL_CONFIG = copy.deepcopy(bigchaindb._config)
@pytest.fixture(scope='function', autouse=True)
def clean_config(monkeypatch):
monkeypatch.setattr('bigchaindb.config', copy.deepcopy(ORIGINAL_CONFIG))
def clean_config(monkeypatch, request):
import bigchaindb
original_config = copy.deepcopy(ORIGINAL_CONFIG)
backend = request.config.getoption('--database-backend')
original_config['database'] = bigchaindb._database_map[backend]
monkeypatch.setattr('bigchaindb.config', original_config)
def test_bigchain_instance_is_initialized_when_conf_provided():
def test_bigchain_instance_is_initialized_when_conf_provided(request):
import bigchaindb
from bigchaindb import config_utils
assert 'CONFIGURED' not in bigchaindb.config
config_utils.set_config({'keypair': {'public': 'a', 'private': 'b'}})
assert bigchaindb.config['CONFIGURED'] is True
b = bigchaindb.Bigchain()
assert b.me
assert b.me_private
def test_bigchain_instance_raises_when_not_configured(monkeypatch):
def test_bigchain_instance_raises_when_not_configured(request, monkeypatch):
import bigchaindb
from bigchaindb import config_utils
from bigchaindb.common import exceptions
assert 'CONFIGURED' not in bigchaindb.config
@ -101,42 +109,64 @@ def test_env_config(monkeypatch):
def test_autoconfigure_read_both_from_file_and_env(monkeypatch, request):
# constants
DATABASE_HOST = 'test-host'
DATABASE_NAME = 'test-dbname'
DATABASE_PORT = 4242
DATABASE_BACKEND = request.config.getoption('--database-backend')
SERVER_BIND = '1.2.3.4:56'
KEYRING = 'pubkey_0:pubkey_1:pubkey_2'
file_config = {
'database': {
'host': 'test-host',
'backend': request.config.getoption('--database-backend')
'host': DATABASE_HOST
},
'backlog_reassign_delay': 5
}
monkeypatch.setattr('bigchaindb.config_utils.file_config', lambda *args, **kwargs: file_config)
monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_NAME': 'test-dbname',
'BIGCHAINDB_DATABASE_PORT': '4242',
'BIGCHAINDB_SERVER_BIND': '1.2.3.4:56',
'BIGCHAINDB_KEYRING': 'pubkey_0:pubkey_1:pubkey_2'})
monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_NAME': DATABASE_NAME,
'BIGCHAINDB_DATABASE_PORT': str(DATABASE_PORT),
'BIGCHAINDB_DATABASE_BACKEND': DATABASE_BACKEND,
'BIGCHAINDB_SERVER_BIND': SERVER_BIND,
'BIGCHAINDB_KEYRING': KEYRING})
import bigchaindb
from bigchaindb import config_utils
config_utils.autoconfigure()
database_rethinkdb = {
'backend': 'rethinkdb',
'host': DATABASE_HOST,
'port': DATABASE_PORT,
'name': DATABASE_NAME,
}
database_mongodb = {
'backend': 'mongodb',
'host': DATABASE_HOST,
'port': DATABASE_PORT,
'name': DATABASE_NAME,
'replicaset': 'bigchain-rs',
}
database = {}
if DATABASE_BACKEND == 'mongodb':
database = database_mongodb
elif DATABASE_BACKEND == 'rethinkdb':
database = database_rethinkdb
assert bigchaindb.config == {
'CONFIGURED': True,
'server': {
'bind': '1.2.3.4:56',
'bind': SERVER_BIND,
'workers': None,
'threads': None,
},
'database': {
'backend': request.config.getoption('--database-backend'),
'host': 'test-host',
'port': 4242,
'name': 'test-dbname',
'replicaset': 'bigchain-rs'
},
'database': database,
'keypair': {
'public': None,
'private': None,
},
'keyring': ['pubkey_0', 'pubkey_1', 'pubkey_2'],
'keyring': KEYRING.split(':'),
'statsd': {
'host': 'localhost',
'port': 8125,
@ -215,7 +245,6 @@ def test_write_config():
('BIGCHAINDB_DATABASE_HOST', 'test-host', 'host'),
('BIGCHAINDB_DATABASE_PORT', 4242, 'port'),
('BIGCHAINDB_DATABASE_NAME', 'test-db', 'name'),
('BIGCHAINDB_DATABASE_REPLICASET', 'test-replicaset', 'replicaset')
))
def test_database_envs(env_name, env_value, config_key, monkeypatch):
import bigchaindb
@ -227,3 +256,18 @@ def test_database_envs(env_name, env_value, config_key, monkeypatch):
expected_config['database'][config_key] = env_value
assert bigchaindb.config == expected_config
def test_database_envs_replicaset(monkeypatch):
# the replica set env is only used if the backend is mongodb
import bigchaindb
monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_REPLICASET':
'test-replicaset'})
bigchaindb.config['database'] = bigchaindb._database_mongodb
bigchaindb.config_utils.autoconfigure()
expected_config = copy.deepcopy(bigchaindb.config)
expected_config['database']['replicaset'] = 'test-replicaset'
assert bigchaindb.config == expected_config

13
tests/test_models.py
View File

@ -163,16 +163,3 @@ class TestBlockModel(object):
public_key = PublicKey(b.me)
assert public_key.verify(expected_block_serialized, block.signature)
def test_validate_already_voted_on_block(self, b, monkeypatch):
from unittest.mock import Mock
from bigchaindb.models import Transaction
tx = Transaction.create([b.me], [([b.me], 1)])
block = b.create_block([tx])
has_previous_vote = Mock()
has_previous_vote.return_value = True
monkeypatch.setattr(b, 'has_previous_vote', has_previous_vote)
assert block == block.validate(b)
assert has_previous_vote.called is True