diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index acd281ca..57b1d12f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,7 +12,17 @@ If you want to file a bug report, suggest a feature, or ask a code-related quest ## How to Contribute Code or Documentation -### Step 0 - Prepare and Familiarize Yourself +### Step 0 - Decide on an Issue to Resolve, or Create One + +We want you to feel like your contributions (pull requests) are welcome, but if you contribute something unnecessary, unwanted, or perplexing, then your experience may be unpleasant. Your pull request may sit gathering dust as everyone scratches their heads wondering what to do with it. + +To prevent that situation, we ask that all pull requests should resolve, address, or fix an existing issue. If there is no existing issue, then you should create one first. That way there can be commentary and discussion first, and you can have a better idea of what to expect when you create a corresponding pull request. + +When you submit a pull request, please mention the issue (or issues) that it resolves, e.g. "Resolves #123". + +Exception: hotfixes and minor changes don't require a pre-existing issue, but please write a thorough pull request description. + +### Step 1 - Prepare and Familiarize Yourself To contribute code or documentation, you need a [GitHub account](https://github.com/signup/free). @@ -23,16 +33,13 @@ Familiarize yourself with how we do coding and documentation in the BigchainDB p * the GitHub Flow (workflow) * [GitHub Guide: Understanding the GitHub Flow](https://guides.github.com/introduction/flow/) * [Scott Chacon's blog post about GitHub Flow](http://scottchacon.com/2011/08/31/github-flow.html) - * Note that we call the main branch `develop` rather than `master` * [semantic versioning](http://semver.org/) -Note: We have a slight variation on the GitHub Flow: we call the default branch `develop` rather than `master`. - -### Step 1 - Fork bigchaindb on GitHub +### Step 2 - Fork bigchaindb on GitHub In your web browser, go to [the BigchainDB repository on GitHub](https://github.com/bigchaindb/bigchaindb) and click the `Fork` button in the top right corner. This creates a new Git repository named `bigchaindb` in _your_ GitHub account. -### Step 2 - Clone Your Fork +### Step 3 - Clone Your Fork (This only has to be done once.) In your local terminal, use Git to clone _your_ `bigchaindb` repository to your local computer. Also add the original GitHub bigchaindb/bigchaindb repository as a remote named `upstream` (a convention): ```text @@ -41,16 +48,37 @@ cd bigchaindb git add upstream git@github.com:bigchaindb/bigchaindb.git ``` -### Step 3 - Fetch and Merge the Latest from `upstream/develop` +### Step 4 - Fetch and Merge the Latest from `upstream/master` -Switch to the `develop` branch locally, fetch all `upstream` branches, and merge the just-fetched `upstream/develop` branch with the local `develop` branch: +Switch to the `master` branch locally, fetch all `upstream` branches, and merge the just-fetched `upstream/master` branch with the local `master` branch: ```text -git checkout develop +git checkout master git fetch upstream -git merge upstream/develop +git merge upstream/master ``` -### Step 4 - Create a New Branch for Each Bug/Feature +### Step 5 - Install the Python module and the CLI + +In order to use and run the source you just cloned from your fork, you need to install BigchainDB on your computer. +The core of BigchainDB is a Python module you can install using the standard [Python packaging tools](http://python-packaging-user-guide.readthedocs.org/en/latest/). +We highly suggest you use `pip` and `virtualenv` to manage your local development. +If you need more information on how to do that, refer to the *Python Packaging User Guide* to [install `pip`](http://python-packaging-user-guide.readthedocs.org/en/latest/installing/#requirements-for-installing-packages) and to [create your first `virtualenv`](http://python-packaging-user-guide.readthedocs.org/en/latest/installing/#creating-virtual-environments). + +Once you have `pip` installed and (optionally) you are in a virtualenv, go to the root of the repository (i.e. where the `setup.py` file is), and type: +```bash +$ pip install -e .[dev] +``` + +This will install the BigchainDB Python module, the CLI, and all the dependencies useful for contributing to the development of BigchainDB. +How? Let's split the command down into its components: + - `pip` is the Python command to install packages + - `install` tells pip to use the *install* action + - `-e` installs a project in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs) + - `.` installs what's in the current directory + - `[dev]` adds some [extra requirements](https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies) to the installation. (If you are curious, open `setup.py` and look for `dev` in the `extras_require` section.) + + +### Step 6 - Create a New Branch for Each Bug/Feature If your new branch is to **fix a bug** identified in a specific GitHub Issue with number `ISSNO`, then name your new branch `bug/ISSNO/short-description-here`. For example, `bug/67/fix-leap-year-crash`. @@ -61,7 +89,7 @@ Otherwise, please give your new branch a short, descriptive, all-lowercase name. git checkout -b new-branch-name ``` -### Step 5 - Make Edits, git add, git commit +### Step 7 - Make Edits, git add, git commit With your new branch checked out locally, make changes or additions to the code or documentation. Remember to: @@ -79,27 +107,27 @@ git commit -m "Short description of new or changed things" You will want to merge changes from upstream (i.e. the original repository) into your new branch from time to time, using something like: ```text git fetch upstream -git merge upstream/develop +git merge upstream/master ``` Once you're done commiting a set of new things and you're ready to submit them for inclusion, please be sure to run all the tests (as per the instructions at the end of our [Python Style Guide](PYTHON_STYLE_GUIDE.md)). -If your addition or change is substantial, then please add a line or two to the [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/CHANGELOG.md), following the guidelines given at the top of that file. +If your addition or change is substantial, then please add a line or two to the [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CHANGELOG.md), following the guidelines given at the top of that file. (When you submit your pull request [following the instructions below], we run all the tests automatically, so we will see if some are failing. If you don't know why some tests are failing, you can still submit your pull request, but be sure to note the failing tests and to ask for help with resolving them.) -### Step 6 - Push Your New Branch to origin +### Step 8 - Push Your New Branch to origin Make sure you've commited all the additions or changes you want to include in your pull request. Then push your new branch to origin (i.e. _your_ remote bigchaindb repository). ```text git push origin new-branch-name ``` -### Step 7 - Create a Pull Request +### Step 9 - Create a Pull Request Go to the GitHub website and to _your_ remote bigchaindb repository (i.e. something like https://github.com/your-user-name/bigchaindb). -See [GitHub's documentation on how to initiate and send a pull request](https://help.github.com/articles/using-pull-requests/). Note that the destination repository should be `bigchaindb/bigchaindb` and the destination branch will be `develop` (usually, and if it's not, then we can change that if necessary). +See [GitHub's documentation on how to initiate and send a pull request](https://help.github.com/articles/using-pull-requests/). Note that the destination repository should be `bigchaindb/bigchaindb` and the destination branch will be `master` (usually, and if it's not, then we can change that if necessary). If this is the first time you've submitted a pull request to BigchainDB, then you must read and accept the Contributor License Agreement (CLA) before we can merge your contributions. That can be found at [https://www.bigchaindb.com/cla](https://www.bigchaindb.com/cla). @@ -115,4 +143,4 @@ Someone will then merge your branch or suggest changes. If we suggsest changes, * [BigchainDB Licenses](./LICENSES.md) * [Contributor License Agreement](https://www.bigchaindb.com/cla) -(Note: GitHub automatically links to CONTRIBUTING.md when a contributor creates an Issue or opens a Pull Request.) \ No newline at end of file +(Note: GitHub automatically links to CONTRIBUTING.md when a contributor creates an Issue or opens a Pull Request.) diff --git a/PYTHON_STYLE_GUIDE.md b/PYTHON_STYLE_GUIDE.md index 4d4d4888..61863860 100644 --- a/PYTHON_STYLE_GUIDE.md +++ b/PYTHON_STYLE_GUIDE.md @@ -55,12 +55,27 @@ x = 'name: {}; score: {}'.format(name, n) we use the `format()` version. The [official Python documentation says](https://docs.python.org/2/library/stdtypes.html#str.format), "This method of string formatting is the new standard in Python 3, and should be preferred to the % formatting described in String Formatting Operations in new code." -## Writing (Python) Tests +## Writing and Running (Python) Unit Tests We write unit tests for our Python code using the [pytest](http://pytest.org/latest/) framework. All tests go in the `bigchaindb/tests` directory or one of its subdirectories. You can use the tests already in there as templates or examples. -The BigchainDB Documentation has a [section explaining how to run all unit tests](http://bigchaindb.readthedocs.org/en/develop/running-unit-tests.html). +You can run all unit tests using: +```text +py.test -v +``` + +or, if that doesn't work, try: +```text +python -m pytest -v +``` + +or: +```text +python setup.py test +``` + +If you want to learn about all the things you can do with pytest, see [the pytest documentation](http://pytest.org/latest/). **Automated testing of pull requests.** We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.) diff --git a/README.md b/README.md index e72a55a8..5c38f639 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,21 @@ +[![Join the chat at https://gitter.im/bigchaindb/bigchaindb](https://badges.gitter.im/bigchaindb/bigchaindb.svg)](https://gitter.im/bigchaindb/bigchaindb?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +[![PyPI](https://img.shields.io/pypi/v/bigchaindb.svg)](https://pypi.python.org/pypi/BigchainDB) +[![Travis branch](https://img.shields.io/travis/bigchaindb/bigchaindb/master.svg)](https://travis-ci.org/bigchaindb/bigchaindb) +[![Codecov branch](https://img.shields.io/codecov/c/github/bigchaindb/bigchaindb/master.svg)](https://codecov.io/github/bigchaindb/bigchaindb?branch=master) +[![Documentation Status](https://readthedocs.org/projects/bigchaindb/badge/?version=stable)](https://bigchaindb.readthedocs.org/en/stable/) + + # BigchainDB A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whitepaper/) explains what that means. -[![Join the chat at https://gitter.im/bigchaindb/bigchaindb](https://badges.gitter.im/bigchaindb/bigchaindb.svg)](https://gitter.im/bigchaindb/bigchaindb?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -[![PyPI](https://img.shields.io/pypi/v/bigchaindb.svg)](https://pypi.python.org/pypi/BigchainDB) -[![Travis branch](https://img.shields.io/travis/bigchaindb/bigchaindb/develop.svg)](https://travis-ci.org/bigchaindb/bigchaindb) -[![Codecov branch](https://img.shields.io/codecov/c/github/bigchaindb/bigchaindb/develop.svg)](https://codecov.io/github/bigchaindb/bigchaindb?branch=develop) -[![Documentation Status](https://readthedocs.org/projects/bigchaindb/badge/?version=develop)](http://bigchaindb.readthedocs.org/en/develop/?badge=develop) ## Quick Start -### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.org/en/develop/installing-server.html) -### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.org/en/develop/installing-server.html#run-bigchaindb-with-docker) -### [The Python Server API by Example](http://bigchaindb.readthedocs.org/en/develop/python-server-api-examples.html) -### [The Python Driver API by Example](http://bigchaindb.readthedocs.org/en/develop/python-driver-api-examples.html) +### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.org/en/master/installing-server.html) +### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.org/en/master/installing-server.html#run-bigchaindb-with-docker) +### [The Python Server API by Example](http://bigchaindb.readthedocs.org/en/master/python-server-api-examples.html) +### [The Python Driver API by Example](http://bigchaindb.readthedocs.org/en/master/python-driver-api-examples.html) ## Links for Everyone * [BigchainDB.com](https://www.bigchaindb.com/) - the main BigchainDB website, including newsletter signup @@ -24,7 +26,7 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit * [Google Group](https://groups.google.com/forum/#!forum/bigchaindb) ## Links for Developers -* [Documentation](http://bigchaindb.readthedocs.org/en/develop/#) - for developers +* [Documentation](http://bigchaindb.readthedocs.org/en/master/) - for developers * [CONTRIBUTING.md](CONTRIBUTING.md) - how to contribute * [Community guidelines](CODE_OF_CONDUCT.md) * [Open issues](https://github.com/bigchaindb/bigchaindb/issues) diff --git a/bigchaindb/__init__.py b/bigchaindb/__init__.py index 414397f6..6a462ab7 100644 --- a/bigchaindb/__init__.py +++ b/bigchaindb/__init__.py @@ -1,49 +1,35 @@ -import os import copy - -def e(key, default=None, conv=None): - '''Get the environment variable `key`, fallback to `default` - if nothing is found. - - Keyword arguments: - key -- the key to look for in the environment - default -- the default value if nothing is found (default: None) - conv -- a callable used to convert the value (default: use the type of the - default value) - ''' - - val = os.environ.get(key, default) - - if conv or default is not None: - conv = conv or type(default) - return conv(val) - - return val +# from functools import reduce +# PORT_NUMBER = reduce(lambda x, y: x * y, map(ord, 'BigchainDB')) % 2**16 +# basically, the port number is 9984 config = { 'server': { - 'bind': e('BIGCHAIN_SERVER_BIND', default='0.0.0.0:5000'), + # Note: this section supports all the Gunicorn settings: + # - http://docs.gunicorn.org/en/stable/settings.html + 'bind': '0.0.0.0:9984', + 'workers': None, # if none, the value will be cpu_count * 2 + 1 + 'threads': None, # if none, the value will be cpu_count * 2 + 1 }, 'database': { - 'host': e('BIGCHAIN_DATABASE_HOST', default='localhost'), - 'port': e('BIGCHAIN_DATABASE_PORT', default=28015), - 'name': e('BIGCHAIN_DATABASE_NAME', default='bigchain') + 'host': 'localhost', + 'port': 28015, + 'name': 'bigchain', }, 'keypair': { - 'public': e('BIGCHAIN_KEYPAIR_PUBLIC'), - 'private': e('BIGCHAIN_KEYPAIR_PRIVATE') + 'public': None, + 'private': None, }, - 'keyring': [ - ], + 'keyring': [], 'statsd': { - 'host': e('BIGCHAIN_STATSD_HOST', default='localhost'), - 'port': e('BIGCHAIN_STATSD_PORT', default=8125), - 'rate': e('BIGCHAIN_STATSD_SAMPLERATE', default=0.01) + 'host': 'localhost', + 'port': 8125, + 'rate': 0.01, }, - 'api_endpoint': 'http://localhost:8008/api/v1', - 'consensus_plugin': e('BIGCHAIN_CONSENSUS_PLUGIN', default='default') + 'api_endpoint': 'http://localhost:9984/api/v1', + 'consensus_plugin': 'default', } # We need to maintain a backup copy of the original config dict in case diff --git a/bigchaindb/client.py b/bigchaindb/client.py index f7b37ad4..67ce51a9 100644 --- a/bigchaindb/client.py +++ b/bigchaindb/client.py @@ -27,8 +27,8 @@ class Client: 3. Reading them from the `config.json` file. Args: - public_key (str): the base58 encoded public key for the ECDSA secp256k1 curve. - private_key (str): the base58 encoded private key for the ECDSA secp256k1 curve. + public_key (str): the base58 encoded public key for the ED25519 curve. + private_key (str): the base58 encoded private key for the ED25519 curve. api_endpoint (str): a URL where rethinkdb is running. format: scheme://hostname:port consensus_plugin (str): the registered name of your installed diff --git a/bigchaindb/commands/bigchain.py b/bigchaindb/commands/bigchain.py index c1be6761..15092db5 100644 --- a/bigchaindb/commands/bigchain.py +++ b/bigchaindb/commands/bigchain.py @@ -2,14 +2,16 @@ import os +import sys import logging import argparse import copy +import json import bigchaindb import bigchaindb.config_utils from bigchaindb import db -from bigchaindb.exceptions import DatabaseAlreadyExists +from bigchaindb.exceptions import DatabaseAlreadyExists, KeypairNotFoundException from bigchaindb.commands.utils import base_parser, start from bigchaindb.processes import Processes from bigchaindb import crypto @@ -21,13 +23,15 @@ logger = logging.getLogger(__name__) def run_show_config(args): """Show the current configuration""" - from pprint import pprint - # TODO Proposal: remove the "hidden" configuration. Only show config. If # the system needs to be configured, then display information on how to # configure the system. - bigchaindb.config_utils.file_config(args.config) - pprint(bigchaindb.config) + bigchaindb.config_utils.autoconfigure(filename=args.config, force=True) + config = copy.deepcopy(bigchaindb.config) + del config['CONFIGURED'] + private_key = config['keypair']['private'] + config['keypair']['private'] = 'x' * 45 if private_key else None + print(json.dumps(config, indent=4, sort_keys=True)) def run_configure(args, skip_if_exists=False): @@ -55,7 +59,7 @@ def run_configure(args, skip_if_exists=False): conf['keypair']['private'], conf['keypair']['public'] = crypto.generate_key_pair() if not args.yes: - for key in ('host', 'port'): + for key in ('bind', ): val = conf['server'][key] conf['server'][key] = input('API Server {}? (default `{}`): '.format(key, val)) or val @@ -73,7 +77,7 @@ def run_configure(args, skip_if_exists=False): def run_init(args): """Initialize the database""" - bigchaindb.config_utils.file_config(args.config) + bigchaindb.config_utils.autoconfigure(filename=args.config, force=True) # TODO Provide mechanism to: # 1. prompt the user to inquire whether they wish to drop the db # 2. force the init, (e.g., via -f flag) @@ -86,18 +90,21 @@ def run_init(args): def run_drop(args): """Drop the database""" - bigchaindb.config_utils.file_config(args.config) + bigchaindb.config_utils.autoconfigure(filename=args.config, force=True) db.drop(assume_yes=args.yes) def run_start(args): """Start the processes to run the node""" - run_configure(args, skip_if_exists=True) - bigchaindb.config_utils.file_config(args.config) + # run_configure(args, skip_if_exists=True) + bigchaindb.config_utils.autoconfigure(filename=args.config, force=True) try: db.init() except DatabaseAlreadyExists: pass + except KeypairNotFoundException: + sys.exit('Cannot start BigchainDB, no keypair found. Did you run `bigchaindb configure`?') + processes = Processes() logger.info('Start bigchaindb main process') processes.start() diff --git a/bigchaindb/commands/bigchain_benchmark.py b/bigchaindb/commands/bigchain_benchmark.py index 96ba0a5b..2bac4c08 100644 --- a/bigchaindb/commands/bigchain_benchmark.py +++ b/bigchaindb/commands/bigchain_benchmark.py @@ -34,7 +34,7 @@ def _run_load(tx_left, stats): def run_load(args): - bigchaindb.config_utils.file_config(args.config) + bigchaindb.config_utils.autoconfigure(filename=args.config, force=True) logger.info('Starting %s processes', args.multiprocess) stats = logstats.Logstats() logstats.thread.start(stats) diff --git a/bigchaindb/config_utils.py b/bigchaindb/config_utils.py index 645780eb..53eb7954 100644 --- a/bigchaindb/config_utils.py +++ b/bigchaindb/config_utils.py @@ -5,12 +5,12 @@ By calling `file_config`, the global configuration (stored in configuration file. Note that there is a precedence in reading configuration values: - - [not yet] command line; - local config file; - environment vars; - - default config file (contained in `bigchain.__init__`). + - default config file (contained in ``bigchaindb.__init__``). """ + import os import copy import json @@ -23,11 +23,33 @@ import bigchaindb from bigchaindb.consensus import AbstractConsensusRules logger = logging.getLogger(__name__) + CONFIG_DEFAULT_PATH = os.environ.setdefault( 'BIGCHAINDB_CONFIG_PATH', os.path.join(os.path.expanduser('~'), '.bigchaindb'), ) +CONFIG_PREFIX = 'BIGCHAINDB' +CONFIG_SEP = '_' + + +def map_leafs(func, mapping): + """Map a function to the leafs of a mapping.""" + + def _inner(mapping, path=None): + if path is None: + path = [] + + for key, val in mapping.items(): + if isinstance(val, collections.Mapping): + _inner(val, path + [key]) + else: + mapping[key] = func(val, path=path+[key]) + + return mapping + + return _inner(copy.deepcopy(mapping)) + # Thanks Alex <3 # http://stackoverflow.com/a/3233356/597097 @@ -43,7 +65,7 @@ def update(d, u): def file_config(filename=None): - """Read a configuration file and merge it with the default configuration. + """Returns the values found in a configuration file. Args: filename (str): the JSON file with the configuration. Defaults to ``None``. @@ -57,13 +79,73 @@ def file_config(filename=None): filename = CONFIG_DEFAULT_PATH with open(filename) as f: - newconfig = json.load(f) + config = json.load(f) - dict_config(newconfig) logger.info('Configuration loaded from `{}`'.format(filename)) + return config -def dict_config(newconfig): + +def env_config(config): + """Return a new configuration with the values found in the environment. + + The function recursively iterates over the config, checking if there is + a matching env variable. If an env variable is found, the func updates + the configuration with that value. + + The name of the env variable is built combining a prefix (``BIGCHAINDB``) + with the path to the value. If the ``config`` in input is: + ``{'database': {'host': 'localhost'}}`` + this function will try to read the env variable ``BIGCHAINDB_DATABASE_HOST``. + """ + + def load_from_env(value, path): + var_name = CONFIG_SEP.join([CONFIG_PREFIX] + list(map(lambda s: s.upper(), path))) + return os.environ.get(var_name, value) + + return map_leafs(load_from_env, config) + + +def update_types(config, reference, list_sep=':'): + """Return a new configuration where all the values types + are aligned with the ones in the default configuration""" + + def _coerce(current, value): + # Coerce a value to the `current` type. + try: + # First we try to apply current to the value, since it + # might be a function + return current(value) + except TypeError: + # Then we check if current is a list AND if the value + # is a string. + if isinstance(current, list) and isinstance(value, str): + # If so, we use the colon as the separator + return value.split(list_sep) + + try: + # If we are here, we should try to apply the type + # of `current` to the value + return type(current)(value) + except TypeError: + # Worst case scenario we return the value itself. + return value + + def _update_type(value, path): + current = reference + + for elem in path: + try: + current = current[elem] + except KeyError: + return value + + return _coerce(current, value) + + return map_leafs(_update_type, config) + + +def dict_config(config): """Merge the provided configuration with the default one. Args: @@ -74,11 +156,11 @@ def dict_config(newconfig): update made to ``bigchaindb.config`` will be lost. """ bigchaindb.config = copy.deepcopy(bigchaindb._config) - update(bigchaindb.config, newconfig) + update(bigchaindb.config, update_types(config, bigchaindb.config)) bigchaindb.config['CONFIGURED'] = True -def write_config(newconfig, filename=None): +def write_config(config, filename=None): """Write the provided configuration to a specific location. Args: @@ -90,18 +172,29 @@ def write_config(newconfig, filename=None): filename = CONFIG_DEFAULT_PATH with open(filename, 'w') as f: - json.dump(newconfig, f) + json.dump(config, f) -def autoconfigure(): - """Run ``file_config`` if the module has not been initialized. - """ - if bigchaindb.config.get('CONFIGURED'): +def autoconfigure(filename=None, config=None, force=False): + """Run ``file_config`` and ``env_config`` if the module has not + been initialized.""" + + if not force and bigchaindb.config.get('CONFIGURED'): + logger.debug('System already configured, skipping autoconfiguration') return + + newconfig = env_config(bigchaindb.config) + try: - file_config() - except FileNotFoundError: - logger.warning('Cannot find your config file. Run `bigchaindb configure` to create one') + newconfig = update(newconfig, file_config(filename=filename)) + except FileNotFoundError as e: + logger.warning('Cannot find config file `%s`.' % e.filename) + + if config: + newconfig = update(newconfig, config) + + dict_config(newconfig) + return newconfig def load_consensus_plugin(name=None): diff --git a/bigchaindb/consensus.py b/bigchaindb/consensus.py index b65ab9ad..277dad3e 100644 --- a/bigchaindb/consensus.py +++ b/bigchaindb/consensus.py @@ -2,7 +2,7 @@ from abc import ABCMeta, abstractmethod import bigchaindb.exceptions as exceptions from bigchaindb import util -from bigchaindb.crypto import hash_data, PublicKey +from bigchaindb import crypto class AbstractConsensusRules(metaclass=ABCMeta): @@ -156,7 +156,7 @@ class BaseConsensusRules(AbstractConsensusRules): transaction['transaction']['input'])) # Check hash of the transaction - calculated_hash = hash_data(util.serialize( + calculated_hash = crypto.hash_data(util.serialize( transaction['transaction'])) if calculated_hash != transaction['id']: raise exceptions.InvalidHash() @@ -185,7 +185,7 @@ class BaseConsensusRules(AbstractConsensusRules): """ # Check if current hash is correct - calculated_hash = hash_data(util.serialize(block['block'])) + calculated_hash = crypto.hash_data(util.serialize(block['block'])) if calculated_hash != block['id']: raise exceptions.InvalidHash() diff --git a/bigchaindb/core.py b/bigchaindb/core.py index f76dd2d6..ea4b2576 100644 --- a/bigchaindb/core.py +++ b/bigchaindb/core.py @@ -1,6 +1,5 @@ import rethinkdb as r import random -import json import rapidjson @@ -41,8 +40,8 @@ class Bigchain(object): host (str): hostname where the rethinkdb is running. port (int): port in which rethinkb is running (usually 28015). dbname (str): the name of the database to connect to (usually bigchain). - public_key (str): the base58 encoded public key for the ECDSA secp256k1 curve. - private_key (str): the base58 encoded private key for the ECDSA secp256k1 curve. + public_key (str): the base58 encoded public key for the ED25519 curve. + private_key (str): the base58 encoded private key for the ED25519 curve. keyring (list[str]): list of base58 encoded public keys of the federation nodes. """ @@ -298,7 +297,7 @@ class Bigchain(object): # Calculate the hash of the new block block_data = util.serialize(block) block_hash = crypto.hash_data(block_data) - block_signature = crypto.PrivateKey(self.me_private).sign(block_data) + block_signature = crypto.SigningKey(self.me_private).sign(block_data).decode() block = { 'id': block_hash, @@ -419,7 +418,7 @@ class Bigchain(object): } vote_data = util.serialize(vote) - signature = crypto.PrivateKey(self.me_private).sign(vote_data) + signature = crypto.SigningKey(self.me_private).sign(vote_data).decode() vote_signed = { 'node_pubkey': self.me, diff --git a/bigchaindb/crypto.py b/bigchaindb/crypto.py index bcbe0863..506cb00e 100644 --- a/bigchaindb/crypto.py +++ b/bigchaindb/crypto.py @@ -1,155 +1,17 @@ # Separate all crypto code so that we can easily test several implementations -import binascii -import base58 - import sha3 -import bitcoin - -from cryptography.hazmat.backends import default_backend -from cryptography.hazmat.primitives.asymmetric import ec -from cryptography.hazmat.primitives import hashes -from cryptography.exceptions import InvalidSignature - - -class PrivateKey(object): - """ - PrivateKey instance - """ - - def __init__(self, key): - """ - Instantiate the private key with the private_value encoded in base58 - """ - private_value = self.decode(key) - private_numbers = self._private_value_to_cryptography_private_numbers(private_value) - self.private_key = self._cryptography_private_key_from_private_numbers(private_numbers) - - def sign(self, data): - """ - Sign data with private key - """ - signer = self.private_key.signer(ec.ECDSA(hashes.SHA256())) - signer.update(data.encode('utf-8')) - signature = signer.finalize() - return binascii.hexlify(signature).decode('utf-8') - - - @staticmethod - def encode(private_value): - """ - Encode the decimal number private_value to base58 - """ - private_value_hex = bitcoin.encode_privkey(private_value, 'hex') - private_value_base58 = base58.b58encode(bytes.fromhex(private_value_hex)) - return private_value_base58 - - @staticmethod - def decode(key): - """ - Decode the base58 private_value to decimale - """ - private_value_hex = binascii.hexlify(base58.b58decode(key)) - private_value = bitcoin.decode_privkey(private_value_hex) - return private_value - - def _private_value_to_public_values(self, private_value): - """ - Return the public values from the private value - """ - public_value_x, public_value_y = bitcoin.privkey_to_pubkey(private_value) - return (public_value_x, public_value_y) - - def _private_value_to_cryptography_private_numbers(self, private_value): - """ - Return an instance of cryptography PrivateNumbers from the decimal private_value - """ - public_value_x, public_value_y = self._private_value_to_public_values(private_value) - public_numbers = PublicKey._public_values_to_cryptography_public_numbers(public_value_x, public_value_y) - private_numbers = ec.EllipticCurvePrivateNumbers(private_value, public_numbers) - return private_numbers - - @staticmethod - def _cryptography_private_key_from_private_numbers(private_numbers): - """ - Return an instace of cryptography PrivateKey from a cryptography instance of PrivateNumbers - """ - return private_numbers.private_key(default_backend()) - - -class PublicKey(object): - - def __init__(self, key): - """ - Instantiate the public key with the compressed public value encoded in base58 - """ - public_value_x, public_value_y = self.decode(key) - public_numbers = self._public_values_to_cryptography_public_numbers(public_value_x, public_value_y) - self.public_key = self._criptography_public_key_from_public_numbers(public_numbers) - - def verify(self, data, signature): - verifier = self.public_key.verifier(binascii.unhexlify(signature), ec.ECDSA(hashes.SHA256())) - verifier.update(data.encode('utf-8')) - try: - verifier.verify() - except InvalidSignature: - return False - - return True - - @staticmethod - def encode(public_value_x, public_value_y): - """ - Encode the public key represented by the decimal values x and y to base58 - """ - public_value_compressed_hex = bitcoin.encode_pubkey([public_value_x, public_value_y], 'hex_compressed') - public_value_compressed_base58 = base58.b58encode(bytes.fromhex(public_value_compressed_hex)) - return public_value_compressed_base58 - - @staticmethod - def decode(public_value_compressed_base58): - """ - Decode the base58 public_value to the decimal x and y values - """ - public_value_compressed_hex = binascii.hexlify(base58.b58decode(public_value_compressed_base58)) - public_value_x, public_value_y = bitcoin.decode_pubkey(public_value_compressed_hex.decode()) - return (public_value_x, public_value_y) - - @staticmethod - def _public_values_to_cryptography_public_numbers(public_value_x, public_value_y): - """ - Return an instance of cryptography PublicNumbers from the decimal x and y values - """ - public_numbers = ec.EllipticCurvePublicNumbers(public_value_x, public_value_y, ec.SECP256K1()) - return public_numbers - - def _criptography_public_key_from_public_numbers(self, public_numbers): - """ - Return an instance of cryptography PublicKey from a cryptography instance of PublicNumbers - """ - return public_numbers.public_key(default_backend()) - - -def generate_key_pair(): - """ - Generate a new key pair and return the pair encoded in base58 - """ - # Private key - private_key = ec.generate_private_key(ec.SECP256K1, default_backend()) - private_value = private_key.private_numbers().private_value - private_value_base58 = PrivateKey.encode(private_value) - - # Public key - public_key = private_key.public_key() - public_value_x, public_value_y = public_key.public_numbers().x, public_key.public_numbers().y - public_value_compressed_base58 = PublicKey.encode(public_value_x, public_value_y) - - return (private_value_base58, public_value_compressed_base58) +from cryptoconditions import ed25519 def hash_data(data): """Hash the provided data using SHA3-256""" - return sha3.sha3_256(data.encode()).hexdigest() +def generate_key_pair(): + sk, pk = ed25519.ed25519_generate_key_pair() + return sk.decode(), pk.decode() + +SigningKey = ed25519.Ed25519SigningKey +VerifyingKey = ed25519.Ed25519VerifyingKey diff --git a/bigchaindb/db/utils.py b/bigchaindb/db/utils.py index d5551833..b0a44d6a 100644 --- a/bigchaindb/db/utils.py +++ b/bigchaindb/db/utils.py @@ -19,6 +19,9 @@ def get_conn(): def init(): + # Try to access the keypair, throws an exception if it does not exist + b = bigchaindb.Bigchain() + conn = get_conn() dbname = bigchaindb.config['database']['name'] @@ -59,7 +62,6 @@ def init(): r.db(dbname).table('bigchain').index_wait().run(conn) logger.info(' - genesis block') - b = bigchaindb.Bigchain() b.create_genesis_block() logger.info('Done, have fun!') diff --git a/bigchaindb/util.py b/bigchaindb/util.py index 822345b8..2da25f73 100644 --- a/bigchaindb/util.py +++ b/bigchaindb/util.py @@ -9,7 +9,7 @@ from datetime import datetime import bigchaindb from bigchaindb import exceptions -from bigchaindb.crypto import PrivateKey, PublicKey, hash_data +from bigchaindb import crypto class ProcessGroup(object): @@ -159,7 +159,7 @@ def create_tx(current_owner, new_owner, tx_input, operation, payload=None): data = None if payload is not None: if isinstance(payload, dict): - hash_payload = hash_data(serialize(payload)) + hash_payload = crypto.hash_data(serialize(payload)) data = { 'hash': hash_payload, 'payload': payload @@ -167,7 +167,7 @@ def create_tx(current_owner, new_owner, tx_input, operation, payload=None): else: raise TypeError('`payload` must be an dict instance') - hash_payload = hash_data(serialize(payload)) + hash_payload = crypto.hash_data(serialize(payload)) data = { 'hash': hash_payload, 'payload': payload @@ -184,7 +184,7 @@ def create_tx(current_owner, new_owner, tx_input, operation, payload=None): # serialize and convert to bytes tx_serialized = serialize(tx) - tx_hash = hash_data(tx_serialized) + tx_hash = crypto.hash_data(tx_serialized) # create the transaction transaction = { @@ -208,10 +208,10 @@ def sign_tx(transaction, private_key): dict: transaction with the `signature` field included. """ - private_key = PrivateKey(private_key) + private_key = crypto.SigningKey(private_key) signature = private_key.sign(serialize(transaction)) signed_transaction = transaction.copy() - signed_transaction.update({'signature': signature}) + signed_transaction.update({'signature': signature.decode()}) return signed_transaction @@ -222,7 +222,7 @@ def create_and_sign_tx(private_key, current_owner, new_owner, tx_input, operatio def check_hash_and_signature(transaction): # Check hash of the transaction - calculated_hash = hash_data(serialize(transaction['transaction'])) + calculated_hash = crypto.hash_data(serialize(transaction['transaction'])) if calculated_hash != transaction['id']: raise exceptions.InvalidHash() @@ -251,7 +251,7 @@ def verify_signature(signed_transaction): signature = data.pop('signature') public_key_base58 = signed_transaction['transaction']['current_owner'] - public_key = PublicKey(public_key_base58) + public_key = crypto.VerifyingKey(public_key_base58) return public_key.verify(serialize(data), signature) diff --git a/bigchaindb/web/server.py b/bigchaindb/web/server.py index 208f2458..021ceab4 100644 --- a/bigchaindb/web/server.py +++ b/bigchaindb/web/server.py @@ -76,6 +76,9 @@ def create_server(settings): if not settings.get('workers'): settings['workers'] = (multiprocessing.cpu_count() * 2) + 1 + if not settings.get('threads'): + settings['threads'] = (multiprocessing.cpu_count() * 2) + 1 + debug = settings.pop('debug', False) app = create_app(debug) standalone = StandaloneApplication(app, settings) diff --git a/codecov.yml b/codecov.yml index 490d0805..27507adb 100644 --- a/codecov.yml +++ b/codecov.yml @@ -1,5 +1,5 @@ codecov: - branch: develop # the branch to show by default + branch: master # the branch to show by default # The help text for bot says: # "the username that will consume any oauth requests @@ -29,4 +29,4 @@ coverage: comment: layout: "header, diff, changes, sunburst, suggestions" - behavior: default \ No newline at end of file + behavior: default diff --git a/deploy-cluster-aws/create_rethinkdb_conf.py b/deploy-cluster-aws/create_rethinkdb_conf.py index 268dd710..4a11b462 100644 --- a/deploy-cluster-aws/create_rethinkdb_conf.py +++ b/deploy-cluster-aws/create_rethinkdb_conf.py @@ -8,7 +8,7 @@ from __future__ import unicode_literals import os import os.path import shutil -from hostlist import hosts_dev +from hostlist import public_dns_names # cwd = current working directory old_cwd = os.getcwd() @@ -22,7 +22,7 @@ shutil.copy2('rethinkdb.conf.template', 'rethinkdb.conf') # Append additional lines to rethinkdb.conf with open('rethinkdb.conf', 'a') as f: f.write('## The host:port of a node that RethinkDB will connect to\n') - for public_dns_name in hosts_dev: + for public_dns_name in public_dns_names: f.write('join=' + public_dns_name + ':29015\n') os.chdir(old_cwd) diff --git a/deploy-cluster-aws/fab_prepare_chain.py b/deploy-cluster-aws/fab_prepare_chain.py deleted file mode 100644 index caa827ad..00000000 --- a/deploy-cluster-aws/fab_prepare_chain.py +++ /dev/null @@ -1,27 +0,0 @@ -# -*- coding: utf-8 -*- - -""" Generating genesis block -""" - -from __future__ import with_statement, unicode_literals - -from fabric import colors as c -from fabric.api import * -from fabric.api import local, puts, settings, hide, abort, lcd, prefix -from fabric.api import run, sudo, cd, get, local, lcd, env, hide -from fabric.api import task, parallel -from fabric.contrib import files -from fabric.contrib.files import append, exists -from fabric.contrib.console import confirm -from fabric.contrib.project import rsync_project -from fabric.operations import run, put -from fabric.context_managers import settings -from fabric.decorators import roles -from fabtools import * - -env.user = 'ubuntu' -env.key_filename = 'pem/bigchaindb.pem' - -@task -def init_bigchaindb(): - run('bigchaindb -y start &', pty = False) diff --git a/deploy-cluster-aws/fabfile.py b/deploy-cluster-aws/fabfile.py index 22af0f04..f385641b 100644 --- a/deploy-cluster-aws/fabfile.py +++ b/deploy-cluster-aws/fabfile.py @@ -1,47 +1,58 @@ # -*- coding: utf-8 -*- - -"""A fabfile with functionality to prepare, install, and configure -bigchaindb, including its storage backend. +"""A Fabric fabfile with functionality to prepare, install, and configure +BigchainDB, including its storage backend (RethinkDB). """ from __future__ import with_statement, unicode_literals -import requests -from time import * -import os -from datetime import datetime, timedelta -import json -from pprint import pprint - -from fabric import colors as c -from fabric.api import * -from fabric.api import local, puts, settings, hide, abort, lcd, prefix -from fabric.api import run, sudo, cd, get, local, lcd, env, hide +from fabric.api import sudo, env from fabric.api import task, parallel -from fabric.contrib import files -from fabric.contrib.files import append, exists -from fabric.contrib.console import confirm -from fabric.contrib.project import rsync_project +from fabric.contrib.files import sed from fabric.operations import run, put from fabric.context_managers import settings -from fabric.decorators import roles -from fabtools import * -from hostlist import hosts_dev +from hostlist import public_dns_names -env.hosts = hosts_dev -env.roledefs = { - "role1": hosts_dev, - "role2": [hosts_dev[0]], - } -env.roles = ["role1"] +# Ignore known_hosts +# http://docs.fabfile.org/en/1.10/usage/env.html#disable-known-hosts +env.disable_known_hosts = True + +# What remote servers should Fabric connect to? With what usernames? env.user = 'ubuntu' +env.hosts = public_dns_names + +# SSH key files to try when connecting: +# http://docs.fabfile.org/en/1.10/usage/env.html#key-filename env.key_filename = 'pem/bigchaindb.pem' +newrelic_license_key = 'you_need_a_real_license_key' + ###################################################################### -# base software rollout +# DON'T PUT @parallel +@task +def set_hosts(hosts): + """A helper function to change env.hosts from the + command line. + + Args: + hosts (str): 'one_node' or 'two_nodes' + + Example: + fab set_hosts:one_node init_bigchaindb + """ + if hosts == 'one_node': + env.hosts = public_dns_names[:1] + elif hosts == 'two_nodes': + env.hosts = public_dns_names[:2] + else: + raise ValueError('Invalid input to set_hosts.' + ' Expected one_node or two_nodes.' + ' Got {}'.format(hosts)) + + +# Install base software @task @parallel def install_base_software(): @@ -59,7 +70,7 @@ def install_base_software(): python3-pip ipython3 sysstat s3cmd') -# RethinkDB +# Install RethinkDB @task @parallel def install_rethinkdb(): @@ -67,7 +78,7 @@ def install_rethinkdb(): with settings(warn_only=True): # preparing filesystem sudo("mkdir -p /data") - # Locally mounted storage (m3.2xlarge, aber auch c3.xxx) + # Locally mounted storage (m3.2xlarge, but also c3.xxx) try: sudo("umount /mnt") sudo("mkfs -t ext4 /dev/xvdb") @@ -91,27 +102,48 @@ def install_rethinkdb(): sudo('chown -R rethinkdb:rethinkdb /data') # copy config file to target system put('conf/rethinkdb.conf', - '/etc/rethinkdb/instances.d/instance1.conf', mode=0600, use_sudo=True) + '/etc/rethinkdb/instances.d/instance1.conf', + mode=0600, + use_sudo=True) # initialize data-dir sudo('rm -rf /data/*') # finally restart instance sudo('/etc/init.d/rethinkdb restart') -# bigchaindb deployment +# Install BigchainDB (from PyPI) @task @parallel def install_bigchaindb(): sudo('python3 -m pip install bigchaindb') -# startup all nodes of bigchaindb in cluster +# Configure BigchainDB @task @parallel -def start_bigchaindb_nodes(): +def configure_bigchaindb(): + run('bigchaindb -y configure', pty=False) + + +# Initialize BigchainDB +# i.e. create the database, the tables, +# the indexes, and the genesis block. +# (This only needs to be run on one node.) +# Call using: +# fab set_hosts:one_node init_bigchaindb +@task +def init_bigchaindb(): + run('bigchaindb init', pty=False) + + +# Start BigchainDB using screen +@task +@parallel +def start_bigchaindb(): sudo('screen -d -m bigchaindb -y start &', pty=False) +# Install and run New Relic @task def install_newrelic(): with settings(warn_only=True): @@ -119,18 +151,18 @@ def install_newrelic(): # sudo('apt-key adv --keyserver hkp://subkeys.pgp.net --recv-keys 548C16BF') sudo('apt-get update') sudo('apt-get -y --force-yes install newrelic-sysmond') - sudo('nrsysmond-config --set license_key=c88af00c813983f8ee12e9b455aa13fde1cddaa8') + sudo('nrsysmond-config --set license_key=' + newrelic_license_key) sudo('/etc/init.d/newrelic-sysmond restart') -############################### -# Security / FirewallStuff next -############################### +########################### +# Security / Firewall Stuff +########################### @task def harden_sshd(): - """Security harden sshd.""" - + """Security harden sshd. + """ # Disable password authentication sed('/etc/ssh/sshd_config', '#PasswordAuthentication yes', @@ -147,7 +179,8 @@ def harden_sshd(): def disable_root_login(): """Disable `root` login for even more security. Access to `root` account is now possible by first connecting with your dedicated maintenance - account and then running ``sudo su -``.""" + account and then running ``sudo su -``. + """ sudo('passwd --lock root') @@ -172,7 +205,7 @@ def set_fw(): ######################################################### -# some helper-functions to handle bad behavior of cluster +# Some helper-functions to handle bad behavior of cluster ######################################################### # rebuild indexes diff --git a/deploy-cluster-aws/launch_ec2_nodes.py b/deploy-cluster-aws/launch_ec2_nodes.py index c531821b..9ebf3026 100644 --- a/deploy-cluster-aws/launch_ec2_nodes.py +++ b/deploy-cluster-aws/launch_ec2_nodes.py @@ -166,26 +166,31 @@ for i, instance in enumerate(instances_with_tag): format(instance.instance_id)) # Get a list of the pubic DNS names of the instances_with_tag -hosts_dev = [] +public_dns_names = [] for instance in instances_with_tag: public_dns_name = getattr(instance, 'public_dns_name', None) if public_dns_name is not None: - hosts_dev.append(public_dns_name) + public_dns_names.append(public_dns_name) # Write a shellscript to add remote keys to ~/.ssh/known_hosts print('Preparing shellscript to add remote keys to known_hosts') with open('add2known_hosts.sh', 'w') as f: f.write('#!/bin/bash\n') - for public_dns_name in hosts_dev: + for public_dns_name in public_dns_names: f.write('ssh-keyscan ' + public_dns_name + ' >> ~/.ssh/known_hosts\n') -# Create a file named hostlist.py containing hosts_dev. +# Create a file named hostlist.py containing public_dns_names. # If a hostlist.py already exists, it will be overwritten. print('Writing hostlist.py') with open('hostlist.py', 'w') as f: f.write('# -*- coding: utf-8 -*-\n') + f.write('"""A list of the public DNS names of all the nodes in this\n') + f.write('BigchainDB cluster/federation.\n') + f.write('"""\n') + f.write('\n') f.write('from __future__ import unicode_literals\n') - f.write('hosts_dev = {}\n'.format(hosts_dev)) + f.write('\n') + f.write('public_dns_names = {}\n'.format(public_dns_names)) # Wait wait_time = 45 diff --git a/deploy-cluster-aws/startup.sh b/deploy-cluster-aws/startup.sh index 84d420cb..9a73f7f8 100755 --- a/deploy-cluster-aws/startup.sh +++ b/deploy-cluster-aws/startup.sh @@ -55,27 +55,33 @@ chmod +x add2known_hosts.sh # (Re)create the RethinkDB configuration file conf/rethinkdb.conf python create_rethinkdb_conf.py -# rollout base packages (dependencies) needed before -# storage backend (rethinkdb) and bigchaindb can be rolled out +# Rollout base packages (dependencies) needed before +# storage backend (RethinkDB) and BigchainDB can be rolled out fab install_base_software -# rollout storage backend (rethinkdb) +# Rollout storage backend (RethinkDB) and start it fab install_rethinkdb -# rollout bigchaindb +# Rollout BigchainDB (but don't start it yet) fab install_bigchaindb -# generate genesis block -# HORST is the last public_dns_name listed in conf/rethinkdb.conf -# For example: -# ec2-52-58-86-145.eu-central-1.compute.amazonaws.com -HORST=`tail -1 conf/rethinkdb.conf|cut -d: -f1|cut -d= -f2` -fab -H $HORST -f fab_prepare_chain.py init_bigchaindb +# Configure BigchainDB on all nodes +fab configure_bigchaindb -# initiate sharding -fab start_bigchaindb_nodes +# TODO Get public keys from all nodes +# using e.g. bigchaindb export-pubkey + +# TODO Add list of public keys to keyring of all nodes +# using e.g. bigchaindb import-pubkey + +# Send a "bigchaindb init" command to one node +# to initialize the BigchainDB database +# i.e. create the database, the tables, +# the indexes, and the genesis block. +fab set_hosts:one_node init_bigchaindb + +# Start BigchainDB on all the nodes using "screen" +fab start_bigchaindb # cleanup rm add2known_hosts.sh - -# DONE diff --git a/docs/source/bigchaindb-cli.md b/docs/source/bigchaindb-cli.md index 108e5846..4186426e 100644 --- a/docs/source/bigchaindb-cli.md +++ b/docs/source/bigchaindb-cli.md @@ -2,10 +2,11 @@ BigchainDB has some Command Line Interfaces (CLIs). One of them is the `bigchaindb` command which we already saw when we first started BigchainDB using: ```text +$ bigchaindb configure $ bigchaindb start ``` -The fist time you run `bigchaindb start`, it creates a default configuration file in `$HOME/.bigchaindb`. You can check that configuration using: +When you run `bigchaindb configure`, it creates a default configuration file in `$HOME/.bigchaindb`. You can check that configuration using: ```text $ bigchaindb show-config ``` @@ -20,3 +21,7 @@ There's another command named `bigchaindb-benchmark`. It's used to run benchmark $ bigchaindb-benchmark -h $ bigchaindb-benchmark load -h ``` + +Note that you can always start `bigchaindb` using a different config file using the `-c` option. +For more information check the help with `bigchaindb -h`. + diff --git a/docs/source/configuration.md b/docs/source/configuration.md new file mode 100644 index 00000000..b5d538e1 --- /dev/null +++ b/docs/source/configuration.md @@ -0,0 +1,226 @@ +# Configuring a BigchainDB Node + +The BigchainDB configuration settings for a particular node are stored on that node in a configuration file at `$HOME/.bigchaindb`. That file doesn't exist by default. (It's not created when installing BigchainDB.) One could create it using a text editor, but it's easiest to use the `bigchaindb configure` command: + +```text +$ bigchaindb configure +``` + +It will ask some questions and generate a new keypair (i.e. a private key and corresponding public key for the node). See below for some additional explanation of the settings and their meanings. To accept a suggested default value, press Enter or Return. If you want to accept all the default values, use the `-y` option when running the command, that is: + +```text +$ bigchaindb -y configure +``` + +## Using a Different Path for the Configuration File + +By default, the configuration settings are stored in `$HOME/.bigchaindb`. If you want to +specify a different path for your configuration file, you can use the `-c` parameter. +This works for every subcommand under the `bigchaindb` executable. + +For example, if you want to **generate** a new configuration file under a +specific path, you can run: + +``` +$ bigchaindb -c local.json configure +$ bigchaindb -c test.json configure +``` + +This will create two new files named `local.json` and `test.json` in your +current working directory. + +From now on, you can refer to those configuration files using the `-c` +parameter; for example: + +``` +$ bigchaindb -c local.json show-config +``` + +will show the configuration for `local.json`. + +If you want to **start** BigchainDB with the `test.json` configuration file, you can use: + +``` +$ bigchaindb -c test.json start +``` + + +## Using Environment Variables to Configure the Node + +Sometimes it's more convenient to use environment variables to configure the +system, for example when using Docker or Heroku. In +that case you can configure the system using environment variables. + +Every configuration parameter can be mapped to an environment variable. The +environment variables available are: + +- `BIGCHAINDB_DATABASE_HOST` defines the RethinkDB database hostname to connect to. +- `BIGCHAINDB_DATABASE_PORT` defines the RethinkDB database port to connect to. +- `BIGCHAINDB_DATABASE_NAME` defines the RethinkDB database name to use. +- `BIGCHAINDB_KEYPAIR_PUBLIC` defines the public key of the BigchainDB node. +- `BIGCHAINDB_KEYPAIR_PRIVATE` defines the private key of the BigchainDB node. +- `BIGCHAINDB_KEYRING` is a colon-separated list of the public keys of all _other_ nodes in the cluster. +- `BIGCHAINDB_STATSD_HOST` defines the hostname of the statsd server for [monitoring](monitoring.html). +- `BIGCHAINDB_STATSD_PORT` defines the port of the statsd server for monitoring. +- `BIGCHAINDB_STATSD_RATE` is a float between `0` and `1` that defines the fraction of transaction operations sampled. +- `BIGCHAINDB_API_ENDPOINT` defines the API endpoint to use (e.g. `http://localhost:9984/api/v1`). +- `BIGCHAINDB_CONSENSUS_PLUGIN` defines the name of the [consensus plugin](consensus.html) to use. +- `BIGCHAINDB_SERVER_BIND` defines where to bind the server socket, the format is `addr:port` (e.g. `0.0.0.0:9984`). +- `BIGCHAINDB_SERVER_WORKERS` defines the [number of workers](http://docs.gunicorn.org/en/stable/settings.html#workers) + to start for the server API. +- `BIGCHAINDB_SERVER_THREADS` defines the [number of threads](http://docs.gunicorn.org/en/stable/settings.html#threads) + to start for the server API. + + +## Order of Precedence in Determining Configuration Values + +All configuration values start with their default values (defined in `bigchaindb.__init__`), but a default value can be overriden by an environment variable, and a value set by an environment variable can be overriden by a value in a local configuration file (`$HOME/.bigchaindb` or the location specified by the `-c` command-line option). + +In summary, there is an order of precedence in reading configuration values: +1. local configuration file +2. environment variables +3. default configuration file (defined in ``bigchaindb.__init__``) + +This means that if the default configuration contains: + +```json +{ + "database": { + "host": "localhost", + "port": 28015 + } +} +``` + +while the local file `local.json` contains: +```json +{ + "database": { + "host": "ec2-xx-xx-xxx-xxx.eu-central-1.compute.amazonaws.com" + } +} + +``` + +and you run this command: +``` +$ BIGCHAINDB_DATABASE_HOST=anotherhost.com \ + BIGCHAINDB_DATABASE_PORT=4242 \ + BIGCHAINDB_KEYRING=pubkey0:pubkey1 \ + bigchaindb -c local.json show-config +``` + +you will get the following values for all the configuration settings: +```json +{ + "api_endpoint": "http://localhost:8008/api/v1", + "consensus_plugin": "default", + "database": { + "host": "ec2-xx-xx-xxx-xxx.eu-central-1.compute.amazonaws.com", + "name": "bigchain", + "port": 4242 + }, + "keypair": { + "private": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", + "public": "nJq6EmdUkvFjQRB5hFvDmvZtv1deb3W3RgmiAq6dyygC" + }, + "keyring": [ + "pubkey0", + "pubkey1" + ], + "server": { + "bind": "0.0.0.0:9984", + "threads": null, + "workers": null + }, + "statsd": { + "host": "localhost", + "port": 8125, + "rate": 0.01 + } +} +``` + + +## Another Example + +As another example, let's assume we **don't** have any configuration file stored in +`$HOME/.bigchaindb`. As you can see, `show-config` displays the default configuration (and a +warning): +``` +$ bigchaindb show-config +WARNING:bigchaindb.config_utils:Cannot find config file `/home/vrde/.bigchaindb`. +{ + "api_endpoint": "http://localhost:9984/api/v1", + "consensus_plugin": "default", + "database": { + "host": "localhost", + "name": "bigchain", + "port": 28015 + }, + "keypair": { + "private": null, + "public": null + }, + "keyring": [], + "server": { + "bind": "0.0.0.0:9984", + "threads": null, + "workers": null + }, + "statsd": { + "host": "localhost", + "port": 8125, + "rate": 0.01 + } +} +``` + +If we try to run the node, the command will fail: + +``` +$ bigchaindb start +WARNING:bigchaindb.config_utils:Cannot find config file `/home/vrde/.bigchaindb`. +Cannot start BigchainDB, no keypair found. Did you run `bigchaindb configure`? +``` + +This is failing as expected: a BigchainDB node needs at least a key pair to work. +We can pass the key pair using environment variables: +``` +$ BIGCHAINDB_KEYPAIR_PUBLIC=26y9EuyGP44JXxqcvF8GbCJGqkiqFXddZzxVjLU3rWbHp \ + BIGCHAINDB_KEYPAIR_PRIVATE=9PkLfHbzXnSSNnb1sSBL73C2MydzKLs5fAHoA4Q7otrG \ + bigchaindb start +``` + +We can also run `show-config` to see how the configuration looks: +``` +$ BIGCHAINDB_KEYPAIR_PUBLIC=26y9EuyGP44JXxqcvF8GbCJGqkiqFXddZzxVjLU3rWbHp \ + BIGCHAINDB_KEYPAIR_PRIVATE=9PkLfHbzXnSSNnb1sSBL73C2MydzKLs5fAHoA4Q7otrG \ + bigchaindb show-config + +WARNING:bigchaindb.config_utils:Cannot find config file `/home/vrde/.bigchaindb`. +{ + "api_endpoint": "http://localhost:9984/api/v1", + "consensus_plugin": "default", + "database": { + "host": "localhost", + "name": "bigchain", + "port": 28015 + }, + "keypair": { + "private": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", + "public": "26y9EuyGP44JXxqcvF8GbCJGqkiqFXddZzxVjLU3rWbHp" + }, + "keyring": [], + "server": { + "bind": "0.0.0.0:9984", + "threads": null, + "workers": null + }, + "statsd": { + "host": "localhost", + "port": 8125, + "rate": 0.01 + } +} +``` diff --git a/docs/source/cryptography.md b/docs/source/cryptography.md index aaa9f7ef..03fe22c3 100644 --- a/docs/source/cryptography.md +++ b/docs/source/cryptography.md @@ -19,8 +19,8 @@ tx_hash = hashlib.sha3_256(data).hexdigest() ## Signature algorithm and keys -The signature algorithm used by BigchainDB is ECDSA with the secp256k1 curve -using the python [cryptography](https://cryptography.io/en/latest/) module. +The signature algorithm used by BigchainDB is [ED25519](https://tools.ietf.org/html/draft-irtf-cfrg-eddsa-04) +using the python [ed25519](https://github.com/warner/python-ed25519) module, overloaded by the [cryptoconditions library](https://github.com/bigchaindb/cryptoconditions). The private key is the base58 encoded hexadecimal representation of private number. The public key is the base58 encoded hexadecimal representation of the diff --git a/docs/source/http-client-server-api.md b/docs/source/http-client-server-api.md index 1520b046..da045fa3 100644 --- a/docs/source/http-client-server-api.md +++ b/docs/source/http-client-server-api.md @@ -2,10 +2,15 @@ When you start Bigchaindb using `bigchaindb start`, an HTTP API is exposed at: -[http://localhost:5000/api/v1/](http://localhost:5000/api/v1/) +- [http://localhost:9984/api/v1/](http://localhost:9984/api/v1/) -Right now, that API can only be accessed from localhost (i.e. not remotely). In the future, we'll enable remote access and explain how that works. See [Issue #149](https://github.com/bigchaindb/bigchaindb/issues/149) on GitHub. -The HTTP API currently exposes two endpoints, one to get information about a specific transaction id, and one to push a transaction to the BigchainDB cluster. Those endpoints are documented at: +Please note that by default the server binds to `0.0.0.0:9984`, hence the API +is exposed to the world. + +The HTTP API currently exposes two endpoints, one to get information about a +specific transaction id, and one to push a transaction to the BigchainDB +cluster. Those endpoints are documented at: + +- [http://docs.bigchaindb.apiary.io/](http://docs.bigchaindb.apiary.io/) -[http://docs.bigchaindb.apiary.io/](http://docs.bigchaindb.apiary.io/) diff --git a/docs/source/index.rst b/docs/source/index.rst index 8a75833f..ec10c228 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -15,6 +15,7 @@ Table of Contents introduction installing-server running-unit-tests + configuration python-server-api-examples bigchaindb-cli http-client-server-api diff --git a/docs/source/installing-server.md b/docs/source/installing-server.md index c168f62d..0d12a875 100644 --- a/docs/source/installing-server.md +++ b/docs/source/installing-server.md @@ -68,14 +68,14 @@ Note: You can use `pip` to upgrade the `bigchaindb` package to the latest versio ### How to Install BigchainDB from Source -BigchainDB (i.e. both the Server and the officially-supported drivers) is in its early stages and being actively developed on its [GitHub repository](https://github.com/bigchaindb/bigchaindb). Contributions are highly appreciated. If you want to help with development, then you'll want to install BigchainDB from source. Here's how. +If you want to install BitchainDB from source because you want to contribute code (i.e. as a BigchainDB developer), then please see the instructions in [the `CONTRIBUTING.md` file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md). -First, clone the public repository: +Otherwise, clone the public repository: ```text $ git clone git@github.com:bigchaindb/bigchaindb.git ``` -Install from the source: +and then install from source: ```text $ python setup.py install ``` @@ -94,10 +94,16 @@ $ rethinkdb Then open a different terminal and run: ```text -$ bigchaindb start +$ bigchaindb -y configure +$ bigchaindb init ``` -During its first run, BigchainDB Server takes care of configuring a single node environment. +That creates a configuration file in `$HOME/.bigchaindb` (documented in [the section on configuration](configuration.html)), initializes the database, creates the tables, creates the indexes, and generates the genesis block. + +You can start BigchainDB Server using: +```text +$ bigchaindb start +``` ## Run BigchainDB with Docker @@ -122,8 +128,22 @@ then do a one-time configuration step to create the config file; it will be stored on your host machine under ` ~/.bigchaindb_docker/config`: ```text $ docker-compose run --rm bigchaindb bigchaindb configure +Starting bigchaindb_rethinkdb-data_1 +Generating keypair +API Server bind? (default `0.0.0.0:9984`): +Database host? (default `localhost`): rethinkdb +Database port? (default `28015`): +Database name? (default `bigchain`): +Statsd host? (default `localhost`): statsd +Statsd port? (default `8125`): +Statsd rate? (default `0.01`): +Ready to go! ``` +As shown above, make sure that you set the database and statsd hosts to their +corresponding service names (`rethinkdb`, `statsd`), defined in`docker-compose.yml` +and `docker-compose-monitor.yml`. + You can then start it up (in the background, as a daemon) using: ```text $ docker-compose up -d diff --git a/docs/source/introduction.md b/docs/source/introduction.md index 3c1230ad..bb7f4e70 100644 --- a/docs/source/introduction.md +++ b/docs/source/introduction.md @@ -7,4 +7,4 @@ BigchainDB is a scalable blockchain database. You can read about its motivations 3. Developers of BigchainDB driver software (SDKs used to develop client software). 4. App developers who are developing client apps to talk to one or more live, operational BigchainDB clusters. They would use one of the BigchainDB drivers. -If you're curious about what's in our roadmap, see [the ROADMAP.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/ROADMAP.md) and [the list of open issues](https://github.com/bigchaindb/bigchaindb/issues). If you want to request a feature, file a bug report, or make a pull request, see [the CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/CONTRIBUTING.md). +If you're curious about what's in our roadmap, see [the ROADMAP.md file](https://github.com/bigchaindb/bigchaindb/blob/master/ROADMAP.md) and [the list of open issues](https://github.com/bigchaindb/bigchaindb/issues). If you want to request a feature, file a bug report, or make a pull request, see [the CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md). diff --git a/docs/source/licenses.md b/docs/source/licenses.md index aa60e2bc..97699c56 100644 --- a/docs/source/licenses.md +++ b/docs/source/licenses.md @@ -1,3 +1,3 @@ # Licenses -Information about how the BigchainDB code and documentation are licensed can be found in [the LICENSES.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/LICENSES.md) (in the root directory of the repository). \ No newline at end of file +Information about how the BigchainDB code and documentation are licensed can be found in [the LICENSES.md file](https://github.com/bigchaindb/bigchaindb/blob/master/LICENSES.md) (in the root directory of the repository). diff --git a/docs/source/python-server-api-examples.md b/docs/source/python-server-api-examples.md index 25fc5b19..2b2bfd86 100644 --- a/docs/source/python-server-api-examples.md +++ b/docs/source/python-server-api-examples.md @@ -11,6 +11,8 @@ We create a digital asset, sign it, write it to a BigchainDB Server instance, re First, make sure you have RethinkDB and BigchainDB _installed and running_, i.e. you [installed them](installing-server.html) and you ran: ```text $ rethinkdb +$ bigchaindb configure +$ bigchaindb init $ bigchaindb start ``` diff --git a/docs/source/release-notes.md b/docs/source/release-notes.md index 1a80c120..5b4f1e88 100644 --- a/docs/source/release-notes.md +++ b/docs/source/release-notes.md @@ -4,6 +4,6 @@ You can find a list of all BigchainDB releases and release notes on GitHub at: [https://github.com/bigchaindb/bigchaindb/releases](https://github.com/bigchaindb/bigchaindb/releases) -The [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/CHANGELOG.md) contains much the same information, but it also has notes about what to expect in the _next_ release. +The [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CHANGELOG.md) contains much the same information, but it also has notes about what to expect in the _next_ release. -We also have [a roadmap document in ROADMAP.md](https://github.com/bigchaindb/bigchaindb/blob/develop/ROADMAP.md). +We also have [a roadmap document in ROADMAP.md](https://github.com/bigchaindb/bigchaindb/blob/master/ROADMAP.md). diff --git a/docs/source/running-unit-tests.md b/docs/source/running-unit-tests.md index 1a4f0e34..e20a0fc2 100644 --- a/docs/source/running-unit-tests.md +++ b/docs/source/running-unit-tests.md @@ -2,29 +2,21 @@ Once you've installed BigchainDB Server, you may want to run all the unit tests. This section explains how. -First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. Before you can run all the unit tests, you must [install BigchainDB from source](installing-server.html#how-to-install-bigchaindb-from-source). +First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. **Before you can run all the unit tests, you must [install BigchainDB from source](installing-server.html#how-to-install-bigchaindb-from-source).** To run all the unit tests, first make sure you have RethinkDB running: + ```text $ rethinkdb ``` then in another terminal, do: + ```text -$ py.test -v +$ python setup.py test ``` -If the above command doesn't work (e.g. maybe you are running in a conda virtual environment), try: -```text -$ python -m pytest -v -``` - -(We write our unit tests using the [pytest](http://pytest.org/latest/) framework.) - -You can also run all unit tests via `setup.py`, using: -```text -$ python setup.py test -``` +(Aside: How does the above command work? The documentation for [pytest-runner](https://pypi.python.org/pypi/pytest-runner) explains. We use [pytest](http://pytest.org/latest/) to write all unit tests.) ### Using docker-compose to Run the Tests diff --git a/setup.py b/setup.py index 0fd5f4b8..8f4cfab6 100644 --- a/setup.py +++ b/setup.py @@ -71,7 +71,7 @@ setup( 'rethinkdb==2.2.0.post4', 'pysha3==0.3', 'pytz==2015.7', - 'cryptography==1.2.3', + 'cryptoconditions==0.1.1', 'statsd==3.2.1', 'python-rapidjson==0.0.6', 'logstats==0.2.1', diff --git a/tests/conftest.py b/tests/conftest.py index 3781b2b6..44de6316 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -19,17 +19,30 @@ CONFIG = { 'name': DB_NAME }, 'keypair': { - 'private': '3i2FDXp87N9ExXSvWxqBAw9EgzoxxGTQNKbtxmWBpTyL', - 'public': '29Tw3ozmSRtN8XNofvsu5RdoQRk9gAonfpkFvRZDmhTPo' + 'private': '31Lb1ZGKTyHnmVK3LUMrAUrPNfd4sE2YyBt3UA4A25aA', + 'public': '4XYfCbabAWVUCbjTmRTFEu2sc3dFEdkse4r6X498B1s8' } } # Test user. inputs will be created for this user. Cryptography Keys -USER_PRIVATE_KEY = 'GmRZxQdQv7tooMijXytQkexKuFN6mJocciJarAmMwTX2' -USER_PUBLIC_KEY = 'r3cEu8GNoz8rYpNJ61k7GqfR8VEvdUbtyHce8u1kaYwh' +USER_PRIVATE_KEY = '8eJ8q9ZQpReWyQT5aFCiwtZ5wDZC4eDnCen88p3tQ6ie' +USER_PUBLIC_KEY = 'JEAkEJqLbbgDRAtMm8YAjGp759Aq2qTn9eaEHUj2XePE' + + +# We need this function to avoid loading an existing +# conf file located in the home of the user running +# the tests. If it's too aggressive we can change it +# later. +@pytest.fixture(scope='function', autouse=True) +def ignore_local_config_file(monkeypatch): + def mock_file_config(filename=None): + raise FileNotFoundError() + + monkeypatch.setattr('bigchaindb.config_utils.file_config', mock_file_config) @pytest.fixture +@pytest.fixture(scope='function', autouse=True) def restore_config(request, node_config): from bigchaindb import config_utils config_utils.dict_config(node_config) diff --git a/tests/db/test_bigchain_api.py b/tests/db/test_bigchain_api.py index 1fed8800..23e73580 100644 --- a/tests/db/test_bigchain_api.py +++ b/tests/db/test_bigchain_api.py @@ -8,7 +8,7 @@ import rethinkdb as r import bigchaindb from bigchaindb import util from bigchaindb import exceptions -from bigchaindb.crypto import PrivateKey, PublicKey, generate_key_pair, hash_data +from bigchaindb import crypto from bigchaindb.voter import Voter from bigchaindb.block import Block @@ -44,7 +44,7 @@ class TestBigchainApi(object): 'operation': 'd', 'timestamp': tx['transaction']['timestamp'], 'data': { - 'hash': hash_data(util.serialize(payload)), + 'hash': crypto.hash_data(util.serialize(payload)), 'payload': payload } } @@ -52,7 +52,7 @@ class TestBigchainApi(object): # assert tx_hash == tx_calculated_hash def test_transaction_signature(self, b): - sk, vk = generate_key_pair() + sk, vk = crypto.generate_key_pair() tx = b.create_transaction(vk, 'b', 'c', 'd') tx_signed = b.sign_transaction(tx, sk) @@ -108,7 +108,7 @@ class TestBigchainApi(object): def test_assign_transaction_multiple_nodes(self, b, user_public_key, user_private_key): # create 5 federation nodes for _ in range(5): - b.federation_nodes.append(generate_key_pair()[1]) + b.federation_nodes.append(crypto.generate_key_pair()[1]) # test assignee for several transactions for _ in range(20): @@ -185,11 +185,11 @@ class TestBigchainApi(object): def test_create_new_block(self, b): new_block = b.create_block([]) - block_hash = hash_data(util.serialize(new_block['block'])) + block_hash = crypto.hash_data(util.serialize(new_block['block'])) assert new_block['block']['voters'] == [b.me] assert new_block['block']['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(new_block['block']), new_block['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(new_block['block']), new_block['signature']) is True assert new_block['id'] == block_hash assert new_block['votes'] == [] @@ -371,8 +371,8 @@ class TestBlockValidation(object): } block_data = util.serialize(block) - block_hash = hash_data(block_data) - block_signature = PrivateKey(b.me_private).sign(block_data) + block_hash = crypto.hash_data(block_data) + block_signature = crypto.SigningKey(b.me_private).sign(block_data) block = { 'id': block_hash, @@ -408,45 +408,6 @@ class TestBlockValidation(object): assert b.is_valid_block(block) -class TestBigchainCrypto(object): - PRIVATE_VALUE = 64328150571824492670917070117568709277186368319388887463636481841106388379832 - PUBLIC_VALUE_X = 48388170575736684074633245566225141536152842355597159440179742847497614196929 - PUBLIC_VALUE_Y = 65233479152484407841598798165960909560839872511163322973341535484598825150846 - - PRIVATE_VALUE_B58 = 'AaAp4xBavbe6VGeQF2mWdSKNM1r6HfR2Z1tAY6aUkwdq' - PUBLIC_VALUE_COMPRESSED_B58 = 'ifEi3UuTDT4CqUUKiS5omgeDodhu2aRFHVp6LoahbEVe' - - def test_private_key_encode(self): - private_value_base58 = PrivateKey.encode(self.PRIVATE_VALUE) - assert private_value_base58 == self.PRIVATE_VALUE_B58 - - def test_private_key_decode(self): - private_value = PrivateKey.decode(self.PRIVATE_VALUE_B58) - assert private_value == self.PRIVATE_VALUE - - def test_public_key_encode(self): - public_value_compressed_base58 = PublicKey.encode(self.PUBLIC_VALUE_X, self.PUBLIC_VALUE_Y) - assert public_value_compressed_base58 == self.PUBLIC_VALUE_COMPRESSED_B58 - - def test_public_key_decode(self): - public_value_x, public_value_y = PublicKey.decode(self.PUBLIC_VALUE_COMPRESSED_B58) - assert public_value_x == self.PUBLIC_VALUE_X - assert public_value_y == self.PUBLIC_VALUE_Y - - def test_sign_verify(self): - message = 'Hello World!' - public_key = PublicKey(self.PUBLIC_VALUE_COMPRESSED_B58) - private_key = PrivateKey(self.PRIVATE_VALUE_B58) - assert public_key.verify(message, private_key.sign(message)) is True - - def test_generate_key_pair(self): - private_value_base58, public_value_compressed_base58 = generate_key_pair() - assert PrivateKey.encode( - PrivateKey.decode(private_value_base58)) == private_value_base58 - assert PublicKey.encode( - *PublicKey.decode(public_value_compressed_base58)) == public_value_compressed_base58 - - class TestBigchainVoter(object): def test_valid_block_voting(self, b): @@ -483,7 +444,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is True assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_invalid_block_voting(self, b, user_public_key): # create queue and voter @@ -524,7 +485,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is False assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_vote_creation_valid(self, b): # create valid block @@ -538,7 +499,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is True assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_vote_creation_invalid(self, b): # create valid block @@ -552,7 +513,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is False assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True class TestBigchainBlock(object): diff --git a/tests/db/test_voter.py b/tests/db/test_voter.py index d8146829..28f7fa59 100644 --- a/tests/db/test_voter.py +++ b/tests/db/test_voter.py @@ -6,7 +6,7 @@ import multiprocessing as mp from bigchaindb import util from bigchaindb.voter import Voter, BlockStream -from bigchaindb.crypto import PublicKey, generate_key_pair +from bigchaindb import crypto class TestBigchainVoter(object): @@ -45,7 +45,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is True assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_valid_block_voting_with_create_transaction(self, b): q_new_block = mp.Queue() @@ -53,7 +53,7 @@ class TestBigchainVoter(object): genesis = b.create_genesis_block() # create a `CREATE` transaction - test_user_priv, test_user_pub = generate_key_pair() + test_user_priv, test_user_pub = crypto.generate_key_pair() tx = b.create_transaction(b.me, test_user_pub, None, 'CREATE') tx_signed = b.sign_transaction(tx, b.me_private) assert b.is_valid_transaction(tx_signed) @@ -87,7 +87,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is True assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_valid_block_voting_with_transfer_transactions(self, b): q_new_block = mp.Queue() @@ -95,7 +95,7 @@ class TestBigchainVoter(object): b.create_genesis_block() # create a `CREATE` transaction - test_user_priv, test_user_pub = generate_key_pair() + test_user_priv, test_user_pub = crypto.generate_key_pair() tx = b.create_transaction(b.me, test_user_pub, None, 'CREATE') tx_signed = b.sign_transaction(tx, b.me_private) assert b.is_valid_transaction(tx_signed) @@ -124,7 +124,7 @@ class TestBigchainVoter(object): assert len(blocks[1]['votes']) == 1 # create a `TRANSFER` transaction - test_user2_priv, test_user2_pub = generate_key_pair() + test_user2_priv, test_user2_pub = crypto.generate_key_pair() tx2 = b.create_transaction(test_user_pub, test_user2_pub, tx['id'], 'TRANSFER') tx2_signed = b.sign_transaction(tx2, test_user_priv) assert b.is_valid_transaction(tx2_signed) @@ -158,7 +158,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is True assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_invalid_block_voting(self, b, user_public_key): # create queue and voter @@ -197,7 +197,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is False assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_vote_creation_valid(self, b): # create valid block @@ -211,7 +211,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is True assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_vote_creation_invalid(self, b): # create valid block @@ -225,7 +225,7 @@ class TestBigchainVoter(object): assert vote['vote']['is_block_valid'] is False assert vote['vote']['invalid_reason'] is None assert vote['node_pubkey'] == b.me - assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True + assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True def test_voter_considers_unvoted_blocks_when_single_node(self, b): # simulate a voter going donw in a single node environment @@ -301,7 +301,7 @@ class TestBlockStream(object): def test_if_federation_size_is_greater_than_one_ignore_past_blocks(self, b): for _ in range(5): - b.federation_nodes.append(generate_key_pair()[1]) + b.federation_nodes.append(crypto.generate_key_pair()[1]) new_blocks = mp.Queue() bs = BlockStream(new_blocks) block_1 = b.create_block([]) diff --git a/tests/test_commands.py b/tests/test_commands.py index 6e731c13..c289f279 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -1,3 +1,4 @@ +import json from argparse import Namespace from pprint import pprint import copy @@ -11,12 +12,6 @@ def mock_run_configure(monkeypatch): monkeypatch.setattr(bigchain, 'run_configure', lambda *args, **kwargs: None) -@pytest.fixture -def mock_file_config(monkeypatch): - from bigchaindb import config_utils - monkeypatch.setattr(config_utils, 'file_config', lambda *args: None) - - @pytest.fixture def mock_write_config(monkeypatch): from bigchaindb import config_utils @@ -65,13 +60,13 @@ def mock_bigchaindb_backup_config(monkeypatch): monkeypatch.setattr('bigchaindb._config', config) -def test_bigchain_run_start(mock_run_configure, mock_file_config, - mock_processes_start, mock_db_init_with_existing_db): +def test_bigchain_run_start(mock_run_configure, mock_processes_start, mock_db_init_with_existing_db): from bigchaindb.commands.bigchain import run_start args = Namespace(config=None, yes=True) run_start(args) +@pytest.mark.skipif(reason="BigchainDB doesn't support the automatic creation of a config file anymore") def test_bigchain_run_start_assume_yes_create_default_config(monkeypatch, mock_processes_start, mock_generate_key_pair, mock_db_init_with_existing_db): import bigchaindb @@ -99,25 +94,27 @@ def test_bigchain_run_start_assume_yes_create_default_config(monkeypatch, mock_p # TODO Please beware, that if debugging, the "-s" switch for pytest will # interfere with capsys. # See related issue: https://github.com/pytest-dev/pytest/issues/128 -def test_bigchain_show_config(capsys, mock_file_config): +@pytest.mark.usefixtures('restore_config') +def test_bigchain_show_config(capsys): from bigchaindb import config from bigchaindb.commands.bigchain import run_show_config + args = Namespace(config=None) _, _ = capsys.readouterr() run_show_config(args) - output_config, _ = capsys.readouterr() - pprint(config) - expected_outout_config, _ = capsys.readouterr() - assert output_config == expected_outout_config + output_config = json.loads(capsys.readouterr()[0]) + del config['CONFIGURED'] + config['keypair']['private'] = 'x' * 45 + assert output_config == config -def test_bigchain_run_init_when_db_exists(mock_file_config, mock_db_init_with_existing_db): +def test_bigchain_run_init_when_db_exists(mock_db_init_with_existing_db): from bigchaindb.commands.bigchain import run_init args = Namespace(config=None) run_init(args) -def test_drop_existing_db(mock_file_config, mock_rethink_db_drop): +def test_drop_existing_db(mock_rethink_db_drop): from bigchaindb.commands.bigchain import run_drop args = Namespace(config=None, yes=True) run_drop(args) diff --git a/tests/utils/test_config_utils.py b/tests/utils/test_config_utils.py index fa63158d..cec7743f 100644 --- a/tests/utils/test_config_utils.py +++ b/tests/utils/test_config_utils.py @@ -60,9 +60,108 @@ def test_load_consensus_plugin_raises_with_invalid_subclass(monkeypatch): from bigchaindb import config_utils monkeypatch.setattr(config_utils, 'iter_entry_points', - lambda *args: [ type('entry_point', - (object), - {'load': lambda: object}) ]) + lambda *args: [type('entry_point', (object), {'load': lambda: object})]) with pytest.raises(TypeError): config_utils.load_consensus_plugin() + + +def test_map_leafs_iterator(): + from bigchaindb import config_utils + + mapping = { + 'a': {'b': {'c': 1}, + 'd': {'z': 44}}, + 'b': {'d': 2}, + 'c': 3 + } + + result = config_utils.map_leafs(lambda x, path: x * 2, mapping) + assert result == { + 'a': {'b': {'c': 2}, + 'd': {'z': 88}}, + 'b': {'d': 4}, + 'c': 6 + } + + result = config_utils.map_leafs(lambda x, path: path, mapping) + assert result == { + 'a': {'b': {'c': ['a', 'b', 'c']}, + 'd': {'z': ['a', 'd', 'z']}}, + 'b': {'d': ['b', 'd']}, + 'c': ['c'] + } + + +def test_update_types(): + from bigchaindb import config_utils + + raw = { + 'a_string': 'test', + 'an_int': '42', + 'a_float': '3.14', + 'a_list': 'a:b:c', + } + + reference = { + 'a_string': 'test', + 'an_int': 42, + 'a_float': 3.14, + 'a_list': ['a', 'b', 'c'], + } + + result = config_utils.update_types(raw, reference) + assert result == reference + + +def test_env_config(monkeypatch): + monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_HOST': 'test-host', + 'BIGCHAINDB_DATABASE_PORT': 'test-port'}) + + from bigchaindb import config_utils + + result = config_utils.env_config({'database': {'host': None, 'port': None}}) + expected = {'database': {'host': 'test-host', 'port': 'test-port'}} + + assert result == expected + + +def test_autoconfigure_read_both_from_file_and_env(monkeypatch): + file_config = { + 'database': {'host': 'test-host'} + } + monkeypatch.setattr('bigchaindb.config_utils.file_config', lambda *args, **kwargs: file_config) + monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_NAME': 'test-dbname', + 'BIGCHAINDB_DATABASE_PORT': '4242', + 'BIGCHAINDB_KEYRING': 'pubkey_0:pubkey_1:pubkey_2'}) + + import bigchaindb + from bigchaindb import config_utils + config_utils.autoconfigure() + + assert bigchaindb.config == { + 'CONFIGURED': True, + 'server': { + 'bind': '0.0.0.0:9984', + 'workers': None, + 'threads': None, + }, + 'database': { + 'host': 'test-host', + 'port': 4242, + 'name': 'test-dbname', + }, + 'keypair': { + 'public': None, + 'private': None, + }, + 'keyring': ['pubkey_0', 'pubkey_1', 'pubkey_2'], + 'statsd': { + 'host': 'localhost', + 'port': 8125, + 'rate': 0.01, + }, + 'api_endpoint': 'http://localhost:9984/api/v1', + 'consensus_plugin': 'default', + } +