Mirroristas/bigchaindb
2
0
Fork 0
mirror of https://github.com/bigchaindb/bigchaindb.git synced 2024-10-13 13:34:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into feat/116/more-solid-webserver

Browse Source
This commit is contained in:
vrde 2016-04-15 15:55:55 +02:00
commit 92ee52bb64
No known key found for this signature in database
GPG Key ID: 6581C7C39B3D397D
37 changed files with 795 additions and 460 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

64
CONTRIBUTING.md
View File

@ -12,7 +12,17 @@ If you want to file a bug report, suggest a feature, or ask a code-related quest
## How to Contribute Code or Documentation
### Step 0 - Prepare and Familiarize Yourself
### Step 0 - Decide on an Issue to Resolve, or Create One
We want you to feel like your contributions (pull requests) are welcome, but if you contribute something unnecessary, unwanted, or perplexing, then your experience may be unpleasant. Your pull request may sit gathering dust as everyone scratches their heads wondering what to do with it.
To prevent that situation, we ask that all pull requests should resolve, address, or fix an existing issue. If there is no existing issue, then you should create one first. That way there can be commentary and discussion first, and you can have a better idea of what to expect when you create a corresponding pull request.
When you submit a pull request, please mention the issue (or issues) that it resolves, e.g. "Resolves #123".
Exception: hotfixes and minor changes don't require a pre-existing issue, but please write a thorough pull request description.
### Step 1 - Prepare and Familiarize Yourself
To contribute code or documentation, you need a [GitHub account](https://github.com/signup/free).
@ -23,16 +33,13 @@ Familiarize yourself with how we do coding and documentation in the BigchainDB p
* the GitHub Flow (workflow)
* [GitHub Guide: Understanding the GitHub Flow](https://guides.github.com/introduction/flow/)
* [Scott Chacon's blog post about GitHub Flow](http://scottchacon.com/2011/08/31/github-flow.html)
* Note that we call the main branch `develop` rather than `master`
* [semantic versioning](http://semver.org/)
Note: We have a slight variation on the GitHub Flow: we call the default branch `develop` rather than `master`.
### Step 1 - Fork bigchaindb on GitHub
### Step 2 - Fork bigchaindb on GitHub
In your web browser, go to [the BigchainDB repository on GitHub](https://github.com/bigchaindb/bigchaindb) and click the `Fork` button in the top right corner. This creates a new Git repository named `bigchaindb` in _your_ GitHub account.
### Step 2 - Clone Your Fork
### Step 3 - Clone Your Fork
(This only has to be done once.) In your local terminal, use Git to clone _your_ `bigchaindb` repository to your local computer. Also add the original GitHub bigchaindb/bigchaindb repository as a remote named `upstream` (a convention):
```text
@ -41,16 +48,37 @@ cd bigchaindb
git add upstream git@github.com:bigchaindb/bigchaindb.git
```
### Step 3 - Fetch and Merge the Latest from `upstream/develop`
### Step 4 - Fetch and Merge the Latest from `upstream/master`
Switch to the `develop` branch locally, fetch all `upstream` branches, and merge the just-fetched `upstream/develop` branch with the local `develop` branch:
Switch to the `master` branch locally, fetch all `upstream` branches, and merge the just-fetched `upstream/master` branch with the local `master` branch:
```text
git checkout develop
git checkout master
git fetch upstream
git merge upstream/develop
git merge upstream/master
```
### Step 4 - Create a New Branch for Each Bug/Feature
### Step 5 - Install the Python module and the CLI
In order to use and run the source you just cloned from your fork, you need to install BigchainDB on your computer.
The core of BigchainDB is a Python module you can install using the standard [Python packaging tools](http://python-packaging-user-guide.readthedocs.org/en/latest/).
We highly suggest you use `pip` and `virtualenv` to manage your local development.
If you need more information on how to do that, refer to the *Python Packaging User Guide* to [install `pip`](http://python-packaging-user-guide.readthedocs.org/en/latest/installing/#requirements-for-installing-packages) and to [create your first `virtualenv`](http://python-packaging-user-guide.readthedocs.org/en/latest/installing/#creating-virtual-environments).
Once you have `pip` installed and (optionally) you are in a virtualenv, go to the root of the repository (i.e. where the `setup.py` file is), and type:
```bash
$ pip install -e .[dev]
```
This will install the BigchainDB Python module, the CLI, and all the dependencies useful for contributing to the development of BigchainDB.
How? Let's split the command down into its components:
- `pip` is the Python command to install packages
- `install` tells pip to use the *install* action
- `-e` installs a project in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)
- `.` installs what's in the current directory
- `[dev]` adds some [extra requirements](https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies) to the installation. (If you are curious, open `setup.py` and look for `dev` in the `extras_require` section.)
### Step 6 - Create a New Branch for Each Bug/Feature
If your new branch is to **fix a bug** identified in a specific GitHub Issue with number `ISSNO`, then name your new branch `bug/ISSNO/short-description-here`. For example, `bug/67/fix-leap-year-crash`.
@ -61,7 +89,7 @@ Otherwise, please give your new branch a short, descriptive, all-lowercase name.
git checkout -b new-branch-name
```
### Step 5 - Make Edits, git add, git commit
### Step 7 - Make Edits, git add, git commit
With your new branch checked out locally, make changes or additions to the code or documentation. Remember to:
@ -79,27 +107,27 @@ git commit -m "Short description of new or changed things"
You will want to merge changes from upstream (i.e. the original repository) into your new branch from time to time, using something like:
```text
git fetch upstream
git merge upstream/develop
git merge upstream/master
```
Once you're done commiting a set of new things and you're ready to submit them for inclusion, please be sure to run all the tests (as per the instructions at the end of our [Python Style Guide](PYTHON_STYLE_GUIDE.md)).
If your addition or change is substantial, then please add a line or two to the [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/CHANGELOG.md), following the guidelines given at the top of that file.
If your addition or change is substantial, then please add a line or two to the [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CHANGELOG.md), following the guidelines given at the top of that file.
(When you submit your pull request [following the instructions below], we run all the tests automatically, so we will see if some are failing. If you don't know why some tests are failing, you can still submit your pull request, but be sure to note the failing tests and to ask for help with resolving them.)
### Step 6 - Push Your New Branch to origin
### Step 8 - Push Your New Branch to origin
Make sure you've commited all the additions or changes you want to include in your pull request. Then push your new branch to origin (i.e. _your_ remote bigchaindb repository).
```text
git push origin new-branch-name
```
### Step 7 - Create a Pull Request
### Step 9 - Create a Pull Request
Go to the GitHub website and to _your_ remote bigchaindb repository (i.e. something like https://github.com/your-user-name/bigchaindb).
See [GitHub's documentation on how to initiate and send a pull request](https://help.github.com/articles/using-pull-requests/). Note that the destination repository should be `bigchaindb/bigchaindb` and the destination branch will be `develop` (usually, and if it's not, then we can change that if necessary).
See [GitHub's documentation on how to initiate and send a pull request](https://help.github.com/articles/using-pull-requests/). Note that the destination repository should be `bigchaindb/bigchaindb` and the destination branch will be `master` (usually, and if it's not, then we can change that if necessary).
If this is the first time you've submitted a pull request to BigchainDB, then you must read and accept the Contributor License Agreement (CLA) before we can merge your contributions. That can be found at [https://www.bigchaindb.com/cla](https://www.bigchaindb.com/cla).
@ -115,4 +143,4 @@ Someone will then merge your branch or suggest changes. If we suggsest changes,
* [BigchainDB Licenses](./LICENSES.md)
* [Contributor License Agreement](https://www.bigchaindb.com/cla)
(Note: GitHub automatically links to CONTRIBUTING.md when a contributor creates an Issue or opens a Pull Request.)
(Note: GitHub automatically links to CONTRIBUTING.md when a contributor creates an Issue or opens a Pull Request.)

19
PYTHON_STYLE_GUIDE.md
View File

@ -55,12 +55,27 @@ x = 'name: {}; score: {}'.format(name, n)
we use the `format()` version. The [official Python documentation says](https://docs.python.org/2/library/stdtypes.html#str.format), "This method of string formatting is the new standard in Python 3, and should be preferred to the % formatting described in String Formatting Operations in new code."
## Writing (Python) Tests
## Writing and Running (Python) Unit Tests
We write unit tests for our Python code using the [pytest](http://pytest.org/latest/) framework.
All tests go in the `bigchaindb/tests` directory or one of its subdirectories. You can use the tests already in there as templates or examples.
The BigchainDB Documentation has a [section explaining how to run all unit tests](http://bigchaindb.readthedocs.org/en/develop/running-unit-tests.html).
You can run all unit tests using:
```text
py.test -v
```
or, if that doesn't work, try:
```text
python -m pytest -v
```
or:
```text
python setup.py test
```
If you want to learn about all the things you can do with pytest, see [the pytest documentation](http://pytest.org/latest/).
**Automated testing of pull requests.** We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.)

22
README.md
View File

@ -1,19 +1,21 @@
[![Join the chat at https://gitter.im/bigchaindb/bigchaindb](https://badges.gitter.im/bigchaindb/bigchaindb.svg)](https://gitter.im/bigchaindb/bigchaindb?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![PyPI](https://img.shields.io/pypi/v/bigchaindb.svg)](https://pypi.python.org/pypi/BigchainDB)
[![Travis branch](https://img.shields.io/travis/bigchaindb/bigchaindb/master.svg)](https://travis-ci.org/bigchaindb/bigchaindb)
[![Codecov branch](https://img.shields.io/codecov/c/github/bigchaindb/bigchaindb/master.svg)](https://codecov.io/github/bigchaindb/bigchaindb?branch=master)
[![Documentation Status](https://readthedocs.org/projects/bigchaindb/badge/?version=stable)](https://bigchaindb.readthedocs.org/en/stable/)
# BigchainDB
A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whitepaper/) explains what that means.
[![Join the chat at https://gitter.im/bigchaindb/bigchaindb](https://badges.gitter.im/bigchaindb/bigchaindb.svg)](https://gitter.im/bigchaindb/bigchaindb?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![PyPI](https://img.shields.io/pypi/v/bigchaindb.svg)](https://pypi.python.org/pypi/BigchainDB)
[![Travis branch](https://img.shields.io/travis/bigchaindb/bigchaindb/develop.svg)](https://travis-ci.org/bigchaindb/bigchaindb)
[![Codecov branch](https://img.shields.io/codecov/c/github/bigchaindb/bigchaindb/develop.svg)](https://codecov.io/github/bigchaindb/bigchaindb?branch=develop)
[![Documentation Status](https://readthedocs.org/projects/bigchaindb/badge/?version=develop)](http://bigchaindb.readthedocs.org/en/develop/?badge=develop)
## Quick Start
### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.org/en/develop/installing-server.html)
### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.org/en/develop/installing-server.html#run-bigchaindb-with-docker)
### [The Python Server API by Example](http://bigchaindb.readthedocs.org/en/develop/python-server-api-examples.html)
### [The Python Driver API by Example](http://bigchaindb.readthedocs.org/en/develop/python-driver-api-examples.html)
### [Install and Run BigchainDB Server](http://bigchaindb.readthedocs.org/en/master/installing-server.html)
### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.org/en/master/installing-server.html#run-bigchaindb-with-docker)
### [The Python Server API by Example](http://bigchaindb.readthedocs.org/en/master/python-server-api-examples.html)
### [The Python Driver API by Example](http://bigchaindb.readthedocs.org/en/master/python-driver-api-examples.html)
## Links for Everyone
* [BigchainDB.com](https://www.bigchaindb.com/) - the main BigchainDB website, including newsletter signup
@ -24,7 +26,7 @@ A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whit
* [Google Group](https://groups.google.com/forum/#!forum/bigchaindb)
## Links for Developers
* [Documentation](http://bigchaindb.readthedocs.org/en/develop/#) - for developers
* [Documentation](http://bigchaindb.readthedocs.org/en/master/) - for developers
* [CONTRIBUTING.md](CONTRIBUTING.md) - how to contribute
* [Community guidelines](CODE_OF_CONDUCT.md)
* [Open issues](https://github.com/bigchaindb/bigchaindb/issues)

52
bigchaindb/__init__.py
View File

@ -1,49 +1,35 @@
import os
import copy
def e(key, default=None, conv=None):
'''Get the environment variable `key`, fallback to `default`
if nothing is found.
Keyword arguments:
key -- the key to look for in the environment
default -- the default value if nothing is found (default: None)
conv -- a callable used to convert the value (default: use the type of the
default value)
'''
val = os.environ.get(key, default)
if conv or default is not None:
conv = conv or type(default)
return conv(val)
return val
# from functools import reduce
# PORT_NUMBER = reduce(lambda x, y: x * y, map(ord, 'BigchainDB')) % 2**16
# basically, the port number is 9984
config = {
'server': {
'bind': e('BIGCHAIN_SERVER_BIND', default='0.0.0.0:5000'),
# Note: this section supports all the Gunicorn settings:
# - http://docs.gunicorn.org/en/stable/settings.html
'bind': '0.0.0.0:9984',
'workers': None, # if none, the value will be cpu_count * 2 + 1
'threads': None, # if none, the value will be cpu_count * 2 + 1
},
'database': {
'host': e('BIGCHAIN_DATABASE_HOST', default='localhost'),
'port': e('BIGCHAIN_DATABASE_PORT', default=28015),
'name': e('BIGCHAIN_DATABASE_NAME', default='bigchain')
'host': 'localhost',
'port': 28015,
'name': 'bigchain',
},
'keypair': {
'public': e('BIGCHAIN_KEYPAIR_PUBLIC'),
'private': e('BIGCHAIN_KEYPAIR_PRIVATE')
'public': None,
'private': None,
},
'keyring': [
],
'keyring': [],
'statsd': {
'host': e('BIGCHAIN_STATSD_HOST', default='localhost'),
'port': e('BIGCHAIN_STATSD_PORT', default=8125),
'rate': e('BIGCHAIN_STATSD_SAMPLERATE', default=0.01)
'host': 'localhost',
'port': 8125,
'rate': 0.01,
},
'api_endpoint': 'http://localhost:8008/api/v1',
'consensus_plugin': e('BIGCHAIN_CONSENSUS_PLUGIN', default='default')
'api_endpoint': 'http://localhost:9984/api/v1',
'consensus_plugin': 'default',
}
# We need to maintain a backup copy of the original config dict in case

4
bigchaindb/client.py
View File

@ -27,8 +27,8 @@ class Client:
3. Reading them from the `config.json` file.
Args:
public_key (str): the base58 encoded public key for the ECDSA secp256k1 curve.
private_key (str): the base58 encoded private key for the ECDSA secp256k1 curve.
public_key (str): the base58 encoded public key for the ED25519 curve.
private_key (str): the base58 encoded private key for the ED25519 curve.
api_endpoint (str): a URL where rethinkdb is running.
format: scheme://hostname:port
consensus_plugin (str): the registered name of your installed

27
bigchaindb/commands/bigchain.py
View File

@ -2,14 +2,16 @@
import os
import sys
import logging
import argparse
import copy
import json
import bigchaindb
import bigchaindb.config_utils
from bigchaindb import db
from bigchaindb.exceptions import DatabaseAlreadyExists
from bigchaindb.exceptions import DatabaseAlreadyExists, KeypairNotFoundException
from bigchaindb.commands.utils import base_parser, start
from bigchaindb.processes import Processes
from bigchaindb import crypto
@ -21,13 +23,15 @@ logger = logging.getLogger(__name__)
def run_show_config(args):
"""Show the current configuration"""
from pprint import pprint
# TODO Proposal: remove the "hidden" configuration. Only show config. If
# the system needs to be configured, then display information on how to
# configure the system.
bigchaindb.config_utils.file_config(args.config)
pprint(bigchaindb.config)
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
config = copy.deepcopy(bigchaindb.config)
del config['CONFIGURED']
private_key = config['keypair']['private']
config['keypair']['private'] = 'x' * 45 if private_key else None
print(json.dumps(config, indent=4, sort_keys=True))
def run_configure(args, skip_if_exists=False):
@ -55,7 +59,7 @@ def run_configure(args, skip_if_exists=False):
conf['keypair']['private'], conf['keypair']['public'] = crypto.generate_key_pair()
if not args.yes:
for key in ('host', 'port'):
for key in ('bind', ):
val = conf['server'][key]
conf['server'][key] = input('API Server {}? (default `{}`): '.format(key, val)) or val
@ -73,7 +77,7 @@ def run_configure(args, skip_if_exists=False):
def run_init(args):
"""Initialize the database"""
bigchaindb.config_utils.file_config(args.config)
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
# TODO Provide mechanism to:
# 1. prompt the user to inquire whether they wish to drop the db
# 2. force the init, (e.g., via -f flag)
@ -86,18 +90,21 @@ def run_init(args):
def run_drop(args):
"""Drop the database"""
bigchaindb.config_utils.file_config(args.config)
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
db.drop(assume_yes=args.yes)
def run_start(args):
"""Start the processes to run the node"""
run_configure(args, skip_if_exists=True)
bigchaindb.config_utils.file_config(args.config)
# run_configure(args, skip_if_exists=True)
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
try:
db.init()
except DatabaseAlreadyExists:
pass
except KeypairNotFoundException:
sys.exit('Cannot start BigchainDB, no keypair found. Did you run `bigchaindb configure`?')
processes = Processes()
logger.info('Start bigchaindb main process')
processes.start()

2
bigchaindb/commands/bigchain_benchmark.py
View File

@ -34,7 +34,7 @@ def _run_load(tx_left, stats):
def run_load(args):
bigchaindb.config_utils.file_config(args.config)
bigchaindb.config_utils.autoconfigure(filename=args.config, force=True)
logger.info('Starting %s processes', args.multiprocess)
stats = logstats.Logstats()
logstats.thread.start(stats)

125
bigchaindb/config_utils.py
View File

@ -5,12 +5,12 @@ By calling `file_config`, the global configuration (stored in
configuration file.
Note that there is a precedence in reading configuration values:
- [not yet] command line;
- local config file;
- environment vars;
- default config file (contained in `bigchain.__init__`).
- default config file (contained in ``bigchaindb.__init__``).
"""
import os
import copy
import json
@ -23,11 +23,33 @@ import bigchaindb
from bigchaindb.consensus import AbstractConsensusRules
logger = logging.getLogger(__name__)
CONFIG_DEFAULT_PATH = os.environ.setdefault(
'BIGCHAINDB_CONFIG_PATH',
os.path.join(os.path.expanduser('~'), '.bigchaindb'),
)
CONFIG_PREFIX = 'BIGCHAINDB'
CONFIG_SEP = '_'
def map_leafs(func, mapping):
"""Map a function to the leafs of a mapping."""
def _inner(mapping, path=None):
if path is None:
path = []
for key, val in mapping.items():
if isinstance(val, collections.Mapping):
_inner(val, path + [key])
else:
mapping[key] = func(val, path=path+[key])
return mapping
return _inner(copy.deepcopy(mapping))
# Thanks Alex <3
# http://stackoverflow.com/a/3233356/597097
@ -43,7 +65,7 @@ def update(d, u):
def file_config(filename=None):
"""Read a configuration file and merge it with the default configuration.
"""Returns the values found in a configuration file.
Args:
filename (str): the JSON file with the configuration. Defaults to ``None``.
@ -57,13 +79,73 @@ def file_config(filename=None):
filename = CONFIG_DEFAULT_PATH
with open(filename) as f:
newconfig = json.load(f)
config = json.load(f)
dict_config(newconfig)
logger.info('Configuration loaded from `{}`'.format(filename))
return config
def dict_config(newconfig):
def env_config(config):
"""Return a new configuration with the values found in the environment.
The function recursively iterates over the config, checking if there is
a matching env variable. If an env variable is found, the func updates
the configuration with that value.
The name of the env variable is built combining a prefix (``BIGCHAINDB``)
with the path to the value. If the ``config`` in input is:
``{'database': {'host': 'localhost'}}``
this function will try to read the env variable ``BIGCHAINDB_DATABASE_HOST``.
"""
def load_from_env(value, path):
var_name = CONFIG_SEP.join([CONFIG_PREFIX] + list(map(lambda s: s.upper(), path)))
return os.environ.get(var_name, value)
return map_leafs(load_from_env, config)
def update_types(config, reference, list_sep=':'):
"""Return a new configuration where all the values types
are aligned with the ones in the default configuration"""
def _coerce(current, value):
# Coerce a value to the `current` type.
try:
# First we try to apply current to the value, since it
# might be a function
return current(value)
except TypeError:
# Then we check if current is a list AND if the value
# is a string.
if isinstance(current, list) and isinstance(value, str):
# If so, we use the colon as the separator
return value.split(list_sep)
try:
# If we are here, we should try to apply the type
# of `current` to the value
return type(current)(value)
except TypeError:
# Worst case scenario we return the value itself.
return value
def _update_type(value, path):
current = reference
for elem in path:
try:
current = current[elem]
except KeyError:
return value
return _coerce(current, value)
return map_leafs(_update_type, config)
def dict_config(config):
"""Merge the provided configuration with the default one.
Args:
@ -74,11 +156,11 @@ def dict_config(newconfig):
update made to ``bigchaindb.config`` will be lost.
"""
bigchaindb.config = copy.deepcopy(bigchaindb._config)
update(bigchaindb.config, newconfig)
update(bigchaindb.config, update_types(config, bigchaindb.config))
bigchaindb.config['CONFIGURED'] = True
def write_config(newconfig, filename=None):
def write_config(config, filename=None):
"""Write the provided configuration to a specific location.
Args:
@ -90,18 +172,29 @@ def write_config(newconfig, filename=None):
filename = CONFIG_DEFAULT_PATH
with open(filename, 'w') as f:
json.dump(newconfig, f)
json.dump(config, f)
def autoconfigure():
"""Run ``file_config`` if the module has not been initialized.
"""
if bigchaindb.config.get('CONFIGURED'):
def autoconfigure(filename=None, config=None, force=False):
"""Run ``file_config`` and ``env_config`` if the module has not
been initialized."""
if not force and bigchaindb.config.get('CONFIGURED'):
logger.debug('System already configured, skipping autoconfiguration')
return
newconfig = env_config(bigchaindb.config)
try:
file_config()
except FileNotFoundError:
logger.warning('Cannot find your config file. Run `bigchaindb configure` to create one')
newconfig = update(newconfig, file_config(filename=filename))
except FileNotFoundError as e:
logger.warning('Cannot find config file `%s`.' % e.filename)
if config:
newconfig = update(newconfig, config)
dict_config(newconfig)
return newconfig
def load_consensus_plugin(name=None):

6
bigchaindb/consensus.py
View File

@ -2,7 +2,7 @@ from abc import ABCMeta, abstractmethod
import bigchaindb.exceptions as exceptions
from bigchaindb import util
from bigchaindb.crypto import hash_data, PublicKey
from bigchaindb import crypto
class AbstractConsensusRules(metaclass=ABCMeta):
@ -156,7 +156,7 @@ class BaseConsensusRules(AbstractConsensusRules):
transaction['transaction']['input']))
# Check hash of the transaction
calculated_hash = hash_data(util.serialize(
calculated_hash = crypto.hash_data(util.serialize(
transaction['transaction']))
if calculated_hash != transaction['id']:
raise exceptions.InvalidHash()
@ -185,7 +185,7 @@ class BaseConsensusRules(AbstractConsensusRules):
"""
# Check if current hash is correct
calculated_hash = hash_data(util.serialize(block['block']))
calculated_hash = crypto.hash_data(util.serialize(block['block']))
if calculated_hash != block['id']:
raise exceptions.InvalidHash()

9
bigchaindb/core.py
View File

@ -1,6 +1,5 @@
import rethinkdb as r
import random
import json
import rapidjson
@ -41,8 +40,8 @@ class Bigchain(object):
host (str): hostname where the rethinkdb is running.
port (int): port in which rethinkb is running (usually 28015).
dbname (str): the name of the database to connect to (usually bigchain).
public_key (str): the base58 encoded public key for the ECDSA secp256k1 curve.
private_key (str): the base58 encoded private key for the ECDSA secp256k1 curve.
public_key (str): the base58 encoded public key for the ED25519 curve.
private_key (str): the base58 encoded private key for the ED25519 curve.
keyring (list[str]): list of base58 encoded public keys of the federation nodes.
"""
@ -298,7 +297,7 @@ class Bigchain(object):
# Calculate the hash of the new block
block_data = util.serialize(block)
block_hash = crypto.hash_data(block_data)
block_signature = crypto.PrivateKey(self.me_private).sign(block_data)
block_signature = crypto.SigningKey(self.me_private).sign(block_data).decode()
block = {
'id': block_hash,
@ -419,7 +418,7 @@ class Bigchain(object):
}
vote_data = util.serialize(vote)
signature = crypto.PrivateKey(self.me_private).sign(vote_data)
signature = crypto.SigningKey(self.me_private).sign(vote_data).decode()
vote_signed = {
'node_pubkey': self.me,

152
bigchaindb/crypto.py
View File

@ -1,155 +1,17 @@
# Separate all crypto code so that we can easily test several implementations
import binascii
import base58
import sha3
import bitcoin
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import hashes
from cryptography.exceptions import InvalidSignature
class PrivateKey(object):
"""
PrivateKey instance
"""
def __init__(self, key):
"""
Instantiate the private key with the private_value encoded in base58
"""
private_value = self.decode(key)
private_numbers = self._private_value_to_cryptography_private_numbers(private_value)
self.private_key = self._cryptography_private_key_from_private_numbers(private_numbers)
def sign(self, data):
"""
Sign data with private key
"""
signer = self.private_key.signer(ec.ECDSA(hashes.SHA256()))
signer.update(data.encode('utf-8'))
signature = signer.finalize()
return binascii.hexlify(signature).decode('utf-8')
@staticmethod
def encode(private_value):
"""
Encode the decimal number private_value to base58
"""
private_value_hex = bitcoin.encode_privkey(private_value, 'hex')
private_value_base58 = base58.b58encode(bytes.fromhex(private_value_hex))
return private_value_base58
@staticmethod
def decode(key):
"""
Decode the base58 private_value to decimale
"""
private_value_hex = binascii.hexlify(base58.b58decode(key))
private_value = bitcoin.decode_privkey(private_value_hex)
return private_value
def _private_value_to_public_values(self, private_value):
"""
Return the public values from the private value
"""
public_value_x, public_value_y = bitcoin.privkey_to_pubkey(private_value)
return (public_value_x, public_value_y)
def _private_value_to_cryptography_private_numbers(self, private_value):
"""
Return an instance of cryptography PrivateNumbers from the decimal private_value
"""
public_value_x, public_value_y = self._private_value_to_public_values(private_value)
public_numbers = PublicKey._public_values_to_cryptography_public_numbers(public_value_x, public_value_y)
private_numbers = ec.EllipticCurvePrivateNumbers(private_value, public_numbers)
return private_numbers
@staticmethod
def _cryptography_private_key_from_private_numbers(private_numbers):
"""
Return an instace of cryptography PrivateKey from a cryptography instance of PrivateNumbers
"""
return private_numbers.private_key(default_backend())
class PublicKey(object):
def __init__(self, key):
"""
Instantiate the public key with the compressed public value encoded in base58
"""
public_value_x, public_value_y = self.decode(key)
public_numbers = self._public_values_to_cryptography_public_numbers(public_value_x, public_value_y)
self.public_key = self._criptography_public_key_from_public_numbers(public_numbers)
def verify(self, data, signature):
verifier = self.public_key.verifier(binascii.unhexlify(signature), ec.ECDSA(hashes.SHA256()))
verifier.update(data.encode('utf-8'))
try:
verifier.verify()
except InvalidSignature:
return False
return True
@staticmethod
def encode(public_value_x, public_value_y):
"""
Encode the public key represented by the decimal values x and y to base58
"""
public_value_compressed_hex = bitcoin.encode_pubkey([public_value_x, public_value_y], 'hex_compressed')
public_value_compressed_base58 = base58.b58encode(bytes.fromhex(public_value_compressed_hex))
return public_value_compressed_base58
@staticmethod
def decode(public_value_compressed_base58):
"""
Decode the base58 public_value to the decimal x and y values
"""
public_value_compressed_hex = binascii.hexlify(base58.b58decode(public_value_compressed_base58))
public_value_x, public_value_y = bitcoin.decode_pubkey(public_value_compressed_hex.decode())
return (public_value_x, public_value_y)
@staticmethod
def _public_values_to_cryptography_public_numbers(public_value_x, public_value_y):
"""
Return an instance of cryptography PublicNumbers from the decimal x and y values
"""
public_numbers = ec.EllipticCurvePublicNumbers(public_value_x, public_value_y, ec.SECP256K1())
return public_numbers
def _criptography_public_key_from_public_numbers(self, public_numbers):
"""
Return an instance of cryptography PublicKey from a cryptography instance of PublicNumbers
"""
return public_numbers.public_key(default_backend())
def generate_key_pair():
"""
Generate a new key pair and return the pair encoded in base58
"""
# Private key
private_key = ec.generate_private_key(ec.SECP256K1, default_backend())
private_value = private_key.private_numbers().private_value
private_value_base58 = PrivateKey.encode(private_value)
# Public key
public_key = private_key.public_key()
public_value_x, public_value_y = public_key.public_numbers().x, public_key.public_numbers().y
public_value_compressed_base58 = PublicKey.encode(public_value_x, public_value_y)
return (private_value_base58, public_value_compressed_base58)
from cryptoconditions import ed25519
def hash_data(data):
"""Hash the provided data using SHA3-256"""
return sha3.sha3_256(data.encode()).hexdigest()
def generate_key_pair():
sk, pk = ed25519.ed25519_generate_key_pair()
return sk.decode(), pk.decode()
SigningKey = ed25519.Ed25519SigningKey
VerifyingKey = ed25519.Ed25519VerifyingKey

4
bigchaindb/db/utils.py
View File

@ -19,6 +19,9 @@ def get_conn():
def init():
# Try to access the keypair, throws an exception if it does not exist
b = bigchaindb.Bigchain()
conn = get_conn()
dbname = bigchaindb.config['database']['name']
@ -59,7 +62,6 @@ def init():
r.db(dbname).table('bigchain').index_wait().run(conn)
logger.info(' - genesis block')
b = bigchaindb.Bigchain()
b.create_genesis_block()
logger.info('Done, have fun!')

16
bigchaindb/util.py
View File

@ -9,7 +9,7 @@ from datetime import datetime
import bigchaindb
from bigchaindb import exceptions
from bigchaindb.crypto import PrivateKey, PublicKey, hash_data
from bigchaindb import crypto
class ProcessGroup(object):
@ -159,7 +159,7 @@ def create_tx(current_owner, new_owner, tx_input, operation, payload=None):
data = None
if payload is not None:
if isinstance(payload, dict):
hash_payload = hash_data(serialize(payload))
hash_payload = crypto.hash_data(serialize(payload))
data = {
'hash': hash_payload,
'payload': payload
@ -167,7 +167,7 @@ def create_tx(current_owner, new_owner, tx_input, operation, payload=None):
else:
raise TypeError('`payload` must be an dict instance')
hash_payload = hash_data(serialize(payload))
hash_payload = crypto.hash_data(serialize(payload))
data = {
'hash': hash_payload,
'payload': payload
@ -184,7 +184,7 @@ def create_tx(current_owner, new_owner, tx_input, operation, payload=None):
# serialize and convert to bytes
tx_serialized = serialize(tx)
tx_hash = hash_data(tx_serialized)
tx_hash = crypto.hash_data(tx_serialized)
# create the transaction
transaction = {
@ -208,10 +208,10 @@ def sign_tx(transaction, private_key):
dict: transaction with the `signature` field included.
"""
private_key = PrivateKey(private_key)
private_key = crypto.SigningKey(private_key)
signature = private_key.sign(serialize(transaction))
signed_transaction = transaction.copy()
signed_transaction.update({'signature': signature})
signed_transaction.update({'signature': signature.decode()})
return signed_transaction
@ -222,7 +222,7 @@ def create_and_sign_tx(private_key, current_owner, new_owner, tx_input, operatio
def check_hash_and_signature(transaction):
# Check hash of the transaction
calculated_hash = hash_data(serialize(transaction['transaction']))
calculated_hash = crypto.hash_data(serialize(transaction['transaction']))
if calculated_hash != transaction['id']:
raise exceptions.InvalidHash()
@ -251,7 +251,7 @@ def verify_signature(signed_transaction):
signature = data.pop('signature')
public_key_base58 = signed_transaction['transaction']['current_owner']
public_key = PublicKey(public_key_base58)
public_key = crypto.VerifyingKey(public_key_base58)
return public_key.verify(serialize(data), signature)

3
bigchaindb/web/server.py
View File

@ -76,6 +76,9 @@ def create_server(settings):
if not settings.get('workers'):
settings['workers'] = (multiprocessing.cpu_count() * 2) + 1
if not settings.get('threads'):
settings['threads'] = (multiprocessing.cpu_count() * 2) + 1
debug = settings.pop('debug', False)
app = create_app(debug)
standalone = StandaloneApplication(app, settings)

4
codecov.yml
View File

@ -1,5 +1,5 @@
codecov:
branch: develop # the branch to show by default
branch: master # the branch to show by default
# The help text for bot says:
# "the username that will consume any oauth requests
@ -29,4 +29,4 @@ coverage:
comment:
layout: "header, diff, changes, sunburst, suggestions"
behavior: default
behavior: default

4
deploy-cluster-aws/create_rethinkdb_conf.py
View File

@ -8,7 +8,7 @@ from __future__ import unicode_literals
import os
import os.path
import shutil
from hostlist import hosts_dev
from hostlist import public_dns_names
# cwd = current working directory
old_cwd = os.getcwd()
@ -22,7 +22,7 @@ shutil.copy2('rethinkdb.conf.template', 'rethinkdb.conf')
# Append additional lines to rethinkdb.conf
with open('rethinkdb.conf', 'a') as f:
f.write('## The host:port of a node that RethinkDB will connect to\n')
for public_dns_name in hosts_dev:
for public_dns_name in public_dns_names:
f.write('join=' + public_dns_name + ':29015\n')
os.chdir(old_cwd)

27
deploy-cluster-aws/fab_prepare_chain.py
View File

@ -1,27 +0,0 @@
# -*- coding: utf-8 -*-
""" Generating genesis block
"""
from __future__ import with_statement, unicode_literals
from fabric import colors as c
from fabric.api import *
from fabric.api import local, puts, settings, hide, abort, lcd, prefix
from fabric.api import run, sudo, cd, get, local, lcd, env, hide
from fabric.api import task, parallel
from fabric.contrib import files
from fabric.contrib.files import append, exists
from fabric.contrib.console import confirm
from fabric.contrib.project import rsync_project
from fabric.operations import run, put
from fabric.context_managers import settings
from fabric.decorators import roles
from fabtools import *
env.user = 'ubuntu'
env.key_filename = 'pem/bigchaindb.pem'
@task
def init_bigchaindb():
run('bigchaindb -y start &', pty = False)

117
deploy-cluster-aws/fabfile.py vendored
View File

@ -1,47 +1,58 @@
# -*- coding: utf-8 -*-
"""A fabfile with functionality to prepare, install, and configure
bigchaindb, including its storage backend.
"""A Fabric fabfile with functionality to prepare, install, and configure
BigchainDB, including its storage backend (RethinkDB).
"""
from __future__ import with_statement, unicode_literals
import requests
from time import *
import os
from datetime import datetime, timedelta
import json
from pprint import pprint
from fabric import colors as c
from fabric.api import *
from fabric.api import local, puts, settings, hide, abort, lcd, prefix
from fabric.api import run, sudo, cd, get, local, lcd, env, hide
from fabric.api import sudo, env
from fabric.api import task, parallel
from fabric.contrib import files
from fabric.contrib.files import append, exists
from fabric.contrib.console import confirm
from fabric.contrib.project import rsync_project
from fabric.contrib.files import sed
from fabric.operations import run, put
from fabric.context_managers import settings
from fabric.decorators import roles
from fabtools import *
from hostlist import hosts_dev
from hostlist import public_dns_names
env.hosts = hosts_dev
env.roledefs = {
"role1": hosts_dev,
"role2": [hosts_dev[0]],
}
env.roles = ["role1"]
# Ignore known_hosts
# http://docs.fabfile.org/en/1.10/usage/env.html#disable-known-hosts
env.disable_known_hosts = True
# What remote servers should Fabric connect to? With what usernames?
env.user = 'ubuntu'
env.hosts = public_dns_names
# SSH key files to try when connecting:
# http://docs.fabfile.org/en/1.10/usage/env.html#key-filename
env.key_filename = 'pem/bigchaindb.pem'
newrelic_license_key = 'you_need_a_real_license_key'
######################################################################
# base software rollout
# DON'T PUT @parallel
@task
def set_hosts(hosts):
"""A helper function to change env.hosts from the
command line.
Args:
hosts (str): 'one_node' or 'two_nodes'
Example:
fab set_hosts:one_node init_bigchaindb
"""
if hosts == 'one_node':
env.hosts = public_dns_names[:1]
elif hosts == 'two_nodes':
env.hosts = public_dns_names[:2]
else:
raise ValueError('Invalid input to set_hosts.'
' Expected one_node or two_nodes.'
' Got {}'.format(hosts))
# Install base software
@task
@parallel
def install_base_software():
@ -59,7 +70,7 @@ def install_base_software():
python3-pip ipython3 sysstat s3cmd')
# RethinkDB
# Install RethinkDB
@task
@parallel
def install_rethinkdb():
@ -67,7 +78,7 @@ def install_rethinkdb():
with settings(warn_only=True):
# preparing filesystem
sudo("mkdir -p /data")
# Locally mounted storage (m3.2xlarge, aber auch c3.xxx)
# Locally mounted storage (m3.2xlarge, but also c3.xxx)
try:
sudo("umount /mnt")
sudo("mkfs -t ext4 /dev/xvdb")
@ -91,27 +102,48 @@ def install_rethinkdb():
sudo('chown -R rethinkdb:rethinkdb /data')
# copy config file to target system
put('conf/rethinkdb.conf',
'/etc/rethinkdb/instances.d/instance1.conf', mode=0600, use_sudo=True)
'/etc/rethinkdb/instances.d/instance1.conf',
mode=0600,
use_sudo=True)
# initialize data-dir
sudo('rm -rf /data/*')
# finally restart instance
sudo('/etc/init.d/rethinkdb restart')
# bigchaindb deployment
# Install BigchainDB (from PyPI)
@task
@parallel
def install_bigchaindb():
sudo('python3 -m pip install bigchaindb')
# startup all nodes of bigchaindb in cluster
# Configure BigchainDB
@task
@parallel
def start_bigchaindb_nodes():
def configure_bigchaindb():
run('bigchaindb -y configure', pty=False)
# Initialize BigchainDB
# i.e. create the database, the tables,
# the indexes, and the genesis block.
# (This only needs to be run on one node.)
# Call using:
# fab set_hosts:one_node init_bigchaindb
@task
def init_bigchaindb():
run('bigchaindb init', pty=False)
# Start BigchainDB using screen
@task
@parallel
def start_bigchaindb():
sudo('screen -d -m bigchaindb -y start &', pty=False)
# Install and run New Relic
@task
def install_newrelic():
with settings(warn_only=True):
@ -119,18 +151,18 @@ def install_newrelic():
# sudo('apt-key adv --keyserver hkp://subkeys.pgp.net --recv-keys 548C16BF')
sudo('apt-get update')
sudo('apt-get -y --force-yes install newrelic-sysmond')
sudo('nrsysmond-config --set license_key=c88af00c813983f8ee12e9b455aa13fde1cddaa8')
sudo('nrsysmond-config --set license_key=' + newrelic_license_key)
sudo('/etc/init.d/newrelic-sysmond restart')
###############################
# Security / FirewallStuff next
###############################
###########################
# Security / Firewall Stuff
###########################
@task
def harden_sshd():
"""Security harden sshd."""
"""Security harden sshd.
"""
# Disable password authentication
sed('/etc/ssh/sshd_config',
'#PasswordAuthentication yes',
@ -147,7 +179,8 @@ def harden_sshd():
def disable_root_login():
"""Disable `root` login for even more security. Access to `root` account
is now possible by first connecting with your dedicated maintenance
account and then running ``sudo su -``."""
account and then running ``sudo su -``.
"""
sudo('passwd --lock root')
@ -172,7 +205,7 @@ def set_fw():
#########################################################
# some helper-functions to handle bad behavior of cluster
# Some helper-functions to handle bad behavior of cluster
#########################################################
# rebuild indexes

15
deploy-cluster-aws/launch_ec2_nodes.py
View File

@ -166,26 +166,31 @@ for i, instance in enumerate(instances_with_tag):
format(instance.instance_id))
# Get a list of the pubic DNS names of the instances_with_tag
hosts_dev = []
public_dns_names = []
for instance in instances_with_tag:
public_dns_name = getattr(instance, 'public_dns_name', None)
if public_dns_name is not None:
hosts_dev.append(public_dns_name)
public_dns_names.append(public_dns_name)
# Write a shellscript to add remote keys to ~/.ssh/known_hosts
print('Preparing shellscript to add remote keys to known_hosts')
with open('add2known_hosts.sh', 'w') as f:
f.write('#!/bin/bash\n')
for public_dns_name in hosts_dev:
for public_dns_name in public_dns_names:
f.write('ssh-keyscan ' + public_dns_name + ' >> ~/.ssh/known_hosts\n')
# Create a file named hostlist.py containing hosts_dev.
# Create a file named hostlist.py containing public_dns_names.
# If a hostlist.py already exists, it will be overwritten.
print('Writing hostlist.py')
with open('hostlist.py', 'w') as f:
f.write('# -*- coding: utf-8 -*-\n')
f.write('"""A list of the public DNS names of all the nodes in this\n')
f.write('BigchainDB cluster/federation.\n')
f.write('"""\n')
f.write('\n')
f.write('from __future__ import unicode_literals\n')
f.write('hosts_dev = {}\n'.format(hosts_dev))
f.write('\n')
f.write('public_dns_names = {}\n'.format(public_dns_names))
# Wait
wait_time = 45

34
deploy-cluster-aws/startup.sh
View File

@ -55,27 +55,33 @@ chmod +x add2known_hosts.sh
# (Re)create the RethinkDB configuration file conf/rethinkdb.conf
python create_rethinkdb_conf.py
# rollout base packages (dependencies) needed before
# storage backend (rethinkdb) and bigchaindb can be rolled out
# Rollout base packages (dependencies) needed before
# storage backend (RethinkDB) and BigchainDB can be rolled out
fab install_base_software
# rollout storage backend (rethinkdb)
# Rollout storage backend (RethinkDB) and start it
fab install_rethinkdb
# rollout bigchaindb
# Rollout BigchainDB (but don't start it yet)
fab install_bigchaindb
# generate genesis block
# HORST is the last public_dns_name listed in conf/rethinkdb.conf
# For example:
# ec2-52-58-86-145.eu-central-1.compute.amazonaws.com
HORST=`tail -1 conf/rethinkdb.conf|cut -d: -f1|cut -d= -f2`
fab -H $HORST -f fab_prepare_chain.py init_bigchaindb
# Configure BigchainDB on all nodes
fab configure_bigchaindb
# initiate sharding
fab start_bigchaindb_nodes
# TODO Get public keys from all nodes
# using e.g. bigchaindb export-pubkey
# TODO Add list of public keys to keyring of all nodes
# using e.g. bigchaindb import-pubkey
# Send a "bigchaindb init" command to one node
# to initialize the BigchainDB database
# i.e. create the database, the tables,
# the indexes, and the genesis block.
fab set_hosts:one_node init_bigchaindb
# Start BigchainDB on all the nodes using "screen"
fab start_bigchaindb
# cleanup
rm add2known_hosts.sh
# DONE

7
docs/source/bigchaindb-cli.md
View File

@ -2,10 +2,11 @@
BigchainDB has some Command Line Interfaces (CLIs). One of them is the `bigchaindb` command which we already saw when we first started BigchainDB using:
```text
$ bigchaindb configure
$ bigchaindb start
```
The fist time you run `bigchaindb start`, it creates a default configuration file in `$HOME/.bigchaindb`. You can check that configuration using:
When you run `bigchaindb configure`, it creates a default configuration file in `$HOME/.bigchaindb`. You can check that configuration using:
```text
$ bigchaindb show-config
```
@ -20,3 +21,7 @@ There's another command named `bigchaindb-benchmark`. It's used to run benchmark
$ bigchaindb-benchmark -h
$ bigchaindb-benchmark load -h
```
Note that you can always start `bigchaindb` using a different config file using the `-c` option.
For more information check the help with `bigchaindb -h`.

226
docs/source/configuration.md Normal file
View File

@ -0,0 +1,226 @@
# Configuring a BigchainDB Node
The BigchainDB configuration settings for a particular node are stored on that node in a configuration file at `$HOME/.bigchaindb`. That file doesn't exist by default. (It's not created when installing BigchainDB.) One could create it using a text editor, but it's easiest to use the `bigchaindb configure` command:
```text
$ bigchaindb configure
```
It will ask some questions and generate a new keypair (i.e. a private key and corresponding public key for the node). See below for some additional explanation of the settings and their meanings. To accept a suggested default value, press Enter or Return. If you want to accept all the default values, use the `-y` option when running the command, that is:
```text
$ bigchaindb -y configure
```
## Using a Different Path for the Configuration File
By default, the configuration settings are stored in `$HOME/.bigchaindb`. If you want to
specify a different path for your configuration file, you can use the `-c` parameter.
This works for every subcommand under the `bigchaindb` executable.
For example, if you want to **generate** a new configuration file under a
specific path, you can run:
```
$ bigchaindb -c local.json configure
$ bigchaindb -c test.json configure
```
This will create two new files named `local.json` and `test.json` in your
current working directory.
From now on, you can refer to those configuration files using the `-c`
parameter; for example:
```
$ bigchaindb -c local.json show-config
```
will show the configuration for `local.json`.
If you want to **start** BigchainDB with the `test.json` configuration file, you can use:
```
$ bigchaindb -c test.json start
```
## Using Environment Variables to Configure the Node
Sometimes it's more convenient to use environment variables to configure the
system, for example when using Docker or Heroku. In
that case you can configure the system using environment variables.
Every configuration parameter can be mapped to an environment variable. The
environment variables available are:
- `BIGCHAINDB_DATABASE_HOST` defines the RethinkDB database hostname to connect to.
- `BIGCHAINDB_DATABASE_PORT` defines the RethinkDB database port to connect to.
- `BIGCHAINDB_DATABASE_NAME` defines the RethinkDB database name to use.
- `BIGCHAINDB_KEYPAIR_PUBLIC` defines the public key of the BigchainDB node.
- `BIGCHAINDB_KEYPAIR_PRIVATE` defines the private key of the BigchainDB node.
- `BIGCHAINDB_KEYRING` is a colon-separated list of the public keys of all _other_ nodes in the cluster.
- `BIGCHAINDB_STATSD_HOST` defines the hostname of the statsd server for [monitoring](monitoring.html).
- `BIGCHAINDB_STATSD_PORT` defines the port of the statsd server for monitoring.
- `BIGCHAINDB_STATSD_RATE` is a float between `0` and `1` that defines the fraction of transaction operations sampled.
- `BIGCHAINDB_API_ENDPOINT` defines the API endpoint to use (e.g. `http://localhost:9984/api/v1`).
- `BIGCHAINDB_CONSENSUS_PLUGIN` defines the name of the [consensus plugin](consensus.html) to use.
- `BIGCHAINDB_SERVER_BIND` defines where to bind the server socket, the format is `addr:port` (e.g. `0.0.0.0:9984`).
- `BIGCHAINDB_SERVER_WORKERS` defines the [number of workers](http://docs.gunicorn.org/en/stable/settings.html#workers)
to start for the server API.
- `BIGCHAINDB_SERVER_THREADS` defines the [number of threads](http://docs.gunicorn.org/en/stable/settings.html#threads)
to start for the server API.
## Order of Precedence in Determining Configuration Values
All configuration values start with their default values (defined in `bigchaindb.__init__`), but a default value can be overriden by an environment variable, and a value set by an environment variable can be overriden by a value in a local configuration file (`$HOME/.bigchaindb` or the location specified by the `-c` command-line option).
In summary, there is an order of precedence in reading configuration values:
1. local configuration file
2. environment variables
3. default configuration file (defined in ``bigchaindb.__init__``)
This means that if the default configuration contains:
```json
{
"database": {
"host": "localhost",
"port": 28015
}
}
```
while the local file `local.json` contains:
```json
{
"database": {
"host": "ec2-xx-xx-xxx-xxx.eu-central-1.compute.amazonaws.com"
}
}
```
and you run this command:
```
$ BIGCHAINDB_DATABASE_HOST=anotherhost.com \
BIGCHAINDB_DATABASE_PORT=4242 \
BIGCHAINDB_KEYRING=pubkey0:pubkey1 \
bigchaindb -c local.json show-config
```
you will get the following values for all the configuration settings:
```json
{
"api_endpoint": "http://localhost:8008/api/v1",
"consensus_plugin": "default",
"database": {
"host": "ec2-xx-xx-xxx-xxx.eu-central-1.compute.amazonaws.com",
"name": "bigchain",
"port": 4242
},
"keypair": {
"private": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"public": "nJq6EmdUkvFjQRB5hFvDmvZtv1deb3W3RgmiAq6dyygC"
},
"keyring": [
"pubkey0",
"pubkey1"
],
"server": {
"bind": "0.0.0.0:9984",
"threads": null,
"workers": null
},
"statsd": {
"host": "localhost",
"port": 8125,
"rate": 0.01
}
}
```
## Another Example
As another example, let's assume we **don't** have any configuration file stored in
`$HOME/.bigchaindb`. As you can see, `show-config` displays the default configuration (and a
warning):
```
$ bigchaindb show-config
WARNING:bigchaindb.config_utils:Cannot find config file `/home/vrde/.bigchaindb`.
{
"api_endpoint": "http://localhost:9984/api/v1",
"consensus_plugin": "default",
"database": {
"host": "localhost",
"name": "bigchain",
"port": 28015
},
"keypair": {
"private": null,
"public": null
},
"keyring": [],
"server": {
"bind": "0.0.0.0:9984",
"threads": null,
"workers": null
},
"statsd": {
"host": "localhost",
"port": 8125,
"rate": 0.01
}
}
```
If we try to run the node, the command will fail:
```
$ bigchaindb start
WARNING:bigchaindb.config_utils:Cannot find config file `/home/vrde/.bigchaindb`.
Cannot start BigchainDB, no keypair found. Did you run `bigchaindb configure`?
```
This is failing as expected: a BigchainDB node needs at least a key pair to work.
We can pass the key pair using environment variables:
```
$ BIGCHAINDB_KEYPAIR_PUBLIC=26y9EuyGP44JXxqcvF8GbCJGqkiqFXddZzxVjLU3rWbHp \
BIGCHAINDB_KEYPAIR_PRIVATE=9PkLfHbzXnSSNnb1sSBL73C2MydzKLs5fAHoA4Q7otrG \
bigchaindb start
```
We can also run `show-config` to see how the configuration looks:
```
$ BIGCHAINDB_KEYPAIR_PUBLIC=26y9EuyGP44JXxqcvF8GbCJGqkiqFXddZzxVjLU3rWbHp \
BIGCHAINDB_KEYPAIR_PRIVATE=9PkLfHbzXnSSNnb1sSBL73C2MydzKLs5fAHoA4Q7otrG \
bigchaindb show-config
WARNING:bigchaindb.config_utils:Cannot find config file `/home/vrde/.bigchaindb`.
{
"api_endpoint": "http://localhost:9984/api/v1",
"consensus_plugin": "default",
"database": {
"host": "localhost",
"name": "bigchain",
"port": 28015
},
"keypair": {
"private": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"public": "26y9EuyGP44JXxqcvF8GbCJGqkiqFXddZzxVjLU3rWbHp"
},
"keyring": [],
"server": {
"bind": "0.0.0.0:9984",
"threads": null,
"workers": null
},
"statsd": {
"host": "localhost",
"port": 8125,
"rate": 0.01
}
}
```

4
docs/source/cryptography.md
View File

@ -19,8 +19,8 @@ tx_hash = hashlib.sha3_256(data).hexdigest()
## Signature algorithm and keys
The signature algorithm used by BigchainDB is ECDSA with the secp256k1 curve
using the python [cryptography](https://cryptography.io/en/latest/) module.
The signature algorithm used by BigchainDB is [ED25519](https://tools.ietf.org/html/draft-irtf-cfrg-eddsa-04)
using the python [ed25519](https://github.com/warner/python-ed25519) module, overloaded by the [cryptoconditions library](https://github.com/bigchaindb/cryptoconditions).
The private key is the base58 encoded hexadecimal representation of private number.
The public key is the base58 encoded hexadecimal representation of the

13
docs/source/http-client-server-api.md
View File

@ -2,10 +2,15 @@
When you start Bigchaindb using `bigchaindb start`, an HTTP API is exposed at:
[http://localhost:5000/api/v1/](http://localhost:5000/api/v1/)
- [http://localhost:9984/api/v1/](http://localhost:9984/api/v1/)
Right now, that API can only be accessed from localhost (i.e. not remotely). In the future, we'll enable remote access and explain how that works. See [Issue #149](https://github.com/bigchaindb/bigchaindb/issues/149) on GitHub.
The HTTP API currently exposes two endpoints, one to get information about a specific transaction id, and one to push a transaction to the BigchainDB cluster. Those endpoints are documented at:
Please note that by default the server binds to `0.0.0.0:9984`, hence the API
is exposed to the world.
The HTTP API currently exposes two endpoints, one to get information about a
specific transaction id, and one to push a transaction to the BigchainDB
cluster. Those endpoints are documented at:
- [http://docs.bigchaindb.apiary.io/](http://docs.bigchaindb.apiary.io/)
[http://docs.bigchaindb.apiary.io/](http://docs.bigchaindb.apiary.io/)

1
docs/source/index.rst
View File

@ -15,6 +15,7 @@ Table of Contents
introduction
installing-server
running-unit-tests
configuration
python-server-api-examples
bigchaindb-cli
http-client-server-api

30
docs/source/installing-server.md
View File

@ -68,14 +68,14 @@ Note: You can use `pip` to upgrade the `bigchaindb` package to the latest versio
### How to Install BigchainDB from Source
BigchainDB (i.e. both the Server and the officially-supported drivers) is in its early stages and being actively developed on its [GitHub repository](https://github.com/bigchaindb/bigchaindb). Contributions are highly appreciated. If you want to help with development, then you'll want to install BigchainDB from source. Here's how.
If you want to install BitchainDB from source because you want to contribute code (i.e. as a BigchainDB developer), then please see the instructions in [the `CONTRIBUTING.md` file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md).
First, clone the public repository:
Otherwise, clone the public repository:
```text
$ git clone git@github.com:bigchaindb/bigchaindb.git
```
Install from the source:
and then install from source:
```text
$ python setup.py install
```
@ -94,10 +94,16 @@ $ rethinkdb
Then open a different terminal and run:
```text
$ bigchaindb start
$ bigchaindb -y configure
$ bigchaindb init
```
During its first run, BigchainDB Server takes care of configuring a single node environment.
That creates a configuration file in `$HOME/.bigchaindb` (documented in [the section on configuration](configuration.html)), initializes the database, creates the tables, creates the indexes, and generates the genesis block.
You can start BigchainDB Server using:
```text
$ bigchaindb start
```
## Run BigchainDB with Docker
@ -122,8 +128,22 @@ then do a one-time configuration step to create the config file; it will be
stored on your host machine under ` ~/.bigchaindb_docker/config`:
```text
$ docker-compose run --rm bigchaindb bigchaindb configure
Starting bigchaindb_rethinkdb-data_1
Generating keypair
API Server bind? (default `0.0.0.0:9984`):
Database host? (default `localhost`): rethinkdb
Database port? (default `28015`):
Database name? (default `bigchain`):
Statsd host? (default `localhost`): statsd
Statsd port? (default `8125`):
Statsd rate? (default `0.01`):
Ready to go!
```
As shown above, make sure that you set the database and statsd hosts to their
corresponding service names (`rethinkdb`, `statsd`), defined in`docker-compose.yml`
and `docker-compose-monitor.yml`.
You can then start it up (in the background, as a daemon) using:
```text
$ docker-compose up -d

2
docs/source/introduction.md
View File

@ -7,4 +7,4 @@ BigchainDB is a scalable blockchain database. You can read about its motivations
3. Developers of BigchainDB driver software (SDKs used to develop client software).
4. App developers who are developing client apps to talk to one or more live, operational BigchainDB clusters. They would use one of the BigchainDB drivers.
If you're curious about what's in our roadmap, see [the ROADMAP.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/ROADMAP.md) and [the list of open issues](https://github.com/bigchaindb/bigchaindb/issues). If you want to request a feature, file a bug report, or make a pull request, see [the CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/CONTRIBUTING.md).
If you're curious about what's in our roadmap, see [the ROADMAP.md file](https://github.com/bigchaindb/bigchaindb/blob/master/ROADMAP.md) and [the list of open issues](https://github.com/bigchaindb/bigchaindb/issues). If you want to request a feature, file a bug report, or make a pull request, see [the CONTRIBUTING.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CONTRIBUTING.md).

2
docs/source/licenses.md
View File

@ -1,3 +1,3 @@
# Licenses
Information about how the BigchainDB code and documentation are licensed can be found in [the LICENSES.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/LICENSES.md) (in the root directory of the repository).
Information about how the BigchainDB code and documentation are licensed can be found in [the LICENSES.md file](https://github.com/bigchaindb/bigchaindb/blob/master/LICENSES.md) (in the root directory of the repository).

2
docs/source/python-server-api-examples.md
View File

@ -11,6 +11,8 @@ We create a digital asset, sign it, write it to a BigchainDB Server instance, re
First, make sure you have RethinkDB and BigchainDB _installed and running_, i.e. you [installed them](installing-server.html) and you ran:
```text
$ rethinkdb
$ bigchaindb configure
$ bigchaindb init
$ bigchaindb start
```

4
docs/source/release-notes.md
View File

@ -4,6 +4,6 @@ You can find a list of all BigchainDB releases and release notes on GitHub at:
[https://github.com/bigchaindb/bigchaindb/releases](https://github.com/bigchaindb/bigchaindb/releases)
The [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/develop/CHANGELOG.md) contains much the same information, but it also has notes about what to expect in the _next_ release.
The [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CHANGELOG.md) contains much the same information, but it also has notes about what to expect in the _next_ release.
We also have [a roadmap document in ROADMAP.md](https://github.com/bigchaindb/bigchaindb/blob/develop/ROADMAP.md).
We also have [a roadmap document in ROADMAP.md](https://github.com/bigchaindb/bigchaindb/blob/master/ROADMAP.md).

18
docs/source/running-unit-tests.md
View File

@ -2,29 +2,21 @@
Once you've installed BigchainDB Server, you may want to run all the unit tests. This section explains how.
First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. Before you can run all the unit tests, you must [install BigchainDB from source](installing-server.html#how-to-install-bigchaindb-from-source).
First of all, if you installed BigchainDB Server using `pip` (i.e. by getting the package from PyPI), then you didn't install the tests. **Before you can run all the unit tests, you must [install BigchainDB from source](installing-server.html#how-to-install-bigchaindb-from-source).**
To run all the unit tests, first make sure you have RethinkDB running:
```text
$ rethinkdb
```
then in another terminal, do:
```text
$ py.test -v
$ python setup.py test
```
If the above command doesn't work (e.g. maybe you are running in a conda virtual environment), try:
```text
$ python -m pytest -v
```
(We write our unit tests using the [pytest](http://pytest.org/latest/) framework.)
You can also run all unit tests via `setup.py`, using:
```text
$ python setup.py test
```
(Aside: How does the above command work? The documentation for [pytest-runner](https://pypi.python.org/pypi/pytest-runner) explains. We use [pytest](http://pytest.org/latest/) to write all unit tests.)
### Using docker-compose to Run the Tests

2
setup.py
View File

@ -71,7 +71,7 @@ setup(
'rethinkdb==2.2.0.post4',
'pysha3==0.3',
'pytz==2015.7',
'cryptography==1.2.3',
'cryptoconditions==0.1.1',
'statsd==3.2.1',
'python-rapidjson==0.0.6',
'logstats==0.2.1',

21
tests/conftest.py
View File

@ -19,17 +19,30 @@ CONFIG = {
'name': DB_NAME
},
'keypair': {
'private': '3i2FDXp87N9ExXSvWxqBAw9EgzoxxGTQNKbtxmWBpTyL',
'public': '29Tw3ozmSRtN8XNofvsu5RdoQRk9gAonfpkFvRZDmhTPo'
'private': '31Lb1ZGKTyHnmVK3LUMrAUrPNfd4sE2YyBt3UA4A25aA',
'public': '4XYfCbabAWVUCbjTmRTFEu2sc3dFEdkse4r6X498B1s8'
}
}
# Test user. inputs will be created for this user. Cryptography Keys
USER_PRIVATE_KEY = 'GmRZxQdQv7tooMijXytQkexKuFN6mJocciJarAmMwTX2'
USER_PUBLIC_KEY = 'r3cEu8GNoz8rYpNJ61k7GqfR8VEvdUbtyHce8u1kaYwh'
USER_PRIVATE_KEY = '8eJ8q9ZQpReWyQT5aFCiwtZ5wDZC4eDnCen88p3tQ6ie'
USER_PUBLIC_KEY = 'JEAkEJqLbbgDRAtMm8YAjGp759Aq2qTn9eaEHUj2XePE'
# We need this function to avoid loading an existing
# conf file located in the home of the user running
# the tests. If it's too aggressive we can change it
# later.
@pytest.fixture(scope='function', autouse=True)
def ignore_local_config_file(monkeypatch):
def mock_file_config(filename=None):
raise FileNotFoundError()
monkeypatch.setattr('bigchaindb.config_utils.file_config', mock_file_config)
@pytest.fixture
@pytest.fixture(scope='function', autouse=True)
def restore_config(request, node_config):
from bigchaindb import config_utils
config_utils.dict_config(node_config)

63
tests/db/test_bigchain_api.py
View File

@ -8,7 +8,7 @@ import rethinkdb as r
import bigchaindb
from bigchaindb import util
from bigchaindb import exceptions
from bigchaindb.crypto import PrivateKey, PublicKey, generate_key_pair, hash_data
from bigchaindb import crypto
from bigchaindb.voter import Voter
from bigchaindb.block import Block
@ -44,7 +44,7 @@ class TestBigchainApi(object):
'operation': 'd',
'timestamp': tx['transaction']['timestamp'],
'data': {
'hash': hash_data(util.serialize(payload)),
'hash': crypto.hash_data(util.serialize(payload)),
'payload': payload
}
}
@ -52,7 +52,7 @@ class TestBigchainApi(object):
# assert tx_hash == tx_calculated_hash
def test_transaction_signature(self, b):
sk, vk = generate_key_pair()
sk, vk = crypto.generate_key_pair()
tx = b.create_transaction(vk, 'b', 'c', 'd')
tx_signed = b.sign_transaction(tx, sk)
@ -108,7 +108,7 @@ class TestBigchainApi(object):
def test_assign_transaction_multiple_nodes(self, b, user_public_key, user_private_key):
# create 5 federation nodes
for _ in range(5):
b.federation_nodes.append(generate_key_pair()[1])
b.federation_nodes.append(crypto.generate_key_pair()[1])
# test assignee for several transactions
for _ in range(20):
@ -185,11 +185,11 @@ class TestBigchainApi(object):
def test_create_new_block(self, b):
new_block = b.create_block([])
block_hash = hash_data(util.serialize(new_block['block']))
block_hash = crypto.hash_data(util.serialize(new_block['block']))
assert new_block['block']['voters'] == [b.me]
assert new_block['block']['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(new_block['block']), new_block['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(new_block['block']), new_block['signature']) is True
assert new_block['id'] == block_hash
assert new_block['votes'] == []
@ -371,8 +371,8 @@ class TestBlockValidation(object):
}
block_data = util.serialize(block)
block_hash = hash_data(block_data)
block_signature = PrivateKey(b.me_private).sign(block_data)
block_hash = crypto.hash_data(block_data)
block_signature = crypto.SigningKey(b.me_private).sign(block_data)
block = {
'id': block_hash,
@ -408,45 +408,6 @@ class TestBlockValidation(object):
assert b.is_valid_block(block)
class TestBigchainCrypto(object):
PRIVATE_VALUE = 64328150571824492670917070117568709277186368319388887463636481841106388379832
PUBLIC_VALUE_X = 48388170575736684074633245566225141536152842355597159440179742847497614196929
PUBLIC_VALUE_Y = 65233479152484407841598798165960909560839872511163322973341535484598825150846
PRIVATE_VALUE_B58 = 'AaAp4xBavbe6VGeQF2mWdSKNM1r6HfR2Z1tAY6aUkwdq'
PUBLIC_VALUE_COMPRESSED_B58 = 'ifEi3UuTDT4CqUUKiS5omgeDodhu2aRFHVp6LoahbEVe'
def test_private_key_encode(self):
private_value_base58 = PrivateKey.encode(self.PRIVATE_VALUE)
assert private_value_base58 == self.PRIVATE_VALUE_B58
def test_private_key_decode(self):
private_value = PrivateKey.decode(self.PRIVATE_VALUE_B58)
assert private_value == self.PRIVATE_VALUE
def test_public_key_encode(self):
public_value_compressed_base58 = PublicKey.encode(self.PUBLIC_VALUE_X, self.PUBLIC_VALUE_Y)
assert public_value_compressed_base58 == self.PUBLIC_VALUE_COMPRESSED_B58
def test_public_key_decode(self):
public_value_x, public_value_y = PublicKey.decode(self.PUBLIC_VALUE_COMPRESSED_B58)
assert public_value_x == self.PUBLIC_VALUE_X
assert public_value_y == self.PUBLIC_VALUE_Y
def test_sign_verify(self):
message = 'Hello World!'
public_key = PublicKey(self.PUBLIC_VALUE_COMPRESSED_B58)
private_key = PrivateKey(self.PRIVATE_VALUE_B58)
assert public_key.verify(message, private_key.sign(message)) is True
def test_generate_key_pair(self):
private_value_base58, public_value_compressed_base58 = generate_key_pair()
assert PrivateKey.encode(
PrivateKey.decode(private_value_base58)) == private_value_base58
assert PublicKey.encode(
*PublicKey.decode(public_value_compressed_base58)) == public_value_compressed_base58
class TestBigchainVoter(object):
def test_valid_block_voting(self, b):
@ -483,7 +444,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is True
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_invalid_block_voting(self, b, user_public_key):
# create queue and voter
@ -524,7 +485,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is False
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_vote_creation_valid(self, b):
# create valid block
@ -538,7 +499,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is True
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_vote_creation_invalid(self, b):
# create valid block
@ -552,7 +513,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is False
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
class TestBigchainBlock(object):

22
tests/db/test_voter.py
View File

@ -6,7 +6,7 @@ import multiprocessing as mp
from bigchaindb import util
from bigchaindb.voter import Voter, BlockStream
from bigchaindb.crypto import PublicKey, generate_key_pair
from bigchaindb import crypto
class TestBigchainVoter(object):
@ -45,7 +45,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is True
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_valid_block_voting_with_create_transaction(self, b):
q_new_block = mp.Queue()
@ -53,7 +53,7 @@ class TestBigchainVoter(object):
genesis = b.create_genesis_block()
# create a `CREATE` transaction
test_user_priv, test_user_pub = generate_key_pair()
test_user_priv, test_user_pub = crypto.generate_key_pair()
tx = b.create_transaction(b.me, test_user_pub, None, 'CREATE')
tx_signed = b.sign_transaction(tx, b.me_private)
assert b.is_valid_transaction(tx_signed)
@ -87,7 +87,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is True
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_valid_block_voting_with_transfer_transactions(self, b):
q_new_block = mp.Queue()
@ -95,7 +95,7 @@ class TestBigchainVoter(object):
b.create_genesis_block()
# create a `CREATE` transaction
test_user_priv, test_user_pub = generate_key_pair()
test_user_priv, test_user_pub = crypto.generate_key_pair()
tx = b.create_transaction(b.me, test_user_pub, None, 'CREATE')
tx_signed = b.sign_transaction(tx, b.me_private)
assert b.is_valid_transaction(tx_signed)
@ -124,7 +124,7 @@ class TestBigchainVoter(object):
assert len(blocks[1]['votes']) == 1
# create a `TRANSFER` transaction
test_user2_priv, test_user2_pub = generate_key_pair()
test_user2_priv, test_user2_pub = crypto.generate_key_pair()
tx2 = b.create_transaction(test_user_pub, test_user2_pub, tx['id'], 'TRANSFER')
tx2_signed = b.sign_transaction(tx2, test_user_priv)
assert b.is_valid_transaction(tx2_signed)
@ -158,7 +158,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is True
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_invalid_block_voting(self, b, user_public_key):
# create queue and voter
@ -197,7 +197,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is False
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_vote_creation_valid(self, b):
# create valid block
@ -211,7 +211,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is True
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_vote_creation_invalid(self, b):
# create valid block
@ -225,7 +225,7 @@ class TestBigchainVoter(object):
assert vote['vote']['is_block_valid'] is False
assert vote['vote']['invalid_reason'] is None
assert vote['node_pubkey'] == b.me
assert PublicKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
assert crypto.VerifyingKey(b.me).verify(util.serialize(vote['vote']), vote['signature']) is True
def test_voter_considers_unvoted_blocks_when_single_node(self, b):
# simulate a voter going donw in a single node environment
@ -301,7 +301,7 @@ class TestBlockStream(object):
def test_if_federation_size_is_greater_than_one_ignore_past_blocks(self, b):
for _ in range(5):
b.federation_nodes.append(generate_key_pair()[1])
b.federation_nodes.append(crypto.generate_key_pair()[1])
new_blocks = mp.Queue()
bs = BlockStream(new_blocks)
block_1 = b.create_block([])

27
tests/test_commands.py
View File

@ -1,3 +1,4 @@
import json
from argparse import Namespace
from pprint import pprint
import copy
@ -11,12 +12,6 @@ def mock_run_configure(monkeypatch):
monkeypatch.setattr(bigchain, 'run_configure', lambda *args, **kwargs: None)
@pytest.fixture
def mock_file_config(monkeypatch):
from bigchaindb import config_utils
monkeypatch.setattr(config_utils, 'file_config', lambda *args: None)
@pytest.fixture
def mock_write_config(monkeypatch):
from bigchaindb import config_utils
@ -65,13 +60,13 @@ def mock_bigchaindb_backup_config(monkeypatch):
monkeypatch.setattr('bigchaindb._config', config)
def test_bigchain_run_start(mock_run_configure, mock_file_config,
mock_processes_start, mock_db_init_with_existing_db):
def test_bigchain_run_start(mock_run_configure, mock_processes_start, mock_db_init_with_existing_db):
from bigchaindb.commands.bigchain import run_start
args = Namespace(config=None, yes=True)
run_start(args)
@pytest.mark.skipif(reason="BigchainDB doesn't support the automatic creation of a config file anymore")
def test_bigchain_run_start_assume_yes_create_default_config(monkeypatch, mock_processes_start,
mock_generate_key_pair, mock_db_init_with_existing_db):
import bigchaindb
@ -99,25 +94,27 @@ def test_bigchain_run_start_assume_yes_create_default_config(monkeypatch, mock_p
# TODO Please beware, that if debugging, the "-s" switch for pytest will
# interfere with capsys.
# See related issue: https://github.com/pytest-dev/pytest/issues/128
def test_bigchain_show_config(capsys, mock_file_config):
@pytest.mark.usefixtures('restore_config')
def test_bigchain_show_config(capsys):
from bigchaindb import config
from bigchaindb.commands.bigchain import run_show_config
args = Namespace(config=None)
_, _ = capsys.readouterr()
run_show_config(args)
output_config, _ = capsys.readouterr()
pprint(config)
expected_outout_config, _ = capsys.readouterr()
assert output_config == expected_outout_config
output_config = json.loads(capsys.readouterr()[0])
del config['CONFIGURED']
config['keypair']['private'] = 'x' * 45
assert output_config == config
def test_bigchain_run_init_when_db_exists(mock_file_config, mock_db_init_with_existing_db):
def test_bigchain_run_init_when_db_exists(mock_db_init_with_existing_db):
from bigchaindb.commands.bigchain import run_init
args = Namespace(config=None)
run_init(args)
def test_drop_existing_db(mock_file_config, mock_rethink_db_drop):
def test_drop_existing_db(mock_rethink_db_drop):
from bigchaindb.commands.bigchain import run_drop
args = Namespace(config=None, yes=True)
run_drop(args)

105
tests/utils/test_config_utils.py
View File

@ -60,9 +60,108 @@ def test_load_consensus_plugin_raises_with_invalid_subclass(monkeypatch):
from bigchaindb import config_utils
monkeypatch.setattr(config_utils,
'iter_entry_points',
lambda *args: [ type('entry_point',
(object),
{'load': lambda: object}) ])
lambda *args: [type('entry_point', (object), {'load': lambda: object})])
with pytest.raises(TypeError):
config_utils.load_consensus_plugin()
def test_map_leafs_iterator():
from bigchaindb import config_utils
mapping = {
'a': {'b': {'c': 1},
'd': {'z': 44}},
'b': {'d': 2},
'c': 3
}
result = config_utils.map_leafs(lambda x, path: x * 2, mapping)
assert result == {
'a': {'b': {'c': 2},
'd': {'z': 88}},
'b': {'d': 4},
'c': 6
}
result = config_utils.map_leafs(lambda x, path: path, mapping)
assert result == {
'a': {'b': {'c': ['a', 'b', 'c']},
'd': {'z': ['a', 'd', 'z']}},
'b': {'d': ['b', 'd']},
'c': ['c']
}
def test_update_types():
from bigchaindb import config_utils
raw = {
'a_string': 'test',
'an_int': '42',
'a_float': '3.14',
'a_list': 'a:b:c',
}
reference = {
'a_string': 'test',
'an_int': 42,
'a_float': 3.14,
'a_list': ['a', 'b', 'c'],
}
result = config_utils.update_types(raw, reference)
assert result == reference
def test_env_config(monkeypatch):
monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_HOST': 'test-host',
'BIGCHAINDB_DATABASE_PORT': 'test-port'})
from bigchaindb import config_utils
result = config_utils.env_config({'database': {'host': None, 'port': None}})
expected = {'database': {'host': 'test-host', 'port': 'test-port'}}
assert result == expected
def test_autoconfigure_read_both_from_file_and_env(monkeypatch):
file_config = {
'database': {'host': 'test-host'}
}
monkeypatch.setattr('bigchaindb.config_utils.file_config', lambda *args, **kwargs: file_config)
monkeypatch.setattr('os.environ', {'BIGCHAINDB_DATABASE_NAME': 'test-dbname',
'BIGCHAINDB_DATABASE_PORT': '4242',
'BIGCHAINDB_KEYRING': 'pubkey_0:pubkey_1:pubkey_2'})
import bigchaindb
from bigchaindb import config_utils
config_utils.autoconfigure()
assert bigchaindb.config == {
'CONFIGURED': True,
'server': {
'bind': '0.0.0.0:9984',
'workers': None,
'threads': None,
},
'database': {
'host': 'test-host',
'port': 4242,
'name': 'test-dbname',
},
'keypair': {
'public': None,
'private': None,
},
'keyring': ['pubkey_0', 'pubkey_1', 'pubkey_2'],
'statsd': {
'host': 'localhost',
'port': 8125,
'rate': 0.01,
},
'api_endpoint': 'http://localhost:9984/api/v1',
'consensus_plugin': 'default',
}