mirror of
https://github.com/bigchaindb/bigchaindb.git
synced 2024-10-13 13:34:05 +00:00
Merge remote-tracking branch 'remotes/bigchaindb/develop' into develop
Conflicts: README.md bigchaindb/core.py
This commit is contained in:
diminator
commit
482cd634f2
@ -7,7 +7,7 @@ contribute to the project.
|
||||
We are committed to making participation in this project a harassment-free
|
||||
experience for everyone, regardless of level of experience, gender, gender
|
||||
identity and expression, sexual orientation, disability, personal appearance,
|
||||
body size, race, ethnicity, age, religion, or nationality.
|
||||
body size, race, ethnicity, age, religion, nationality, or species--no picking on Wrigley for being a buffalo!
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
|
||||
@ -18,8 +18,8 @@ To contribute code or documentation, you need a [GitHub account](https://github.
|
||||
|
||||
Familiarize yourself with how we do coding and documentation in the BigchainDB project, including:
|
||||
|
||||
* our Python Style Guide (coming soon)
|
||||
* [our documentation strategy](./docs/README.md) (including code documentation)
|
||||
* [our Python Style Guide](PYTHON_STYLE_GUIDE.md) (includes how we write tests)
|
||||
* [our documentation strategy](./docs/README.md) (including in-code documentation)
|
||||
* the GitHub Flow (workflow)
|
||||
* [GitHub Guide: Understanding the GitHub Flow](https://guides.github.com/introduction/flow/)
|
||||
* [Scott Chacon's blog post about GitHub Flow](http://scottchacon.com/2011/08/31/github-flow.html)
|
||||
@ -63,27 +63,29 @@ git checkout -b new-branch-name
|
||||
|
||||
### Step 5 - Make Edits, git add, git commit
|
||||
|
||||
With your new branch checked out locally, make changes or additions to the code or documentation, git add them, and git commit them.
|
||||
With your new branch checked out locally, make changes or additions to the code or documentation. Remember to:
|
||||
|
||||
* follow [our Python Style Guide](PYTHON_STYLE_GUIDE.md).
|
||||
* write and run tests for any new or changed code. There's a section in [our Python Style Guide](PYTHON_STYLE_GUIDE.md) about writing and running tests.
|
||||
* add or update documentation as necessary. Follow [our documentation strategy](./docs/README.md).
|
||||
|
||||
As you go, git add and git commit your changes or additions, e.g.
|
||||
```text
|
||||
git add new-or-changed-file
|
||||
git add new-or-changed-file-1
|
||||
git add new-or-changed-file-2
|
||||
git commit -m "Short description of new or changed things"
|
||||
```
|
||||
|
||||
Remember to write tests for new code. If you don't, our code (test) coverage will go down, and we won't be able to accept your code. (We have some hard checks that run on all new pull requests and code coverage is one of them.)
|
||||
|
||||
Please run all existing tests to make sure you didn't break something. Do:
|
||||
```text
|
||||
py.test -v
|
||||
```
|
||||
|
||||
Remember to write or modify documentation to reflect your additions or changes.
|
||||
|
||||
You will want to merge changes from upstream (i.e. the original repository) into your new branch from time to time, using something like:
|
||||
```text
|
||||
git fetch upstream
|
||||
git merge upstream/develop
|
||||
```
|
||||
|
||||
Once you're done commiting a set of new things and you're ready to submit them for inclusion, please be sure to run all the tests (as per the instructions at the end of our [Python Style Guide](PYTHON_STYLE_GUIDE.md)).
|
||||
|
||||
(When you submit your pull request [following the instructions below], we run all the tests automatically, so we will see if some are failing. If you don't know why some tests are failing, you can still submit your pull request, but be sure to note the failing tests and to ask for help with resolving them.)
|
||||
|
||||
### Step 6 - Push Your New Branch to origin
|
||||
|
||||
Make sure you've commited all the additions or changes you want to include in your pull request. Then push your new branch to origin (i.e. _your_ remote bigchaindb repository).
|
||||
|
||||
102
PYTHON_STYLE_GUIDE.md
Normal file
102
PYTHON_STYLE_GUIDE.md
Normal file
@ -0,0 +1,102 @@
|
||||
# Python Style Guide
|
||||
|
||||
This guide starts out with our general Python coding style guidelines and ends with a section on how we write & run (Python) tests.
|
||||
|
||||
## General Python Coding Style Guidelines
|
||||
|
||||
Our starting point is [PEP8](https://www.python.org/dev/peps/pep-0008/), the standard "Style Guide for Python Code." Many Python IDEs will check your code against PEP8. (Note that PEP8 isn't frozen; it actually changes over time, but slowly.)
|
||||
|
||||
BigchainDB uses Python 3.4+, so you can ignore all PEP8 guidelines specific to Python 2.
|
||||
|
||||
### Python Docstrings
|
||||
|
||||
PEP8 says some things about docstrings, but not what to put in them or how to structure them. [PEP257](https://www.python.org/dev/peps/pep-0257/) was one proposal for docstring conventions, but we prefer [Google-style docstrings](https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments) instead: they're easier to read and the [napoleon extension](http://www.sphinx-doc.org/en/stable/ext/napoleon.html) for Sphinx lets us turn them into nice-looking documentation. Here are some references on Google-style docstrings:
|
||||
|
||||
* [Google's docs on Google-style docstrings](https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments)
|
||||
* [napoleon's docs include an overview of Google-style docstrings](http://sphinxcontrib-napoleon.readthedocs.org/en/latest/index.html)
|
||||
* [Example Google-style docstrings](http://sphinxcontrib-napoleon.readthedocs.org/en/latest/example_google.html) (from napoleon's docs)
|
||||
|
||||
### Maximum Line Length
|
||||
|
||||
PEP8 has some [maximum line length guidelines](https://www.python.org/dev/peps/pep-0008/#id17), starting with "Limit all lines to a maximum of 79 characters" but "for flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters."
|
||||
|
||||
We discussed this at length, and it seems that the consensus is: _try_ to keep line lengths less than 79/72 characters, unless you have a special situation where longer lines would improve readability. (The basic reason is that 79/72 works for everyone, and BigchainDB is an open source project.) As a hard limit, keep all lines less than 119 characters (which is the width of GitHub code review).
|
||||
|
||||
### Single or Double Quotes?
|
||||
|
||||
Python lets you use single or double quotes. PEP8 says you can use either, as long as you're consistent. We try to stick to using single quotes, except in cases where using double quotes is more readable. For example:
|
||||
```python
|
||||
print('This doesn\'t look so nice.')
|
||||
print("Doesn't this look nicer?")
|
||||
```
|
||||
|
||||
### Breaking Strings Across Multiple Lines
|
||||
|
||||
Should we use parentheses or slashes (`\`) to break strings across multiple lines, i.e.
|
||||
```python
|
||||
my_string = ('This is a very long string, so long that it will not fit into just one line '
|
||||
'so it must be split across multiple lines.')
|
||||
# or
|
||||
my_string = 'This is a very long string, so long that it will not fit into just one line ' \
|
||||
'so it must be split across multiple lines.'
|
||||
```
|
||||
|
||||
It seems the preference is for slashes, but using parentheses is okay too. (There are good arguments either way. Arguing about it seems like a waste of time.)
|
||||
|
||||
### Using the % operator or `format()` to Format Strings
|
||||
|
||||
Given the choice:
|
||||
```python
|
||||
x = 'name: %s; score: %d' % (name, n)
|
||||
# or
|
||||
x = 'name: {}; score: {}'.format(name, n)
|
||||
```
|
||||
|
||||
we use the `format()` version. The [official Python documentation says](https://docs.python.org/2/library/stdtypes.html#str.format), "This method of string formatting is the new standard in Python 3, and should be preferred to the % formatting described in String Formatting Operations in new code."
|
||||
|
||||
|
||||
## Writing (Python) Tests
|
||||
|
||||
We write tests for our Python code using the [pytest](http://pytest.org/latest/) framework.
|
||||
|
||||
All tests go in the `bigchaindb/tests` directory or one of its subdirectories. You can use the tests already in there as templates or examples.
|
||||
|
||||
### Standard Ways to Run All Tests
|
||||
|
||||
To run all the tests, first make sure you have RethinkDB running:
|
||||
```text
|
||||
$ rethinkdb
|
||||
```
|
||||
|
||||
then in another terminal, do:
|
||||
```text
|
||||
$ py.test -v
|
||||
```
|
||||
|
||||
If that doesn't work (e.g. maybe you are running in a conda virtual environment), try:
|
||||
```text
|
||||
$ python -m pytest -v
|
||||
```
|
||||
|
||||
You can also run all tests via `setup.py`, using:
|
||||
```text
|
||||
$ python setup.py test
|
||||
```
|
||||
|
||||
### Using `docker-compose` to Run the Tests
|
||||
|
||||
You can use `docker-compose` to run the tests. (You don't have to start RethinkDB first: `docker-compose` does that on its own, when it reads the `docker-compose.yml` file.)
|
||||
|
||||
First, build the images (~once), using:
|
||||
```text
|
||||
$ docker-compose build
|
||||
```
|
||||
|
||||
then run the tests using:
|
||||
```text
|
||||
$ docker-compose run --rm bigchaindb py.test -v
|
||||
```
|
||||
|
||||
### Automated Testing of All Pull Requests
|
||||
|
||||
We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.)
|
||||
94
README.md
94
README.md
@ -1,11 +1,14 @@
|
||||
# BigchainDB - Interledger fork
|
||||
|
||||
A scalable blockchain database. [The whitepaper](https://www.bigchaindb.com/whitepaper/) explains what that means.
|
||||
|
||||
[](https://gitter.im/bigchaindb/bigchaindb?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
[](https://pypi.python.org/pypi/BigchainDB)
|
||||
[](https://travis-ci.org/bigchaindb/bigchaindb)
|
||||
[](https://codecov.io/github/bigchaindb/bigchaindb?branch=develop)
|
||||
[](http://bigchaindb.readthedocs.org/en/develop/?badge=develop)
|
||||
|
||||
|
||||
This fork provides basic functionality for supporting the interledger protocol, see http://interledger.org/
|
||||
|
||||
The edits are found under interledger/core.py and interledger/tests/test_connector.py
|
||||
@ -27,76 +30,29 @@ $3> python3 interledger/tests/test_connector.py
|
||||
- [ ] multi-(big)chain(db) instantiation + network path optimization
|
||||
|
||||
|
||||
## Documentation
|
||||
## <a name="gettingstarted"></a>Quick Start
|
||||
|
||||
Documentation is available at https://bigchaindb.readthedocs.org/
|
||||
### [Install & Run BigchainDB](http://bigchaindb.readthedocs.org/en/develop/installing.html)
|
||||
### [Run BigchainDB with Docker](http://bigchaindb.readthedocs.org/en/develop/installing.html#run-bigchaindb-with-docker)
|
||||
### [Getting Started (Tutorial)](http://bigchaindb.readthedocs.org/en/develop/getting-started.html)
|
||||
|
||||
## <a name="gettingstarted"></a>Getting started
|
||||
## Links for Everyone
|
||||
* [BigchainDB.com](https://www.bigchaindb.com/) - the main BigchainDB website, including newsletter signup
|
||||
* [Whitepaper](https://www.bigchaindb.com/whitepaper/) - outlines the motivations, goals and core algorithms of BigchainDB
|
||||
* [Roadmap](ROADMAP.md)
|
||||
* [Blog](https://medium.com/the-bigchaindb-blog)
|
||||
* [Twitter](https://twitter.com/BigchainDB)
|
||||
* [Google Group](https://groups.google.com/forum/#!forum/bigchaindb)
|
||||
|
||||
### Install RethinkDB
|
||||
## Links for Developers
|
||||
* [Documentation](http://bigchaindb.readthedocs.org/en/develop/#) - for developers
|
||||
* [CONTRIBUTING.md](CONTRIBUTING.md) - how to contribute
|
||||
* [Community guidelines](CODE_OF_CONDUCT.md)
|
||||
* [Open issues](https://github.com/bigchaindb/bigchaindb/issues)
|
||||
* [Open pull requests](https://github.com/bigchaindb/bigchaindb/pulls)
|
||||
* [Gitter chatroom](https://gitter.im/bigchaindb/bigchaindb)
|
||||
|
||||
#### On Ubuntu
|
||||
```sh
|
||||
# install rethinkdb https://rethinkdb.com/docs/install/ubuntu/
|
||||
$ source /etc/lsb-release && echo "deb http://download.rethinkdb.com/apt $DISTRIB_CODENAME main" | sudo tee /etc/apt/sources.list.d/rethinkdb.list
|
||||
$ wget -qO- http://download.rethinkdb.com/apt/pubkey.gpg | sudo apt-key add -
|
||||
$ sudo apt-get update
|
||||
$ sudo apt-get install rethinkdb
|
||||
|
||||
# start rethinkdb
|
||||
$ rethinkdb
|
||||
```
|
||||
|
||||
#### On other platforms
|
||||
To install RethinkDB on other platform, please refer to [the official documentation](https://rethinkdb.com/docs/install/).
|
||||
|
||||
### Install BigchainDB
|
||||
```sh
|
||||
$ pip install bigchaindb
|
||||
```
|
||||
|
||||
### Running BigchainDB
|
||||
Currently BigchainDB only supports Python 3.4+
|
||||
|
||||
|
||||
Start the main process. If it's the first time `bigchaindb` will generate a default
|
||||
configuration file for you.
|
||||
```sh
|
||||
$ bigchaindb start
|
||||
```
|
||||
|
||||
Generate some tests transactions:
|
||||
|
||||
```sh
|
||||
$ bigchaindb-benchmark load # add '-m' if you want to use all your cores
|
||||
```
|
||||
|
||||
To know more about the bigchain command run
|
||||
```sh
|
||||
$ bigchaindb -h
|
||||
```
|
||||
|
||||
#### Importing `BigchainDB` from the interpreter (python/ipython)
|
||||
Make sure your `rethinkdb` process is running.
|
||||
|
||||
```python
|
||||
>>> from bigchaindb import Bigchain
|
||||
>>> b = Bigchain()
|
||||
>>> b.me
|
||||
'2B8C8PJxhycFzn4wncRhBNmMWwE5Frr9nLBUa1dGGxj5W'
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
BigchainDB creates a default configuration file on `$HOME/.bigchaindb` on the
|
||||
first run.
|
||||
|
||||
```sh
|
||||
$ bigchaindb show-config
|
||||
```
|
||||
|
||||
#### Testing
|
||||
|
||||
```
|
||||
$ py.test -v
|
||||
```
|
||||
## Legal
|
||||
* [Licenses](LICENSES.md) - open source & open content
|
||||
* [Imprint](https://www.bigchaindb.com/imprint/)
|
||||
* [Contact Us](https://www.bigchaindb.com/contact/)
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
import os
|
||||
import copy
|
||||
|
||||
from bigchaindb.core import Bigchain # noqa
|
||||
|
||||
|
||||
def e(key, default=None, conv=None):
|
||||
@ -35,10 +34,17 @@ config = {
|
||||
'private': e('BIGCHAIN_KEYPAIR_PRIVATE')
|
||||
},
|
||||
'keyring': [
|
||||
]
|
||||
],
|
||||
'statsd': {
|
||||
'host': e('BIGCHAIN_STATSD_HOST', default='localhost'),
|
||||
'port': e('BIGCHAIN_STATSD_PORT', default=8125),
|
||||
'rate': e('BIGCHAIN_STATSD_SAMPLERATE', default=0.01)
|
||||
}
|
||||
}
|
||||
|
||||
# We need to maintain a backup copy of the original config dict in case
|
||||
# the user wants to reconfigure the node. Check ``bigchaindb.config_utils``
|
||||
# for more info.
|
||||
_config = copy.deepcopy(config)
|
||||
from bigchaindb.core import Bigchain # noqa
|
||||
|
||||
|
||||
@ -4,11 +4,16 @@ import queue
|
||||
|
||||
import rethinkdb as r
|
||||
|
||||
import bigchaindb
|
||||
from bigchaindb import Bigchain
|
||||
from bigchaindb.monitor import Monitor
|
||||
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
monitor = Monitor()
|
||||
|
||||
|
||||
class Block(object):
|
||||
|
||||
@ -52,6 +57,7 @@ class Block(object):
|
||||
b = Bigchain()
|
||||
|
||||
while True:
|
||||
monitor.gauge('tx_queue_gauge', self.q_tx_to_validate.qsize(), rate=bigchaindb.config['statsd']['rate'])
|
||||
tx = self.q_tx_to_validate.get()
|
||||
|
||||
# poison pill
|
||||
|
||||
@ -45,7 +45,7 @@ def run_configure(args, skip_if_exists=False):
|
||||
if config_file_exists and not args.yes:
|
||||
want = input('Config file `{}` exists, do you want to override it? '
|
||||
'(cannot be undone) [y/n]: '.format(config_path))
|
||||
if not want:
|
||||
if want != 'y':
|
||||
return
|
||||
|
||||
# Patch the default configuration with the new values
|
||||
@ -59,6 +59,10 @@ def run_configure(args, skip_if_exists=False):
|
||||
val = conf['database'][key]
|
||||
conf['database'][key] = input('Database {}? (default `{}`): '.format(key, val)) or val
|
||||
|
||||
for key in ('host', 'port', 'rate'):
|
||||
val = conf['statsd'][key]
|
||||
conf['statsd'][key] = input('Statsd {}? (default `{}`): '.format(key, val)) or val
|
||||
|
||||
bigchaindb.config_utils.write_config(conf, config_path)
|
||||
print('Ready to go!')
|
||||
|
||||
|
||||
@ -10,6 +10,9 @@ import bigchaindb
|
||||
from bigchaindb import config_utils
|
||||
from bigchaindb import exceptions
|
||||
from bigchaindb.crypto import hash_data, PublicKey, PrivateKey, generate_key_pair
|
||||
from bigchaindb.monitor import Monitor
|
||||
|
||||
monitor = Monitor()
|
||||
|
||||
|
||||
class GenesisBlockAlreadyExistsError(Exception):
|
||||
@ -68,6 +71,7 @@ class Bigchain(object):
|
||||
def reconnect(self):
|
||||
return r.connect(host=self.host, port=self.port, db=self.dbname)
|
||||
|
||||
@monitor.timer('create_transaction', rate=bigchaindb.config['statsd']['rate'])
|
||||
def create_transaction(self, current_owners, new_owners, tx_input, operation, payload=None):
|
||||
"""Create a new transaction
|
||||
|
||||
@ -144,6 +148,7 @@ class Bigchain(object):
|
||||
transaction (dict): transaction to sign.
|
||||
private_key (str): base58 encoded private key to create a signature of the transaction.
|
||||
public_key (str): (optional) base58 encoded public key to identify each signature of a multisig transaction.
|
||||
|
||||
Returns:
|
||||
dict: transaction with the `signature` field included.
|
||||
|
||||
@ -203,6 +208,7 @@ class Bigchain(object):
|
||||
return False
|
||||
return True
|
||||
|
||||
@monitor.timer('write_transaction', rate=bigchaindb.config['statsd']['rate'])
|
||||
def write_transaction(self, signed_transaction):
|
||||
"""Write the transaction to bigchain.
|
||||
|
||||
@ -298,7 +304,6 @@ class Bigchain(object):
|
||||
The transaction that used the `txid` as an input if it exists else it returns `None`
|
||||
|
||||
"""
|
||||
|
||||
# checks if an input was already spent
|
||||
# checks if the bigchain has any transaction with input `transaction_id`
|
||||
response = r.table('bigchain').concat_map(lambda doc: doc['block']['transactions'])\
|
||||
@ -340,6 +345,7 @@ class Bigchain(object):
|
||||
|
||||
return owned
|
||||
|
||||
@monitor.timer('validate_transaction', rate=bigchaindb.config['statsd']['rate'])
|
||||
def validate_transaction(self, transaction):
|
||||
"""Validate a transaction.
|
||||
|
||||
@ -380,11 +386,12 @@ class Bigchain(object):
|
||||
raise exceptions.TransactionOwnerError('current_owner `{}` does not own the input `{}`'.format(
|
||||
transaction['transaction']['current_owners'], transaction['transaction']['input']))
|
||||
|
||||
# check if the input was already spent
|
||||
# check if the input was already spent by a transaction other then this one.
|
||||
spent = self.get_spent(tx_input['id'])
|
||||
if spent:
|
||||
raise exceptions.DoubleSpend('input `{}` was already spent'.format(
|
||||
transaction['transaction']['input']))
|
||||
if spent['id'] != transaction['id']:
|
||||
raise exceptions.DoubleSpend('input `{}` was already spent'.format(
|
||||
transaction['transaction']['input']))
|
||||
|
||||
# Check hash of the transaction
|
||||
calculated_hash = hash_data(self.serialize(transaction['transaction']))
|
||||
@ -452,6 +459,7 @@ class Bigchain(object):
|
||||
|
||||
return block
|
||||
|
||||
@monitor.timer('validate_block')
|
||||
# TODO: check that the votings structure is correctly constructed
|
||||
def validate_block(self, block):
|
||||
"""Validate a block.
|
||||
@ -496,6 +504,7 @@ class Bigchain(object):
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
@monitor.timer('write_block')
|
||||
def write_block(self, block, durability='soft'):
|
||||
"""Write a block to bigchain.
|
||||
|
||||
@ -528,7 +537,7 @@ class Bigchain(object):
|
||||
if blocks_count:
|
||||
raise GenesisBlockAlreadyExistsError('Cannot create the Genesis block')
|
||||
|
||||
payload = {'message': 'Hello World from the Bigchain'}
|
||||
payload = {'message': 'Hello World from the BigchainDB'}
|
||||
transaction = self.create_transaction(self.me, self.me, None, 'GENESIS', payload=payload)
|
||||
transaction_signed = self.sign_transaction(transaction, self.me_private)
|
||||
|
||||
|
||||
30
bigchaindb/monitor.py
Normal file
30
bigchaindb/monitor.py
Normal file
@ -0,0 +1,30 @@
|
||||
import statsd
|
||||
from platform import node
|
||||
|
||||
import bigchaindb
|
||||
from bigchaindb import config_utils
|
||||
|
||||
class Monitor(statsd.StatsClient):
|
||||
"""Set up statsd monitoring
|
||||
|
||||
"""
|
||||
def __init__(self, *args, **kwargs):
|
||||
"""Overrides statsd client, fixing prefix to messages and loading configuration
|
||||
|
||||
Args:
|
||||
*args: arguments (identical to Statsclient)
|
||||
**kwargs: keyword arguments (identical to Statsclient)
|
||||
"""
|
||||
config_utils.autoconfigure()
|
||||
|
||||
if not kwargs:
|
||||
kwargs = {}
|
||||
|
||||
# set prefix, parameters from configuration file
|
||||
if 'prefix' not in kwargs:
|
||||
kwargs['prefix'] = '{hostname}.'.format(hostname=node())
|
||||
if 'host' not in kwargs:
|
||||
kwargs['host'] = bigchaindb.config['statsd']['host']
|
||||
if 'port' not in kwargs:
|
||||
kwargs['port'] = bigchaindb.config['statsd']['port']
|
||||
super().__init__(*args, **kwargs)
|
||||
25
docker-compose-monitor.yml
Normal file
25
docker-compose-monitor.yml
Normal file
@ -0,0 +1,25 @@
|
||||
influxdb:
|
||||
image: tutum/influxdb
|
||||
ports:
|
||||
- "8083:8083"
|
||||
- "8086:8086"
|
||||
expose:
|
||||
- "8090"
|
||||
- "8099"
|
||||
environment:
|
||||
PRE_CREATE_DB: "telegraf"
|
||||
|
||||
grafana:
|
||||
image: bigchaindb/grafana-bigchaindb-docker
|
||||
tty: true
|
||||
ports:
|
||||
- "3000:3000"
|
||||
links:
|
||||
- influxdb:localhost
|
||||
|
||||
statsd:
|
||||
image: bigchaindb/docker-telegraf-statsd
|
||||
ports:
|
||||
- "8125:8125/udp"
|
||||
links:
|
||||
- influxdb:localhost
|
||||
@ -1,3 +1,4 @@
|
||||
Sphinx==1.3.5
|
||||
sphinxcontrib-napoleon==0.4.4
|
||||
sphinx-rtd-theme>=0.1.9
|
||||
recommonmark
|
||||
|
||||
BIN
docs/source/_static/monitoring_system_diagram.png
Normal file
BIN
docs/source/_static/monitoring_system_diagram.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
@ -28,6 +28,9 @@ from recommonmark.parser import CommonMarkParser
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
import sphinx_rtd_theme
|
||||
|
||||
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
'sphinx.ext.intersphinx',
|
||||
@ -119,7 +122,8 @@ todo_include_todos = False
|
||||
|
||||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||
# a list of builtin themes.
|
||||
html_theme = 'alabaster'
|
||||
html_theme = 'sphinx_rtd_theme'
|
||||
|
||||
|
||||
# Theme options are theme-specific and customize the look and feel of a theme
|
||||
# further. For a list of options available for each theme, see the
|
||||
@ -127,7 +131,7 @@ html_theme = 'alabaster'
|
||||
#html_theme_options = {}
|
||||
|
||||
# Add any paths that contain custom themes here, relative to this directory.
|
||||
#html_theme_path = []
|
||||
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
|
||||
|
||||
# The name for this set of Sphinx documents. If None, it defaults to
|
||||
# "<project> v<release> documentation".
|
||||
|
||||
@ -21,6 +21,7 @@ Table of Contents
|
||||
models
|
||||
json-serialization
|
||||
developer-interface
|
||||
monitoring
|
||||
contributing
|
||||
faq
|
||||
release-notes
|
||||
|
||||
@ -2,6 +2,14 @@
|
||||
|
||||
We're developing BigchainDB Server ("BigchainDB") on Ubuntu 14.04, but it should work on any OS that runs RethinkDB Server and Python 3.4+. (BigchainDB is built on top of RethinkDB Server.)
|
||||
|
||||
BigchainDB Server is intended to be run on each server in a large distributed cluster of servers; it's not very useful running by itself on a single computer. That's _why_ we're developing it on Ubuntu 14.04: it's one of the more common server operating systems.
|
||||
|
||||
Mac OS X users may get the best results [running BigchainDB Server with Docker](https://bigchaindb.readthedocs.org/en/develop/installing.html#run-bigchaindb-with-docker).
|
||||
|
||||
Windows users may get the best results [running BigchainDB Server in a VM with Vagrant](https://bigchaindb.readthedocs.org/en/develop/installing.html#how-to-install-bigchaindb-on-a-vm-with-vagrant).
|
||||
|
||||
(Right now, there are no BigchainDB clients/drivers. Those will be able to run on a much larger array of operating systems. They're coming soon.)
|
||||
|
||||
## Install and Run RethinkDB Server
|
||||
|
||||
The RethinkDB documentation has instructions for how to install RethinkDB Server on a variety of operating systems. Do that (using their instructions for your OS): [Install RethinkDB Server](http://rethinkdb.com/docs/install/).
|
||||
@ -19,12 +27,20 @@ If you don't already have it, then you should [install Python 3.4+](https://www.
|
||||
|
||||
BigchainDB has some OS-level dependencies. In particular, you need to install the OS-level dependencies for the Python **cryptography** package. Instructions for installing those dependencies on your OS can be found in the [cryptography package documentation](https://cryptography.io/en/latest/installation/).
|
||||
|
||||
On Ubuntu 14.04, we found that the following was enough (YMMV):
|
||||
On Ubuntu 14.04, we found that the following was enough:
|
||||
```text
|
||||
$ sudo apt-get update
|
||||
$ sudo apt-get install libffi-dev g++ libssl-dev python3-dev
|
||||
```
|
||||
|
||||
On Fedora 23, we found that the following was enough (tested in February 2015):
|
||||
```text
|
||||
$ sudo dnf update
|
||||
$ sudo dnf install libffi-devel gcc-c++ redhat-rpm-config python3-devel openssl-devel
|
||||
```
|
||||
|
||||
(If you're using a version of Fedora before version 22, you may have to use `yum` instead of `dnf`.)
|
||||
|
||||
With OS-level dependencies installed, you can install BigchainDB with `pip` or from source.
|
||||
|
||||
### How to Install BigchainDB with `pip`
|
||||
@ -91,7 +107,7 @@ For those who like using Docker and wish to experiment with BigchainDB in
|
||||
non-production environments, we currently maintain a `dockerfile` that can be
|
||||
used to build an image for `bigchaindb`, along with a `docker-compose.yml` file
|
||||
to manage a "standalone node", consisting mainly of two containers: one for
|
||||
RethinkDB, and another for `bigchaindb`.
|
||||
RethinkDB, and another for BigchainDB.
|
||||
|
||||
Assuming you have `docker` and `docker-compose` installed, you would proceed as
|
||||
follows.
|
||||
@ -112,27 +128,21 @@ stored on your host machine under ` ~/.bigchaindb_docker/config`:
|
||||
$ docker-compose run --rm bigchaindb bigchaindb configure
|
||||
```
|
||||
|
||||
You can load test transactions via:
|
||||
You can then start it up (in the background, as a daemon) using:
|
||||
```text
|
||||
$ docker-compose up -d
|
||||
```
|
||||
|
||||
then you can load test transactions via:
|
||||
```text
|
||||
$ docker-compose run --rm bigchaindb bigchaindb-benchmark load
|
||||
```
|
||||
|
||||
You should then be able to start `bigchaindb`, via:
|
||||
```text
|
||||
$ docker-compose run --rm bigchaindb bigchaindb start
|
||||
```
|
||||
If you're on Linux, you can probably view the RethinkDB dashboard at:
|
||||
|
||||
or
|
||||
```text
|
||||
$ docker-compose up
|
||||
```
|
||||
[http://localhost:58080/](http://localhost:58080/)
|
||||
|
||||
You should be able to view the RethinkDB dashboard at:
|
||||
```text
|
||||
http://docker_host:58080/
|
||||
```
|
||||
|
||||
where `docker_host` is the IP or hostname of the machine running the Docker
|
||||
engine. If you are developing on Linux, this most likely will be `localhost`,
|
||||
whereas if you are running docker-machine (e.g.: on Mac OS X) this will be the
|
||||
If that doesn't work, then replace `localhost`
|
||||
with the IP or hostname of the machine running the Docker engine.
|
||||
If you are running docker-machine (e.g.: on Mac OS X) this will be the
|
||||
IP of the Docker machine (`docker-machine ip machine_name`).
|
||||
|
||||
32
docs/source/monitoring.md
Normal file
32
docs/source/monitoring.md
Normal file
@ -0,0 +1,32 @@
|
||||
# Monitoring
|
||||
|
||||
BigchainDB uses [StatsD](https://github.com/etsy/statsd) for monitoring. We require some additional infrastructure to take full advantage of its functionality:
|
||||
|
||||
* an agent to listen for metrics: [Telegraf](https://github.com/influxdata/telegraf),
|
||||
* a time-series database: [InfluxDB](https://influxdata.com/time-series-platform/influxdb/), and
|
||||
* a frontend to display analytics: [Grafana](http://grafana.org/).
|
||||
|
||||
We put each of those inside its own Docker container. The whole system is illustrated below.
|
||||
|
||||
|
||||
|
||||
For ease of use, we've created a Docker [_Compose file_](https://docs.docker.com/compose/compose-file/) (named `docker-compose-monitor.yml`) to define the monitoring system setup. To use it, just go to to the top `bigchaindb` directory and run:
|
||||
```text
|
||||
$ docker-compose -f docker-compose-monitor.yml build
|
||||
$ docker-compose -f docker-compose-monitor.yml up
|
||||
```
|
||||
|
||||
then point a browser tab to:
|
||||
|
||||
[http://localhost:3000/dashboard/script/bigchaindb_dashboard.js](http://localhost:3000/dashboard/script/bigchaindb_dashboard.js)
|
||||
|
||||
The login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](installing.html#run-bigchaindb) and load some test transactions:
|
||||
```text
|
||||
$ bigchaindb-benchmark load
|
||||
```
|
||||
|
||||
then refresh the page after a few seconds.
|
||||
|
||||
If you're not interested in monitoring, don't worry: BigchainDB will function just fine without any monitoring setup.
|
||||
|
||||
Feel free to modify the [custom Grafana dashboard](https://github.com/rhsimplex/grafana-bigchaindb-docker/blob/master/bigchaindb_dashboard.js) to your liking!
|
||||
1
setup.py
1
setup.py
@ -26,6 +26,7 @@ docs_require = [
|
||||
'recommonmark>=0.4.0',
|
||||
'Sphinx>=1.3.5',
|
||||
'sphinxcontrib-napoleon>=0.4.4',
|
||||
'sphinx-rtd-theme>=0.1.9',
|
||||
]
|
||||
|
||||
setup(
|
||||
|
||||
@ -45,6 +45,7 @@ def inputs(user_public_key):
|
||||
def test_remove_unclosed_sockets():
|
||||
pass
|
||||
|
||||
|
||||
class TestBigchainApi(object):
|
||||
|
||||
def test_create_transaction(self, b):
|
||||
@ -58,7 +59,6 @@ class TestBigchainApi(object):
|
||||
with pytest.raises(TypeError):
|
||||
b.create_transaction('a', 'b', 'c', 'd', payload=[])
|
||||
|
||||
|
||||
def test_transaction_hash(self, b):
|
||||
payload = {'cats': 'are awesome'}
|
||||
tx = b.create_transaction('a', 'b', 'c', 'd', payload)
|
||||
@ -166,8 +166,8 @@ class TestBigchainApi(object):
|
||||
b.create_genesis_block()
|
||||
|
||||
genesis_blocks = list(r.table('bigchain')
|
||||
.filter(r.row['block_number'] == 0)
|
||||
.run(b.conn))
|
||||
.filter(r.row['block_number'] == 0)
|
||||
.run(b.conn))
|
||||
|
||||
assert len(genesis_blocks) == 1
|
||||
|
||||
@ -349,6 +349,24 @@ class TestTransactionValidation(object):
|
||||
assert tx_valid_signed == b.validate_transaction(tx_valid_signed)
|
||||
assert tx_valid_signed == b.is_valid_transaction(tx_valid_signed)
|
||||
|
||||
@pytest.mark.usefixtures('inputs')
|
||||
def test_valid_non_create_transaction_after_block_creation(self, b, user_public_key, user_private_key):
|
||||
input_valid = b.get_owned_ids(user_public_key).pop()
|
||||
tx_valid = b.create_transaction(user_public_key, 'b', input_valid, 'd')
|
||||
|
||||
tx_valid_signed = b.sign_transaction(tx_valid, user_private_key)
|
||||
assert tx_valid_signed == b.validate_transaction(tx_valid_signed)
|
||||
assert tx_valid_signed == b.is_valid_transaction(tx_valid_signed)
|
||||
|
||||
# create block
|
||||
block = b.create_block([tx_valid_signed])
|
||||
assert b.is_valid_block(block)
|
||||
b.write_block(block, durability='hard')
|
||||
|
||||
# check that the transaction is still valid after being written to the bigchain
|
||||
assert tx_valid_signed == b.validate_transaction(tx_valid_signed)
|
||||
assert tx_valid_signed == b.is_valid_transaction(tx_valid_signed)
|
||||
|
||||
|
||||
class TestBlockValidation(object):
|
||||
|
||||
@ -357,7 +375,7 @@ class TestBlockValidation(object):
|
||||
|
||||
# change block hash
|
||||
block.update({'id': 'abc'})
|
||||
with pytest.raises(exceptions.InvalidHash) as excinfo:
|
||||
with pytest.raises(exceptions.InvalidHash):
|
||||
b.validate_block(block)
|
||||
|
||||
@pytest.mark.skipif(reason='Separated tx validation from block creation.')
|
||||
@ -368,7 +386,6 @@ class TestBlockValidation(object):
|
||||
tx_invalid = b.create_transaction('a', 'b', valid_input, 'c')
|
||||
|
||||
block = b.create_block([tx_invalid])
|
||||
assert invalid_transactions == [tx_invalid]
|
||||
|
||||
# create a block with invalid transactions
|
||||
block = {
|
||||
@ -520,7 +537,6 @@ class TestBigchainVoter(object):
|
||||
time.sleep(1)
|
||||
voter.kill()
|
||||
|
||||
|
||||
# retrive block from bigchain
|
||||
bigchain_block = r.table('bigchain').get(block['id']).run(b.conn)
|
||||
|
||||
@ -765,4 +781,3 @@ class TestBigchainBlock(object):
|
||||
|
||||
def test_duplicated_transactions(self):
|
||||
pytest.skip('We may have duplicates in the initial_results and changefeed')
|
||||
|
||||
|
||||
@ -3,7 +3,6 @@ import time
|
||||
import rethinkdb as r
|
||||
import multiprocessing as mp
|
||||
|
||||
from bigchaindb import Bigchain
|
||||
from bigchaindb.voter import Voter, BlockStream
|
||||
from bigchaindb.crypto import PublicKey
|
||||
|
||||
@ -35,6 +34,47 @@ class TestBigchainVoter(object):
|
||||
.order_by(r.asc((r.row['block']['timestamp'])))
|
||||
.run(b.conn))
|
||||
|
||||
# validate vote
|
||||
assert len(blocks[1]['votes']) == 1
|
||||
vote = blocks[1]['votes'][0]
|
||||
|
||||
assert vote['vote']['voting_for_block'] == block['id']
|
||||
assert vote['vote']['previous_block'] == genesis['id']
|
||||
assert vote['vote']['is_block_valid'] is True
|
||||
assert vote['vote']['invalid_reason'] is None
|
||||
assert vote['node_pubkey'] == b.me
|
||||
assert PublicKey(b.me).verify(b.serialize(vote['vote']), vote['signature']) is True
|
||||
|
||||
def test_valid_block_voting_with_create_transaction(self, b):
|
||||
q_new_block = mp.Queue()
|
||||
|
||||
genesis = b.create_genesis_block()
|
||||
|
||||
# create a `CREATE` transaction
|
||||
test_user_priv, test_user_pub = b.generate_keys()
|
||||
tx = b.create_transaction(b.me, test_user_pub, None, 'CREATE')
|
||||
tx_signed = b.sign_transaction(tx, b.me_private)
|
||||
assert b.is_valid_transaction(tx_signed)
|
||||
|
||||
# create valid block
|
||||
block = b.create_block([tx_signed])
|
||||
# assert block is valid
|
||||
assert b.is_valid_block(block)
|
||||
b.write_block(block, durability='hard')
|
||||
|
||||
# create queue and voter
|
||||
voter = Voter(q_new_block)
|
||||
|
||||
# vote
|
||||
voter.start()
|
||||
# wait for vote to be written
|
||||
time.sleep(1)
|
||||
voter.kill()
|
||||
|
||||
# retrive block from bigchain
|
||||
blocks = list(r.table('bigchain')
|
||||
.order_by(r.asc((r.row['block']['timestamp'])))
|
||||
.run(b.conn))
|
||||
|
||||
# validate vote
|
||||
assert len(blocks[1]['votes']) == 1
|
||||
@ -47,6 +87,76 @@ class TestBigchainVoter(object):
|
||||
assert vote['node_pubkey'] == b.me
|
||||
assert PublicKey(b.me).verify(b.serialize(vote['vote']), vote['signature']) is True
|
||||
|
||||
def test_valid_block_voting_with_transfer_transactions(self, b):
|
||||
q_new_block = mp.Queue()
|
||||
|
||||
b.create_genesis_block()
|
||||
|
||||
# create a `CREATE` transaction
|
||||
test_user_priv, test_user_pub = b.generate_keys()
|
||||
tx = b.create_transaction(b.me, test_user_pub, None, 'CREATE')
|
||||
tx_signed = b.sign_transaction(tx, b.me_private)
|
||||
assert b.is_valid_transaction(tx_signed)
|
||||
|
||||
# create valid block
|
||||
block = b.create_block([tx_signed])
|
||||
# assert block is valid
|
||||
assert b.is_valid_block(block)
|
||||
b.write_block(block, durability='hard')
|
||||
|
||||
# create queue and voter
|
||||
voter = Voter(q_new_block)
|
||||
|
||||
# vote
|
||||
voter.start()
|
||||
# wait for vote to be written
|
||||
time.sleep(1)
|
||||
voter.kill()
|
||||
|
||||
# retrive block from bigchain
|
||||
blocks = list(r.table('bigchain')
|
||||
.order_by(r.asc((r.row['block']['timestamp'])))
|
||||
.run(b.conn))
|
||||
|
||||
# validate vote
|
||||
assert len(blocks[1]['votes']) == 1
|
||||
|
||||
# create a `TRANSFER` transaction
|
||||
test_user2_priv, test_user2_pub = b.generate_keys()
|
||||
tx2 = b.create_transaction(test_user_pub, test_user2_pub, tx['id'], 'TRANSFER')
|
||||
tx2_signed = b.sign_transaction(tx2, test_user_priv)
|
||||
assert b.is_valid_transaction(tx2_signed)
|
||||
|
||||
# create valid block
|
||||
block = b.create_block([tx2_signed])
|
||||
# assert block is valid
|
||||
assert b.is_valid_block(block)
|
||||
b.write_block(block, durability='hard')
|
||||
|
||||
# create queue and voter
|
||||
voter = Voter(q_new_block)
|
||||
|
||||
# vote
|
||||
voter.start()
|
||||
# wait for vote to be written
|
||||
time.sleep(1)
|
||||
voter.kill()
|
||||
|
||||
# retrive block from bigchain
|
||||
blocks = list(r.table('bigchain')
|
||||
.order_by(r.asc((r.row['block']['timestamp'])))
|
||||
.run(b.conn))
|
||||
|
||||
# validate vote
|
||||
assert len(blocks[2]['votes']) == 1
|
||||
|
||||
vote = blocks[2]['votes'][0]
|
||||
|
||||
assert vote['vote']['voting_for_block'] == block['id']
|
||||
assert vote['vote']['is_block_valid'] is True
|
||||
assert vote['vote']['invalid_reason'] is None
|
||||
assert vote['node_pubkey'] == b.me
|
||||
assert PublicKey(b.me).verify(b.serialize(vote['vote']), vote['signature']) is True
|
||||
|
||||
def test_invalid_block_voting(self, b, user_public_key):
|
||||
# create queue and voter
|
||||
@ -66,7 +176,6 @@ class TestBigchainVoter(object):
|
||||
assert not b.is_valid_block(block)
|
||||
b.write_block(block, durability='hard')
|
||||
|
||||
|
||||
# vote
|
||||
voter.start()
|
||||
time.sleep(1)
|
||||
@ -167,7 +276,6 @@ class TestBigchainVoter(object):
|
||||
time.sleep(1)
|
||||
voter.kill()
|
||||
|
||||
|
||||
# retrive blocks from bigchain
|
||||
blocks = list(r.table('bigchain')
|
||||
.order_by(r.asc((r.row['block']['timestamp'])))
|
||||
@ -187,7 +295,6 @@ class TestBigchainVoter(object):
|
||||
pass
|
||||
|
||||
|
||||
|
||||
class TestBlockStream(object):
|
||||
|
||||
def test_if_federation_size_is_greater_than_one_ignore_past_blocks(self, b):
|
||||
@ -219,7 +326,6 @@ class TestBlockStream(object):
|
||||
assert bs.get() == block_1
|
||||
assert bs.get() == block_2
|
||||
|
||||
|
||||
def test_if_old_blocks_get_should_return_old_block_first(self, b):
|
||||
# create two blocks
|
||||
block_1 = b.create_block([])
|
||||
|
||||
@ -60,6 +60,7 @@ def mock_bigchaindb_backup_config(monkeypatch):
|
||||
config = {
|
||||
'keypair': {},
|
||||
'database': {'host': 'host', 'port': 12345, 'name': 'adbname'},
|
||||
'statsd': {'host': 'host', 'port': 12345, 'rate': 0.1},
|
||||
}
|
||||
monkeypatch.setattr('bigchaindb._config', config)
|
||||
|
||||
@ -143,3 +144,22 @@ def test_run_configure_when_config_does_not_exist(monkeypatch,
|
||||
args = Namespace(config='foo', yes=True)
|
||||
return_value = run_configure(args)
|
||||
assert return_value is None
|
||||
|
||||
|
||||
def test_run_configure_when_config_does_exist(monkeypatch,
|
||||
mock_write_config,
|
||||
mock_generate_key_pair,
|
||||
mock_bigchaindb_backup_config):
|
||||
value = {}
|
||||
def mock_write_config(newconfig, filename=None):
|
||||
value['return'] = newconfig
|
||||
|
||||
from bigchaindb.commands.bigchain import run_configure
|
||||
monkeypatch.setattr('os.path.exists', lambda path: True)
|
||||
monkeypatch.setattr('builtins.input', lambda question: '\n')
|
||||
monkeypatch.setattr('bigchaindb.config_utils.write_config', mock_write_config)
|
||||
|
||||
args = Namespace(config='foo', yes=None)
|
||||
run_configure(args)
|
||||
assert value == {}
|
||||
|
||||
|
||||
14
tests/test_monitor.py
Normal file
14
tests/test_monitor.py
Normal file
@ -0,0 +1,14 @@
|
||||
from platform import node
|
||||
|
||||
|
||||
def test_monitor_class_init_defaults():
|
||||
import bigchaindb
|
||||
from bigchaindb.monitor import Monitor
|
||||
monitor = Monitor()
|
||||
assert monitor
|
||||
assert len(monitor._addr) == 2
|
||||
# TODO get value from config
|
||||
# assert monitor._addr[0] == bigchaindb.config['statsd']['host']
|
||||
assert monitor._addr[0] == '127.0.0.1'
|
||||
assert monitor._addr[1] == bigchaindb.config['statsd']['port']
|
||||
assert monitor._prefix == node() + '.'
|
||||
@ -5,7 +5,7 @@ import pytest
|
||||
import bigchaindb
|
||||
|
||||
|
||||
ORIGINAL_CONFIG = copy.deepcopy(bigchaindb.config)
|
||||
ORIGINAL_CONFIG = copy.deepcopy(bigchaindb._config)
|
||||
|
||||
|
||||
@pytest.fixture(scope='function', autouse=True)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user