diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c0975b0c..8e3a517b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,8 +18,8 @@ To contribute code or documentation, you need a [GitHub account](https://github. Familiarize yourself with how we do coding and documentation in the BigchainDB project, including: -* our Python Style Guide (coming soon) -* [our documentation strategy](./docs/README.md) (including code documentation) +* [our Python Style Guide](PYTHON_STYLE_GUIDE.md) (includes how we write tests) +* [our documentation strategy](./docs/README.md) (including in-code documentation) * the GitHub Flow (workflow) * [GitHub Guide: Understanding the GitHub Flow](https://guides.github.com/introduction/flow/) * [Scott Chacon's blog post about GitHub Flow](http://scottchacon.com/2011/08/31/github-flow.html) @@ -63,27 +63,29 @@ git checkout -b new-branch-name ### Step 5 - Make Edits, git add, git commit -With your new branch checked out locally, make changes or additions to the code or documentation, git add them, and git commit them. +With your new branch checked out locally, make changes or additions to the code or documentation. Remember to: + +* follow [our Python Style Guide](PYTHON_STYLE_GUIDE.md). +* write and run tests for any new or changed code. There's a section in [our Python Style Guide](PYTHON_STYLE_GUIDE.md) about writing and running tests. +* add or update documentation as necessary. Follow [our documentation strategy](./docs/README.md). + +As you go, git add and git commit your changes or additions, e.g. ```text -git add new-or-changed-file +git add new-or-changed-file-1 +git add new-or-changed-file-2 git commit -m "Short description of new or changed things" ``` -Remember to write tests for new code. If you don't, our code (test) coverage will go down, and we won't be able to accept your code. (We have some hard checks that run on all new pull requests and code coverage is one of them.) - -Please run all existing tests to make sure you didn't break something. Do: -```text -py.test -v -``` - -Remember to write or modify documentation to reflect your additions or changes. - You will want to merge changes from upstream (i.e. the original repository) into your new branch from time to time, using something like: ```text git fetch upstream git merge upstream/develop ``` +Once you're done commiting a set of new things and you're ready to submit them for inclusion, please be sure to run all the tests (as per the instructions at the end of our [Python Style Guide](PYTHON_STYLE_GUIDE.md)). + +(When you submit your pull request [following the instructions below], we run all the tests automatically, so we will see if some are failing. If you don't know why some tests are failing, you can still submit your pull request, but be sure to note the failing tests and to ask for help with resolving them.) + ### Step 6 - Push Your New Branch to origin Make sure you've commited all the additions or changes you want to include in your pull request. Then push your new branch to origin (i.e. _your_ remote bigchaindb repository). diff --git a/PYTHON_STYLE_GUIDE.md b/PYTHON_STYLE_GUIDE.md new file mode 100644 index 00000000..d416ad65 --- /dev/null +++ b/PYTHON_STYLE_GUIDE.md @@ -0,0 +1,102 @@ +# Python Style Guide + +This guide starts out with our general Python coding style guidelines and ends with a section on how we write & run (Python) tests. + +## General Python Coding Style Guidelines + +Our starting point is [PEP8](https://www.python.org/dev/peps/pep-0008/), the standard "Style Guide for Python Code." Many Python IDEs will check your code against PEP8. (Note that PEP8 isn't frozen; it actually changes over time, but slowly.) + +BigchainDB uses Python 3.4+, so you can ignore all PEP8 guidelines specific to Python 2. + +### Python Docstrings + +PEP8 says some things about docstrings, but not what to put in them or how to structure them. [PEP257](https://www.python.org/dev/peps/pep-0257/) was one proposal for docstring conventions, but we prefer [Google-style docstrings](https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments) instead: they're easier to read and the [napoleon extension](http://www.sphinx-doc.org/en/stable/ext/napoleon.html) for Sphinx lets us turn them into nice-looking documentation. Here are some references on Google-style docstrings: + +* [Google's docs on Google-style docstrings](https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments) +* [napoleon's docs include an overview of Google-style docstrings](http://sphinxcontrib-napoleon.readthedocs.org/en/latest/index.html) +* [Example Google-style docstrings](http://sphinxcontrib-napoleon.readthedocs.org/en/latest/example_google.html) (from napoleon's docs) + +### Maximum Line Length + +PEP8 has some [maximum line length guidelines](https://www.python.org/dev/peps/pep-0008/#id17), starting with "Limit all lines to a maximum of 79 characters" but "for flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters." + +We discussed this at length, and it seems that the consensus is: try to keep line lengths less than 79/72 characters, unless you have a special situation where longer lines would improve readability. (The basic reason is that 79/72 works for everyone, and BigchainDB is an open source project.) + +### Single or Double Quotes? + +Python lets you use single or double quotes. PEP8 says you can use either, as long as you're consistent. We try to stick to using single quotes, except in cases where using double quotes is more readable. For example: +```python +print('This doesn\'t look so nice.') +print("Doesn't this look nicer?") +``` + +### Breaking Strings Across Multiple Lines + +Should we use parentheses or slashes (`\`) to break strings across multiple lines, i.e. +```python +my_string = ('This is a very long string, so long that it will not fit into just one line ' + 'so it must be split across multiple lines.') +# or +my_string = 'This is a very long string, so long that it will not fit into just one line ' \ + 'so it must be split across multiple lines.' +``` + +It seems the preference is for slashes, but using parentheses is okay too. (There are good arguments either way. Arguing about it seems like a waste of time.) + +### Using the % operator or `format()` to Format Strings + +Given the choice: +```python +x = 'name: %s; score: %d' % (name, n) +# or +x = 'name: {}; score: {}'.format(name, n) +``` + +we use the `format()` version. The [official Python documentation says](https://docs.python.org/2/library/stdtypes.html#str.format), "This method of string formatting is the new standard in Python 3, and should be preferred to the % formatting described in String Formatting Operations in new code." + + +## Writing (Python) Tests + +We write tests for our Python code using the [pytest](http://pytest.org/latest/) framework. + +All tests go in the `bigchaindb/tests` directory or one of its subdirectories. You can use the tests already in there as templates or examples. + +### Standard Ways to Run All Tests + +To run all the tests, first make sure you have RethinkDB running: +```text +$ rethinkdb +``` + +then in another terminal, do: +```text +$ py.test -v +``` + +If that doesn't work (e.g. maybe you are running in a conda virtual environment), try: +```text +$ python -m pytest -v +``` + +You can also run all tests via `setup.py`, using: +```text +$ python setup.py test +``` + +### Using `docker-compose` to Run the Tests + +You can use `docker-compose` to run the tests. (You don't have to start RethinkDB first: `docker-compose` does that on its own, when it reads the `docker-compose.yml` file.) + +First, build the images (~once), using: +```text +$ docker-compose build +``` + +then run the tests using: +```text +$ docker-compose run --rm bigchaindb py.test -v +``` + +### Automated Testing of All Pull Requests + +We use [Travis CI](https://travis-ci.com/), so that whenever someone creates a new BigchainDB pull request on GitHub, Travis CI gets the new code and does _a bunch of stuff_. You can find out what we tell Travis CI to do in [the `.travis.yml` file](.travis.yml): it tells Travis CI how to install BigchainDB, how to run all the tests, and what to do "after success" (e.g. run `codecov`). (We use [Codecov](https://codecov.io/) to get a rough estimate of our test coverage.) diff --git a/bigchaindb/commands/bigchain.py b/bigchaindb/commands/bigchain.py index 33d37822..35b2de2f 100644 --- a/bigchaindb/commands/bigchain.py +++ b/bigchaindb/commands/bigchain.py @@ -45,7 +45,7 @@ def run_configure(args, skip_if_exists=False): if config_file_exists and not args.yes: want = input('Config file `{}` exists, do you want to override it? ' '(cannot be undone) [y/n]: '.format(config_path)) - if not want: + if want != 'y': return # Patch the default configuration with the new values diff --git a/docker-compose-monitor.yml b/docker-compose-monitor.yml index ee7fe3a4..6865931d 100644 --- a/docker-compose-monitor.yml +++ b/docker-compose-monitor.yml @@ -10,7 +10,7 @@ influxdb: PRE_CREATE_DB: "telegraf" grafana: - image: rhsimplex/grafana-bigchaindb-docker + image: bigchaindb/grafana-bigchaindb-docker tty: true ports: - "3000:3000" @@ -18,7 +18,7 @@ grafana: - influxdb:localhost statsd: - image: rhsimplex/docker-telegraf-statsd + image: bigchaindb/docker-telegraf-statsd ports: - "8125:8125/udp" links: diff --git a/docs/source/_static/monitoring_system_diagram.png b/docs/source/_static/monitoring_system_diagram.png new file mode 100644 index 00000000..6b654eac Binary files /dev/null and b/docs/source/_static/monitoring_system_diagram.png differ diff --git a/docs/source/installing.md b/docs/source/installing.md index ff51539c..9ff24298 100644 --- a/docs/source/installing.md +++ b/docs/source/installing.md @@ -2,6 +2,14 @@ We're developing BigchainDB Server ("BigchainDB") on Ubuntu 14.04, but it should work on any OS that runs RethinkDB Server and Python 3.4+. (BigchainDB is built on top of RethinkDB Server.) +BigchainDB Server is intended to be run on each server in a large distributed cluster of servers; it's not very useful running by itself on a single computer. That's _why_ we're developing it on Ubuntu 14.04: it's one of the more common server operating systems. + +Mac OS X users may get the best results [running BigchainDB Server with Docker](https://bigchaindb.readthedocs.org/en/develop/installing.html#run-bigchaindb-with-docker). + +Windows users may get the best results [running BigchainDB Server in a VM with Vagrant](https://bigchaindb.readthedocs.org/en/develop/installing.html#how-to-install-bigchaindb-on-a-vm-with-vagrant). + +(Right now, there are no BigchainDB clients/drivers. Those will be able to run on a much larger array of operating systems. They're coming soon.) + ## Install and Run RethinkDB Server The RethinkDB documentation has instructions for how to install RethinkDB Server on a variety of operating systems. Do that (using their instructions for your OS): [Install RethinkDB Server](http://rethinkdb.com/docs/install/). @@ -91,7 +99,7 @@ For those who like using Docker and wish to experiment with BigchainDB in non-production environments, we currently maintain a `dockerfile` that can be used to build an image for `bigchaindb`, along with a `docker-compose.yml` file to manage a "standalone node", consisting mainly of two containers: one for -RethinkDB, and another for `bigchaindb`. +RethinkDB, and another for BigchainDB. Assuming you have `docker` and `docker-compose` installed, you would proceed as follows. @@ -112,27 +120,21 @@ stored on your host machine under ` ~/.bigchaindb_docker/config`: $ docker-compose run --rm bigchaindb bigchaindb configure ``` -You can load test transactions via: +You can then start it up (in the background, as a daemon) using: +```text +$ docker-compose up -d +``` + +then you can load test transactions via: ```text $ docker-compose run --rm bigchaindb bigchaindb-benchmark load ``` -You should then be able to start `bigchaindb`, via: -```text -$ docker-compose run --rm bigchaindb bigchaindb start -``` +If you're on Linux, you can probably view the RethinkDB dashboard at: -or -```text -$ docker-compose up -``` +[http://localhost:58080/](http://localhost:58080/) -You should be able to view the RethinkDB dashboard at: -```text -http://docker_host:58080/ -``` - -where `docker_host` is the IP or hostname of the machine running the Docker -engine. If you are developing on Linux, this most likely will be `localhost`, -whereas if you are running docker-machine (e.g.: on Mac OS X) this will be the +If that doesn't work, then replace `localhost` +with the IP or hostname of the machine running the Docker engine. +If you are running docker-machine (e.g.: on Mac OS X) this will be the IP of the Docker machine (`docker-machine ip machine_name`). diff --git a/docs/source/monitoring.md b/docs/source/monitoring.md index 436a42aa..4d61cbc4 100644 --- a/docs/source/monitoring.md +++ b/docs/source/monitoring.md @@ -1,21 +1,31 @@ # Monitoring -BigchainDB uses [statsd](https://github.com/etsy/statsd) for monitoring. To fully take advantage of this functionality requires some additional infrastructure: an agent to listen for metrics (e.g. [telegraf](https://github.com/influxdata/telegraf)), a time-series database (e.g. [influxdb](https://influxdata.com/time-series-platform/influxdb/)), and a frontend to display analytics (e.g. [Grafana](http://grafana.org/)). +BigchainDB uses [StatsD](https://github.com/etsy/statsd) for monitoring. We require some additional infrastructure to take full advantage of its functionality: -For ease of use, we've provided a docker compose file that sets up all these services for testing. Simply run in the BigchainDB directory: +* an agent to listen for metrics: [Telegraf](https://github.com/influxdata/telegraf), +* a time-series database: [InfluxDB](https://influxdata.com/time-series-platform/influxdb/), and +* a frontend to display analytics: [Grafana](http://grafana.org/). +We put each of those inside its own Docker container. The whole system is illustrated below. + +![BigchainDB monitoring system diagram: Application metrics flow from servers running BigchainDB to Telegraf to InfluxDB to Grafana](./_static/monitoring_system_diagram.png) + +For ease of use, we've created a Docker [_Compose file_](https://docs.docker.com/compose/compose-file/) (named `docker-compose-monitor.yml`) to define the monitoring system setup. To use it, just go to to the top `bigchaindb` directory and run: ```text $ docker-compose -f docker-compose-monitor.yml build $ docker-compose -f docker-compose-monitor.yml up ``` -and point a browser tab to `http://localhost:3000/dashboard/script/bigchaindb_dashboard.js`. Login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](installing.html#run-bigchaindb) and load some test transactions: +then point a browser tab to: +[http://localhost:3000/dashboard/script/bigchaindb_dashboard.js](http://localhost:3000/dashboard/script/bigchaindb_dashboard.js) + +The login and password are `admin` by default. If BigchainDB is running and processing transactions, you should see analytics—if not, [start BigchainDB](installing.html#run-bigchaindb) and load some test transactions: ```text $ bigchaindb-benchmark load ``` -and refresh the page after a few seconds. +then refresh the page after a few seconds. If you're not interested in monitoring, don't worry: BigchainDB will function just fine without any monitoring setup. diff --git a/tests/test_commands.py b/tests/test_commands.py index faa455d0..57ebdbe9 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -144,3 +144,22 @@ def test_run_configure_when_config_does_not_exist(monkeypatch, args = Namespace(config='foo', yes=True) return_value = run_configure(args) assert return_value is None + + +def test_run_configure_when_config_does_exist(monkeypatch, + mock_write_config, + mock_generate_key_pair, + mock_bigchaindb_backup_config): + value = {} + def mock_write_config(newconfig, filename=None): + value['return'] = newconfig + + from bigchaindb.commands.bigchain import run_configure + monkeypatch.setattr('os.path.exists', lambda path: True) + monkeypatch.setattr('builtins.input', lambda question: '\n') + monkeypatch.setattr('bigchaindb.config_utils.write_config', mock_write_config) + + args = Namespace(config='foo', yes=None) + run_configure(args) + assert value == {} +