Merge pull request #589 from bigchaindb/feat/551/ansible-run-ntp-daemon

Feat/551/ansible run ntp daemon

.gitignore

@@ -73,3 +73,7 @@ deploy-cluster-aws/hostlist.py
 deploy-cluster-aws/ssh_key.py
 benchmarking-tests/hostlist.py
 benchmarking-tests/ssh_key.py
+
+# Ansible-specific files
+ntools/one-m/ansible/hosts
+ntools/one-m/ansible/ansible.cfg

docs/source/appendices/consensus.md

@@ -71,7 +71,7 @@ class SillyConsensusRules(BaseConsensusRules):
 
 ## Packaging a plugin
 
-BigchainDB uses [setuptool's entry_point](https://pythonhosted.org/setuptools/setuptools.html) system to provide the plugin functionality. Any custom plugin needs to add this section to the `setup()` call in their `setup.py`:
+BigchainDB uses [setuptools](https://setuptools.readthedocs.io/en/latest/)' entry_points to provide the plugin functionality. Any custom plugin needs to add this section to the `setup()` call in their `setup.py`:
 
 ```python
 entry_points={

docs/source/clusters-feds/aws-testing-cluster.md

@@ -28,7 +28,7 @@ What did you just install?
 * "[Fabric](http://www.fabfile.org/) is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks."
 * [fabtools](https://github.com/ronnix/fabtools) are "tools for writing awesome Fabric files"
 * [requests](http://docs.python-requests.org/en/master/) is a Python package/library for sending HTTP requests
-* "[Boto](https://boto3.readthedocs.org/en/latest/) is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2." (`boto3` is the name of the latest Boto package.)
+* "[Boto](https://boto3.readthedocs.io/en/latest/) is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2." (`boto3` is the name of the latest Boto package.)
 * [The aws-cli package](https://pypi.python.org/pypi/awscli), which is an AWS Command Line Interface (CLI).
 
 

docs/source/clusters-feds/set-up-a-federation.md

@@ -19,7 +19,7 @@ The federation must decide some things before setting up the initial cluster (in
 2. What will the replication factor be? (It must be 3 or more for [RethinkDB failover](https://rethinkdb.com/docs/failover/) to work.)
 3. Which node will be responsible for sending the commands to configure the RethinkDB database?
 
-Once those things have been decided, each node operator can begin [setting up their BigchainDB (production) node](../prod-node-setup-mgmt/index.html).
+Once those things have been decided, each node operator can begin setting up their BigchainDB (production) node.
 
 Each node operator will eventually need two pieces of information from all other nodes in the federation:
 

docs/source/dev-and-test/running-unit-tests.md

@@ -16,7 +16,7 @@ then in another terminal, do:
 $ python setup.py test
 ```
 
-(Aside: How does the above command work? The documentation for [pytest-runner](https://pypi.python.org/pypi/pytest-runner) explains. We use [pytest](http://pytest.org/latest/) to write all unit tests.)
+(Aside: How does the above command work? The documentation for [pytest-runner](https://pypi.python.org/pypi/pytest-runner) explains. We use [pytest](http://docs.pytest.org/en/latest/) to write all unit tests.)
 
 
 ### Using docker-compose to Run the Tests

docs/source/index.rst

@@ -12,7 +12,7 @@ Table of Contents
    node-cluster-fed
    nodes/index
    dev-and-test/index
-   prod-node-setup-mgmt/index
+   prod-node-depl-tplt/index
    server-reference/index
    drivers-clients/index
    clusters-feds/index

docs/source/nodes/setup-run-node.md

@@ -2,7 +2,7 @@
 
 If you want to set up a BigchainDB node that's intended to be one of the nodes in a BigchainDB cluster (i.e. where each node is operated by a different member of a federation), then this page is for you, otherwise see [elsewhere](../introduction.html).
 
-This is a page of general guidelines for setting up a node. It says nothing about how to upgrade software, storage, processing, etc. or other details of node management. That will be added in the future, in [the section on production node setup & management](../prod-node-setup-mgmt/index.html). Once that section is more complete, this page will probably be deleted.
+This is a page of general guidelines for setting up a node. It says nothing about how to upgrade software, storage, processing, etc. or other details of node management. That will be added in the future, in [the section on production node setup & management](../prod-node-depl-tplt/index.html). Once that section is more complete, this page will probably be deleted.
 
 
 ## Get a Server

docs/source/prod-node-depl-tplt/index.rst

@@ -0,0 +1,12 @@
+Production Node Deployment Template
+===================================
+
+.. toctree::
+   :maxdepth: 1
+
+   overview
+   install-terraform
+   install-ansible
+   prov-one-m-aws
+   prov-one-m-azure
+   start-one-m-node

docs/source/prod-node-depl-tplt/install-ansible.md

@@ -0,0 +1,11 @@
+# Install Ansible
+
+The Ansible documentation has [installation instructions](https://docs.ansible.com/ansible/intro_installation.html). Note the control machine requirements: at the time of writing, Ansible required Python 2.6 or 2.7. (Support for Python 3 [is a goal of Ansible 2.2](https://github.com/ansible/ansible/issues/15976#issuecomment-221264089).)
+
+For example, you could create a special Python 2.x virtualenv named `ansenv` and then install Ansible in it:
+```text
+cd repos/bigchaindb/ntools
+virtualenv -p /usr/local/lib/python2.7.11/bin/python ansenv
+source ansenv/bin/activate
+pip install ansible
+```

docs/source/prod-node-depl-tplt/install-terraform.md

@@ -2,7 +2,10 @@
 
 The [Terraform documentation has installation instructions](https://www.terraform.io/intro/getting-started/install.html) for all common operating systems.
 
-Note: Hashicorp (the company behind Terraform) will try to convince you that running Terraform on their servers (inside Atlas) would be great. **While that might be true for many, it is not true for BigchainDB.** BigchainDB federations are supposed to be decentralized, and if everyone used Atlas, that would be a point of centralization. If you don't want to run Terraform on your local machine, you could install it on a cloud machine under your control (e.g. on AWS).
+If you don't want to run Terraform on your local machine, you can install it on a cloud machine under your control (e.g. on AWS).
+
+Note: Hashicorp has an enterprise version of Terraform called "Terraform Enterprise." You can license it by itself or get it as part of Atlas. If you decide to license Terraform Enterprise or Atlas, be sure to install it on your own hosting (i.e. "on premise"), not on the hosting provided by Hashicorp. The reason is that BigchainDB clusters are supposed to be decentralized. If everyone used Hashicorp's hosted Atlas, then that would be a point of centralization.
+
 
 ## Ubuntu Installation Tips
 

docs/source/prod-node-depl-tplt/overview.md

@@ -0,0 +1,14 @@
+# Overview
+
+A BigchainDB production node has more components and requirements than a dev/test node. Those are outlined in the [BigchainDB Nodes](../nodes/index.html) section.
+
+You can provision and deploy a production node (to meet the requirments) using whatever tools you prefer.
+
+This section documents a template (example), showing how one could use certain tools to provision and deploy a prodution node. Feel free to ignore this section or use it to help you with your preferred tools.
+
+In this section, we use:
+
+* [Terraform](https://www.terraform.io/) to provision infrastructure such as AWS instances, storage and security groups, and
+* [Ansible](https://www.ansible.com/) to manage the software and files on that infrastructure (configuration management).
+
+If you notice something that could be done better, let us know (e.g. by creating an issue on GitHub).

docs/source/prod-node-depl-tplt/prov-one-m-aws.md

@@ -19,7 +19,7 @@ It should ask you the value of `ssh_key_name`.
 
 It figured out the plan by reading all the `.tf` Terraform files in the directory.
 
-If you don't want to be asked for the `ssh_key_name`, you can change the default value of `ssh_key_name` or [you can set an environmen variable](https://www.terraform.io/docs/configuration/variables.html) named `TF_VAR_ssh_key_name`.
+If you don't want to be asked for the `ssh_key_name`, you can change the default value of `ssh_key_name` (in the file `variables.tf`) or [you can set an environmen variable](https://www.terraform.io/docs/configuration/variables.html) named `TF_VAR_ssh_key_name`.
 
 
 ## Provision
@@ -34,7 +34,7 @@ Terraform will report its progress as it provisions all the resources. Once it's
 At this point, there is no software installed on the instance except for Ubuntu 14.04 and whatever else came with the Amazon Machine Image (AMI) specified in the configuration. The next step is to use Ansible to install and configure all the necessary software.
 
 
-## (Optional) "Destroy"
+## Optional: "Destroy"
 
 If you want to shut down all the resources just provisioned, you must first disable termination protection on the instance:
 

docs/source/prod-node-depl-tplt/prov-one-m-azure.md

@@ -0,0 +1,3 @@
+# Provision a One-Machine Node on Azure
+
+This is just a placeholder so you can see the future structure of these docs.

docs/source/prod-node-depl-tplt/start-one-m-node.md

@@ -0,0 +1,45 @@
+# Start a One-Machine Node
+
+In this step, we will install, configure and run all the software necessary to run BigchainDB, all on one machine.
+
+
+## Create an Ansible Inventory File
+
+An Ansible "inventory" file is a file which lists all the hosts (machines) you want to manage using Ansible. (Ansible will communicate with them via SSH.) Right now, we only want to manage one host.
+
+First, determine the public IP address of the host (i.e. something like `192.0.2.128`).
+
+Then create a one-line text file named `hosts` by doing this:
+```text
+# cd to the directory .../bigchaindb/ntools/one-m/ansible
+echo "192.0.2.128" > hosts
+```
+
+but replace `192.0.2.128` with the IP address of the host.
+
+
+## Run the Ansible Playbook
+
+The next step is to run the Ansible playbook `one-m-node.yml`:
+```text
+# cd to the directory .../bigchaindb/ntools/one-m/ansible
+ansible-playbook -i hosts --private-key ~/.ssh/<key-name> one-m-node.yml
+```
+
+where `<key-name>` should be replaced by the name of the SSH private key you created earlier (for SSHing to the host machine at your cloud hosting provider).
+
+What did you just do? Running that playbook ensures all the software necessary for a one-machine BigchainDB node is installed, configured, and running properly. You can run that playbook on a regular schedule to ensure that the system stays properly configured. If something is okay, it does nothing; it only takes action when something is not as-desired.
+
+Note: At the time of writing, the playbook only installs, configures and runs an NTP daemon, but more is coming soon.
+
+
+## Optional: Create an Ansible Config File
+
+The above command (`ansible-playbook -i ...`) is fairly long. You can omit the optional arguments if you put their values in an [Ansible configuration file](https://docs.ansible.com/ansible/intro_configuration.html) (config file) instead. There are many places where you can put a config file, but to make one specifically for the "one-m" case, you should put it in `.../bigchaindb/ntools/one-m/ansible/`. In that directory, create a file named `ansible.cfg` with the following contents:
+```text
+[defaults]
+private_key_file = $HOME/.ssh/<key-name>
+inventory = hosts
+```
+
+where, as before, `<key-name>` must be replaced.

docs/source/prod-node-setup-mgmt/index.rst

@@ -1,10 +0,0 @@
-Production Node Setup & Management
-==================================
-
-.. toctree::
-   :maxdepth: 1
-
-   overview
-   install-terraform
-   prov-one-m-aws
-   

docs/source/prod-node-setup-mgmt/overview.md

@@ -1,13 +0,0 @@
-# Overview
-
-Deploying and managing a production BigchainDB node is much more involved than working with a dev/test node:
-
-* There are more components in a production node; see [the page about node components](../nodes/node-components.html)
-* Production nodes need more security
-* Production nodes need monitoring
-* Production nodes need maintenance, e.g. software upgrades, scaling
-
-Thankfully, there are tools to help! We use:
-
-* [Terraform](https://www.terraform.io/) to provision infrastructure such as AWS instances, storage and security groups
-* [Ansible](https://www.ansible.com/) to manage the software installed on that infrastructure (configuration management)

ntools/one-m/ansible/group_vars/all

@@ -0,0 +1,5 @@
+---
+# ansible/group_vars/all
+# Variables in this file are for *all* host groups (i.e. all hosts)
+
+example_var: 23

ntools/one-m/ansible/one-m-node.yml

@@ -0,0 +1,11 @@
+---
+# This playbook deploys a BigchainDB node in one machine (one-m).
+
+- name: Ensure a one-machine BigchainDB node is configured properly
+  hosts: all
+  remote_user: ubuntu
+
+  roles:
+    - bcdb_base
+    - ntp
+    # TODO: upgrade pip and setuptools, see https://github.com/bobbyrenwick/ansible-pip 

ntools/one-m/ansible/roles/bcdb_base/tasks/main.yml

@@ -0,0 +1,25 @@
+---
+# ansible/roles/bcdb_base/tasks/main.yml
+
+# Note: "become: true" basically means "become root user for this task" i.e. sudo <cmd>
+# See https://docs.ansible.com/ansible/become.html
+
+- name: Do the equivalent of "sudo apt-get update"
+  apt: update_cache=yes
+  become: true
+
+- name: Configure all unpacked but unconfigured packages
+  shell: /usr/bin/dpkg --configure -a
+  become: true
+
+- name: Attempt to correct a system with broken dependencies in place
+  shell: /usr/bin/apt-get -y -f install
+  become: true
+
+- name: Ensure the LATEST git g++ python3-dev are installed
+  apt: name={{item}} state=latest
+  become: true
+  with_items:
+    - git
+    - g++
+    - python3-dev

ntools/one-m/ansible/roles/ntp/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+# ansible/roles/common/handlers/main.yml
+
+- name: restart ntp
+  service: name=ntpd state=restarted

ntools/one-m/ansible/roles/ntp/tasks/main.yml

@@ -0,0 +1,24 @@
+---
+# ansible/roles/ntp/tasks/main.yml
+
+- name: Ensure ntpdate is not installed (and uninstall it if necessary)
+  apt: name=ntpdate state=absent
+  become: true
+
+- name: Ensure the LATEST ntp is installed and do "sudo apt-get update" 
+  apt: name=ntp state=latest update_cache=yes
+  become: true
+
+- name: Retrieve facts about the file /etc/ntp.conf
+  stat: path=/etc/ntp.conf
+  register: ntp_conf_file
+
+- name: Fail when /etc/ntp.conf doesn't exist
+  fail: msg="The NTP config file /etc/ntp.conf doesn't exist'"
+  when: ntp_conf_file.stat.exists == False
+
+# For now, we assume the default /etc/ntp.conf file is okay
+
+- name: Ensure the ntp service is now started and should start on boot (enabled=yes)
+  service: name=ntp state=started enabled=yes
+  become: true