Merge pull request #600 from bigchaindb/feat/552/ansible-setup-storage-for-rethinkdb

Ansible playbook now sets up storage for the underlying DB

docs/source/prod-node-depl-tplt/prov-one-m-aws.md

@@ -44,6 +44,9 @@ If you want to shut down all the resources just provisioned, you must first disa
 
 Terraform should "destroy" (i.e. terminate or delete) all the AWS resources you provisioned above.
 
+If it fails (e.g. because of an attached and mounted EBS volume), then you can terminate the instance using the EC2 console: **Actions** > **Instance State** > **Terminate** > **Yes, Terminate**. Once the instance is terminated, you should still do `terraform destroy` to make sure that all the other resources are destroyed.
+
+
 ## See Also
 
 * The [Terraform Documentation](https://www.terraform.io/docs/)

docs/source/prod-node-depl-tplt/start-one-m-node.md

@@ -30,7 +30,7 @@ where `<key-name>` should be replaced by the name of the SSH private key you cre
 
 What did you just do? Running that playbook ensures all the software necessary for a one-machine BigchainDB node is installed, configured, and running properly. You can run that playbook on a regular schedule to ensure that the system stays properly configured. If something is okay, it does nothing; it only takes action when something is not as-desired.
 
-Note: At the time of writing, the playbook only installs, configures and runs an NTP daemon, but more is coming soon.
+Note: At the time of writing (Aug. 25, 2016), the playbook isn't complete, so not all of the above happens yet.
 
 
 ## Optional: Create an Ansible Config File

ntools/one-m/ansible/one-m-node.yml

@@ -8,4 +8,4 @@
   roles:
     - bcdb_base
     - ntp
-    # TODO: upgrade pip and setuptools, see https://github.com/bobbyrenwick/ansible-pip 
+    - db_storage

ntools/one-m/ansible/roles/bcdb_base/tasks/main.yml

@@ -4,22 +4,44 @@
 # Note: "become: true" basically means "become root user for this task" i.e. sudo <cmd>
 # See https://docs.ansible.com/ansible/become.html
 
-- name: Do the equivalent of "sudo apt-get update"
-  apt: update_cache=yes
-  become: true
-
-- name: Configure all unpacked but unconfigured packages
-  shell: /usr/bin/dpkg --configure -a
-  become: true
-
-- name: Attempt to correct a system with broken dependencies in place
-  shell: /usr/bin/apt-get -y -f install
-  become: true
-
-- name: Ensure the LATEST git g++ python3-dev are installed
-  apt: name={{item}} state=latest
+# Note: update_cache=yes means it will do the equivalent of
+# sudo apt-get update before the operation.
+- name: Ensure the latest BigchainDB-required Ubuntu packages are installed
+  apt: name={{item}} state=latest update_cache=yes
   become: true
   with_items:
     - git
     - g++
     - python3-dev
+    - python3-setuptools  # mainly for easy_install3, which is used to get latest pip3
+
+# This should make both pip and pip3 be pip version >=8.1.2 (python 3.4).
+# See the comments about this below.
+- name: Ensure the latest pip/pip3 is installed, using easy_install3
+  easy_install: executable=easy_install3 name=pip state=latest
+  become: true
+
+- name: Ensure the latest setuptools (Python package) is installed
+  pip: executable=pip3 name=setuptools state=latest
+  become: true
+
+# Notes about getting the latest version of pip3:
+#
+# The first way I tried to get the latest pip3 (commented-out below) didn't work.
+# The first task works, but then the attempt to do
+# the equivalent of "pip install -U pip" fails. "Found existing installation"
+# and it didn't want to uninstall it
+
+# Installing the python3-pip package installs a Python 3 version of pip named pip3
+#- name: Ensure the latest python-pip and python3-pip Ubuntu packages are installed
+#  apt: name={{item}} state=latest update_cache=yes
+#  become: true
+#  with_items:
+#    - python-pip
+#    - python3-pip
+#
+#- name: Ensure pip is the latest version
+#  pip: executable=pip name=pip state=latest
+#
+#- name: Ensure pip3 is the latest version
+#  pip: executable=pip3 name=pip state=latest

ntools/one-m/ansible/roles/db_storage/tasks/main.yml

@@ -0,0 +1,29 @@
+---
+# ansible/roles/db_storage/tasks/main.yml
+
+#- name: Ensure the /data directory (for DB storage) exists
+#  file: path=/data state=directory
+
+- name: Format the block storage device at /dev/xvdp with an ext4 file system
+  filesystem: fstype=ext4 dev=/dev/xvdp
+  become: true
+
+# Note that this also modifies /etc/fstab so the mount will persist through a crash.
+# To better understand the /etc/fstab fields/columns, see:
+# http://man7.org/linux/man-pages/man5/fstab.5.html
+# https://tinyurl.com/jmmsyon = the soure code of the mount module
+- name: Ensure /data dir exists and is mounted + update /etc/fstab
+  mount:
+    name=/data
+    src=/dev/xvdp
+    fstype=ext4
+    opts="defaults,nofail,nobootwait"
+    dump=0
+    passno=2
+    state=mounted
+  become: true
+
+# Modify the I/O scheduler? Is that even a good idea?
+# Must do this in /sys/block/xvdp/queue/scheduler
+# and also with grub (so the I/O scheduler stays changed on reboot)
+# Example: https://gist.github.com/keithchambers/80b60559ad83cebf1672

ntools/one-m/ansible/roles/ntp/tasks/main.yml

@@ -5,7 +5,7 @@
   apt: name=ntpdate state=absent
   become: true
 
-- name: Ensure the LATEST ntp is installed and do "sudo apt-get update" 
+- name: Ensure the latest ntp is installed
   apt: name=ntp state=latest update_cache=yes
   become: true
 
@@ -13,6 +13,8 @@
   stat: path=/etc/ntp.conf
   register: ntp_conf_file
 
+- debug: var=ntp_conf_file.stat.exists
+
 - name: Fail when /etc/ntp.conf doesn't exist
   fail: msg="The NTP config file /etc/ntp.conf doesn't exist'"
   when: ntp_conf_file.stat.exists == False

ntools/one-m/aws/variables.tf

@@ -3,7 +3,7 @@ variable "aws_region" {
 }
 
 variable "aws_instance_type" {
-  default = "m4.xlarge"
+  default = "m4.large"
 }
 
 variable "root_storage_in_GiB" {