Merge pull request #1652 from bigchaindb/resolve-issue-1649

Removed old Terraform & Ansible templates
This commit is contained in:
Troy McConaghy 2017-07-17 12:08:34 +02:00 committed by GitHub
commit f576894b4c
20 changed files with 0 additions and 628 deletions

View File

@ -19,8 +19,6 @@ Appendices
commands
aws-setup
aws-testing-cluster
template-terraform-aws
template-ansible
azure-quickstart-template
generate-key-pair-for-ssh
firewall-notes

View File

@ -1,84 +0,0 @@
# Template: Ansible Playbook to Run a BigchainDB Node on an Ubuntu Machine
This page explains how to use [Ansible](https://www.ansible.com/) to install, configure and run all the software needed to run a one-machine BigchainDB node on a server running Ubuntu 16.04.
**Note: We're not actively maintaining the associated Ansible files (e.g. playbooks). They are RethinkDB-specific, even though we now recommend using MongoDB. You may find the old Ansible stuff useful nevertheless, which is why we moved this page to the Appendices rather than deleting it.**
## Install Ansible
The Ansible documentation has [installation instructions](https://docs.ansible.com/ansible/intro_installation.html). Note the control machine requirements: at the time of writing, Ansible required Python 2.6 or 2.7. ([Python 3 support is coming](https://docs.ansible.com/ansible/python_3_support.html): "Ansible 2.2 features a tech preview of Python 3 support." and the latest version, as of January 31, 2017, was 2.2.1.0. For now, it's probably best to use it with Python 2.)
For example, you could create a special Python 2.x virtualenv named `ansenv` and then install Ansible in it:
```text
cd repos/bigchaindb/ntools
virtualenv -p /usr/local/lib/python2.7.11/bin/python ansenv
source ansenv/bin/activate
pip install ansible
```
## About Our Example Ansible Playbook
Our example Ansible playbook installs, configures and runs a basic BigchainDB node on an Ubuntu 16.04 machine. That playbook is in `.../bigchaindb/ntools/one-m/ansible/one-m-node.yml`.
When you run the playbook (as per the instructions below), it ensures all the necessary software is installed, configured and running. It can be used to get a BigchainDB node set up on a bare Ubuntu 16.04 machine, but it can also be used to ensure that everything is okay on a running BigchainDB node. (If you run the playbook against a host where everything is okay, then it won't change anything on that host.)
## Create an Ansible Inventory File
An Ansible "inventory" file is a file which lists all the hosts (machines) you want to manage using Ansible. (Ansible will communicate with them via SSH.) Right now, we only want to manage one host.
First, determine the public IP address of the host (i.e. something like `192.0.2.128`).
Then create a one-line text file named `hosts` by doing this:
```text
# cd to the directory .../bigchaindb/ntools/one-m/ansible
echo "192.0.2.128" > hosts
```
but replace `192.0.2.128` with the IP address of the host.
## Run the Ansible Playbook(s)
The latest Ubuntu 16.04 AMIs from Canonical don't include Python 2 (which is required by Ansible), so the first step is to run a small Ansible playbook to install Python 2 on the managed node:
```text
# cd to the directory .../bigchaindb/ntools/one-m/ansible
ansible-playbook -i hosts --private-key ~/.ssh/<key-name> install-python2.yml
```
where `<key-name>` should be replaced by the name of the SSH private key you created earlier (for SSHing to the host machine at your cloud hosting provider).
The next step is to run the Ansible playbook named `one-m-node.yml`:
```text
# cd to the directory .../bigchaindb/ntools/one-m/ansible
ansible-playbook -i hosts --private-key ~/.ssh/<key-name> one-m-node.yml
```
What did you just do? Running that playbook ensures all the software necessary for a one-machine BigchainDB node is installed, configured, and running properly. You can run that playbook on a regular schedule to ensure that the system stays properly configured. If something is okay, it does nothing; it only takes action when something is not as-desired.
## Some Notes on the One-Machine Node You Just Got Running
* It ensures that the installed version of RethinkDB is the latest. You can change that by changing the installation task.
* It uses a very basic RethinkDB configuration file based on `bigchaindb/ntools/one-m/ansible/roles/rethinkdb/templates/rethinkdb.conf.j2`.
* If you edit the RethinkDB configuration file, then running the Ansible playbook will **not** restart RethinkDB for you. You must do that manually. (You can stop RethinkDB using `sudo /etc/init.d/rethinkdb stop`; run the playbook to get RethinkDB started again. This assumes you're using init.d, which is what the Ansible playbook assumes. If you want to use systemd, you'll have to edit the playbook accordingly, and stop RethinkDB using `sudo systemctl stop rethinkdb@<name_instance>`.)
* It generates and uses a default BigchainDB configuration file, which it stores in `~/.bigchaindb` (the default location).
* If you edit the BigchainDB configuration file, then running the Ansible playbook will **not** restart BigchainDB for you. You must do that manually. (You could stop it using `sudo killall -9 bigchaindb`. Run the playbook to get it started again.)
## Optional: Create an Ansible Config File
The above command (`ansible-playbook -i ...`) is fairly long. You can omit the optional arguments if you put their values in an [Ansible configuration file](https://docs.ansible.com/ansible/intro_configuration.html) (config file) instead. There are many places where you can put a config file, but to make one specifically for the "one-m" case, you should put it in `.../bigchaindb/ntools/one-m/ansible/`. In that directory, create a file named `ansible.cfg` with the following contents:
```text
[defaults]
private_key_file = $HOME/.ssh/<key-name>
inventory = hosts
```
where, as before, `<key-name>` must be replaced.
## Next Steps
You could make changes to the Ansible playbook (and the resources it uses) to make the node more production-worthy. See [the section on production node assumptions, components and requirements](../production-nodes/index.html).

View File

@ -1,84 +0,0 @@
# Template: Using Terraform to Provision an Ubuntu Machine on AWS
This page explains a way to use [Terraform](https://www.terraform.io/) to provision an Ubuntu machine (i.e. an EC2 instance with Ubuntu 16.04) and other resources on [AWS](https://aws.amazon.com/). That machine can then be used to host a one-machine BigchainDB node, for example.
**Note: We're not actively maintaining the associated Terraform files. You may find them useful nevertheless, which is why we moved this page to the Appendices rather than deleting it.**
## Install Terraform
The [Terraform documentation has installation instructions](https://www.terraform.io/intro/getting-started/install.html) for all common operating systems.
If you don't want to run Terraform on your local machine, you can install it on a cloud machine under your control (e.g. on AWS).
Note: Hashicorp has an enterprise version of Terraform called "Terraform Enterprise." You can license it by itself or get it as part of Atlas. If you decide to license Terraform Enterprise or Atlas, be sure to install it on your own hosting (i.e. "on premise"), not on the hosting provided by Hashicorp. The reason is that BigchainDB clusters are supposed to be decentralized. If everyone used Hashicorp's hosted Atlas, then that would be a point of centralization.
**Ubuntu Installation Tips**
If you want to install Terraform on Ubuntu, first [download the .zip file](https://www.terraform.io/downloads.html). Then install it in `/opt`:
```text
sudo mkdir -p /opt/terraform
sudo unzip path/to/zip-file.zip -d /opt/terraform
```
Why install it in `/opt`? See [the answers at Ask Ubuntu](https://askubuntu.com/questions/1148/what-is-the-best-place-to-install-user-apps).
Next, add `/opt/terraform` to your path. If you use bash for your shell, then you could add this line to `~/.bashrc`:
```text
export PATH="/opt/terraform:$PATH"
```
After doing that, relaunch your shell or force it to read `~/.bashrc` again, e.g. by doing `source ~/.bashrc`. You can verify that terraform is installed and in your path by doing:
```text
terraform --version
```
It should say the current version of Terraform.
## Get Set Up to Use Terraform
First, do the [basic AWS setup steps outlined in the Appendices](../appendices/aws-setup.html).
Then go to the `.../bigchaindb/ntools/one-m/aws/` directory and open the file `variables.tf`. Most of the variables have sensible default values, but you can change them if you like. In particular, you may want to change `aws_region`. (Terraform looks in `~/.aws/credentials` to get your AWS credentials, so you don't have to enter those anywhere.)
The `ssh_key_name` has no default value, so Terraform will prompt you every time it needs it.
To see what Terraform will do, run:
```text
terraform plan
```
It should ask you the value of `ssh_key_name`.
It figured out the plan by reading all the `.tf` Terraform files in the directory.
If you don't want to be asked for the `ssh_key_name`, you can change the default value of `ssh_key_name` (in the file `variables.tf`) or [you can set an environmen variable](https://www.terraform.io/docs/configuration/variables.html) named `TF_VAR_ssh_key_name`.
## Use Terraform to Provision Resources
To provision all the resources specified in the plan, do the following. **Note: This will provision actual resources on AWS, and those cost money. Be sure to shut down the resources you don't want to keep running later, otherwise the cost will keep growing.**
```text
terraform apply
```
Terraform will report its progress as it provisions all the resources. Once it's done, you can go to the Amazon EC2 web console and see the instance, its security group, its elastic IP, and its attached storage volumes (one for the root directory and one for RethinkDB storage).
At this point, there is no software installed on the instance except for Ubuntu 16.04 and whatever else came with the Amazon Machine Image (AMI) specified in the Terraform configuration (files).
The next step is to install, configure and run all the necessary software for a BigchainDB node. You could use [our example Ansible playbook](template-ansible.html) to do that.
## Optional: "Destroy" the Resources
If you want to shut down all the resources just provisioned, you must first disable termination protection on the instance:
1. Go to the EC2 console and select the instance you just launched. It should be named `BigchainDB_node`.
2. Click **Actions** > **Instance Settings** > **Change Termination Protection** > **Yes, Disable**
3. Back in your terminal, do `terraform destroy`
Terraform should "destroy" (i.e. terminate or delete) all the AWS resources you provisioned above.
If it fails (e.g. because of an attached and mounted EBS volume), then you can terminate the instance using the EC2 console: **Actions** > **Instance State** > **Terminate** > **Yes, Terminate**. Once the instance is terminated, you should still do `terraform destroy` to make sure that all the other resources are destroyed.

View File

@ -1 +0,0 @@
This directory contains tools for provisioning, deploying and managing a BigchainDB node (on AWS, Azure or wherever).

View File

@ -1,5 +0,0 @@
---
# ansible/group_vars/all
# Variables in this file are for *all* host groups (i.e. all hosts)
example_var: 23

View File

@ -1,15 +0,0 @@
---
# This playbook ensures Python 2 is installed on the managed node.
# This is inspired by https://gist.github.com/gwillem/4ba393dceb55e5ae276a87300f6b8e6f
- hosts: all
gather_facts: false
remote_user: ubuntu
pre_tasks:
- name: Install Python 2
raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
become: true
# action: setup will gather facts after python2 has been installed
- action: setup

View File

@ -1,12 +0,0 @@
---
# This playbook deploys a BigchainDB node in one machine (one-m).
- name: Ensure a one-machine BigchainDB node is configured properly
hosts: all
remote_user: ubuntu
roles:
- ntp
- db_storage
- rethinkdb
- bigchaindb

View File

@ -1,3 +0,0 @@
#!/bin/bash
bigchaindb -y start &
disown

View File

@ -1,83 +0,0 @@
---
# ansible/roles/bigchaindb/tasks/main.yml
# Note: "become: true" basically means "become root user for this task" i.e. sudo <cmd>
# See https://docs.ansible.com/ansible/become.html
# Note: update_cache=yes means it will do the equivalent of
# sudo apt-get update before the operation.
- name: Ensure the latest BigchainDB-required Ubuntu packages are installed
apt: name={{item}} state=latest update_cache=yes
become: true
with_items:
- make
- git
- g++
- libffi-dev
- python3-dev
- python3-pip
- python3-setuptools
- name: Ensure the latest setuptools (Python package) is installed
pip: executable=pip3 name=setuptools state=latest
become: true
# This should make both pip and pip3 be pip version >=8.1.2 (python 3.4).
# See the comments about this below.
#- name: Ensure the latest pip/pip3 is installed, using easy_install3
# easy_install: executable=easy_install3 name=pip state=latest
# become: true
- name: Install BigchainDB from PyPI using sudo pip3 install bigchaindb
pip: executable=pip3 name=bigchaindb state=latest
become: true
- name: Gather facts about the file ~/.bigchaindb
stat: path={{ ansible_env.HOME }}/.bigchaindb
register: home_bigchaindb_config_file
- name: If ~/.bigchaindb doesn't exist, generate a default BigchainDB config file there
shell: bigchaindb -y configure rethinkdb
when: not home_bigchaindb_config_file.stat.exists
- name: Look up all processes with 'bigchaindb' in their name
shell: pgrep bigchaindb
ignore_errors: true
changed_when: false
register: pgrep_bigchaindb
# pgrep_bigchaindb.rc (return code) should be 0 if a bigchaindb process is already running
- name: Ensure a copy of start_bigchaindb.sh is on the remote host
copy: src=start_bigchaindb.sh dest=/tmp/start_bigchaindb.sh mode=0775
become: true
# Running BigchainDB in the background from Ansible is tricky, see:
# https://superuser.com/questions/870871/run-a-remote-script-application-in-detached-mode-in-ansible
- name: If BigchainDB isn't running then run it
command: /tmp/start_bigchaindb.sh
async: 45
poll: 0
when: pgrep_bigchaindb.rc != 0
# Notes about getting the latest version of pip3:
#
# The first way I tried to get the latest pip3 (commented-out below) didn't work.
# The first task works, but then the attempt to do
# the equivalent of "pip install -U pip" fails. "Found existing installation"
# and it didn't want to uninstall it
# Installing the python3-pip package installs a Python 3 version of pip named pip3
#- name: Ensure the latest python-pip and python3-pip Ubuntu packages are installed
# apt: name={{item}} state=latest update_cache=yes
# become: true
# with_items:
# - python-pip
# - python3-pip
#
#- name: Ensure pip is the latest version
# pip: executable=pip name=pip state=latest
#
#- name: Ensure pip3 is the latest version
# pip: executable=pip3 name=pip state=latest

View File

@ -1,31 +0,0 @@
---
# ansible/roles/db_storage/tasks/main.yml
#- name: Ensure the /data directory (for DB storage) exists
# file: path=/data state=directory
- name: Format the block storage device at /dev/xvdp with an ext4 file system
filesystem: fstype=ext4 dev=/dev/xvdp
become: true
# Note that this also modifies /etc/fstab so the mount will persist through a crash.
# To better understand the /etc/fstab fields/columns, see:
# http://man7.org/linux/man-pages/man5/fstab.5.html
# https://tinyurl.com/jmmsyon = the soure code of the mount module
# Note: It seems the "nobootwait" option is gone in Ubuntu 16.04. See
# https://askubuntu.com/questions/786928/ubuntu-16-04-fstab-fails-with-nobootwait
- name: Ensure /data dir exists and is mounted + update /etc/fstab
mount:
name=/data
src=/dev/xvdp
fstype=ext4
opts="defaults,nofail"
dump=0
passno=2
state=mounted
become: true
# Modify the I/O scheduler? Is that even a good idea?
# Must do this in /sys/block/xvdp/queue/scheduler
# and also with grub (so the I/O scheduler stays changed on reboot)
# Example: https://gist.github.com/keithchambers/80b60559ad83cebf1672

View File

@ -1,5 +0,0 @@
---
# ansible/roles/ntp/handlers/main.yml
- name: restart ntp
service: name=ntp state=restarted

View File

@ -1,28 +0,0 @@
---
# ansible/roles/ntp/tasks/main.yml
- name: Ensure ntpdate is not installed (and uninstall it if necessary)
apt: name=ntpdate state=absent
become: true
- name: Ensure the latest ntp is installed
apt: name=ntp state=latest update_cache=yes
become: true
- name: Retrieve facts about the file /etc/ntp.conf
stat: path=/etc/ntp.conf
register: ntp_conf_file
- debug: var=ntp_conf_file.stat.exists
- name: Fail when /etc/ntp.conf doesn't exist
fail: msg="The NTP config file /etc/ntp.conf doesn't exist'"
when: ntp_conf_file.stat.exists == False
# For now, we assume the default /etc/ntp.conf file is okay
- name: Ensure the ntp service is now started and should start on boot (enabled=yes)
service: name=ntp state=started enabled=yes
become: true
# All notified handlers are executed now (only once each, and only if notified)

View File

@ -1,72 +0,0 @@
---
# ansible/roles/rethinkdb/tasks/main.yml
# Note: the .list extension will be added to the rethinkdb filename automatically
# Note: xenial is the $DISTRIB_CODENAME for Ubuntu 16.04
- name: >
Ensure RethinkDB's APT repository for Ubuntu xenial is present
in /etc/apt/sources.list.d/rethinkdb.list
apt_repository:
repo='deb http://download.rethinkdb.com/apt xenial main'
filename=rethinkdb
state=present
become: true
- name: Ensure RethinkDB's public GPG key is in the set of APT keys
apt_key: url=http://download.rethinkdb.com/apt/pubkey.gpg state=present
become: true
- name: Ensure the latest rethinkdb package is installed
apt: name=rethinkdb state=latest update_cache=yes
become: true
- name: Ensure the /data directory's owner and group are both 'rethinkdb'
file: path=/data state=directory owner=rethinkdb group=rethinkdb
become: true
- name: Gather facts about the file /tmp/created_on_run1
stat: path=/tmp/created_on_run1
register: file_created_on_run1
- name: if the file /tmp/created_on_run1 doesn't exist then create /data/delete_me
file: path=/data/delete_me state=touch owner=rethinkdb group=rethinkdb
become: true
when: not file_created_on_run1.stat.exists
- name: if the file /tmp/created_on_run1 doesn't exist then do sudo rm -rf /data/*
shell: rm -rf /data/*
become: true
when: not file_created_on_run1.stat.exists
- name: if the file /tmp/created_on_run1 doesn't exist then create it
file: path=/tmp/created_on_run1 state=touch
become: true
when: not file_created_on_run1.stat.exists
# To enable starting RethinkDB on boot (on init.d systems),
# the config file must be put in /etc/rethinkdb/instances.d/
# See https://www.rethinkdb.com/docs/start-on-startup/
# Note: This task does NOT have a notify: rethinkdb restart
# A task to ensure RethinkDB is started comes later.
- name: >
Generate a RethinkDB config file from rethinkdb.conf.j2 and put it in
/etc/rethinkdb/instances.d/instance1.conf
template:
src=rethinkdb.conf.j2
dest=/etc/rethinkdb/instances.d/instance1.conf
owner=root
group=root
mode=0664
become: true
register: config_file
- name: Ensure rethinkdb is now started
service: name=rethinkdb state=started
become: true
register: rethinkdb_started
- debug: var=rethinkdb_started
- debug: msg="The RethinkDB config file changed while RethinkDB was already running.
RethinkDB was not stopped and restarted. You must do that manually."
when: config_file.changed and (not rethinkdb_started.changed)

View File

@ -1,11 +0,0 @@
directory=/data
runuser=rethinkdb
rungroup=rethinkdb
bind=all
# The IP address of localhost is 127.0.0.1
#bind-http=127.0.0.1
# direct-io
# List of *other* hostnames in the cluster:
# join=hostname1:29015
# join=hostname2:29015
# etc.

View File

@ -1,21 +0,0 @@
# Each AWS region has a different AMI name
# even though the contents are the same.
# This file has the mapping from region --> AMI name.
#
# These are all Ubuntu 16.04 LTS AMIs
# with Arch = amd64, Instance Type = hvm:ebs-ssd
# from https://cloud-images.ubuntu.com/locator/ec2/
# as of Jan. 31, 2017
variable "amis" {
type = "map"
default = {
eu-west-1 = "ami-d8f4deab"
eu-central-1 = "ami-5aee2235"
us-east-1 = "ami-6edd3078"
us-west-2 = "ami-7c803d1c"
ap-northeast-1 = "ami-eb49358c"
ap-southeast-1 = "ami-b1943fd2"
ap-southeast-2 = "ami-fe71759d"
sa-east-1 = "ami-7379e31f"
}
}

View File

@ -1,6 +0,0 @@
# You can get the value of "ip_address" after running terraform apply using:
# $ terraform output ip_address
# You could use that in a script, for example
output "ip_address" {
value = "${aws_eip.ip.public_ip}"
}

View File

@ -1,6 +0,0 @@
provider "aws" {
# An AWS access_key and secret_key are needed; Terraform looks
# for an AWS credentials file in the default location.
# See https://tinyurl.com/pu8gd9h
region = "${var.aws_region}"
}

View File

@ -1,47 +0,0 @@
# One instance (virtual machine) on AWS:
# https://www.terraform.io/docs/providers/aws/r/instance.html
resource "aws_instance" "instance" {
ami = "${lookup(var.amis, var.aws_region)}"
instance_type = "${var.aws_instance_type}"
tags {
Name = "BigchainDB_node"
}
ebs_optimized = true
key_name = "${var.ssh_key_name}"
vpc_security_group_ids = ["${aws_security_group.node_sg1.id}"]
root_block_device = {
volume_type = "gp2"
volume_size = "${var.root_storage_in_GiB}"
delete_on_termination = true
}
# Enable EC2 Instance Termination Protection
disable_api_termination = true
}
# This EBS volume will be used for database storage (not for root).
# https://www.terraform.io/docs/providers/aws/r/ebs_volume.html
resource "aws_ebs_volume" "db_storage" {
type = "gp2"
availability_zone = "${aws_instance.instance.availability_zone}"
# Size in GiB (not GB!)
size = "${var.DB_storage_in_GiB}"
tags {
Name = "BigchainDB_db_storage"
}
}
# This allocates a new elastic IP address, if necessary
# and then associates it with the above aws_instance
resource "aws_eip" "ip" {
instance = "${aws_instance.instance.id}"
vpc = true
}
# This attaches the instance to the EBS volume for RethinkDB storage
# https://www.terraform.io/docs/providers/aws/r/volume_attachment.html
resource "aws_volume_attachment" "ebs_att" {
# Why /dev/sdp? See https://tinyurl.com/z2zqm6n
device_name = "/dev/sdp"
volume_id = "${aws_ebs_volume.db_storage.id}"
instance_id = "${aws_instance.instance.id}"
}

View File

@ -1,93 +0,0 @@
# It might be better to:
# 1. start by only allowing SSH on port 22 (in the security group)
# 2. use SSH to set up a proper firewall on the (virtual) machine
# 3. add a second security group with more ports open
resource "aws_security_group" "node_sg1" {
name_prefix = "BigchainDB_"
description = "Single-machine BigchainDB node security group"
tags = {
Name = "BigchainDB_one-m"
}
# Allow all outbound traffic
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
# SSH
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# DNS
ingress {
from_port = 53
to_port = 53
protocol = "udp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTP is used by some package managers
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# NTP daemons use port 123 but the request will
# come from inside the firewall so a response is expected
# SNMP (e.g. for server monitoring)
ingress {
from_port = 161
to_port = 161
protocol = "udp"
cidr_blocks = ["0.0.0.0/0"]
}
# HTTPS is used when installing RethinkDB
# and by some package managers
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# Future: Don't allow port 8080 for the RethinkDB web interface.
# Use a SOCKS proxy or reverse proxy instead.
ingress {
from_port = 8080
to_port = 8080
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# BigchainDB Client-Server REST API
ingress {
from_port = 9984
to_port = 9984
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# Port 28015 doesn't have to be open to the outside
# since the RethinkDB client and server are on localhost
# RethinkDB intracluster communications use port 29015
ingress {
from_port = 29015
to_port = 29015
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}

View File

@ -1,19 +0,0 @@
variable "aws_region" {
default = "eu-central-1"
}
variable "aws_instance_type" {
default = "m4.large"
}
variable "root_storage_in_GiB" {
default = 10
}
variable "DB_storage_in_GiB" {
default = 30
}
variable "ssh_key_name" {
# No default. Ask as needed.
}